MicroMator: Open and Flexible Software for Reactive Microscopy

11 Microscopy image analysis has recently made enormous progress both in terms of accuracy and speed 12 thanks to machine learning methods. This greatly facilitates the online adaptation of microscopy 13 experimental plans using real‐time information of the observed systems and their environments. Here 14 we report MicroMator, an open and flexible software for defining and driving reactive microscopy 15 experiments, and present applications to single‐cell control and single‐cell recombination. 16


Introduction
Software for microscopy automation are essential to support reproducible high-throughput microscopy experiments 1 .Samples can now be routinely imaged using complex spatial and temporal patterns.Yet, in the overwhelming majority of cases, executions of experiments are still cast in stone at the beginning, with little to no possibility for human or computer-driven interventions during the experiments.This is all the more surprising given that image analysis has recently made a giant leap in terms of accuracy and rapidity thanks to deep learning methods.Software empowering microscopy with real-time adaptation capabilities is needed to exploit the full potential of automated microscopes.
Several dedicated software solutions have been developed for applications requiring real-time analysis, most notably efficient scanning of large and complex samples (eg Refs [2][3][4][5] ) and real-time control of cellular processes (eg Refs [6][7][8][9][10][11][12] ).However only a handful generic solutions have been proposed so far [13][14][15][16] .AutomatedFCS is a Fiji plugin that is part of the Microscopy Pipeline Constructor.Its potential has been demonstrated but the tool applies only to Zeiss confocal microscopes 15 .The three other tools are all extensions of µManager, a widely-used software 17,18 controlling a large range of microscopy hardware.However, no in-depth case studies demonstrating their practical applicability -and showing possible limitations-have been reported so far.

MicroMator Software
To fill this gap, we developed MicroMator, a modular open-source Python software to support reactive microscopy, and demonstrate its potential via two challenging case studies.In MicroMator, events play a fundamental role (Fig 1A).They consist of Triggers and Effects.They can be defined by the user in a flexible manner.Examples of triggers include "at the 10th frame", "if more than 100 cells are in the field of view", "if the fluorescence of the 3rd newborn cell exceeds 100 a.u." and "if message !update position=10 frame=last is received from Discord".Examples of effects include changing a microscope configuration, sending light in the field of view with a given pattern, actuating a microfluidic pump, or starting an optimization routine.Microscopy experiments are defined by a main image acquisition loop, that serves as a backbone for the experiment, and by event creation functions (Fig 1B).Naturally, the main acquisition loop itself can be modified by event effects in the course of the experiment.For controlling hardware, MicroMator primarily uses the powerful Python API of µManager MMCorePy, but can also use other dedicated Python or web-based APIs provided by vendors, as done for our CellAsic ONIX microfluidic platform.Various types of analysis can be performed using dedicated software modules, such as on-line image analysis or real-time control and optimization.Communication modules can be used to interface MicroMator with digital distribution platforms such as Discord to track experiment progress and potential issues.Lastly, MicroMator leverages Python's multiprocessing module to perform computations concurrently and possesses an extensive and customizable logging system, gathering logs of all modules in a unique file and fostering reproducible research (see SI Text).
We also provide SegMator, a software which uses U-Net for bright-field yeast cell segmentation (provided by DeLTA 19 ), and tracking using TrackPy 21 .U-Net is a convolutional neural network with a structure that excels at image segmentation 20 .U-Net can analyze dense fields of cells in a few seconds and with good accuracy (see SI Text and Fig S1 and S2).
To showcase the full potential of reactive experiments performed with MicroMator, we designed experiments in which cellular processes are controlled in real-time.Single-cell stimulations are computed on line based on the cell state and/or position, demonstrating that reactive loops can be implemented at the level of individual cells.

Model predictive control of gene expression at the single-cell level in yeast
For our first application, we use the EL222 optogenetic system and the mScarletI fluorescent reporter to engineer light-responsive yeast cells (Fig 2A).Our goal is to compare different model predictive control (MPC) strategies for controlling the expression levels of a protein in a cell population.Using real-time imaging, segmentation and cell tracking, different cells can be stimulated differently in the field of view using a digital micromirror device (DMD).The cellular response was then characterized for different light stimulation profiles in the same experiment (Fig 2B).Only the most central part of the cell receives significant light stimulation.This erosion of the stimulation region helps improving the precision of single-cell light stimulations in dense cell regions because of illumination bleedthrough of DMD systems (see SI Text and Fig S3).We then developed and calibrated an "average cell" (deterministic) and a "single cell" (stochastic) model of light-driven gene expression (SI Text and Fig S4 , and S5).
In open loop control, the average cell model is used to precompute a temporal pattern of light stimulation so that cells follow a target behavior.This light pattern is then applied to all cells in the field of view ( Defining control performance as the time averaged deviation to target, we found that the single-cell control method leads to a modest reduction of error of the population averaged fluorescence but to a drastic improvement of the average error of the single-cell fluorescence (Fig 2F ).

Patterns of recombined yeast cells
For our second application, we constructed a light-driven artificial recombination system in yeast and employed different light stimulation strategies to obtain various structures of recombined cells.
We again used the EL222 optogenetic induction system but this time to drive the expression of the Cre recombinase.The Cre recombinase induces the expression of a fluorescent reporter, mCerulean, via an amplification step using the ATAF1 transcription factor (Fig 3A & 3B).
Firstly, we tried to create islets of recombined cells.To this end, we dynamically searched for cells that were far from any previously-targeted cell, and targeted these cells for recombination.To maximize the chances that the chosen cells do recombine, we tracked each chosen cell and targeted it repeatedly with light stimulations.Our strategy was effective in creating isolated micro-colonies of recombined cells (Fig 3C).Analysis of the lineage trees of targeted cells and non-targeted cells confirmed that recombined cells have a slow growth phenotype.
Secondly, we applied a ring-like recombination signal.More specifically, every cell that was in the designated zone at any moment throughout the experiment has been targeted for recombination.As a result, we did obtain a ring-like pattern of recombined cells (Fig 3D).Experimental and biological limitations can been revealed by the analysis of the tails of the distributions of the recombination readout (i.e., mCerulean fluorescence) within the cell populations (Fig 3D ).For example, we found that some cells have been erroneously targeted for recombination because of tracking issues, and that only a few cells have not shown the recombined phenotype at the end of the experiment despite having being effectively targeted for recombination (Fig 3D and S8).

Significance
We presented MicroMator together with two challenging applications demonstrating how this software can help using automated microscopy platforms to their full potential.Each application goes beyond the state of the art.Indeed, we showed the first application of single-cell model predictive control using a full stochastic model of gene expression and the first demonstration of cell recombination driven at the single-cell level, enabling single-cell resolution patterns.
Yet, we foresee that reactiveness in microscopy will primarily be used to enhance and automate classical experiments.Examples of simple use cases abound: triggering autofocus only when needed, dynamically adjusting the imaging condition to the signal strength, identifying novel regions of interest, or following the course of experiments via easily accessible online services (e.g.warning messages sent on Discord), to provide but a few examples.Thanks to its modular nature and to its use of a simple but powerful event system, MicroMator capacities can be conveniently expanded to drive novel hardware or perform a wide range of analyses.MicroMator is a relatively simple software extension that significantly empowers laboratory equipment that is present in most quantitative biology laboratory worldwide.MicroMator is a relatively simple software extension that significantly enhances heavily-used research equipment found in most quantitative biology laboratories.for targeted fluorescence imaging and for optogenetic stimulations.A CellASIC ONIX2 system (Merck) was used together with the Y04C CellASIC microfluidic plates to grow yeast cells in monolayers.Media flow was maintained by a 7.5 kPa pressure gradient.The media was the same as for pre-culture.The temperature was maintained at 30 °C by an opaque environmental box and a temperature controller 2000-2, both from PECON.
The microscope was operated using MicroMator.

Model predictive control of gene expression
To compare single-cell and population control strategies, we developed stochastic and deterministic models of gene expression.Both have been calibrated with respect to the dataset presented in Fig 2B and Fig S4 .For population control, we used the deterministic model, assumed Gaussian measurement noise and used a Kalman filter for state estimation.Each model assumes a deterministic delay between the time the light signal is applied and the time protein production is effective.For MPC, fluorescence measurements were taken every 6 minutes and we considered receding time horizons of 24 minutes.The controller explores the set of light stimulation profiles in which a 1000ms light stimulation is either applied or not for each measurement time interval, and selects the profile minimizing mean square deviations.For tracking purposes, brightfield measurements were taken every 3 minutes.For single-cell control, we used the stochastic model and simulated the cell behavior using a finite state projection approximation.For each and every cell, state estimation is performed using a Bayesian approach which conditions the probability distribution for each cell on the most recent measurement, and light stimulation profiles are selected using the approach outlined above and the expected absolute deviation as selection criterion.More information is provided in SI Text.
Fig 2C).In closed loop population-based control, the average cell model and the average of the measured fluorescence of cells are used by classical state estimators and model predictive controllers to compute in real-time the appropriate light stimulation to drive the mean fluorescence to its target (Fig 2D).Finally, in closed loop single-cell control, a stochastic model of gene expression and single-cell fluorescence measurements are used by advanced state estimators and controllers to compute in real-time the appropriate light stimulations to drive the fluorescence of each and every cell in the field of view to its target (Fig 2E).This control problem is quite challenging and needs to be solved for hundreds of cells in parallel.Advanced methods for numerical simulation and state estimation were essential (see SI Text and Fig S6).

Figure 1 .
Figure 1.MicroMator overview.A. Modular software architecture.MicroMator consists of an extensible set of modules that control various hardware and software aspects of microscopy experiments and of a central core that handles user-defined events.It is written in the high-level programming language Python.B. Event-based reactive microscopy workflow.Imaging can be followed by online analysis of the samples.This typically involves segmentation, tracking, quantification of cell properties, and possibly advanced additional computations.Effects may then be triggered based on the result of the analysis.These may include the physical actuation of the hardware or the initiation of communications or of additional computations.

Figure 2 .
Figure 2. Control gene expression at the single cell level in yeast.A. The red fluorescent protein mScarletI is placed under the control of the light-responsive transcription factor EL222. B. To efficiently characterize cell responses to light stimulations, cells in the field of view are partitioned in 3 groups, each group being stimulated with a different temporal profile.Bright-field images are segmented and cells are tracked.Then, based on their groups, cells are stimulated during the appropriate time with eroded masks.The temporal evolution of the mean mScarletI fluorescence of the cells in the three groups is shown with envelopes indicating one standard deviation.C. Open-loop control experiment in which a model of the response of the cell population is used to precompute a light stimulation profile that drives the cell population to the target behavior.The application of the light profile leads to significant deviations from the target of the individual cell trajectories.D. Closed-loop control experiment in which the same model is used jointly with real-time observations of the population state to decide which light profile to apply to all cells, using a receding horizon strategy.E. A stochastic model of individual cell response is used jointly with single-cell observations to decide which light profile to apply to each cell.F. The different strategies have similar performances to drive the mean fluorescence to its target, but the single-cell feedback strategy is significantly better to drive individual cells to their target profiles.

Figure 3 .
Figure 3. Patterns of recombined yeast cells.A. Upon light exposure, the Cre recombinase is expressed and triggers recombination, leading to the expression of ATAF1 and then of mCerulean-Far1M.Stars indicate nuclear localization of the protein.B. Targeted cells are stimulated for 1 second every 6 minutes until the end of the experiment.Fluorescence levels emitted by targeted cells can be recorded.At the end of the experiment, all cells are imaged and a recombined or nonrecombined phenotype is attributed.C. Cells are dynamically selected such that no target cells are close to each other.Cell lineages of targeted and non-targeted cells can be manually reconstructed and statistics can be extracted.D. A ring-like region in the field of view is selected at the beginning of the experiment and all cells entering the designated region at some time point are targeted for recombination.The distributions of the fluorescence levels of the targeted and non-targeted cells can be computed at the end of the experiment.The vast majority of cells present the expected phenotype and outliers can be further analyzed.