Sylvestre P J T Bachollet

Yuriy Shpinov

Fanny Broch

Hela Benaissa

Arnaud Gautier

Nicolas Pietrancosta

Jean-Maurice Mallet

Blaise Dumat
email: blaise.dumat@ens.psl.eu

An expanded palette of fluorogenic HaloTag probes with enhanced contrast for targeted cellular imaging

Keywords: Fluorogenic probes, molecular rotors, HaloTag, Fluorescence imaging

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Fluorescence imaging has become an invaluable tool to study the structure and dynamics of living systems due to its high sensitivity (down to the single molecule), high temporal and spatial resolution and to its biocompatibility, allowing non-invasive imaging of samples in physiological conditions. Numerous fluorescent molecular probes have been developed over time and organic chemistry offers virtually endless possibilities to tailor fluorophores for each application. [START_REF] Lavis | Bright Ideas for Chemical Biology[END_REF] Nonetheless, since the discovery of the Green Fluorescent Protein (GFP), fluorescent proteins (FPs) have taken a prominent place in the field of bioimaging thanks to the unparalleled targeting selectivity of genetic encoding. A large variety of FPs has since been developed with emission spanning the whole visible spectrum and optimized optical and biochemical properties paving the way to multicolor and complex imaging experiments. [START_REF] Day | The Fluorescent Protein Palette: Tools for Cellular Imaging[END_REF] More recently, hybrid chemogenetic systems have appeared, that associate a genetically encoded self-labeling protein tag with a small molecular fluorophore linked either covalently or non-covalently to the protein. [START_REF] Los | HaloTag: A Novel Protein Labeling Technology for Cell Imaging and Protein Analysis[END_REF][START_REF] Keppler | A General Method for the Covalent Labeling of Fusion Proteins with Small Molecules in Vivo[END_REF][START_REF] Gautier | An Engineered Protein Tag for Multiprotein Labeling in Living Cells[END_REF][START_REF] Hoelzel | Visualizing and Manipulating Biological Processes by Using HaloTag and SNAP-Tag Technologies[END_REF][START_REF] Hori | Development of Fluorogenic Probes for Quick No-Wash Live-Cell Imaging of Intracellular Proteins[END_REF] Such combination benefits from the genetic targeting of FPs and from the structural diversity and versatility of organic chromophores and it offers better spatial and temporal control over the fluorescent labeling compared to fluorescent proteins that are continuously expressed and fluorescent. It may nonetheless suffer from common problems associated with molecular probes, including the risk of off-target binding, yielding non-specific signal, and the need to wash the excess of dye. To alleviate these issues, advanced chemogenetic reporters are built on fluorogenic probes that only become fluorescent when bound to the cognate protein tags. [START_REF] Plamont | Small Fluorescence-Activating and Absorption-Shifting Tag for Tunable Protein Imaging in Vivo[END_REF][START_REF] Bruchez | Dark Dyes-Bright Complexes: Fluorogenic Protein Labeling[END_REF][START_REF] Gao | Development of Fluorogenic Probes for Rapid High-Contrast Imaging of Transient Nuclear Localization of Sirtuin 3[END_REF][START_REF] Szent-Gyorgyi | Fluorogen-Activating Single-Chain Antibodies for Imaging Cell Surface Proteins[END_REF] Considerable efforts have thus been put in the development of fluorogens with diverse optical properties and on their association with protein tags, in particular the most commonly used SNAP-tag [START_REF] Leng | A Wash-Free SNAP-Tag Fluorogenic Probe Based on the Additive Effects of Quencher Release and Environmental Sensitivity[END_REF][START_REF] Yu | Protein Sensing in Living Cells by Molecular Rotor-Based Fluorescence-Switchable Chemical Probes[END_REF][START_REF] Sun | Development of SNAP-Tag Fluorogenic Probes for Wash-Free Fluorescence Imaging[END_REF][START_REF] Jung | A SNAP-Tag Fluorogenic Probe Mimicking the Chromophore of the Red Fluorescent Protein Kaede[END_REF] and HaloTag. [START_REF] Liu | The Cation-π Interaction Enables a Halo-Tag Fluorogenic Probe for Fast No-Wash Live Cell Imaging and Gel-Free Protein Quantification[END_REF][START_REF] Clark | Light-Up "Channel Dyes" for Haloalkane-Based Protein Labeling in Vitro and in Bacterial Cells[END_REF][START_REF] Kang | Structure-Guided Synthesis of a Protein-Based Fluorescent Sensor for Alkyl Halides[END_REF][START_REF] Wang | A General Strategy to Develop Cell Permeable and Fluorogenic Probes for Multicolour Nanoscopy[END_REF][START_REF] Grimm | A General Method to Improve Fluorophores for Live-Cell and Single-Molecule Microscopy[END_REF][START_REF] Bachollet | Fluorogenic Protein Probes with Red and Near-Infrared Emission for Genetically Targeted Imaging[END_REF] The latter is, to date, one of the most useful tag for covalent protein labeling with high selectivity and fast reaction kinetics. Its chemically simple and lipophilic chloroalkane ligand also facilitates the synthesis of the dedicated probes as well as their cell permeability. [START_REF] Wilhelm | Kinetic and Structural Characterization of the Self-Labeling Protein Tags HaloTag7, SNAP-Tag, and CLIP-Tag[END_REF] The HaloTag is derived from a bacterial alkane dehalogenase and was built and optimized (up to the last iteration HaloTag7) to bind xanthene dyes and especially rhodamine derivatives with high reaction rates (k2 = 1.9 x 10 7 M -1 s -1 for TMR-Halo at 25°C). [START_REF] Los | HaloTag: A Novel Protein Labeling Technology for Cell Imaging and Protein Analysis[END_REF][START_REF] Encell | Development of a Dehalogenase-Based Protein Fusion Tag Capable of Rapid, Selective and Covalent Attachment to Customizable Ligands[END_REF] This has driven a lot of the research on HaloTag probes towards rhodamines and, more particularly, fluorogenic Si-Rhodamines taking advantage of the polarity-sensitive spirocyclization reaction. [START_REF] Wang | A General Strategy to Develop Cell Permeable and Fluorogenic Probes for Multicolour Nanoscopy[END_REF][START_REF] Grimm | A General Method to Improve Fluorophores for Live-Cell and Single-Molecule Microscopy[END_REF][START_REF] Wilhelm | Kinetic and Structural Characterization of the Self-Labeling Protein Tags HaloTag7, SNAP-Tag, and CLIP-Tag[END_REF][START_REF] Lukinavičius | Fluorogenic Probes for Multicolor Imaging in Living Cells[END_REF][START_REF] Grimm | A General Method to Fine-Tune Fluorophores for Live-Cell and in Vivo Imaging[END_REF][START_REF] Frei | HaloTag9 : An Engineered Protein Tag to Improve Fluorophore Performance[END_REF] This approach certainly yields highly efficient fluorescent chemogenetic reporters but has narrowed their chemical diversity. Alternative strategies to develop fluorogenic HaloTag probes have relied on the intrinsic viscosity and/or polarity-sensitive emission of dipolar fluorophores. [START_REF] Liu | The Cation-π Interaction Enables a Halo-Tag Fluorogenic Probe for Fast No-Wash Live Cell Imaging and Gel-Free Protein Quantification[END_REF][START_REF] Clark | Light-Up "Channel Dyes" for Haloalkane-Based Protein Labeling in Vitro and in Bacterial Cells[END_REF][START_REF] Fares | A Molecular Rotor-Based Halo-Tag Ligand Enables a Fluorogenic Proteome Stress Sensor to Detect Protein Misfolding in Mildly Stressed Proteome[END_REF] The approach was pioneered by Kool and coworkers who developed a dimethylamino-styrylpyridinium dye with a viscosity-sensitive emission that was activated upon reaction with HaloTag thanks to the immobilization inside the protein binding channel. [START_REF] Clark | Light-Up "Channel Dyes" for Haloalkane-Based Protein Labeling in Vitro and in Bacterial Cells[END_REF] These "molecular rotor" structures allow exploring a much wider chemical space: the fluorescence activation mechanism is applicable to a large variety of dipolar structures with electron-donating and withdrawing groups connected through a flexible conjugated bond and is commonly used to develop biomolecular probes. [START_REF] Hoche | The Origin of the Solvent Dependence of Fluorescence Quantum Yields in Dipolar Merocyanine Dyes[END_REF][START_REF] Xie | Identification of Optimal Fluorescent Probes for G-Quadruplex Nucleic Acids through Systematic Exploration of Mono-And Distyryl Dye Libraries[END_REF][START_REF] Shaya | Turnon Fluorene Push-Pull Probes with High Brightness and Photostability for Visualizing Lipid Order in Biomembranes[END_REF][START_REF] Péresse | Next-Generation Fluorogen-Based Reporters and Biosensors for Advanced Bioimaging[END_REF][START_REF] Klymchenko | Solvatochromic and Fluorogenic Dyes as Environment-Sensitive Probes: Design and Biological Applications[END_REF][START_REF] Dumat | Vinyl-Triphenylamine Dyes, a New Family of Switchable Fluorescent Probes for Targeted Two-Photon Cellular Imaging: From DNA to Protein Labeling[END_REF] Based on this strategy, we have previously reported a series of red and far-red emitting fluorogenic HaloTag probes (Figure 1A). [START_REF] Bachollet | Fluorogenic Protein Probes with Red and Near-Infrared Emission for Genetically Targeted Imaging[END_REF] After reaction with HaloTag, the probes are covalently tethered to the protein and exhibit large enhancements of their fluorescence emissions compared to their free form in water (up to 156-fold). The two most promising fluorogens Red-Halo2 and NIR-Halo1 enabled wash-free imaging of the nucleus and actin in live cells by targeting HaloTag-NLS and HaloTag-LifeAct fusion proteins respectively. We nonetheless observed a low nonspecific signal in the cytoplasm that may prove detrimental when imaging low-abundant or cytosolic proteins.

Herein, taking advantage of the versatile design, we report an expanded palette of HaloTag fluorogens designed to cover a wider emission range from green to red and to minimize the non-specific signal. The new set of probes was characterized in vitro by absorption and fluorescence spectroscopies and in cellular imaging by confocal microscopy. Some new analyses were also performed on the previously reported Red-Halo2 and NIR-Halo1 probes for comparison purposes. The photophysical properties were studied in vitro in glycerol and in presence of HaloTag to confirm their viscosity-dependent emission and fluorogenic reaction with the protein tag. The selectivity of the probes in wash-free imaging experiments was then studied in cells and, taking advantage of the superior contrast of the newly designed probes, we applied them to image of a variety of cellular organelles and proteins.

Results and discussion

Design and synthesis. The previously reported Red-Halo and NIR-Halo probes cover an emissive range from red to far-red. In order to cover a lower wavelength range, we have modified Red-Halo2 to introduce two lesser deficient electron-withdrawing groups, 2-cyanoacetic acid or imidazolinone, to form, respectively, the green and yellow emitting derivatives CCVJ-Halo and Y-Halo (Figure 1B). 2-Cyanoacetic acid gives the CCVJ fluorophore which is a classical molecular rotor with a green emission used as a fluorogenic protein probe and presents a readily available carboxylic acid function for conjugation with the HaloTag ligand. [START_REF] Yu | Protein Sensing in Living Cells by Molecular Rotor-Based Fluorescence-Switchable Chemical Probes[END_REF][START_REF] Iwaki | Antibodies for Fluorescent Molecular Rotors[END_REF][START_REF] Goh | Molecular Rotors as Conditionally Fluorescent Labels for Rapid Detection of Biomolecular Interactions[END_REF] CCVJ was already coupled to HaloTag but with a different ligand unit which does not readily make the probe fluorogenic upon reaction and was instead used to detect protein aggregation. [START_REF] Fares | A Molecular Rotor-Based Halo-Tag Ligand Enables a Fluorogenic Proteome Stress Sensor to Detect Protein Misfolding in Mildly Stressed Proteome[END_REF] Imidazolinone is the electron withdrawing group found in the GFP chromophore and, in aminated derivatives, it was reported to afford yellow-emitting compounds. [START_REF] Baranov | Red-Shifted Fluorescent Aminated Derivatives of a Conformationally Locked GFP Chromophore[END_REF][START_REF] Ye | A General Strategy to Control Viscosity Sensitivity of Molecular Rotor-Based Fluorophores[END_REF] In addition to the emission color-tuning, the second goal was to minimize the unspecific signal observed with Red-Halo2 and NIR-Halo1 due to the partial activation and/or accumulation of the fluorogens in cytoplasmic organelles. Factors governing the non-specific binding of a dye to intracellular components are difficult to predict and depend in part on the net charge (e.g. positively charged compounds binding to mitochondria) and on the lipophilic/hydrophilic balance. The Halo dyes are neutral structures with poor water solubility and we hypothesized that their lipophilic character is one of the driving force of the non-specific binding by leading the probes to accumulate into lipid structures such as the endoplasmic reticulum (ER) membrane. We have thus set out to develop analogs of Red-Halo2 with lower lipophilicity. As a first strategy, we substituted the aniline ring of Red-Halo2 with an ethyl or a triethyleneglycol chain through an ester coupling to afford Red-Halo2-Et or Red-Halo2-PEG respectively (Figure 1B). The ethyl ester is expected to induce only a minute decrease in the lipophilicity of Red-Halo2, thanks to the polar ester function, while the addition of a PEG chain is known to increase the hydrophilicity. In a second approach, we have introduced an azetidine donating amino group instead of the julolidine (Figure 1). Azetidine was previously reported as an efficient locked amino donating group since the 4-membered cycle strain prevents the formation of a "twisted intramolecular charge transfer" (TICT) state and it may advantageously replace julolidine while being slightly less lipophilic. [START_REF] Grimm | A General Method to Improve Fluorophores for Live-Cell and Single-Molecule Microscopy[END_REF] Azetidine is also less electron-donating than julolidine and yielded the blue-shifted orange-emitting Orange-Halo. Using inexpensive building blocks and reagents, the probes can be obtained in straightforward 2 to 4 steps synthetic procedures detailed in the supplementary information (Scheme S1-S4).

Photophysical properties and reaction with HaloTag.

With their flexible dipolar structures, all probes display typical molecular rotor features and we have previously evidenced the fluorogenic behavior of Red-Halo2 and NIR-Halo1 in presence of proteins. [START_REF] Bachollet | Fluorogenic Protein Probes with Red and Near-Infrared Emission for Genetically Targeted Imaging[END_REF] We have nonetheless confirmed the molecular rotor properties (i.e. the viscosity-dependent emission) for the whole series of probes (of the two ester derivatives Red-Halo2-PEG and Red-Halo2-Et, only the latter was studied) by measuring their spectral properties in water and in glycerol (Figure S1 and Table 1). As expected from the rational design, all probes are weakly fluorescent in water but their emission is strongly enhanced (from 28 to 390-fold) in glycerol with quantum yields ranging from 9 to 20 %, in accordance with an increase in viscosity (Table 1). The largest fluorescence enhancement in glycerol is observed for Orange-Halo and is due to the combination of a shift in the excitation wavelength with the fluorescence activation: although the azetidine is slightly less electron-donating than the julolidine, Orange-Halo is expected to have spectral properties close to those of Red-Halo2 but its absorption in water is unexpectedly blue-shifted (409 nm). In glycerol however, a large bathochromic shift is observed resulting in an absorption spectrum more akin to the parent Red-Halo2 (Figure S1, Table 1). The solubility in water was assessed by UV-vis absorption spectroscopy. Between 0.5 and 20 µM, the probes obey the Lambert-Beer law with the exception of Red-Halo2-Et and NIR-Halo1 which show solubility limit around 13 µM and 1 µM respectively (Figure S2A&B). Beyond a 1 µM concentration, the absorption spectrum of NIR-Halo1 exhibits a sharp blue-shifted absorption band that may be attributed to aggregation as is observed for H-aggregates in cyanine dyes (Figure S2C). The unexpectedly blue-shifted absorption of Orange-Halo in aqueous buffer may suggest that it also forms soluble aggregates, but the transition between the free and aggregated form is not visible in the studied concentration range. The viscosity-sensitive emission and good fluorescence brightness observed in glycerol confirm the potential of the probes to be used as fluorogenic biomolecular probes and we next studied the reaction of the probes with a GST-HaloTag (GST-HT) protein in vitro and the resulting fluorescence properties of the formed complexes (Figure 2 and Table 1). All probes undergo a fluorogenic reaction reaction with GST-HT, which allowed us to measure the reaction rate constants by following the increase of fluorescence over time (Figure 2A&B). With the exception of Y-Halo, the reactions obey a classical pseudo-first-order exponential law and the calculated second order reaction rate constants are comprised between 0.29 10 3 and 5.3 10 3 M -1 s -1 (Table 1). The slowest reaction rates were observed for Orange-Halo and Red-Halo2-Et which can be explained in part by solubility issues: the more hydrophilic Red-Halo2-PEG displays the fastest reaction rate, an order of magnitude higher than that of its ethyl ester counterpart. The possible aggregation of Orange-Halo in water may also explain its slow reaction kinetics. The reaction with Y-Halo is almost instantaneous with an immediate sharp increase of fluorescence that is followed by a slower decrease of intensity that stabilizes after a few minutes (Figure 2B). It is possible that Y-Halo undergoes a first non-covalent association with the protein yielding a highly fluorescent complex followed by the covalent enzymatic reaction that leads to a compound with lower fluorescence. Alternatively, it may also adopt several conformations with different emissive properties once bound to the protein. Apart from Y-Halo, the reaction rates of the probes are slower than that of rhodamine-based HaloTag ligands but they are in the same range or faster than that observed for other reported solvatochromic HaloTag probes [START_REF] Liu | The Cation-π Interaction Enables a Halo-Tag Fluorogenic Probe for Fast No-Wash Live Cell Imaging and Gel-Free Protein Quantification[END_REF][START_REF] Kang | Structure-Guided Synthesis of a Protein-Based Fluorescent Sensor for Alkyl Halides[END_REF] and for alternative self-labeling protein tags such as SNAP-tag, CLIP-tag and PYP-tag. [START_REF] Keppler | A General Method for the Covalent Labeling of Fusion Proteins with Small Molecules in Vivo[END_REF][START_REF] Gautier | An Engineered Protein Tag for Multiprotein Labeling in Living Cells[END_REF][START_REF] Gao | Development of Fluorogenic Probes for Rapid High-Contrast Imaging of Transient Nuclear Localization of Sirtuin 3[END_REF][START_REF] Leng | A Wash-Free SNAP-Tag Fluorogenic Probe Based on the Additive Effects of Quencher Release and Environmental Sensitivity[END_REF] After reaction with HaloTag, the spectral properties of the bound probes are essentially similar to those in glycerol but with lower fluorogenicities (Figure 2C&D, Table 1). CCVJ-Halo and Y-Halo display comparable 12 to 15-fold fluorescence activation and while Red-Halo2-Et, Red-Halo2-PEG and Orange-Halo perform slightly better, the five new probes have lower fluorescence enhancement factors than their parent compound Red-Halo2 (156-fold). These lower values are due to their lower fluorescence brightnesses but also, for some compounds and in particular Y-Halo, to higher residual fluorescence in water (Figure S3). Despite a similar fluorescence activation mechanism, the probes display large differences in fluorogenicities and fluorescence quantum yields that do not correlate with the relative properties observed in glycerol (Table 1). The green and yellow probes in particular are dimmer in HaloTag than in glycerol and the two-step binding reaction of Y-Halo may for instance indicate that, of several possible conformations within the protein, the most stable is not necessarily the most emissive.

Molecular dynamics. To better understand the properties of the hybrid reporters, we performed molecular dynamics on the fluorogen/HaloTag complexes over 1 ns with a 10 ps step, generating 100 poses that were clustered to identify representative structures of the various possible conformations (Figure 3 and Figure S4). Similar calculations were previously performed for Red-Halo2 and NIR-Halo1. [START_REF] Bachollet | Fluorogenic Protein Probes with Red and Near-Infrared Emission for Genetically Targeted Imaging[END_REF] The calculations yielded at least two clusters for each complex, showing that the fluorogens are indeed able to adopt several conformations within HaloTag. The results also highlight that the probes have few interactions with the protein and remain in contact with the solvent. Consequently, they retain an important structural mobility as evidenced by the RMSF values of the fluorogen moieties (Figure 3). The main interaction between HaloTag and the fluorogens occurs with the F144 residue via π-stacking with the aniline ring or with the electron-withdrawing group. That interaction is however not always favorable and can lead to twisted non-planar conformations that are not favorable for fluorescence emission. Overall, these observations explain the differences in properties observed between glycerol and HaloTag: the probes are not fully immobilized within the protein and there is only a partial activation due to viscosity and hydrophobic effects on the surface of the protein. This partial activation was relatively efficient for Red-Halo2, with little difference between HaloTag and glycerol but less so for other derivatives and in particular for CCVJ-Halo, Y-Halo and Orange-Halo. This suggests that the optimization of the interaction with HaloTag, through variation of the ligand length and/or mutation of the binding site, may lead to much brighter complexes. Biophysical and photophysical characterization in cellular imaging. Even with a targeted fluorogenic probe, some nonspecific signal is inevitable and can arise from the spontaneous accumulation and partial activation of the excess probe in cellular organelles with a favorable hydrophobic environment and/or increased viscosity (e.g. ER, membrane). This non-specific signal will grow stronger with the probe concentration but will also depend on the relative amount of expressed HaloTag protein which will vary from one experiment to another depending on the transfection level. Some cytosolic signal can also originate from the specific reaction of the probe with newly synthesized proteins not yet addressed. To demonstrate the applicability of our HaloTag fluorogens, we evaluated and compared their selectivity and imaging contrast in wash-free live-cell imaging experiments. Hela cells transiently expressing the HaloTag-NLS protein were stained with 0.5 µM of Halo probe and were imaged after 30 to 60 minutes without washing steps or medium change. After incubation, the cells showed a good viability with no sign of cytotoxicity and all four new probes were successful in selectively staining the nucleus (Figure 4A). For each fluorogen, we have compared the specific signal (S) in the nuclei with the nonspecific signal (NS) in the cytoplasm of an adjacent untransfected cell. The contrast ratio (S/NS) is given on each panel of figure 4A. For a more precise analysis, we have calculated the contrast on a larger set of cells from several independent experiments and the distribution of the results is given as a box plot (Figure 4B and Table S1). The specific signal (and thus the related contrast) is highly dependent on the expression level from one cell to another, which causes a large dispersion of the values, but average and maximum contrast values inform on the relative performance of the probes. Amongst the Red-Halo2 analogs, the azetidine derivative Orange-Halo and Red-Halo2-Et give an improved contrast compared to the parent compound Red-Halo2 but not the more hydrophilic Red-Halo2-PEG. These observations are confirmed by the comparison of the fluorescence signal generated by each analog in wild-type Hela cells under identical imaging conditions (Figure S5). Red-Halo2-PEG yields the same non-specific cytoplasmic signal as Red-Halo2 while that of Orange-Halo and Red-Halo2-Et is respectively 3 and 6-fold lower which is well in line with the relative distribution of the S/NS ratios. The diverging properties of Red-Halo2-PEG (higher brightness, faster reaction but lower imaging contrast) and Red-Halo2-Et (slower reaction and higher contrast) show that the substitution of the aniline ring is a promising but not easily rationalized strategy to improve the properties of the Halo probes. And, contrary to our initial assumption, the non-specific fluorescence signal in cells is not directly or at least not only linked to the hydrophilic/lipophilic balance of the probes. The green and yellow emitting derivatives, CCVJ-Halo and Y-Halo also exhibit improved S/NS ratios that reach remarkably high values for the latter. Y-Halo yields almost no recordable unspecific signal which, despite its moderate brightness once bound to HaloTag, makes it a very promising fluorogenic reporter. To further evidence the increased contrast of the new probes we have successfully carried out wash-free imaging with the two most promising probes Y-Halo and Red-Halo2-Et at higher concentrations of 1 and 5 µM (Figure S6). At 1 µM, both probes still yield very contrasted images and, at 5 µM, the nonspecific signal becomes visible but the contrast remains satisfactory. These results stress the difficulty of designing efficient fluorogenic probes and the necessity to balance in vitro and cellular properties: a high fluorescence enhancement factor in vitro is an important figure of merit for a fluorogen, yet it does not always translate into a high contrast in cellular imaging. Here, despite its superior optical properties measured in cuvette, Red-Halo2 is also the most prone to off-target activation in cells yielding a higher non-specific signal. On the other hand, the new probes, with the exception of Red-Halo2-PEG, exhibit improved contrast values that are not directly correlated to the relative brightness or fluorescence enhancement values measured in vitro. We finally investigated the photostability of the probes in confocal microscopy and compared it to that of Halo-TMR, a commercial tetramethylrhodamine fluorescent HaloTag ligand. Under continuous illumination for 10 to 15 minutes, the HaloTag fluorogens exhibited a good photostability which appears related to the nature of the electron withdrawing group. All compounds derived from Red-Halo2 with a rhodanine group suffered higher photobleaching while the CCVJ and imidazolinone derivatives exhibited an excellent photostability higher than that of Halo-TMR (Figure 5). The poorest photostability was observed for NIR-Halo1, which can be explained by the extended π-system and the presence of a dimethylamino electron-donating group instead of the julolidine. Overall, the HaloTag fluorogens are robust fluorophores able to sustain high excitation intensities with low photobleaching, which compensates partly for their moderate brightness. Targeted subcellular imaging. In order to take advantage of their increased contrast, we have used the palette of novel HaloTag probes to image of variety of cellular components in addition to the nuclear labeling discussed above (Figure 6 and Figure S7). The parent compound Red-Halo2 remains efficient to image the nucleus via the targeting of Histone 2B or the cytoskeleton (microtubules). Although microtubule bundles are clearly visible, the labeling of microtubules using MAP4-HaloTag also yielded a significant cytoplasmic signal (Figure 6D and Figure S7). The plasmid is derived from the MAP4-FAST plasmid that generated a satisfactory microtubules labeling [START_REF] Plamont | Small Fluorescence-Activating and Absorption-Shifting Tag for Tunable Protein Imaging in Vivo[END_REF] and it is possible that the larger HaloTag protein interferes with the microtubule binding. Except for this limitation, the new fluorogens proved successful in imaging the endoplasmic reticulum (ER), the golgi apparatus, H2B, the outer membrane and actin. In addition to providing a wider range of excitation and emission wavelength, the new probes, thanks to their improved selectivity, enabled the imaging of additional cellular targets, including cytoplasmic organelles.

Multicolor imaging. CCVJ-Halo, Y-Halo and Orange-Halo display absorption spectra centered on 458 nm, 489 nm and 487 nm respectively, which allows for an optimal excitation with the 458 and 488 nm Argon laser lines commonly available on microscopes. The Red-Halo2 derivatives and NIR-Halo1 can be efficiently excited with the 488 or 514 nm laser lines. Due to their flexible structures that increase the number of available vibrational energy levels, the Halo probes inherently display broad emission peaks. The more flexible structure NIR-Halo1 displays the broadest peak, with a FWHM of 68 nm and the emission peaks of the Red-Halo2 analogs have comparable, albeit slightly narrower, breadth (Table 1). Although considerable progress has been made in the analysis and spectral deconvolution of multicolor images, these large emission ranges may render their discrimination from other fluorophores more difficult in multiplexed experiments. On the other hand, the green and yellow derivatives, CCVJ-Halo and Y-Halo, exhibit narrower emission peaks with FWHM of 54 and 50 nm that could make them more easily applicable in multicolor imaging. As an example, we have successfully combined Y-Halo and Mitotracker Deep Red FM in Hela cells expressing Halo-NLS (Figure 7). Both probes can be selectively excited and their emission collected without any cross-talk.

Conclusion

The fluorogenic HaloTag probes reported here enrich a palette of hybrid chemogenetic reporters with good photostability, low nonspecific signal and diversified spectral properties that cover the green to far-red emission range. Despite these promising results, comparison with glycerol and theoretical studies show that the molecular rotors are only partially activated after reaction with the protein by lack of a defined hydrophobic binding pocket. Recent reports suggest that discrete mutations of HaloTag may be performed to better fit solvatochromic or molecular rotor probes. [START_REF] Kang | Structure-Guided Synthesis of a Protein-Based Fluorescent Sensor for Alkyl Halides[END_REF][START_REF] Miró-Vinyals | HaloTag Engineering for Enhanced Fluorogenicity and Kinetics with a Styrylpyridium Dye[END_REF] In parallel to rational molecular design, engineering of the HaloTag protein may thus be envisioned to improve the association with the fluorogens and reach higher quantum yields and faster reaction kinetics.

Experimental procedures

Probes synthesis and characterization. Detailed synthetic procedures and chemical analyses are available in the supplementary information.

Photophysical properties with HaloTag. The fluorogenic dyes were dissolved in DMSO at a stock concentration of 0.5 mM. The interaction of the probes with Halotag was assessed by incubating the dye at 1 µM (0.4 µL of stock solution) concentration with 1.3 µM (ca. 6 µL of commercial stock solution) of protein in a 200 µL working volume of pH 7.4 Phosphate buffer (10 mM phosphate with 100 mM NaCl). Fluorescence spectra were recorded at regular time intervals until completion of the reaction. The different photophysical properties of the protein-bound probes were then measured.

Kinetics study of the labeling reactions.

The second-order rate constants of the reactions were calculated by monitoring the fluorescence intensity during the reaction of the probes with a large excess of protein to assume a pseudo first-order kinetics law according to the following procedure. GST-HT (2 µM) was added to a 50 µL solution of Halo probe (100 nM) in pH 7.4 Phosphate buffer (10 mM Phosphate, 100 mM NaCl). The fluorescence intensity (excitation and emission at the maximum absorption and emission wavelengths of each probe) was then recorded over time until completion of the reaction with a 5 s to 20 s step depending on the probe. The second order rate constants were then calculated according to a previously reported model: [START_REF] Hori | Development of Fluorogenic Probes for Quick No-Wash Live-Cell Imaging of Intracellular Proteins[END_REF] the fluorescence was converted to the labeled fraction according to the following equation:

Labeled fraction = (F-F0)/(Fmax-F0)

Where Fmax and F0 are the maximum and initial fluorescence intensity respectively. Using Origin 2019 software, the data was fitted to the following equation to calculate the pseudo first-order rate constant kobs: Labeled fraction = 1-exp(-kobst) The second order rate constant k2 was then obtained by dividing kobs with the protein concentration. Molecular dynamics. Molecular dynamics are performed using similar conditions than previously described. [START_REF] Bachollet | Fluorogenic Protein Probes with Red and Near-Infrared Emission for Genetically Targeted Imaging[END_REF] Briefly, 3D models of HaloTag coupled to the fluorogen compounds were generated based on a 3D structure of Halotag (pdb entry: 4KAF), refined, minimization and equilibrated by molecular dynamics simulations (1-ns runs) using the CHARMm forcefield. Bonds involving hydrogen atoms were constrained using the SHAKE algorithm.

[48] The Ramachandran plot of the final 3D model showed 88% ( 259/295) favored, 7% (21/295) allowed, 5% (15/295) disallowed residues for Red-Halo2-Et and, respectively, 88% ( 259/295), 8% (24/295), 4% (12/295) for Orange-Halo, and 85% ( 251/295), 11% (33/295), 4% (11/295) for Y-Halo, and 86% ( 265/295), 9% (28/295), 5% (16/295) for CCVJ-Halo. A solvent box with a periodic boundary of 9 Å was added to the model. Solvation was completed with 0.145 M KCl using the solvation protocol implemented in Discovery Studio (Dassault Systèmes BIOVIA, Discovery Studio, 2020, San Diego: Dassault Systèmes, 2019). Molecular Biology. Detailed construction of the plasmids used in this study are described in the supplementary information.

Confocal microscopy. Hela cells were grown overnight on 8-well polymer µslides from Ibidi (#1.5 polymer coverslip, tissue culture treated) at 20k to 50k cells/well in 300 µL of MEM culture medium (Gibco) supplemented with 10% fcs, sodium pyruvate and non-essential amino acids. Cells were then transfected with the desired plasmid using Fugene 6 (Promega Corp.) according to the manufacturer's protocol (200 ng of DNA per well with a 3:1 fugene to DNA ratio). After 24 hours, the medium was changed and replaced with DMEM (no phenol red) and the cells were incubated 30-60 minutes with the dyes and imaged live on a Zeiss LSM710 laser scanning confocal microscope equipped with a Plan apochromat 40X/1.4 NA objective. For photostability measurements, images were acquired under a continuous illumination of 37.5 kW/cm 2 measured in the focal plane over 15 minutes at a resolution of 1024x1024 and a pixel dwell time of 0.52µs. Images were acquired using Zen 2009 software and then processed using ImageJ.

Multicolor imaging with mitotracker Deep Red FM and Y-Halo. Cells were grown according to the protocol above and then labeled Mitotracker Deep Red FM according to the manufacturer protocol. Briefly, Mitotracker was diluted at 100 nM in DMEM (no phenol red) and the cells were incubated for 30 min with the resulting solution. The medium was changed to pure DMEM and the cells were then incubated with Y-Halo according to the protocol above. For imaging conditions, see the caption of Figure 5.

Figure 1 .

 1 Figure 1. Design and structures of the fluorogenic HaloTag probes. (A) structures of two of the previously reported probes with their fluorescence emission spectra and fluorescence enhancement upon reaction with HaloTag and general schematic description of the fluorogenic reaction with HaloTag. (B) Structural modifications and structures of the new probes.

Figure 2 .

 2 Figure 2. Reaction of the Halo probes with GST-HaloTag. (A,B) Reaction kinetics of the Halo probes with GST-HT. [dye] = 100 nM, [GST-HT] = 2 µM, T = 298 K. Data was fitted to an exponential rise function (red lines) with the exception of Y-Halo. Normalized absorption (C) and emission (D) of the Halo probes bound to HaloTag. [Dye] = 1 µM, [GST-HT] = 1.3 µM.

Figure 3 .

 3 Figure 3. Molecular dynamics of the HaloTag/fluorogen complexes. Representative structures of the major conformation cluster and root mean square fluctuations "RMSF" of the structures.

Figure 4 .

 4 Figure 4. Selectivity and contrast of the Halo probes in cellular imaging. (A) Live confocal imaging of Hela cells transiently expressing a HaloTag-NLS protein and incubated with 0.5 µM of fluorogenic Halo probe. On each panel is displayed the contrast ratio between the average nuclear intensity and the average cytoplasmic signal measured in a nontransfected cell. λexc = 514 nm except for CCVJ-Halo (458 nm) and Y-Halo (488 nm). Scale bar: 20 µm (B) Ratio of the specific (S = average intensity in the nucleus) over the non-specific signal (NS = averaged intensity in a cytosolic region of a non-transfected cell). Box and whiskers plot with center lines and squares showing the median and average values for n cells respectively. Box delimits the 25 th and 75 th percentiles and the whiskers show the max and min values. CCVJ-Halo: n = 64 from four experiments, Y-Halo: n = 91 from three experiments, Orange-Halo: n = 91 from three experiments, Red-Halo2-Et: n = 94 from four experiments, Red-Halo2-PEG: n = 92 from two experiments, Red-Halo2: n = 54 from three experiments, NIR-Halo1: n = 72 from four experiments.

Figure 5 .

 5 Figure 5. Photostability of the HaloTag fluorogens. Evolution of the normalized intensity in the nuclei of Hela cells transfected with HaloTag-NLS and incubated with 0.5 µM of HaloTag fluorogens under continuous illumination for 10 to 15 minutes. The photostability of Halo-TMR was also measured for comparison purposes.

Figure 6 .

 6 Figure 6. Targeted organelles and proteins imaging. Live wash-free confocal imaging of Hela cells transiently expressing H2B-HaloTag (nucleus), MAP4-HaloTag (microtubules), PDGFR-HaloTag (external membrane), ER-HaloTag (endoplasmic reticulum), HaloTag-Lifeact (actin) or Golgi-HaloTag (Golgi apparatus) and incubated with 0.5 µM of CCVJ-Halo (A), Y-Halo (B) or Orange-Halo (C) Red-Halo2-Et (D,E) and Red-Halo2 (F). Each panel diplays the fluorescence image (left) and the merged fluorescence and brightfield images (right). Non-transfected cells are visible on the merged images to evidence the selective labeling. λexc = 514 nm except for CCVJ-Halo (458 nm) and Y-Halo (488 nm) Scale bar: 10 µm.

Figure 7 .

 7 Figure 7. Multicolor imaging. Live confocal imaging of Hela cells expressing HaloTag-NLS and incubated with 100 nM of Mitotracker Deep Red FM and 0.5 µM of Y-Halo. (A) yellow channel λexc = 488, collection: 495 -600 nm, (B) red channel λexc = 633, collection: 640 -799 nm, (C) transmission and (D) composite image. Scale bar: 30 µm.

Table 1 . Photophysical properties of the Halo probes in glycerol and in presence of HaloTag

 1 . λabs and λem: maximum absorption and emission wavelength, ε: molar absorptivity coefficient, FWHM: Full width at Half maximum of the emission spectrum, ΦF: fluorescence quantum yield, ε•ΦF: brightness, k2: second order reaction rate constant, F/F0: Fluorescence enhancement factor where F and F0 are the integrated fluorescence intensities in glycerol or HaloTag and in water respectively. Measurement conditions: 1 µM dye with 1.3 µM GST-HT in pH 7.4 phosphate buffer containing 10 mM Phosphate and 100 mM NaCl.

	Molecule	Solvent/	λabs	ε	λem	FWHM	ΦF	ε•ΦF	k2 c	F/F0
		target	(nm)	(10 3 M -1 cm -1 )	(nm)	(nm)			(10 3 M -1 s -1 )
		Glycerol	455	33.9	495		0.19	6441		129
	CCVJ-Halo	HaloTag a	458	31.8	498	54	0.014	445	1.0 ± 0.1	15
		buffer	443	22.7	505
		Glycerol	487	26.2	556		0.099 2594		28
	Y-Halo	HaloTag a	489	34.9	562	50	0.015	524	n.d.	12
		buffer	483	24.3	548
		Glycerol	486	15.2	578		0.14	2128		390
	Orange-Halo	HaloTag a	487	16.2	574	64	0.020	324	0.45 ± 0.03	48
		buffer	413	18.5	590
		Glycerol	520	14.9	595		0.086 1281		32
	Red-Halo2-Et	HaloTag a	504	14.5	588	65	0.050	725	0.29 ± 0.04	32
		buffer	490	14.7	601
	Red-Halo2-PEG	HaloTag a buffer	525 530	32.7 14.4	599	61	0.055 1800	5.3 ± 0.3	22
		Glycerol	521	22.9	602		0.20	4580		298
	Red-Halo2 b	HaloTag a	515	21.0	592	60	0.17	3600	1.3 ± 0.2	156
		buffer	483	14.8	609
		Glycerol	527	27.5	664		0.091 2503		225
	NIR-Halo1 b	HaloTag a	526	25.8	671	68	0.053 1400	1.7 ± 0.04	130
		buffer	480	22.8	684
	a

b Data taken from reference 21 except for k2 values. c Average of duplicate experiments.

Acknowledgements

This work was supported by the Agence Nationale de la Recherche (ANR-18-CE44-0006) and the European Research Council (ERC-2016-CoG-724705 FLUOSWITCH).

Supplementary information

Supplementary figures, materials and methods, chemical synthesis and molecular biology procedures are available in supplementary information.


Theodore Aouad

Hugues Talbot

Binary Multi Channel Morphological Neural Network

Keywords: Binary morphology, deep learning, binarized neural networks

Neural networks and particularly Deep learning have been comparatively little studied from the theoretical point of view. Conversely, Mathematical Morphology is a discipline with solid theoretical foundations. We combine these domains to propose a new type of neural architecture that is theoretically more explainable. We introduce a Binary Morphological Neural Network (BiMoNN) built upon the convolutional neural network. We design it for learning morphological networks with binary inputs and outputs. We demonstrate an equivalence between BiMoNNs and morphological operators that we can use to binarize entire networks. These can learn classical morphological operators and show promising results on a medical imaging application.

Introduction

While demonstrating considerable success in applications, there are few theoretical results in Deep Learning. Many elements are not well understood, with networks operating as black boxes, hindering critical applications such as medical imaging or robotics. Conversely, Mathematical Morphology (MM) is a computer vision discipline with solid theoretical foundations. In this article, we propose to combine ideas from the two domains to construct a new type of neural architecture that is theoretically more justified and explainable. As argued in [START_REF] Angulo | Some open questions on morphological operators and representations in the deep learning era[END_REF] combining the two fields is promising, as MM can se used to construct simpler and more understandable deep neural networks. Using MM is more natural than convolutional neural networks (ConvNets) for some specific tasks. For example, ConvNets are not designed to deal with binary images, while MM is constructed on set-representation of images. Also, as stated in [9], designing a correct sequence of morphological operators can be complicated and time-consuming.

Past researchers have proposed to learn both operators and structuring elements, e.g using the max-plus definition of dilations and erosion [12,[START_REF] Franchi | Deep morphological networks[END_REF]. Other approaches introduce differentiable approximations of the max and min operators, such as the adaptive morphological layer [14], the PConv layer [START_REF] Masci | A learning framework for morphological operators using counter-harmonic mean[END_REF], and the most recent LM orph and SM orph layers [9]. However, these methods deal with grey-scale morphology and grey-scale images.

We introduces the Binary Morphological Neural Network (BiMoNN), a learnable morphological network with binary inputs and output, inspired by the structure of ConvNets. We replace the convolution by elementary operations from the set of erosions, dilations, anti-erosions, and anti-dilations: we can learn the type of operation and its associated structuring element as well as any intersection/union of these elementary operations. we show equivalence properties between our neural network and morphological operators. They can be used both as explainable results or as a way to binarize the network: once these equivalences are respected, the morphological operation can replace the neural network. Booleans weights can replace real-valued weights, improving the inference's computing efficiency [START_REF] Hubara | Binarized neural networks[END_REF][START_REF] Kim | Bitwise neural networks[END_REF]15]. We demonstrate how we managed to learn some classical morphological operators as well as an intermediate step of a challenging medical imaging problem. A preliminary version of our work is available at [START_REF] Aouad | Binary morphological neural network[END_REF]. Our code is publicly available online at https://github.com/TheodoreAouad/Bimonn_DGMM2022.

Method

Notations

Let N ˚be the set of non zero natural integers. Let d P N ˚the dimension of the image (usually d " 2 or d " 3). If a ď b P Z, we denote the discrete interval as rra, bss " ra, bs X Z. Let N I P N ˚and Ω I " rr´N I , N I ss d be the support of binary images I Ă Ω I . Let Ω S " rr´N S , N S ss d be the support of structuring elements (short: SE) S Ă Ω S . If X Ă Z d , we denote X " t´x | x P Xu the symmetric of X with respect to the origin. For a binary image I Ă Ω I , we define its complementation I C " Ω I z I. For a SE S Ă Ω S , if there is no ambiguity, we define its complementation as S C " Ω S z S. For a set X Ă Z d , we denote its indicator function 1 X : Z d Þ Ñ R such that 1 X pxq " 1 if x P X, else 1 X pxq " 0. If 1 I P R Ω I and 1 S P R Ω S we denote the convolutional product

1 I f 1 S : k P Ω I Þ Ñ ř iPΩ S |k´iPΩ I p1 I pk
the white top-hat by S T w pI, Sq " I z pI ˝Sq " I X pI ˝Sq C

While not exhaustive, this list is representative of many useful morphological operators. These operators are obtained through composition and intersection or union of dilations, erosions, anti-dilations, and anti-erosions. Therefore, it is enough to learn these elementary operations, their SEs, and their aggregation.

Binary Structuring Element Neuron

First, we design a single neuron that replaces the convolution operation: it can learn the dilation, erosion, anti-dilation, anti-erosion, and their SEs.

Definition

Proposition 1 shows that we can express a dilation and erosion exactly using a thresholded convolution.

Proposition 1 (Morphological operators from convolution). Let S Ă Ω S be a binary SE and X Ă Ω I be a binary image.

X ' S " ˆ1X f 1 S ě 1 ˙" ˆ1X f 1 S ´1 ě 0 ˙(9) X a S " ˆ1X f 1 S ě |S| ˙" ˆ1X f 1 S ´|S| ě 0 ˙(10)
We stress that the dilations and erosions only differ by a scalar, 1 for the dilation and |S| for the erosion. Given S, we can learn the operation using only this scalar. Taking inspiration from these expressions, we relax S into real weights W P R Ω S . We denote the softplus function by f `: x P R Þ Ñ lnp1 `exppxqq. Let ξ : R Þ Ñ r0, 1s be a smooth increasing function such that ξpxq Ñ xÑ´8 0 and ξpxq Ñ xÑ`8 1. In practice, ξpxq " 1 2 tanhpxq `1 2 . Definition 2 (BiSE neuron). Let W P R Ω S be a weight matrix, b P R a bias and p P R a scaling number. We define a BiSE (Binary Structuring Element) neuron as follow:

ϵ W,b,p : x P r0, 1s Z d Þ Ñ ξpprx f f `pW q ´f `pbqsq P r0, 1s Z d (11)
This expression approximates the thresholded convolution of proposition 1. Indeed, we apply the convolution and subtract a bias. If this expression is negative, we want the result to be 0, else we want the result to be 1. Therefore, we multiply it by a scaling factor and threshold it. The higher |p| is, the closer the output is to 0 or 1. We enforce the weights and bias to be positive: we will explain that later.

The BiSE neuron is a convolution layer with the weights and bias forced to be positive, with a smooth thresholding function as activation. In practice, all the operations are differentiable with real inputs and outputs, not binary. Thus, we define almost binary images, with pixels value either close to 0 or close to 1.

Definition 3 (Almost Binary Image

). We say an image I P r0, 1s Ω S is almost binary if there exists u ă v P r0, 1s such that IpΩ S qXsu, vr" H. We denote this set Ipu, vq.

Morphological equivalence

The following Theorem states the conditions ensuring that the BiSE neuron is equivalent to a morphological operator. We denote ϵ W,b,`8 : x P r0, 1s

Ω I Þ Ñ px f W ą bq P Ω I and ϵ W,b,´8 : x P r0, 1s Ω I Þ Ñ px f W ă bq P Ω I .
Theorem 1 (Dilation / Erosion Equivalence). Let W P R Ω S be a set of weights, b P R a bias and p P R a scaling factor. Let S Ă Ω S be a candidate SE. Given an almost binary input in Ipu, vq ϵ W,b,`8 is a dilation by S if and only if ϵ W,b,´8 is an anti-dilation by S if and only if

ÿ iPS C , wiě0 w i `u ÿ iPS , wiě0 w i ď b ă v min iPS w i `ÿ iPΩ , wiď0 w i (12) -ϵ W,
w i ´p1 ´uq min iPS w i ď b ă v ÿ iPS , wiě0 w i `ÿ iPΩ , wiď0 w i (13)
If one of these expressions is fulfilled, we say that the BiSE neuron is activated. Then we have @i P S , w i ě 0 and b ě 0, and the output of the BiSE ϵ W,b,p is almost binary.

Equations 12 and 13 check if a candidate S Ă Ω S activates the BiSE or not. If the BiSE is activated, given an almost binary input, the output is almost binary. Therefore, applying a BiSE on an almost binary image is the same as applying the morphological operator to the binary image. Finally, we can replace the entire BiSE neuron with the associated morphological operation in the activation case. The sign of p determines if complementation is applied or not.

Theorem 1 justifies a posteriori our enforcing of the weights and bias to be positive in the BiSE definition 2: the goal is to achieve activation, which is only possible for positive parameters.

The BiSE can only be activated by at most one S Ă Ω S simultaneously. We can recover this S with proposition 2.

Proposition 2 (Linear Check).

Let us assume the BiSE is activated for almost binary images in Ipu, vq. Let b P R be the BiSE bias, let W P R Ω S be the weights. Then there exists τ P R such that S " ti P

Ω | W piq ě τ u. -If the BiSE is a dilation / anti-dilation τ " 1 v `b ´ÿ iPS C , wiď0 w i ˘(14)
-If the BiSE is an erosion / anti-erosion

τ " 1 1 ´u `ÿ iPΩ , wiě0 w i ´b˘( 15)
We can check the two SEs associated with each threshold to recover the activation operation. If none works, then by contraposition, the BiSE is not activated. This is done in Op|Ω|q operations.

Binarization

Formalism The purpose of binarization is to replace the real parameters pW, b, pq with binary parameters to improve inference efficiency [15]. If the activation inequalities of Theorem 1 are respected, we saw that we could replace the BiSE with the morphological operation. That is a form of binarization: floating point numbers are no longer required. However, these inequalities are not necessarily respected. How to binarize the BiSE neuron in this case? We want to find the closest morphological operation. Let pW ˚, b ˚, p ˚q P R Ω S ˆR`ˆR be the learned parameters. Let A : pS, opq P PpΩ S q ˆtdilation, erosionu Þ Ñ ApS, opq " tpW, bq P R |Ω S |`1 | f op S pW q ď b ă g op S pW qu, with f op S and g op S the bounds defined in Theorem 1. We want to define a dissimilarity function d : pS, opq, pW, bq P pPpΩ S q ˆtdilation, erosionuq ˆpR |Ω S | ˆRq Þ Ñ dpApS, opq, pW, bqq, and we want to find argmin pS,opqPPpΩ S qˆtdilation,erosionu dpApS, opq, pW ˚, b ˚qq (16)

Choice of dissimilarity A first choice is to use: dpApS, opq, pW, bqq " max ´0, f op S pW q ´b, b ´gop S pW q ¯(17)

The search for pS, opq P PpΩ S q ˆtdilation, erosionu has exponential complexity Op2 |Ω S | ). To simplify this search, we take inspiration from proposition 2: if the BiSE is activated, we know that S " ti | w i ě τ u for a certain τ . Therefore, we reduce the search to all S of this thresholded form, with τ P tw i | i P Ω S u. This reduces the complexity to Op|Ω S |q.

Binary Structuring Element Layer

One strength of ConvNets is their ability to learn multiple filters per layer, which we also want to ensure. In a convolutional layer, each final channel is a sum of one filter per input channel. In our case, the final result is either a union or an intersection of the elementary operators (dilation, erosion, anti-dilation, antierosion). Therefore, we want a layer that can learn the union or intersection of any combination of inputs. Let us consider n binary images x 1 , ..., x n Ă Ω. Let C Ă rr1, nss. Then the intersection and union are given as:

1 Ş iPC xi " ´ÿ iPC 1 xi ě |C| ¯(18) 1 Ť iPC xi " ´ÿ iPC 1 xi ě 1 ¯(19)
As with the BiSE, we can use a single scalar to discriminate between the union or the intersection. To learn the set C, we can use a parameter β i for each image. The following definition ensues: Definition 4 (LUI). Let β " pβ 1 , ..., β c q P R c . Let b P R be a bias and p P R a scaling factor. We define the LUI (Layer Intersection Union) as a thresholded linear combination:

LUI β,b,p : x P pZ d q c Þ Ñ ξ ˆp´c ÿ i"1 f `pβ i qx i ´f `pbq ¯˙P Z d (20)
A LUI layer can learn any intersection or union of any number of almost binary inputs. It is a particular case of BiSE layer. In this case, the SE support would be Ω lui " t0u d ˆrr1, nss. Learning the intersection is equivalent to learning an erosion, and learning the union is equivalent to learning a dilation. Then, we can deduce the following Theorem, by using the BiSE Theorem 1. We denote by I "

Ś n k"1 Ipu k , v k q the set of images with n almost binary channels.

Theorem 2 (LUI intersection / union equivalence).

Let n P N ˚and C Ă rr1, nss. Let b P R. Let pu i ă v i q P r0, 1s 2n . Let β P R n . -LUI β,b,`8 is an intersection by C if and only if

n ÿ k"1 , β k ě0 β k ´min kPC " p1 ´uk qβ k ı ď b ă ÿ kPC , β k ě0 β k v k `n ÿ k"1 , β k ď0 β k (21) -LUI β,b,`8 is a union by C if and only if ÿ kPC , β k ě0 β k u k `ÿ kPrr1,nss z C , β k ě0 β k ď b ă min kPC pβ k v k q `n ÿ k"1 , β k ď0 β k (22)
If one of these inequalities is respected, we say that the LUI is activated. Then, we have @k P C , β k ě 0 and b ě 0 and the output of the LUI is almost binary.

As for the BiSE, if the LUI is activated, the set C can be found by thresholding the pβ k q k for a specific value. If the LUI is not activated, it is still possible to binarize with the same method described for the BiSE in section 2.2.

We combine the BiSE neurons and the LUI in order to be able to learn the morphological operators and aggregate them as unions or intersections.

Definition 5 (BiSEL). A BiSEL (BiSE Layer) is the combination of multiple

BiSE and multiple LUI. Let pϵ n,k q be N ˚K BiSE and pLUI k q k be K LUI. Then we define a BiSEL as:

ϕ : x P pZ d q n Þ Ñ ˆLUI k "
`ϵn,k px n q ˘nı ˙k P pZ d q K (23)

The LUI and BiSE are called the BiSEL's elements. If every element is activated, we say that the BiSEL is totally activated.

If the BiSEL is activated, given almost binary inputs, the outputs of the BiSEL are also almost binary.

The BiSEL follows the same logic as a convolutional layer. In ConvNets, at each layer, we have k filters. Each filter applies one convolution to each channel, and then we apply a linear combination to all convoluted channels. In BiSEL, we have K filters (the number of LUI). For each filter, we apply a morphological operation to a channel, and then we aggregate by taking the intersection or union of any channels. A schema of BiSEL can be found in figure 1. Binarizing a BiSEL is equivalent to binarizing each of its element. If an element is activated, it is easy to binarize it. For those who are not, we must approximate them.

The parameters are the BiSE weights tW n,k u n,k Ă R Ω S , the BiSE biases tb n,k u n,k Ă R, the LUI parameters tβ n,k u n,k Ă R and the LUI biases tb l k u k Ă R. There are N ¨Kp|Ω S | `2q `K parameters.

BiMoNN

The BiSEL is akin to a complete convolution layer with multiple filters. We can simply stack multiple BiSELs to create a network similar to a convolutional neural network. Definition 6 (BiMoNN). Let ϕ 1 , ..., ϕ L be L BiSEL. We define the Binary Morphological Neural Network (BiMoNN) as:

BiMoNN " ϕ L ˝... ˝ϕ1 (24)
We say that the BiMoNN is totally activated if all the composing BiSELs are totally activated.

The BiMoNN can learn the composition of any intersection or union of dilation, erosion, anti-dilation, and anti-erosion. That includes but is not limited to opening, closing, black top-hat, and white top-hat.

There are two drawbacks to our current formulation. First, the input has the same dimension as the output. ConvNets can be combined in multi-scale architectures, which compresses the data. Currently, we cannot solve classification tasks with our method. Second, we can only learn predeternined composition of operators. However, some morphological operators like skeletonization or reconstruction need an undetermined number of iterations before convergence. Incorporating these features is left for further work.

Learning phase The BiMoNN is fully differentiable, and we learn its parameters with classical deep learning techniques. Let L : Ω 2 I Ñ R be a differentiable loss function. Depending on our interpretation of the outside domain Z d z Ω I , the effects on the border can vary. Thus, the loss is computed by avoiding the borders, whose size is half of the kernel size Ω S in each direction. We try classical segmentation losses: the dice loss [11], the binary cross-entropy (BCE), and the mean squared error (MSE).

Then, given a dataset of N labeled images pX, Y q, we minimize the error

1 N
ř N i"1 LpBiMoNNpx i q, y i q using Adam [START_REF] Kingma | Going beyond pconvolutions to learn grayscale morphological operators[END_REF]. We compute the loss gradient with the backpropagation algorithm [13]. In practice, we replace f `pbq by f `pbq `0.5 to help with training for all BiSE and LUI operators. The convolution weights W follow the kaiming uniform initialization [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF], and the biases begin at f `p2q`0.5 " 0.63. We initialize The BiSE and LUI scaling factors p at 0. That may not be optimal, as it could bias the BiSE towards some operations and Sels. Could we find other unbiased methods toward a particular operation or Sel ? This is left for further research.

Binarization Once the training is done, we can binarize each element independently to binarize the whole BiMoNN. Theorems 1 and 2 give activation conditions regarding a specific set of almost binary images Ipu, vq. Given a layer l, the range ru l , v l s depends on the activation of the layer l ´1 and only exists if all the layers ϕ 1 , ..., ϕ l´1 are totally activated. This ideal situation does not occur in most cases. In practice, we binarize the network sequentially layer by layer, from the first to the last, and compute the almost binary range ru l , v l s along the way. If a layer ϕ l is not totally activated, we use the approximation described in 2.2: then, the next range becomes ru l`1 , v l`1 s " r0, 1s.

Experiments

Classical morphological operators

We will be using the three datasets described in [START_REF] Aouad | Binary morphological neural network[END_REF]: Diskorect, a set of customgenerated images composed of ellipses and rectangles that create as many dif-ferent situations as possible; thresholded MNIST upsized to 50 ˆ50, and its complementation that we call inverted MNIST. Examples are shown in figure 3.

Experiment description

We learn the following operators: dilation, erosion, opening, closing, black top-hat, and white top-hat. The 3 SEs used can be seen on table 2. For erosion and dilation on MNIST, we use smaller SEs of size 5 ˆ5. Else for all other operations, SEs are of size 7 ˆ7. The experimental protocol is as follows: let γ be one of these morphological operators. The input-output couples are tpx i , γpx i qq | x i P datasetu. The hyperparameters are the loss used, the learning rate, and the batch size. For each result, see the code for the hyperparameters used. We measure the DICE score [START_REF] Dice | Measures of the amount of ecologic association between species[END_REF] between the predictions and targets. Axial Spondyloarthritis (axSpA) is the most common auto-immune rheumatic disease. It is painful and debilitating if untreated or incorrectly diagnosed. The disease can be detected using MRIs: the inflammation is located on the sacroiliac joint. Given the sacrum and the iliac shapes, the task is to detect this joint region. We call this task axSpA.

The dataset is composed of around 1100 images with categorical images of size about 512ˆ512 with two values: one for the iliac and one for the sacrum. We transform them into binary images by separating each categorical value into one channel: final images have 2 channels. The target is defined as the joint region. Let S a " rr´20, 20ss ˆt0u. The target operation is γ a : X P t0, 1u 512ˆ512ˆ2 " pX 1 'S a qXpX 2 'S a q, with X i the i´channel. The resulting input-output couples are tpx i , γ a px i qq | x i P axSpA datasetu (see figure 3d). Results on all operations are depicted in table 1.

Table 1: DICE Results on morphological operations. R means real weights, B means after binarization (or approximate). Bold means DICE ą 0.8.

Parallel operations

In this section, we study the model's ability to learn the complementation and intersection with top-hat transform. We manage to learn the black top-hat for the Diskorect and inverted MNIST. For the cross, the binarization is perfect (see figure 4). The upper branch learns the complementation of the identity, and the bottom branch learns the closing with two sequential anti-dilations. The white top-hat only worked on Diskorect. The binarization decreases performance.

AxSpA regions of interest

We attempt to learn the joint region. We want to learn the intersection of two dilations of size 41ˆ41. We try three different architectures that can theoretically learn this operation (see figure 5). They go from wide and shallow to deeper and narrower. Architectures 1 and 2 learn the operation well, reaching an excellent DICE (1 and 0.9). Architecture 1's weights are close to the target, and the binarization yields good results (DICE=0.98). On the other hand, architecture 2 learns wider dilations, and its binarization is quite bad (DICE=0.7). Note that there are multiple ways for this architecture to learn the target. Finally, the deeper network failed to learn anything. That may be due to the initialization: the output is constant and stuck in a zero-grad zone, which is a case of vanishing gradient. This application is hopeful in the study of axSpA. This model could be used with classical convolutional neural networks to improve MRI analysis by exploiting the ROI's detection using segmentation results.

Conclusion

We create a neural network designed to work on binary elements. It comes with explainable results that can be used to binarize the network, improving its embedded applicability for inference. We manage to learn the erosion, dilation, opening, and closing of small structuring elements. We can learn sequential operations and more complex ones that need the intersection of multiple operators. Even with bigger structuring elements, we learn appropriate behaviors for concrete applications in medical imaging. However, there are still unanswered questions. Why does the training differ for dual operations? How can we avoid vanishing gradient in deeper networks? How can we reduce the dimension to create a classification model? Can we redesign the model to take into account undetermined number of iterations? How can we binarize more efficiently to preserve the performance, especially if there are multiple channels?

3 ) 4 ) 5 ) 6 )

 3456 ´iq1 S piqqq. Definition 1. Let S Ă Ω S be a SE. Let I Ă Ω I be a binary image. We define the following morphological operators. the dilation by S δ S pIq " I ' S " ď sPS pI `sq (1) the erosion by S ε S pIq " I a Š " the anti-dilation by S δ S pIq C (the anti-erosion by S ε S pIq C (the opening by S I ˝S " pI a Šq ' S (the closing by S I ' S " pI ' Sq a Š (the black top-hat by S T b pI, Sq " pI ' Sq z I " pI ' Sq X I C

Fig. 1 :

 1 Fig. 1: Schema of BiSEL. Input x with 3 channels. Output ϕpxq with 2 channels.

Fig. 2 :

 2 Fig. 2: Target SEs

  Fig. 3: Datasets example

Fig. 4 :Fig. 5 :

 45 Fig. 4: Learned model for black top-hat with cross

  11. Milletari, F., Navab, N., Ahmadi, S.A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation (2016) 12. Mondal, R., Dey, M.S., Chanda, B.: Image restoration by learning morphological opening-closing network. Mathematical Morphology-Theory and Applications 4(1), 87-107 (2020) 13. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning Representations by Backpropagating Errors. Nature 323(6088), 533-536 (1986). https://doi.org/10. 1038/323533a0, http://www.nature.com/articles/323533a0 14. Shen, Y., Zhong, X., Shih, F.Y.: Deep morphological neural networks. arXiv preprint arXiv:1909.01532 (2019) 15. Simons, T., Lee, D.J.: A review of binarized neural networks. Electronics 8(6), 661 (2019)

Fig. 6 :Fig. 7 :Fig. 8 : 2 Fig. 9 :

 67829 Fig. 6: Bernoulli Denoising Set Up

Table 2 :

 2 Results on erosion and dilation. DICE error (1 ´DICE) is presented for each case: R before binarization, B after binarization. ✓ indicates that the neuron is activated.

Sequential operations

For dilation and erosion, the SEs are perfectly learned. The perfect DICE (=1) is reached in almost all cases. Even when the BiSEs are not activated, the approximate binarizations yield good results. Performance can decrease (see disk erosion on inverted MNIST), or increase (see disk erosion on MNIST). On opening and closing, we reach perfect dice for most cases. All SEs can be seen on the code repository. We manage to recover the target SEs for most operations except on the opening on the inverted MNIST. We want to test the following hypothesis H: dual operators learn similarly on dual datasets. On Diskorect, for any image X in this dataset, then X C is also in it. Therefore H implies that opening and closing should have similar results. Already the hypothesis is rejected if we look at the stick opening vs. the stick closing: one succeeds and the other fails. On MNIST, H implies that the closing on MNIST should behave the same as the opening on inverted MNIST and vice versa. That is not observed and refutes further the hypothesis. Note that the model learns two anti-dilations for the closing, which is equivalent to learning dilation followed by erosion.

We compare our results to LM orph and SM orph [9]. They reach almost perfect DICE for erosion and dilation, but recovering the SEs from the learned weights is not straightforward. Moreover, on the opening and closing, the LM orph presents stability issues, and the SM orph fails to converge correctly to the proper operations. Those results are not surprising as these networks were designed for grey-scale morphology.