domain: phys chim

text:
Tal Binyamin

Laurent Pedesseau

Sergei Remennik

Amal Sawahreh

Jacky Even

Lioz Etgar
email: lioz.etgar@mail.huji.ac.il

Fully inorganic mixed cation lead halide perovskite nanoparticles: a study at the atomic level

Mixed cation perovskites are currently the most efficient perovskite materials used in perovskite solar cells. Mixing two cations inside a perovskite structure results in enhanced flexibility when designing interesting material properties. Moreover, using two inorganic cations in the same perovskite maintains the advantage of fully inorganic structures. A fascinating subject to investigate is therefore the nanoscale synthesis and the properties of such mixed inorganic cation perovskites. In this work we mixed Rb and Cs inorganic atoms inside perovskite nanoparticles. We explored down to the atomic resolution different Rb and Cs concentrations and performed the chemical mapping of single nanoparticles. At medium concentrations, the Rb atoms are observed in the core of the particles, whereas the Cs atoms are located in the shell region, forming core shell structures. However, if there are high concentrations of Rb, a phase separation occurs because bulk perovskite based solely on Rb cations is not stable at room temperature. Density functional theory calculations support our experimental observations by showing that a stable nanoparticle is formed when the Rb atoms are located inside the particle and not on the surface. Our work demonstrates the importance of understanding the perovskite structure at the atomic level, leading to the formation of mixed cation bulk perovskites and nanoparticles, and to improved perovskite stability. A new phase of cesium lead bromide (Cs 6 Pb 5 Br 16 ) related to the Rb 6 Pb 5 Cl 16 structure is also reported.

Introduction

Metal halide perovskites have attracted much attention in the last decade due to their use as a light harvester in solar cells. In this capacity, these perovskites have achieved remarkable efficiencies in a short time. [START_REF]NREL transforming energy[END_REF] The lead halide perovskites are known for their large absorption coefficients, long charge carrier diffusion lengths, low exciton binding energies, tunable band gaps, and the possibility of preparing them from solution. 2, [START_REF] Miyata | Direct Measurement of the Exciton Binding Energy and Effective Masses for Charge Carriers in Organic-inorganic Tri-Halide Perovskites[END_REF][START_REF] Filip | Steric Engineering of Metal-Halide Perovskites with Tunable Optical Band Gaps[END_REF][START_REF] Eperon | Formamidinium Lead Trihalide: A Broadly Tunable Perovskite for Efficient Planar Heterojunction Solar Cells[END_REF][START_REF] Zhao | Solid-State Mesostructured Perovskite CH 3 NH 3 PbI 3 Solar Cells: Charge Transport, Recombination, and Diffusion Length[END_REF][START_REF] Park | Perovskite Solar Cells: An Emerging Photovoltaic Technology[END_REF][START_REF] De Roo | Highly Dynamic Ligand Binding and Light Absorption Coefficient of Cesium Lead Bromide Perovskite Nanocrystals[END_REF] This perovskite's chemical formula is AMX 3 , where A is a monovalent cation (organic or inorganic), M is a divalent cation, and X is a halide. The methylammonium lead iodide is among the most common hybrid organic-inorganic perovskites currently being used. When the organic cation is replaced by an inorganic cation, a fully inorganic perovskite is formed. The main option here, due to size limitations, is the cesium cation (Cs + ), which can form cesium lead halide perovskite. CsPbI 3 is not stable in its bulk form, leading to a yellow non-perovskite phase; however, nanoparticles (NPs) of CsPbI 3 are more stable. [START_REF] Huang | Lead Halide Perovskite Nanocrystals in the Research Spotlight: Stability and Defect Tolerance[END_REF][START_REF] Pan | Bidentate Ligand-Passivated CsPbI3 Perovskite Nanocrystals for Stable Near-Unity Photoluminescence Quantum Yield and Efficient Red Light-Emitting Diodes[END_REF] An interesting option is to use a perovskite composition having two cations at the A site, the so-called mixed cation perovskites. [START_REF] Baek | Development of Mixed-Cation Cs x Rb 1-x PbX 3 Perovskite Quantum Dots and Their Full-Color Film with High Stability and Wide Color Gamut[END_REF] The mixed cation perovskites in their bulk form have achieved a high photovoltaic performance for several structures of solar cells. [START_REF] Saliba | Incorporation of Rubidium Cations into Perovskite Solar Cells Improves Photovoltaic Performance[END_REF][START_REF] Saliba | Cesium-Containing Triple Cation Perovskite Solar Cells: Improved Stability, Reproducibility and High Efficiency[END_REF][START_REF] Yi | Entropic Stabilization of Mixed A-Cation ABX3 Metal Halide Perovskites for High Performance Perovskite Solar Cells[END_REF] Another important feature of mixed cation perovskite is its ability to influence the band gap of the perovskite in a gentler way than the halides do, owing to modifying the angle at the X-Pb-X framework. [START_REF] Singh | Stabilizing the Efficiency Beyond 20% with a Mixed Cation Perovskite Solar Cell Fabricated in Ambient Air under Controlled Humidity[END_REF][START_REF] Mcmeekin | A Mixed-Cation Lead Mixed-Halide Perovskite Absorber for Tandem Solar Cells[END_REF] Most attention has been directed to investigating the bulk properties of metal lead halide perovskites, whereas scant efforts have been devoted to elucidating the nano-scale properties of the perovskite material. In the last few years, however, the synthesis and characterization of nanoscale inorganic and hybrid perovskites have rapidly progressed. [START_REF] Amgar | Inorganic and Hybrid Organo-Metal Perovskite Nanostructures: Synthesis, Properties, and Applications[END_REF] The first synthesis of Cs lead halide was reported by Protesescu et al. [START_REF] Protesescu | Nanocrystals of Cesium Lead Halide Perovskites (CsPbX3 , X = Cl, Br, and I): Novel Optoelectronic Materials Showing Bright Emission with Wide Color Gamut[END_REF] Cs lead halide perovskite NPs have a direct band gap. The tunability of their band gap energies is feasible via altering the composition by changing the halides, Cl, Br, and I. In addition, it was reported that the anion exchange could also be applied to control the absorption wavelength. [START_REF] Nedelcu | Fast Anion-Exchange in Highly Luminescent Nanocrystals of Cesium Lead Halide Perovskites (CsPbX 3 , X = Cl, Br, I)[END_REF][START_REF] Akkerman | Tuning the Optical Properties of Cesium Lead Halide Perovskite Nanocrystals by Anion Exchange Reactions[END_REF] The diameter of Cs lead halide perovskite NPs can be up to about 12-15nm; Cs lead halide perovskite NPs therefore exhibit an energy gap tunability in the visible spectrum (400nm-700nm wavelength). CsPbX 3 NPs have narrow emissions and high quantum yields up to 90%. [START_REF] Trots | High-Temperature Structural Evolution of Caesium and Rubidium Triiodoplumbates[END_REF][START_REF] Sessolo | Perovskite Luminescent Materials[END_REF] To date, there have been reports on several CsPbX 3 NP synthesis and surface chemistry methods, including studying their optoelectronic properties. [START_REF] Imran | Simultaneous Cationic and Anionic Ligand Exchange For Colloidally Stable CsPbBr3 Nanocrystals[END_REF][START_REF] Imran | Benzoyl Halides as Alternative Precursors for the Colloidal Synthesis of Lead-Based Halide Perovskite Nanocrystals[END_REF][START_REF] Krieg | Colloidal CsPbX3 (X = Cl, Br, I) Nanocrystals 2.0: Zwitterionic Capping Ligands for Improved Durability and Stability[END_REF] However, there have been only a few reports on mixed cation perovskite NPs in the nanoscale regime. A mixing of organic and inorganic cations (FA and Cs) in perovskite NPs leads to FAdoped CsPbI 3 with a cubic shape and uniform size of 10-15nm. These NPs exhibit better A c c e p t e d M a n u s c r i p t from the red to the NIR region. [START_REF] Protesescu | Dismantling the "Red Wall" of Colloidal Perovskites: Highly Luminescent Formamidinium and Formamidinium-Cesium Lead Iodide Nanocrystals[END_REF] In addition, mixing two inorganic cations in the same NPs was recently reported by us [START_REF] Amgar | Near Ultra-Violet to Mid-Visible Band Gap Tuning of Mixed Cation Rb x Cs 1-x PbX 3 (X = Cl or Br) Perovskite Nanoparticles[END_REF] ; Rb was incorporated into Cs-based NPs, forming all inorganic mixed cation perovskite NPs. The addition of Rb atoms slightly increases the NP band gap, shifting the NP absorbance and PL to shorter wavelengths. Moreover, NP's size decreases when Rb atoms are introduced into them.

However, obtaining a more detailed understanding of the Rb distribution inside the NPs is impossible using indirect characterization techniques such as optical absorption and PL. In this work we used high angle annular dark field (HAADF) imaging, combined with energy dispersive spectrometry (EDS) -spectrum imaging (SI) to study in detail the atomic distributions of the Rb and Cs atoms inside single perovskite NPs. It is a challenging task to analyze the perovskite materials with this high-resolution tool, since these materials are very sensitive to electron beams. We vary the Rb concentration within the Rb x Cs 1-x PbBr 3 NPs, when X=0, 0.2, 0.4, 0.6, and 0.8 including the case of pure Rb lead chloride NPs. Apparently the Rb atoms prefer to be located in the core of the nanoparticle; the Cs atoms remain on the nanoparticle's shell at medium Rb concentrations, whereas at higher Rb concentrations, a phase separation occurs.

Results and discussion

Here we present a detailed structural study of mixed inorganic cation lead halide perovskite NPs at the atomic level. We focused our investigations on mixed cation perovskite NPs having an Rb x Cs 1-x PbBr 3 composition (when X=0, 0.2, 0.4, 0.6, and 0.8). The HAADF detector was used to observe the atomic resolution images; EDS SI allows one to obtain detailed elemental maps of the NPs.

In our previous report we synthesized mixed cation perovskite NPs, along with the ability to tune their optical properties. When Rb atoms are introduced into the CsPbBr 3 NPs, a blue shift of the absorbance and photoluminescence (PL) is observed. For example, the PL wavelength of pure Cs (x=0) is equal to 516 nm, whereas when X=0.2 (some Rb is introduced into the NP) the PL wavelength is shifted to 508 nm (table 1). Since the ionic radius of Rb + is smaller than that of Cs + , the perovskite's framework becomes distorted. In this work we went down to the atomic level, in order to analyze the atomic distribution and chemical composition of single particles. In particular, we detected the Rb to Cs ratios and their distribution over the NPs.

Using the STEM elemental line scan, it is possible to identify the elements as a function of their position inside the particle. The EDS elemental analysis is based on measuring the A c c e p t e d M a n u s c r i p t characteristic X-ray fluorescence emitted due to the interactions of high velocity electrons with sample atoms; each atom has a different signal due to the different energies of the atoms' shells.

Notably the EDS analysis of Rb appears to be very challenging in the presence of Cs and Pb atoms. The main Rb signal derives from two shells: L (~1.69 KeV) and K (~13.37 KeV). The Rb L shell overlaps with the K shell signal of Si K (~1.73 KeV), Br L γ2 (~1.77 KeV), and the L β3 (~1.60 KeV) shell, whereas the K shell overlaps with both Br K β (~13.29 KeV) and Pb Lγ (~14.75 KeV). Detection of small Rb traces by EDS is therefore almost impossible in such systems.

Figure 1 shows the case of pure CsPbBr 3 NPs (i.e. x=0). Figure 1a presents an atomic resolution HAADF STEM image of single NPs; the fast Fourier transform (FFT) of the image can be seen in the inset of Figure 1a. An analysis of the crystallographic structure of the NPs, using the FFT, is presented in Table 1, which shows the d-spacing values and the corresponding Miller indices (5 0 0) and ( 4 Here the PL shifts slightly to shorter wavelengths compared with pure Cs NPs (508 nm compared to 516 nm), as presented in Table 1 and Figure 2b.

With low Rb concentrations, it is very difficult to detect the Rb signal. We performed a reference experiment in which we tracked the Rb signal from pure Cs-based NPs (see Figure 1S in the supporting information). The Rb signal is within the error limit of the measurements for both X=0 and X=0.2; Figure 2d shows that the Rb signal is five times smaller than the Cs signal.

Despite the low Rb signals, the presence of the Rb atoms is indicated, as can be seen in the elemental map in Figure 2c, which shows green dots located mainly inside the NP.

Figure 2a shows HAADF STEM images and a FFT pattern of a single NP. The FFT provides the Miller indices (4 2 2) and (4 4 4) corresponding to Rb 0.2 Cs 0.8 PbBr 3 . Importantly, the HAADF STEM image shows some white spots within the NPs. T. Udayabhaskararao et al. [START_REF] Udayabhaskararao | Growth, and Structural Transformations of Perovskite Nanocrystals[END_REF] associated these spots with Pb 0 NPs. In this work, using HADDF the white spots could also be associated with Pb 0 particles (Figure S2). Figure 3 shows the case of a higher Rb concentration, Rb 0.4 Cs 0.6 PbBr 3 , when X=0.4. Here, owing to the larger amount of Rb, it is possible to observe by elemental analysis the two cations (i.e., Cs + and Rb + ) in the same NP. The Rb signal can be easily recognized and it is equal in intensity to that of Cs.

Figure 3a shows the HAADF image of the NPs when X=0.4 Rb and with 0.6 of Cs; the inset presents the FFT of a single NP; this provides us with the d-spacing values and the corresponding Miller indices (3 3 2) and (4 5 2) (see Table 1). Importantly, when X=0.4 and X=0.6, the samples show a high stability under the HR-TEM electron beam.

The PL and absorbance when X=0.4 (Figure 3b) indicate that  max =496 nm, which is further blue-shifted in relation to pure CsPbBr 3 . The elemental composition of a single NP is observed in Figures 3c and3d. The arrow denotes the scanned area, which is shown in the graph in Figure 3d. The ratios of Cs/(Cs+Rb) and Rb/(Cs+Rb) are presented in Figure 3d, which clearly shows a mirror image of Rb and Cs. As shown in Figure 3d, a Cs atom is replaced by a Rb atom and vice versa. Interestingly, based on the elemental map in Figure 3c, the Rb atoms are located in the center of the NP, whereas the Cs atoms are located on the surface of the NP, forming a "core-shell"-like structure. With larger particles (i.e., 10-12nm), it is easier to recognize the "core-shell" structure, whereas with small NPs the signal is weaker and the particles are thinner, which makes it harder to recognize this structure, even though Rb appears to be at the inner region of NP and Cs on the NP shell .

The pure Rb-based lead bromide perovskite is not stable as bulk materials as well as NPs are, as reported previously [START_REF] Miyata | Direct Measurement of the Exciton Binding Energy and Effective Masses for Charge Carriers in Organic-inorganic Tri-Halide Perovskites[END_REF] . On the other hand, CsPbBr 3 NPs have a stable perovskite phase and can be easily synthesized. Moreover, the high surface energy of NP forces its shell to be more energetically stable than its core. Therefore, in order to obtain mixed cation NPs that are stable, Cs will preferably be located on the shell of the NP; Rb will be located in the core of the NP in order to stabilize this composite structure.

In order to support this observation, density functional theory (DFT) modeling [START_REF] Hohenberg | Inhomogeneous Electron Gas[END_REF][START_REF] Kohn | Self-Consistent Equations Including Exchange and Correlation Effects[END_REF] was performed (see the computational details in the SI). The modeling starts from the bulk structures of CsPbCl 3 and CsPbBr 3 by optimizing them until the residual forces are below 0.005eV/Å. The models were constructed to illustrate the substitution effect of Cs by Rb cations in CsPbCl 3 . A supercell of 98 atoms with a vacuum of 200Å was considered by using the SIESTA package. [START_REF] Soler | The SIESTA Method for Ab Initio Order-N Materials Simulation[END_REF][START_REF] Artacho | The SIESTA Method; Developments and Applicability[END_REF] Three types of substitution in the initial supercell have been simulated, mimicking X=0.125, 0.25, and 0.375 concentrations; they are hereafter referred to as 7Cs:1Rb, 6Cs:2Rb, and 5Cs:3Rb, respectively. As discussed in the SI, each case leads to a A c c e p t e d M a n u s c r i p t stable structure. Figure 6b shows side views of the three most stable structures having three different compositions. When X=0.125 (7Cs:1Rb) the simulation shows that a single Rb cation prefers to be located far from the surface (E= -263.5 meV). For X=0.25 (6Cs:2Rb) a pair of Rb cations again prefers to be far from the surface but in a position somewhat closer to the surface than a single Rb cation (E= -227.5 meV) is. Lastly, when X=0.375 (5Cs:3Rb), three Rb cations were introduced in the supercell and pos5 was found to be the most stable configuration (E= -176.1 meV). In fact, it confirms the tendency shown by 1 and then 2 substitutions. The case of 5Cs:3Rb indicates that a stable structure can be achieved by combining a single Rb cation in the core of the particle and a pair of Rb cations on the subsurface. This result again supports the experimental observations of cases when X=0.4 and X=0.6, where the Rb atoms prefer to be far from the surface of the nanoparticle. Regarding Rb, when X=0.6 with absorbance ( max =491 nm), a slightly blue-shifted PL appears, as can be seen in Figure 4b. The atomic resolution HAADF image and the corresponding FFT are observed in Figure 4a. The Miller indices for this composition are indicated in Table 1. The same occurs when X=0.4; also here the Rb phase is located at the core of the NP, whereas the Cs phase is located in the shell, as can be clearly seen in Figures 4c and4d Increasing the Rb concentration to X=0.8 results in additional PL blue-shifts and absorbance ( max =468 nm, Figure 5b). The STEM image in Figure 5a shows white spots located within a single NP; this indicates phase separation, which may appear due to the large amount of Rb compared to Cs. This is similar to when X=0.2, where white spots were also be observed.

However, here, since Rb alone cannot form a stable perovskite phase, it begins to separate from NP. It is unlikely that both Cs and Rb will be present in the same NP. In the case of X=0.2 due to the small amount of Rb, some of it can remain inside the NP. Figure 5c shows the existence of phase separation, where green (which denotes Rb) particles and magenta (which denotes Cs) particles, are separated from each other.

In large concentrations of Rb, we expect to see Rb in the NP shell and Cs in the NP core, since the nucleation begins with Rb, which is in excess. However, the amount of Cs is too small in a single NP; therefore, it cannot stabilize the structure that supports the observed phase separation. Moreover, a line scan on a single NP (Figures 5c and5d) indicates that most of the Lastly, pure Rb-based NPs were synthesized and studied. The synthesis was based on our previous report [START_REF] Amgar | Novel Rubidium Lead Chloride Nanocrystals: Synthesis and Characterization[END_REF] on Rb 6 Pb 5 Cl 16 . The Rb lead halide NPs cannot be synthesized with bromide (at the halide position) due to the larger ionic radius of bromide, compared with the chloride ion. Figure 6a shows a single Rb 6 Pb 5 Cl 16 NP and its corresponding FFT. The white spots can be observed already in a single NP; the STEM image in Figure 6c shows several white spots for each NP, as was reported by us previously [START_REF] Hohenberg | Inhomogeneous Electron Gas[END_REF] . The Rb 6 Pb 5 Cl 16 NPs form a tetragonal phase, which is a non-perovskite phase based on our X-ray diffraction analysis [START_REF] Amgar | Novel Rubidium Lead Chloride Nanocrystals: Synthesis and Characterization[END_REF] . Monzel et al. [START_REF] Monzel | Zur Neuuntersuchung des Phasendiagramms RbCl/PbCl 2[END_REF] reported on this phase; it was shown that the perovskite phase RbPbCl 3 cannot exist at room temperature, whereas the Rb 6 Pb 5 Cl 16 phase can. The only option to observe a tetragonal perovskite phase based on RbPbCl 3 is heating above 320º C; however, below that temperature this phase does not exist. At higher temperatures, a cubic perovskite phase can also be observed. Based on our previous report, [START_REF] Amgar | Novel Rubidium Lead Chloride Nanocrystals: Synthesis and Characterization[END_REF] the white spots located on each NP are associated with the Pb 0 or Pb 2 O 3 phases. These white spots already appear in fresh samples of Rb 6 Pb 5 Cl 16 NPs, as opposed to when X=0, 0.2, and 0.8; these white spots usually appear because of the electron beam radiation. Interestingly, it is worth mentioning that this Rb 6 Pb 5 Cl 16 structure was also observed with pure CsPbBr 3 NPs; here Cs was used instead of the Rb cation, i.e., Cs 6 Pb 5 Br 16 . This is the first time that this phase was observed in the synthesis of CsPbBr 3 NPs (see Figure S4). Clearly, owing to the bigger size of Cs compared with Rb, this structure can be formed with Br instead of Cl, as in the case of Rb. This interesting structure opens the way to investigate new phases of cesium lead bromide. Figure 6d summarizes the observation from the different Rb + concentrations in the NPs.

Starting, from X=0 which refers to the pure CsPbBr 3 , until the X=0.8 where the Rb + concentration is much higher than the Cs + concertation which possess phase separation. Figure S3 shows the full width half maximum (FWHM) based on the PL spectra of the different Rb concentrations. The PL measurement detects a macro size area of the sample, which means that it takes into account the PL from many NPs. Keeping this in mind, Figure S3 shows that in both the cases when X=0.4 and X=0.6 the FWHM is larger than the other concentrations.

Both concentrations show the "core-shell" structure, which clearly indicates that Rb is incorporated inside the NPs, whereas in the other concentrations Rb was hardly detected or was phase separated. Therefore, when X=0, 0.2, and 0.8, the FWHM value is smaller, since it mainly detects the pure Cs lead bromide NPs, whereas when X=0.4 and 0.6, it detects the coreshell particles that contain Rb and Cs together in the same NPs; therefore, the FWHM increases. This observation indicates the importance of analyzing single NPs at their atomic level.

Experimental Chemicals

Cesium carbonate (Cs 2 CO 3 , 99.9%, Sigma-Aldrich), rubidium carbonate (Rb 2 CO 3 , 99%, Sigma-Aldrich), lead (II) chloride (PbCl 2 , 98%, Sigma-Aldrich), lead(II) bromide (PbBr 2 , ≥98%, Sigma-Aldrich), oleic acid (OA, 90%, Sigma-Aldrich), oleylamine (OLAM, 70%, Sigma-Aldrich), 1-octadecene (ODE, 90%, Sigma-Aldrich), 2-propanol (≥99.8%, Sigma-Aldrich), trioctylphosphine (TOP, 97%, Strem), and hexane (not pure, Gadot) were purchased and used as received, without any further purification.

Preparation of Rb/Cs-oleate

The Rb/Cs-oleate precursor was prepared according to a previously published procedure by Protesescu et al. and by us [START_REF] Protesescu | Nanocrystals of Cesium Lead Halide Perovskites (CsPbX3 , X = Cl, Br, and I): Novel Optoelectronic Materials Showing Bright Emission with Wide Color Gamut[END_REF][START_REF] Amgar | Near Ultra-Violet to Mid-Visible Band Gap Tuning of Mixed Cation Rb x Cs 1-x PbX 3 (X = Cl or Br) Perovskite Nanoparticles[END_REF] In a 100 mL 3-neck flask, different molar ratios of Cs 2 CO 3 /Rb 2 CO 3 (total of 1.228 mmol) were mixed with 625 μL of oleic acid (OA) and 7.5 mL A c c e p t e d M a n u s c r i p t of 1-octadecene (ODE). The solution was degassed for 1 h under vacuum conditions at 120 °C and then heated to 150 °C under Argon flow.

Synthesis of Rb x Cs 1-x PbX 3 (X = Cl, Br) NPs

The NPs were synthesized according to our previous report. [START_REF] Amgar | Near Ultra-Violet to Mid-Visible Band Gap Tuning of Mixed Cation Rb x Cs 1-x PbX 3 (X = Cl or Br) Perovskite Nanoparticles[END_REF] First, 0.188 mmol of PbX 2 was mixed with 0.5 mL of OA, 0.5 mL of OLA, and 5 mL of ODE in an additional 100 mL 3-neck flask (for X=Cl, 1 ml of TOP was added to the flask). The solution was degassed for 1 h under vacuum at 120 °C and then heated to 150 °C under an Argon flow. The reaction was carried out by injecting 0.4 mL of the Rb/Cs-oleate precursor solution into the PbX 2 precursor solution using a preheated syringe. The reaction was quenched using an ice bath after a few seconds.

Isopropanol was added to the crude solution in a volume ratio of 2 : 1 and the NCs were centrifuged at 6000 rpm for 10 min, the participate was dispersed in hexane, isopropanol was added again in a volume ratio of 1 : 1, and the NCs were centrifuged again at 6000 rpm for 10 min. The purified NCs were dispersed in hexane for further characterization.

Synthesis of Rb 6 Pb 5 Cl 16 NPs

Rb 6 Pb 5 Cl 16 NPs were synthesized according to our previously reported procedure. [START_REF] Amgar | Novel Rubidium Lead Chloride Nanocrystals: Synthesis and Characterization[END_REF] Briefly, 0.138 g of Rb 2 CO 3 was mixed with 625 μl of OA and 7.5 ml of ODE in a 100 ml three-neck flask. The solution was degassed for 1 h under vacuum at 120 °C and then heated to 150 °C under argon flow.

Next, 0.052 g of PbCl 2 was mixed with 0.5 ml of OA, 0.5 ml of OLA, 1 ml of TOP, and 5 ml of ODE in an additional 100 ml three-neck flask. The mixture was degassed for 1 h under vacuum at 120 °C and then heated to 150 °C under argon flow. Then 0.4 ml of the Rb-oleate precursor solution was injected into the PbCl 2 precursor solution. The reaction was quenched after a few seconds using an ice bath. Isopropanol was added to the crude solution in a volume ratio of 1:1 and the NCs were centrifuged at 6000 rpm for 10 min. The last stage was repeated once and then the purified NCs were dispersed in hexane for further characterization.

High-resolution transmission electron microscopy (HRTEM)

Scanning TEM high angle annular dark field (STEM-HAADF) images and the FFT of atomic resolution images were captured with a probe-corrected high-resolution scanning transmission electron microscope, Themis Z, produced by Thermo Fisher Scientific, and operated at an accelerating voltage of 300Kv (it can also operate at 80Kv). Elemental analysis of NPs was performed with a Super-X EDS (Energy Dispersive X-Ray Spectroscopy) detector.

Samples were prepared as follows: 2 μL of the NC dispersion were dropped on a thin copper grid coated with an ultra-thin amorphous carbon film on holey carbon, and then the solvent was evaporated using a vacuum chamber.

Conclusions

In this work, we synthesized full inorganic mixed cation perovskite NPs. We introduced Cs + and Rb + cations at different concentrations into these NPs. STEM-HAADF and FFT analysis were used to obtain an atomic level resolution of these NPs. It was found that in cases when X=0.4 and 0.6, Rb is located in the core of the nanoparticle, whereas Cs was observed in the shell. Increasing the concentration to X=0.8 revealed that phase separation exists between Rb and Cs. DFT calculations showed that a stable nanoparticle is observed when Rb is located far from the NP surface; however, when Rb atoms are located on the NP surface, a non-stable structure is observed, which leads to phase separation. Moreover, pure Rb NPs were synthesized, forming a non-perovskite structure with Pb 0 as the nucleation sites. This work reveals the importance of atomic level investigations to better understand the properties and stability of these fundamentally interesting mixed cation perovskite NPs. The research here can generally advance our understanding of the bulk mixed cation perovskites. Moreover, it can

  Figure 1d shows the detected elements as a function of the distance. It can be seen that there is no clear difference exists between the intensities of the elements. In this NP orientation (zone axis [100]) the Cs atoms are ordered as a monoatomic column; Pb and Br lead to mixed columns and finally, Br monoatomic columns are also present. However, Pb and Br can hardly be recognized because Br is the lighter element in the structure.

Figure 1 :

 1 Figure 1: (a) STEM image of CsPbBr 3 NPs with atomic resolution. Inset: FFT of a single particle. (b) Absorbance and photoluminescence spectra of the CsPbBr 3 NPs. (c) HAADF STEM image; the arrow denotes the scanned area. (d) The arrow in Figure 1c denotes the chemical elemental analysis as a function of the distance. The black lines denote where each NP starts and ends.

  A c c e p t e d M a n u s c r i p t Increasing the amount of Rb in the NPs results in mixed inorganic cation perovskite NPs, as shown in Figure 2, when X=0.2 where Rb 0.2 Cs 0.8 PbBr 3 .

Figure 2 :

 2 Figure 2: (a) HAADF STEM of Rb 0.2 Cs 0.8 PbBr 3 NPs. Inset: FFT of a single particle. (b) Absorbance and photoluminescence spectra of Rb 0.2 Cs 0.8 PbBr 3 NPs. (c) EDS maps of Rb and Cs; Rb is denoted in green and Cs in magenta. The arrow shows the integrated profile as presented in Figure d. (d) Rb/(Cs+Rb) and Cs/(Cs+Rb) ratios as a function of the distance.

  A c c e p t e d M a n u s c r i p t

Figure 3 :

 3 Figure 3: (a) HAADF STEM of Rb 0.4 Cs 0.6 PbBr 3 NPs. Inset: FFT of a single particle. (b) Absorbance and photoluminescence spectra of the Rb 0.4 Cs 0.6 PbBr 3 NPs. (c) EDS maps indicating Rb in green and Cs in magenta. The arrow shows the profile position as presented in Figure d. (d) Rb/(Cs+Rb) and Cs/(Cs+Rb) ratios as a function of the distance.

  . The arrow in Figure 4c corresponds to the line scan in Figure 4d, which indicates the higher concentration of Cs in Page 7 of 16 ACS Paragon Plus Environment Chemistry of Materials

AFigure 4 :

 4 Figure 4: (a) Rb 0.6 Cs 0.4 PbBr 3 NPs with atomic resolution (by a HAADF detector). Inset: FFT of a single particle. (b) Absorbance and photoluminescence spectra of the Rb 0.6 Cs 0.4 PbBr 3 NPs. (c) EDS in the TEM denoting Rb in green and Cs in magenta. The arrow shows the EDS profile as presented in Figure d. (d) The Rb/(Cs+Rb) and Cs/(Cs+Rb) ratios as a function of the distance. The black line denotes where one NP is and the other starts.

  A c c e p t e d M a n u s c r i p t nicely illustrated by the DFT modeling; Pos1 in Figure S8 shows a case where three Rb cations are located on the NP surface; this results in the most unstable structure compared with the other cases.

Figure 5 :

 5 Figure 5: (a) Rb 0.8 Cs 0.2 PbBr 3 NPs (HAADF). Inset: FFT of a single particle. (b) Absorbance and photoluminescence spectra of Rb 0.8 Cs 0.2 PbBr 3 NPs. (c) EDS in the TEM; Rb is denoted in green and Cs in magenta. The arrow shows the scanned area, as presented in Figure d. (d) The Rb/(Cs+Rb) and Cs/(Cs+Rb) ratios as a function of the distance.

  A c c e p t e d M a n u s c r i p t

Figure 6 :

 6 Figure 6: (a) HR-TEM image of single Rb 6 Pb 5 Cl 16 NP and the corresponding electron diffraction (inset). (b) Side profiles for the three most stable structures of three different compositions. The cases 1Cs:1Rb, 2Cs:2Rb, and 3Cs:3Rb correspond to when X=0.125, X=0.25, and X=0.375, respectively and with Rb x Cs (1-x) PbY 3 (Y=Cl, Br). (c) HR-TEM of Rb 6 Pb 5 Cl 16 NPs; the white spots indicate other phases present in these NPs. (d) Schematic illustration of increasing Rb concentration in the NPs. Magenda represent the Cs + and green the Rb + .

  A c c e p t e d M a n u s c r i p t

A c c e

  p t e d M a n u s c r i p t The ab initio simulations were performed on HPC resources of TGCC and CINES under allocation 2019-A0060906724 sponsored by GENCI (Grand Equipement National de Calcul Intensif). L.E. would like to acknowledge the Israel Science Foundation grant No. 937/18. J. E. is a senior member of the Institut Universitaire de France.

Table 1 :

 1 Miller indices, the PL max and the full width half maximum (FWHM) for different additions of Rb.

Acknowledgments

This work was supported within the framework of the Israeli-French scientific cooperation (Joint Research Projects 2019-2021 entitled: ALLPOA (From Atomic Level to layered Perovskite for Optoelectronic Applications)) by the Ministry of Science & Technology of the State of Israel (MOST) and France's Centre National de la Recherche Scientifique (CNRS).

Absorbance spectra were recorded using a Jasco V-670 spectrophotometer. Photoluminescence (PL) measurements were performed using an L-shaped spectrofluorometer (Edinburgh Instruments FL920). The Cl-and Br-based NPs were excited at 320 and 400 nm, respectively. The emission was collected at 90° in the range of 350-450 nm for Cl and 400-600 nm for Br.

Computational Details

The computational details can be found in the supporting information.

ACS Paragon Plus Environment

Chemistry of Materials

A c c e p t e d M a n u s c r i p t

open the way for new directions to improve the stability of halide perovskites and control their properties.

Supporting information

Supporting information is available free of charge via the internet at http://pubs.acs.org. The supporting information includes: Additional data extracted from the STEM photos and the EDS elemental maps. A Detailed explanation of the theoretical calculations and theory and the calculated theoretical energies of 1Cs:1Rb, 2Cs:2Rb and 3Cs:3Rb at different positions.

TOC


Florian Lemarchand

Eduardo Fernandes Montesuma

Maxime Pelcat

Erwan Nogues

OPENDENOISING: AN EXTENSIBLE BENCHMARK FOR BUILDING COMPARATIVE STUDIES OF IMAGE DENOISERS

Keywords:

Image denoising has recently taken a leap forward due to machine learning. However, image denoisers, both expert-based and learning-based, are mostly tested on well-behaved generated noises (usually Gaussian) rather than on real-life noises, making performance comparisons difficult in real-world conditions. This is especially true for learning-based denoisers which performance depends on training data. Hence, choosing which method to use for a specific denoising problem is difficult.

This paper proposes a comparative study of existing denoisers, as well as an extensible open tool that makes it possible to reproduce and extend the study. MWCNN is shown to outperform other methods when trained for a real-world image interception noise, and additionally is the second least compute hungry of the tested methods. To evaluate the robustness of conclusions, three test sets are compared. A Kendall's Tau correlation of only 60% is obtained on methods ranking between noise types, demonstrating the need for a benchmarking tool.

INTRODUCTION

Image denoising, as a sub-domain of image restoration, is an extensively studied problem. The objective of a denoiser is to restore a cleaned image signal x from an observation y considered to be a noisy or corrupted version of an original image x. y is generated by a noise function H such that y = H(x). If the noise is additive, H is a sum and y = x + η, where η is a realisation of H. There exist plenty of noise models to represent H among which examples of frequently used models are additive Gaussian, Poisson, salt and pepper or speckle noises. State of the art denoisers are constantly progressing in terms of noise elimination level [START_REF] Dabov | Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering[END_REF][START_REF] Zhang | Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising[END_REF][START_REF] Liu | Multilevel Wavelet-CNN for Image Restoration[END_REF]. However, most techniques are tailored for and evaluated on a given noise distribution, exploiting its probabilistic properties to distinguish it from the signal of interest. On the specific case of Gaussian additive noise, current denoisers are approaching theoretical bounds [START_REF] Chatterjee | Is Denoising Dead?[END_REF].

Besides the largely addressed well-behaved noise models, image restoration is also concerned by more complex noise distributions. While these distributions are application specific, they are real-world cases directly issued from identified technical needs such as image interception in difficult conditions.

In a context where new methods are constantly appearing, it is challenging to fairly compare emerging methods to previous ones. Moreover, when a real-world noise needs to be eliminated, it is difficult to determine which of method is the best for the given noise characteristics. Even if most state of the art methods are evaluated on the de-facto standard databases (e.g. BSD [START_REF] Martin | A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[END_REF], 12 well-known images as Lenna, Cameraman, etc.), methods addressing specific noises and image types have to be evaluated on tailored databases.

In this context, the contributions of this paper are:

• An extensible and open-source benchmark for comparing image restoration methods.

• A comparative study of current denoisers on mixture and interception noise elimination, as a use case for the benchmark.

The paper is organized as follows. Section 2 presents state of the art methods for image denoising as well as existing solutions to benchmark them. Section 3 describes the proposed benchmark. The comparative study, covering six restoration methods, is proposed in Section 4. Section 5 concludes the paper.

RELATED WORK

Related Work on Benchmarks of Image Denoisers

An active research domain in complex noise restoration is photograph restoration. This domain aims at removing a noise introduced by sensor hardware defects. Supervised datasets can be built by calibrating a sensor and hence obtaining pairs of clean and noisy samples. Darmstadt [START_REF] Plotz | Benchmarking Denoising Algorithms with Real Photographs[END_REF] and PolyU [START_REF] Xu | Real-world Noisy Image Denoising: A New Benchmark[END_REF] datasets are such datasets. Authors propose to use their datasets as a means for benchmarking denoising algorithms. This work is complementary to our proposed benchmark that can adapt to different datasets.

Open-source projects have been created to benchmark denoising methods. The University of Toronto proposes a benchmark1 to provide reproducibility for a method proposed in 2010 [START_REF] Estrada | Stochastic Image Denoising[END_REF]. This benchmark is tailored to the solution and not built to be extended. Another unpublished benchmark exists that implements denoising as well as other restoration algorithms such as super-resolution 2 . The benchmark is limited to learning-based, Python-implemented and pre-trained methods. The latter limitation drastically reduces the use of such benchmark for complex noises. Indeed, most state of the art methods, when delivered trained, are trained on well-behaved noise. This paper proposes a benchmark extensible in several aspects. Indeed, the user can introduce his datasets, denoising methods, and metrics.

Related Work on Image Denoisers

Image denoising techniques are as old as image sensors whose defects they counteract. Current denoising solutions are either expertbased denoisers, human crafted based on an expertise of artifacts or Block-Matching 3D (BM3D) [START_REF] Dabov | Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering[END_REF] is a state-of-the-art expertbased method for Additive White Gaussian Noise (AWGN) removal. When introduced in 2007, BM3D clearly outperformed the previous state of the art TNRD method by a good margin of 2dB. BM3D performs block matching to find patches with similar content in the image and uses collaborative filtering, thresholding and Wiener filtering into the transform domain to restore the image.

Authors of [START_REF] Vincent | Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion[END_REF] were the firsts to propose an encoding/decoding model with denoising criterion. Their proposal named stacked autoencoder learns to map the noisy image to a latent space (encoding) and projects back the latent representation to the input space (decoding) to produce the denoised image. More recent auto-encoders have been proposed such as RED [START_REF] Mao | Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections[END_REF] that adds convolutionnal layers and uses skip connections to better keep image priors throughout the encoding/decoding pipeline.

Following these premises of denoising auto-encoders, several Convolutional Neural Network (CNN) methods have emerged such as Denoising Convolutional Neural Network (DnCNN) [START_REF] Zhang | Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising[END_REF]. DnCNN exploits residual learning, i.e. it learns to isolate the noise H from the corrupted sample to later remove this noise instead of directly recovering the latent clean signal. Multi-level Wavelet Convolutionnal Neural Network (MWCNN) [START_REF] Liu | Multilevel Wavelet-CNN for Image Restoration[END_REF] is also CNN-based. Its novelty lies in the symmetrical use of wavelet and inverse wavelet transforms into the contracting and expanding parts of a U-Net [START_REF] Ronneberger | U-Net: Convolutional Networks for Biomedical Image Segmentation[END_REF] architecture. The use of wavelet enables safe subsampling with no information loss providing a better recovering of textures and sharp structures.

The most recent learning-based methods are less supervised, i.e. they require less noisy/clean image pairs to train. In Noise2Void [START_REF] Krull | Noise2void -Learning Denoising From Single Noisy Images[END_REF] and Noise2Noise [START_REF] Lehtinen | Noise2noise: Learning Image Restoration without Clean Data[END_REF], authors propose a tactic to train a denoising model using a single noise realisation. Authors introduce the idea of blind-spot masking during training. They claim that the essential advantage of that strategy is to avoid to learn the identity due to the masking of the central value of the receptive field.

These methods are evaluated on well-behaved noises (typically AWGN) for the tests to be easily reproducible. Only Noise2Void is evaluated on medical images subject to complex noise. In the following, our open benchmark is proposed to assess denoisers' quality.

PROPOSED BENCHMARK

Considering the above discussed issues with existing benchmarks, we propose the OpenDenoising benchmark illustrated in Fig. 1. It is an open-source tool with tutorials and documentation3 released under a CeCILL-C license. OpenDenoising is implemented in Python and has been designed for extensions. Adding a new denoiser to the benchmark is a matter of minutes following a tutorial and opens for comparison with the built-in methods evoked in Section 4. For learning-based methods, the application is compatible and tested with most major frameworks (Tensorflow, Keras, Pytorch, Matlab). For learning-based training and evaluation, it is possible to use one or several datasets either supervised or not. Any scalar metric being coded in Python can be used in the benchmark. Several preprocessing functions, e.g. for data augmentation, are provided, and custom functions can be introduced.

The user chooses whether a training is required for a method and in that case selects training parameters. Once the training is launched, monitors can be output by the benchmark to observe the learning phase. When trained models are available, evaluation is launched with custom or built-in metrics. The results are outlined using custom or built-in plots and/or stored as images or csv summaries.

As an example of OpenDenoising versatility, it would be possible to extends the benchmark to classification methods only implementing custom evaluations metrics. Other potential usages of OpenDenoising include: study the extensibility of methods to new applications (see Section 4), study the strategies for re-training offthe-shelf methods (from scratch or with fine-tuning), and tune hyperparameters. The experimental results presented in the next section exploit OpenDenoising to build a comparative study of state of the art denoisers on various types of noise.

A COMPARATIVE STUDY OF DENOISERS

In this section, we apply top-ranking denoisers to images with various noises. For comparison fairness of training-based methods, no data augmentation is made and the same training datasets are used. This setup excepted, methods are trained (when applicable) using original papers parameters and training strategies. Four noise types with increasing complexity are exploited to observe the behavior of the studied denoisers. Peak Signal to Noise Ratio (PSNR) and Structural Similarity (SSIM) are used to respectively evaluate the objective and subjective quality of the denoised image.

Gaussian Noise

First, Gaussian noise is denoised to test the methods in their original conditions. Denoisers are evaluated on a common noisy dataset corrupted with AWGN. The underlying data is made of 10k natural images extracted from the ImageNet [START_REF] Deng | Im-ageNet: A large-scale hierarchical image database[END_REF] on Fig. 4. Fig. 3 shows in boxplots the 10th, 25th, 75th and 90th percentiles of PSNR results as well as the median. To focus on difficult noises, the maximum noise level commonly found in papers is picked, namely σ = 50. Experimental results, shown on the first line of Table 1, are coherent with the published ones, though slightly under because no data augmentation is applied. On Gaussian noise, RED30 outperforms other methods (by a limited 0.15dB ∆PSNR and 1% ∆SSIM) but it is also the most costly in terms of number of parameters.

Mixture Noise

Complicating the denoising task, a mixture noise is then studied. This mixture noise is constructed through the successive corruption of the samples by the previously used AWGN (σ = 50) and an additional salt and pepper corruption (20% of corrupted pixels, half 0, half maximum). The clean data is conserved and this noise mixture roughly models the behaviour of an image sensor introducing Gaussian noise because of its hardware non-uniformity and salt and pepper due to pixel defects. Fig. 3 shows that learning-based methods perform consistently better than BM3D. BM3D is here used out of its original objective (i.e. Gaussian denoising) and thus performs poorly. Another information brought by mixture noise is that Noise2Void clearly underperforms compared to other learning-based methods. This is not surprising considering the addition of salt and pepper noise that damages the spatial coherence used as a hypothesis in the Noise2Void strategy. RED10, RED30, DnCNN and MWCNN have close performances with a narrow victory for RED30 (0.08dB ∆PSNR).

Interception Noise

A real-world complex noise is now studied, generated by intercepting images from Electro Magnetic (EM) emanations. Electronic devices produce EM emanations that not only interfere with radio devices but also compromise the data they handle. A third party performing a side-channel analysis can recover internal information from both analog [START_REF] Van Eck | Electromagnetic radiation from video display units: An eavesdropping risk?[END_REF], and digital circuits [START_REF] Kuhn | Compromising Emanations of LCD TV Sets[END_REF]. Following an eavesdropping procedure [START_REF] Lemarchand | Electro-Magnetic Side-Channel Attack Through Learned Denoising and Classification[END_REF], it is possible to build a supervised dataset made of pairs of reference images, originally displayed on a screen, and their intercepted noisy versions. Interception strongly damages the images and denoising is necessary to interpret their content. For reproducibility, we released the dataset used for this study 4 . It contains more than 120k samples.

To study the noise complexity, the intercepted clean samples are also artificially corrupted using AWGN with σ = 50. The resulting samples are called interception-like. As shown in Table 1, most methods perform well on that denoising task. The clean content of the intercepted samples contains black characters printed on a white background. The latent clean distribution of samples is thus not an issue to denoise with learning-based methods, Noise2Void excluded. Only BM3D and Noise2Void have problems with background restoration (see Fig. 4). This phenomena is due to the correlation of the samples content and the noising process. The background of the clean samples is fully white. When applying AWGN, half of the noise coefficients are negative and samples are clipped to an integer format. Thus, the assumption of a Gaussian distribution does not hold, leading to poor restoration results with non-supervised methods, unable to adapt.

Table 1, Fig. 3 and Fig. 4 show the results of denoising methods applied to interception noise. Metrics drop for all methods on this complex noise. Noise2Void does not manage to denoise at all. As explained in the original paper, Noise2Void has difficulties with noise correlated between several pixels which is here the case. BM3D is built for AWGN and is not trainable, hence the poor results in that case. Others learned methods like RED10 and DnCNN produce interesting denoising but perceptual results of Fig. 4 show hardly interpretable samples, not revealed by SSIM. RED30 and MWCNN are the best-performing methods for interception noise removal but still with some extant artifacts.

Discussion

Different conclusions can be drawn from the above experiments using OpenDenoising and 4 different datasets.

First, Fig. 3 shows that the performance ranking between methods strongly depends on noise type, training dataset and evaluation dataset. When calculating Kendall's Tau correlation coefficient [START_REF] Javaid | Fidelity metrics for estimation models[END_REF], a value of 60% is obtained between Gaussian noise ranking and Interception noise ranking, ranking based on the mean PSNR. This correlation coefficient, while being high -Kendall's Tau is in [-1; 1]; -1 and 1 respectively meaning fully discordant and fully concordant rankings -, shows the need for a benchmark such as the proposed. It automates the comparison process and the selection of a given method for a denoising problem. As an example, it would be a wrong choice to pick RED30 instead of MWCNN for interception restoration based on the original paper evaluations (e.g. Gaussian evaluation). MWCNN is indeed both more efficient and less computationally intensive than RED30, as shown in Fig. 2.

Results of Table 1, Fig. 3 and Fig. 4 show a growing gap between expert-based and learning-based method as the complexity of the denoising increase. This can be explained by the flexibility of learning-based models and the advanced information brought by a supervised training. This is evidenced by the low performance of the non-supervised Noise2Void on Mixture and Interception noises.

As stated in several studies, PSNR and SSIM do not suit well the assessment of interception restoration. Fig. 4 shows that while evaluation metrics on interception noise are reasonably good (PSNR/SSIM values around 20dB/0.9), the perceptual quality is poor. The explanation lies in the latent content of the samples, made of black characters on a white background. A good background restoration is sufficient to raise good evaluation metrics. This issue is evoked in [START_REF] Johnson | Perceptual Losses for Real-Time Style Transfer and Super-Resolution[END_REF] where authors propose a different evaluation metric to overcome the problem.

CONCLUSIONS

In this paper, the OpenDenoising tool has been proposed. OpenDenoising benchmarks image denoisers and aims at comparing methods on a common ground in terms of datasets, training parameters and evaluation metrics. Supporting several languages and learning frameworks, OpenDenoising is also extensible and open-source. The second contribution of the paper is a comparative study of image restoration in the case of a complex noise source.

Three major conclusions arise from the comparative study. First, the difference in terms of performance between expert-based and learning-based methods rises as the complexity of the noise grows. Second, the ranking of methods is strongly impacted by the nature of the noises. Finally, MWCNN proves to be the best method for the considered real-world interception restoration task. It slightly outperforms DnCNN and RED30 while being substantially faster.

These results show that restoring an image from a complex noise is not universally solved by a single method and that choosing a denoiser requires automated testing. Our future work includes the design of new denoising methods for intercepted images.

Fig. 1 .

 1 Fig. 1. OpenDenoising Block diagram. Users can tune datasets, metrics, denoising models and evaluation functions. OpenDenoising produces denoised samples as well as performance metrics. of statistical noise properties, or learning-based denoisers leveraging on latent image priors extracted from data.Block-Matching 3D (BM3D)[START_REF] Dabov | Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering[END_REF] is a state-of-the-art expertbased method for Additive White Gaussian Noise (AWGN) removal. When introduced in 2007, BM3D clearly outperformed the previous state of the art TNRD method by a good margin of 2dB. BM3D performs block matching to find patches with similar content in the image and uses collaborative filtering, thresholding and Wiener filtering into the transform domain to restore the image.Authors of[START_REF] Vincent | Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion[END_REF] were the firsts to propose an encoding/decoding model with denoising criterion. Their proposal named stacked autoencoder learns to map the noisy image to a latent space (encoding) and projects back the latent representation to the input space (decoding) to produce the denoised image. More recent auto-encoders have been proposed such as RED[START_REF] Mao | Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections[END_REF] that adds convolutionnal layers and uses skip connections to better keep image priors throughout the encoding/decoding pipeline.Following these premises of denoising auto-encoders, several Convolutional Neural Network (CNN) methods have emerged such as Denoising Convolutional Neural Network (DnCNN)[START_REF] Zhang | Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising[END_REF]. DnCNN exploits residual learning, i.e. it learns to isolate the noise H from the corrupted sample to later remove this noise instead of directly recovering the latent clean signal. Multi-level Wavelet Convolutionnal Neural Network (MWCNN)[START_REF] Liu | Multilevel Wavelet-CNN for Image Restoration[END_REF] is also CNN-based. Its novelty lies in the symmetrical use of wavelet and inverse wavelet transforms into the contracting and expanding parts of a U-Net[START_REF] Ronneberger | U-Net: Convolutional Networks for Biomedical Image Segmentation[END_REF] architecture. The use of wavelet enables safe subsampling with no information loss providing a better recovering of textures and sharp structures.The most recent learning-based methods are less supervised, i.e. they require less noisy/clean image pairs to train. In Noise2Void[START_REF] Krull | Noise2void -Learning Denoising From Single Noisy Images[END_REF] and Noise2Noise[START_REF] Lehtinen | Noise2noise: Learning Image Restoration without Clean Data[END_REF], authors propose a tactic to train a denoising model using a single noise realisation. Authors introduce the idea of blind-spot masking during training. They claim that the essential advantage of that strategy is to avoid to learn the identity due to the masking of the central value of the receptive field.These methods are evaluated on well-behaved noises (typically AWGN) for the tests to be easily reproducible. Only Noise2Void is evaluated on medical images subject to complex noise. In the following, our open benchmark is proposed to assess denoisers' quality.

Fig. 2 .

 2 Fig. 2. Inference time (log scale) for different denoisers. Image resolution is 256 × 256. Noise2Void is the fastest method by almost 10 folds. MWCNN, RED10, DnCNN and RED30 are close to each other. BM3D is the slowest with an inference time over one second. Setup: Intel Xeon W-2125 CPU and Nvidia GTX1080 Ti GPU.

Fig. 3 .

 3 Fig. 3. PSNR of denoised images on (a) Gaussian noise, (b) Mixture and (c) Interception noise. Outliers are not displayed.

Fig. 4 .

 4 Fig. 4. From top to bottom, one sample per dataset is shown noisy (left), denoised with different denoisers (middle) and clean (right). PSNR/SSIM are displayed for each sample. Images with the best compromise between PSNR and SSIM metrics are yellow-boxed.

Table 1 .

 1 evaluation set. Average PSNR and SSIM are shown in Table1 and example images displayed Average evaluation of PSNR and SSIM metrics on test sets for, from top to bottom row: AWGN with noise level σ = 50; Mixture noise made of AWGN with noise level σ = 50 and salt and pepper 20% ; Interception-Like noise being interception reference samples noised with AWGN with noise level σ = 50 ; Interception noise. The test set for each noise is made of 200 samples excluded from training set.

		Dataset	No Denoising BM3D RED10 RED30 DnCNN-B MWCNN Noise2Void
		Gaussian	14.96	23.90	25.52	25.82	25.67	25.49	23.41
	PSNR	Mixture Interception-Like	10.58 17.16	18.29 22.04	24.25 51.56	24.58 52.08	24.50 51.66	24.30 51.16	19.93 21.70
		Interception	9.46	9.61	22.59	23.46	23.04	23.66	9.46
		Gaussian	0.24	0.67	0.71	0.73	0.72	0.72	0.62
	SSIM	Mixture Interception-Like	0.12 0.11	0.31 0.98	0.67 0.99	0.68 0.99	0.68 0.99	0.68 0.99	0.51 0.98
		Interception	0.32	0.73	0.94	0.96	0.95	0.96	0.47

www.cs.utoronto.ca/ ˜strider/Denoise/Benchmark/

https://github.com/titsitits/ open-image-restoration

https://github.com/opendenoising/benchmark

https://github.com/opendenoising/interception_ dataset

ACKNOWLEDGEMENTS

This work is supported by the "Pôle d'Excellence Cyber", initiative of the French Ministry of the Armed Forces and Bretagne council.


Florian Lemarchand

Cyril Marlin

Florent Montreuil

Erwan Nogues

Maxime Pelcat

ELECTRO-MAGNETIC SIDE-CHANNEL ATTACK THROUGH LEARNED DENOISING AND CLASSIFICATION

Keywords: Electro-Magnetic Side-Channel, Denoising, Automation

This paper proposes an upgraded Electro Magnetic (EM) sidechannel attack that automatically reconstructs the intercepted data. A novel system is introduced, running in parallel with leakage signal interception and catching compromising data on the fly. Leveraging on deep learning and Character Recognition (CR) the proposed system retrieves more than 57% of characters present in intercepted signals regardless of signal type: analog or digital. The building of the learning database is detailed and the resulting data made publicly available. The solution is based on Software-Defined Radio (SDR) and Graphics Processing Unit (GPU) architectures. It can be easily deployed onto existing information systems to detect compromising data leakage that should be kept secret.

INTRODUCTION

All electronic devices produce Electro Magnetic (EM) emanations that not only interfere with radio devices but also compromise the data handled by the information system. A third party may perform a side-channel analysis and recover the original information, hence compromising the system privacy. While pioneering work of the domain focused on analog signals [START_REF] Van Eck | Electromagnetic radiation from video display units: An eavesdropping risk?[END_REF], recent studies extend the eavesdropping exploit using an EM side-channel attack to digital signals and embedded circuits [START_REF] Kuhn | Compromising Emanations of LCD TV Sets[END_REF]. The attacker's profile is also taking on a new dimension with the increased performance of Software-Defined Radio (SDR). With recent advances in radio equipment, an attacker can leverage on advanced signal processing to further stretch the limits of the side-channel attack using EM emanations [START_REF] Genkin | Synesthesia: Detecting Screen Content via Remote Acoustic Side Channels[END_REF]. With the fast evolution of deep neural networks, an attacker can extract patterns or even the full structured content of the intercepted data with a high degree of confidence and a limited execution time.

In this paper, a learning-based method is proposed with the specialization of Mask R-CNN [START_REF] He | Mask R-CNN[END_REF] as a denoiser and classifier. A complete system is demonstrated, embedding SDR and deep-learning, that detects and recovers leaked information at a distance of several tens of meters. It provides an automated solution where the data is interpreted directly. The solution is compared to other system setups.

The paper is organized as follows. Section 2 presents existing methods to recover information from EM emanations. Section 3 describes the proposed method for automatic character retrieval. Experimental results and detailed performances are exposed in Section 4. Section 5 concludes the paper. This work is supported by the "Pôle d'Excellence Cyber", initiative of the French Ministry of the Armed Forces and the Bretagne council.

RELATED WORK

This paper focuses on two areas: EM side channel attacks on information systems and learning-based techniques that can recover information from noisy environments.

Van Eck et al. [START_REF] Van Eck | Electromagnetic radiation from video display units: An eavesdropping risk?[END_REF] published the first technical reports revealing how involuntary emissions originating from electronics devices can be exploited to compromise data. While the original work of the domain targeted Cathode Ray Tube (CRT) screens and analog signals, Kuhn et al. [START_REF] Kuhn | Compromising Emanations of LCD TV Sets[END_REF] propose to use side-channel attacks to extract confidential data from Liquid Crystal Displays (LCDs), targeting digital data. Subsequently, other types of systems have been attacked. Vuagnoux et al. [START_REF] Vuagnoux | Compromising Electromagnetic Emanations of Wired and Wireless Keyboards[END_REF] extend the principle of EM side-channel attack to capture data from keyboards and, in their recent work, Hayashi et al. present interception methods based on SDR targeting laptops, tablets [START_REF] Hayashi | A Threat for Tablet PCs in Public Space: Remote Visualization of Screen Images Using EM Emanation[END_REF] and smartphones [START_REF] Hayashi | Remote Visualization of Screen Images Using a Pseudo-Antenna That Blends Into the Mobile Environment[END_REF]. Due to their low cost, SDRs increase the potential of attacks from military organizations to hackers. SDR also opens up new post-processing opportunities that improve attacks. De Meulemeester et al. [START_REF] Meulemeester | Synchronization retrieval and image reconstruction of a video display unit exploiting its compromising emanations[END_REF] leverage on SDR to enhance the performance of the attack and automatically find the structure of the captured data. When the intercepted emanation is originally 2D, retrieving the synchronization parameters of the targeted information system enables the captured EM signal to be transformed from a vector to an image, reconstructing the 2-dimensional sensitive visual information. This reconstruction process is called the rastering.

When retrieving visual information from an EM signal, an important part of the original information is lost through the leakage and interception process. This loss leads to a drop of the Signal to Noise Ratio (SNR) and a deterioration of spatial coherence into the reconstructed samples in the case of image data. Hence, denoising methods are needed. Image denoising by signal processing techniques has been extensively studied since it is an important step in many computer vision applications. BM3D [START_REF] Dabov | Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering[END_REF], proposed by Dabov et al., is a the state-of-the-art methods for Additive White Gaussian Noise (AWGN) removal using non-learned processing. BM3D uses thresholding and Wiener filtering into the transform domain. Block-Matching 3D (BM3D) is used in the experiments of Section 4.

Deep learning algorithms have recently stood out from the crowd for solving many signal processing problems. These trained models have an extreme ability to fit complex problems. Recent Graphics Processing Unit (GPU) architectures have been optimized to support deep learning workloads and have fostered ever deeper networks, mining structured information from data and providing results where expert-based algorithms fail. The spread of deep learning has occurred in the domain of image denoising and several models initially developed for other applications have been turned into denoisers. Denoising Convolutional Neural Network (DnCNN) [START_REF] Zhang | Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising[END_REF] is a Convolutional Neural Network (CNN) designed to blindly remove AWGN, without prior knowledge on noise level. Others techniques such as denoising autoencoders [START_REF] Vincent | Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion[END_REF][START_REF] Ronneberger | U-Net: Convolutional Networks for Biomedical Image Segmentation[END_REF] are able to denoise images without restriction on the type of noise. Autoencoders algorithms learn to map their input to a latent space (encoding) and project back the latent representation to the input space (decoding). Autoencoders learn a denoising model by minimizing a loss function which evaluates the difference between the autoencoder output and the reference. Advanced methods, such as Noise2Noise [START_REF] Lehtinen | Noise2noise: Learning Image Restoration without Clean Data[END_REF], infer denoising strategies without any clean input reference data. Noise2Noise algorithm learns a representation of the noise by looking only at noisy samples.

Learning-based models perform well in various denoising but with strong hypothesis regarding the distribution of the noise to be withdrawn [START_REF] Lemarchand | OpenDenoising: an Extensible Benchmark for Building Comparative Studies of Image Denoisers[END_REF]. AWGN assumption is often used. In the considered problem, certain components of the noise are non-randomly distributed and have a spatial coherence (between pixels). Additionally, information is damaged (partially lost and spread over several pixels) by the interception/rastering process. None of the previously exposed methods is tailored for such noise and distortion natures, calling for a novel experimental setup.

Conventional approaches exist to protect devices from eavesdropping. Such approaches appear under different code names such as TEMPEST [START_REF]NACSIM 5000 TEMPEST FUN-DAMENTALS[END_REF] or Emission Security (EMSEC) and consist of shielding devices [START_REF] Kuhn | Compromising Emanations of LCD TV Sets[END_REF] to nullify the emanations, or using fonts that minimize the EM emanations [START_REF] Kuhn | Soft Tempest: Hidden Data Transmission Using Electromagnetic Emanations[END_REF]. However, these approaches are either costly solutions or technically hard to use in practice especially when it comes to ensure the data privacy throughout the lifecycle of a complex information system. The next section details the proposed method to enhance the EM side-channel attack.

PROPOSED SIDE-CHANNEL ATTACK

System Description

Figure 1 shows the proposed end-to-end solution. The method automatically reconstructs leaked visual information from compromising emanations. The setup is composed of two main elements. At first the antenna and SDR processing capture in the Radio Frequency (RF) domain the leaked information originating from the displayed video. Then, the demodulated signal is processed by the host computer, recovering a noisy version of the original image [START_REF] Kuhn | Compromising Emanations of LCD TV Sets[END_REF] leaving room for advanced image processing techniques. On top of proposing an end-to-end solution from the capture to the data itself, the method uses a learning-based approach. It exploits the capturing compromising signals and recognized automatically the leaked data. A first step based on a Mask R-CNN (Mask R-CNN) architecture embeds the following: denoising, segmentation, character detection/localization, and character recognition. A second step postprocesses the Mask R-CNN output. A Hough transform is done for text line detection and a Bitap algorithm [START_REF] Myers | A Fast Bit-vector Algorithm for Approximate String Matching Based on Dynamic Programming[END_REF] is applied to approximate match information. This setup detects several forms of compromising emanations (analog or digital) and automatically triggers an alarm if critical information is leaking. Next sections detail how the method is trained and integrated.

Training Dataset Construction

A substantial effort has been made on building a process that semiautomatically generates and labels datasets for supervised training. Each sample image is made up of a uniform background on which varied characters are printed. Using that process, an open data corpus of 123.610 labeled samples, specific to the problem at hand, has been created to further be used as training, validation and test datasets. This dataset is available online 1 to train denoiser architectures in difficult conditions.

The proposed setup, to be trained, denoises the intercepted sample images and extracts their content, i.e. the detected characters and their positions. The input space that should be covered by the training dataset is large and three main types of interception variability can be observed. Firstly, interception induces an important loss of the information originally existing in the intercepted data. The noise level is directly linked to the distance between the antenna and the target. Several noise levels are generated by adding RF attenuation after the antenna. That loss itself causes inconsistencies in the rasterizing stage. Secondly, EM emanations can come from different sources, using different technologies, implying in turn different intercepted samples for the same reference image. The dataset covers Video Graphics Array (VGA), Display Port (DP)-to-Digital Visual Interface (DVI) and High-Definition Multimedia Interface (HDMI) cables and connectors. Besides this unwanted variability, a synthetic third type of variability is introduced to solve the character retrieval. Many different characters are introduced in the corpus to be displayed on the attacked screen. They range from 11 to 70 points in size and they are both digits and letters, and letters are both upper and lower cases. Varied fonts, character colors and background colors, as well as varied character positions in the sample are used. Considering these different sources of variability, the dataset is built trying to get an equi-representation of the different interception conditions.

The choice has been made to display on the target screen a sample containing patches of size 256 × 256 pixels (top-left image of Figure 2). For building the dataset, having multiple patches speeds the process up because smaller samples can be derived from a single screen interception and more variability can be introduced in the dataset. The main challenge when creating the dataset lies in the samples acquisition itself. Indeed, once intercepted, the samples are not directly usable. The interception process outputs samples such as the one of Figure 2 (middle-top) where intercepted characters are not aligned (temporally and spatially) with respective reference samples. An automated method is introduced that uses the porches, artificially colored in red in Figure 2 (middle-top), to align spatially samples. Porches are detected using brute-force search of large horizontal and vertical gradients (to find vertical and horizontal porches, respectively). A validation step ensures the temporal alignment, based on the insertion of a QRCode in the upper-left patch. If the QRCode is similar between the reference and the intercepted image, the image patches are introduced in the dataset.

Data augmentation [START_REF] Mikolajczyk | Data augmentation for improving deep learning in image classification problem[END_REF] is used to enhance the dataset coverage area. It is done onto patches to add variability into the dataset and reinforce its learning capacity. Conventional methods are applied to raw samples to linearly transform them (Gaussian and median blur, salt and pepper noise, color inversion and contrast normalization).

Implemented Solution to Catch Compromising Data

In order to automate the interception of compromising data, the Mask R-CNN has been turned into a denoiser and classifier. The implementation is based on the one proposed by W. Abdulla2 . Other learning-based and expert-based signal processing methods, discussed in Section 4.2, are also implemented to assess the quality of the proposed framework. Mask R-CNN is a framework adapted from the previous Faster R-CNN [START_REF] Ren | Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks[END_REF]. The network consists of two stages. The first stage, also known as backbone network, is a ResNet101 convolutional network [START_REF] He | Deep Residual Learning for Image Recognition[END_REF] Mask R-CNN is not originally designed to be used for denoising but rather for instance segmentation. However, it fits well the targeted problem. Indeed, the problem is similar to a segmentation where signal has to be separated from noise. As a consequence, when properly feeding a trained Mask R-CNN network with noisy samples containing characters, one obtains lists of labels (i.e. characters recognition), as well as their bounding boxes (characters localization) and binary masks representing the content of the original clean sample. The setup of the classification branch allows to be language-independent and to add classes other than characters.

Two strategies can be employed to exploit Mask R-CNN components for the problem. The first idea is to draw the output masks of Mask R-CNN segmentation (Figure 3 left-hand side) and request an OCR to retrieve characters from the masks. A second possibility is to make use of the classification faculty of Mask R-CNN (Figure 3 right-hand side) and obtain a list of labels without using an OCR engine. The second method using the classifier of Mask R-CNN proves to be better in practice, as shown in Section 4.2.

The training strategy is to initialize the training process using pre-trained weights [START_REF] Mahajan | Exploring the Limits of Weakly Supervised Pretraining[END_REF] for the MS COCO [START_REF] Lin | Microsoft COCO: Common Objects in Context[END_REF] dataset. First, the weights of the backbone are frozen and the head is trained to adapt to the application. Then, the weights of the backbone are relaxed and both backbone and head are trained together until convergence. This process is done to ensure the convergence and speed up training.

EXPERIMENTAL RESULTS

Experimental Setup

The experimental setup is defined as follows: the eavesdropped display is 10 meters away from the interception antenna. A RF attenuator is inserted after the antenna. It ranges from 0 dB to 24 dB to simulate higher interception radius and generate a wide range of noise values. Compromising emanations are issued either by a VGA display, a DP-to-DVI cable or an HDMI connector. The interception system is depicted in Figure 1: the antenna is bilog, the SDR device automatically recovering parameters [START_REF] Meulemeester | Synchronization retrieval and image reconstruction of a video display unit exploiting its compromising emanations[END_REF] is an Ettus X310 receiving with a 100 MHz bandwidth to recover the compromised information with a fine granularity [START_REF] Kuhn | Compromising Emanations of LCD TV Sets[END_REF]. The host computer running post-processing has a linux operating system, an Intel R Xeon R W-2125 Central Processing Unit (CPU) and an Nvidia GTX 1080 Ti GPU. The host computer rasters the compromising data using the CPU while the proposed learning-based denoiser/classifier runs also on the GPU.

Performance Comparison Between Data Catchers

The purpose of the exposed method is to analyze compromising emanations. Once a signal is detected and rasterized, intercepted emanations should be classified into compromising or not. Figure 4 illustrates the outputs of different implemented denoisers. More examples are available at 3 . It is proposed to assess the data leak according to the ability of a model to retrieve original A ratio between the number of characters that a method correctly classifies from an intercepted sample, and the true number of characters in the corresponding clean reference is used as a metric.

The quality assessment method is the following. First, a sample containing a large number of characters is pseudo-randomly generated (similar to dataset construction). The sample is displayed on the eavesdropped screen and EM emanations are intercepted. The proposed denoising/retrieval is applied and the obtained results are compared to the reference sample. The method using Mask R-CNN produces directly a list of retrieved characters. Other methods, implemented to compare the efficiency of the proposal, use denoising in combination with the Tesseract [START_REF] Smith | An Overview of the Tesseract OCR Engine[END_REF] OCR. Tesseract is a well performing OCR engine, retrieving characters from images. It produces a list of characters retrieved from a denoised sample. As the output of Tesseract is of the same type as the output of Mask R-CNN classification, metrics can be extracted to fairly compare methods.

An end-to-end evaluation is used measuring the quality of characters classification. A F-score classically used to evaluate classification model is computed using precision and recall. precision is the number of true positives divided by the number of all positives. recall is the number of true positives divided by the number of relevant samples, the set of relevant samples being the union of true positives and false negatives. For simplification and not use an alignment process, a true positive is chosen here to be the recognition of a character truly existing in the reference sample. version of Mask R-CNN using its own classifier is better than the Tesseract OCR engine applied on Mask R-CNN segmentation mask output. It is also interesting to look at precision and recall scores that compose the F-score. Both Mask R-CNN methods perform better than other methods for the two indices. Precision is almost the same for both methods, meaning that they both present the same ratio of good decision. The difference lies in the recall score. The 0.42 recall score of the version using Tesseract is lower than the 0.57 score of the method using its own classifier, indicating that the latter version miss less characters. The main advantage of the Mask R-CNN is that the processing tasks to solve the final aim of textual information recovery are jointly optimized.

Another key performance indicator of learning-based algorithms is inference time (Table 2). The proposed implementation using Mask R-CNN infers results from an input sample of resolution 1200 × 1900 in 4.04s in average. This inference time, although lower than BM3D latency, is admittedly higher than other neural networks and hardly real-time. Nevertheless, the inference time of Mask R-CNN includes all the denoising/OCR process and provides a largely better retrieval score. In the context of a continuous listening of EM emanations, it provides an acceptable trade-off between processing time and interception performance. The optimization of the inference time could be considered as a future work with the recent advances in accelerating neural network inference [START_REF] Zhang | Caffeine: towards uniformed representation and acceleration for deep convolutional neural networks[END_REF][START_REF] He | AMC: AutoML for Model Compression and Acceleration on Mobile Devices[END_REF].

CONCLUSIONS

Handling data while ensuring trust and privacy is challenging for information system designers. This paper presents how the attack surface can be enlarged with the introduction of deep learning in an EM side-channel attack. The proposed method uses Mask R-CNN as denoiser and it automatically recovers more than 57% of a leaked information for a wide range of interception distances. The proposal is software-based, and runs on the host computer of an off-the-shelf SDR platform.

Fig. 1 :

 1 Fig.1: Experimental setup: the attacked system includes an eavesdropped screen (1) displaying sensitive information. It is connected to an information system (2). An interception chain including an SDR receiver (3) sends samples to a host computer (4) that implements signal processing including a deep learning denoiser and Character Recognition (CR).

1Fig. 2 :

 2 Fig. 2: A reference sample is displayed on the target screen (topleft). The interception module outputs uncalibrated samples. Vertical and horizontal porchs (red) helps alignment and porch withdrawal (top-right). Samples are rescaled and split into patches to obtain the same layout than the reference set.

Fig. 3 :

 3 Fig. 3: The output of Mask R-CNN may be used in two ways. The segmentation can be drawn (left) and further processed by an Optical Character Recognition (OCR), or the Mask R-CNN classifier can directly infer the sample content (right) and propose some display and confidence information.

Fig. 4 :

 4 Fig. 4: Three samples (left, middle, right) displayed different stages of the interception/denoising pipeline. From top to bottom: the reference patch displayed on the screen; the patch after rasterization (raw patch); the patches denoised with BM3D, autoencoder and Mask R-CNN. returns bounding box coordinates for each of the RoIs. The main difference between Faster R-CNN and Mask R-CNN lies in an additional Fully Convolutional Network (FCN) branch [21] running in parallel with the classification and extracting a binary mask for each RoI to provide a more accurate localization of the object of interest.Mask R-CNN is not originally designed to be used for denoising but rather for instance segmentation. However, it fits well the targeted problem. Indeed, the problem is similar to a segmentation where signal has to be separated from noise. As a consequence, when properly feeding a trained Mask R-CNN network with noisy samples containing characters, one obtains lists of labels (i.e. characters recognition), as well as their bounding boxes (characters localization) and binary masks representing the content of the original clean sample. The setup of the classification branch allows to be language-independent and to add classes other than characters.Two strategies can be employed to exploit Mask R-CNN components for the problem. The first idea is to draw the output masks of Mask R-CNN segmentation (Figure3left-hand side) and request an OCR to retrieve characters from the masks. A second possibility is to make use of the classification faculty of Mask R-CNN (Figure3right-hand side) and obtain a list of labels without using an OCR engine. The second method using the classifier of Mask R-CNN proves to be better in practice, as shown in Section 4.2.The training strategy is to initialize the training process using pre-trained weights[START_REF] Mahajan | Exploring the Limits of Weakly Supervised Pretraining[END_REF] for the MS COCO[START_REF] Lin | Microsoft COCO: Common Objects in Context[END_REF] dataset. First, the weights of the backbone are frozen and the head is trained to adapt to the application. Then, the weights of the backbone are relaxed and both backbone and head are trained together until convergence. This process is done to ensure the convergence and speed up training.

Table 1

 1 presents the results of different data catchers on a test set of 12563 patches. All denoising methods are tested using Tesseract, and compared to Mask R-CNN classification used as OCR. Tesseract is first applied to raw (non-denoised) samples as a point of reference. BM3D is the only expert-based denoising solution tested. Noise2Noise, AutoEncoder, RaGAN and UNet are different deep learning networks configured as denoisers. As shown in Table 1, Mask R-CNN classification outperforms all other methods. The

	3 https://github.com/opendenoising/extension

Table 1 :

 1 Character recognition performance for several data catchers using either denoising and Tesseract, or Mask R-CNN (Mask R-CNN) classification. Mask R-CNN classifier outperforms others methods with a 0.68 F-score on the test set.

	Denoiser	OCR	Inference Timing (s)
	Raw		0.19
	BM3D Autoencoder	Tesseract	21.8 1.15
	Mask R-CNN		4.22
	Mask R-CNN Mask R-CNN	4.04

Table 2 :

 2 Inference time for several data catchers using Tesseract or Mask R-CNN classification as OCR. Input resolution is 1200 × 1900 and it is processed using a split in 28 patches. Mask R-CNN classifier is slower than the autoencoder but still faster than BM3D.

https://github.com/matterport/Mask_RCNN