G Martinez

J Wyzula

I Mohelsky

C Faugeras

M Orlita

M Potemski

A Riedel

R Hey

K J Friedland

Polaronic Interaction in a single modulation doped GaAs quantum well revisited with the Feynman-Hellwarth-Iddings-Platzman approximation

Keywords: 67.De, 71.30.Fs, 78.66.Fd I

Absolute far-infrared magnetotransmission experiments have been performed in magnetic fields up to 33.5 T on a series of single GaAs quantum wells doped with different electron concentrations. The transmission spectra have been simulated with a multilayer dielectric model. The imaginary part of the optical response function which reveals singular features related to the electron-phonon interactions has been extracted. In addition to the expected polaronic effects due to the longitudinaloptical phonon of GaAs additional interactions with interface phonons are observed. The main interaction is analyzed quantitatively with the Feynman-Hellwarth-Iddings-Platzman model which is shown to predict correctly the concentration of carriers beyond which the Fröhlich interaction is completely screened.

Pacs number: 78.

INTRODUCTION

Phonons, electrons and their interaction play a major role in the fundamental elementary excitations in solids. In general, that interaction occurs via the deformation potential experienced by the electrons moving across the vibrating nuclei. However, in a polar material, another fundamental interaction, the Fröhlich interaction, can exist between electrons and the longitudinal optical (LO) phonon giving rise to polaronic effects (PE), the subject of many reports [1][2][3] . This process, driven by the free electron-dipole interaction, is quite strong in undoped material but very sensitive to the concentration of free carriers which screen the interaction. On a theoretical ground, PE were first studied by Lee and Pines 4 and later by Feynman 5 . It was realized that this kind of interaction could not be properly handled by perturbation theory applied to the Hamiltonian of the problem, and required a global treatment. Such an approach was proposed by Feynman, Hellwarth, Iddings and Platzman 6 , and referred to, in the following, as the FHIP model. This model allows to solve analytically the path-integral method 5 applied to the problem and, in turn, to evaluate 7 quantitatively the contribution of the Fröhlich interaction to the imaginary part of the dielectric function. It is the goal of our paper to compare the predictions of the FHIP model with experimental results.

In quasi-two-dimensional GaAs (Q2D) based structures, there is evidence of polaronic effects 8 but few experiments 9,10 have focused on cyclotron resonance (CR) in the energy region above the reststrahlen band (RB) of GaAs (this requires for GaAs a magnetic field strength beyond 23 T): these experimental conditions are essential to bring the CR transition energy in resonance with that of the LO phonon of the material, providing therefore a direct access to the imaginary part of the interaction. In the present case, a careful analysis of the optical transmission data obtained on a series of single GaAs quantum wells, imbedded between two GaAs/AlAs superlattices, as a function of magnetic fields up to 33.5 T allows a quantitative experimental evaluation of the imaginary part of the polaron interaction. These experimental results will be compared to the predictions of the FHIP model.

II. CHARACTERIZATION OF SAMPLES AND EXPERIMENTAL PROCEDURE

All samples discussed in this study have been grown by molecular beam epitaxy (MBE) on GaAs substrates. The active part of each sample is composed of a single doped GaAs quantum well (QW) with a width d w of 13 nm sandwiched between two GaAs-AlAs short period superlattices with the Si-n-type doping performed symmetrically on both sides of the QW 11 . The δ-doping is located in one of the GaAs layer of the superlattice at various distances from the QW allowing for varying the electron concentration N s in the QW. We have studied a series of samples with N s varying from 2.0 × 10 11 cm -2 to 9.0 × 10 11 cm -2 as measured, at low temperatures, by transport techniques. The mobility depends on the carrier concentration but always exceeds 10 6 cm 2 /(V.s). For most of our samples, the optical transmission in the infra-red is completely obscured in the RB energy of the GaAs substrate, a region particularly important for our study. To circumvent this problem, one can grow, at the first step of the MBE process, a thick layer of AlAs which allows one, by chemical selective etching, to remove the epi-layer from the substrate. This epi-layer breaks into small flakes, floating at the surface of the chemical bath, which are transferred to a semi-insulating wedged Si substrate. This is called the lift-off process. The clear ad-vantage of the lift-off structures for infra-red studies has been previously demonstrated 12 . We have been able to investigate two lift-off samples with different carrier concentrations. For these samples, the carrier concentration was measured before the lift-off process. In general N s can also be evaluated by fitting at low fields the intensity of the CR transition: for non lift-off samples, the agreement between transport and optical data is very good within experimental errors. However, N s for lift-off samples, as deduced from optical data, is found significantly lower (about 10 percent) than the value obtained by transport data. Therefore the lift-off process relaxes strains in the epi-layer modifying the transfer of electrons between the dopants and the QW. The mobility, however, is not significantly affected because, within the experimental errors, the line-width of the cyclotron resonance at low fields is unchanged. The larger flakes have a typical "diameter" size of about 1.5 mm and have been used to perform absolute magneto-optical transmission measurements. We are going to compare these results with some other ones obtained on non lift-off samples and already partly published 13 . This will allow us to perform a complete consistent treatment of the data for all samples.

Far infra-red magneto-optical experiments have been performed at magnetic field strengths up to 33.5 T and at a fixed temperature of 1.8 K. The sample insert was cooled down with a top black polyethylene window to keep the carrier concentration N s constant under illumination 14 . The light (provided and analyzed by a Fourier transform spectrometer) was delivered to the sample by means of light-pipe optics. All experiments were performed with non-polarized light, in the Faraday geometry with the wave vector of the incoming light parallel to the magnetic field direction and perpendicular to the plane of the samples. For lift-off samples, care has been taken to maintain the epilayer plane in the Faraday geometry. The spectral resolution was kept constant for most of the samples and equal to 2 cm -1 (≈ 0.25 meV). A Si bolometer was placed directly beneath the sample to detect the transmitted radiation. The response of this bolometer is strongly dependent on the magnetic field. Therefore, in order to measure the absolute transmission TA (B, ω), we used a rotating sample-holder and measured, for each value of B, a reference spectrum through either a hole for as-grown samples or a hole covered by a piece of Si substrate for lift-off samples (in both cases the diameter of the hole is adjusted to that of the sample). These spectra are, in turn, normalized with respect to TA(0, ω) to obtain an absolute relative transmission spectrum TR(B, ω)= TA(B, ω)/TA(0, ω) which only displays the magnetic field dependent features. With that presentation, any observed deviation of the intensity of the CR absorption with respect to that measured at low fields reveals, at the corresponding energy, an interaction which has to be explained.

III. EXPERIMENTAL RESULTS FOR LOWER DOPED SAMPLES

In Fig. 1, we display, for magnetic fields higher than 19 T, the relative transmission spectra obtained for samples (a), (b) and (c) with a carrier concentration N s = 2.0 × 10 11 cm -2 , 3.0 × 10 11 cm -2 and 4.7 × 10 11 cm -2 respectively. Samples (a) and (b) are as-grown samples whereas sample (c) has been etched and transferred to Si. We immediately note the additional wealth of information obtained with the lift-off sample (c).

The observed strong decrease of the CR line "intensity" for energies higher than hω LO of GaAs is the very first characteristic of the Fröhlich interaction: this decrease corresponds to a broadening of the CR line with a width related directly to the imaginary part of the interaction. In addition, one also notices that the signal diminishes around hω TO of AlAs and also around hω TO of GaAs for sample (c). This tells us that, in order to simulate correctly the spectra, one needs to adopt a multi-layer dielectric model 15 which includes the appropriate dielectric functions of all undoped barrier layers and the dielectric function ε of the doped QW. For the doped QW the CR active part ε a , for instance, is written as:

ε a = ε L - ω 2 p ω[ω -(ω N P -Re(Σ)) + ı(η + Im(Σ)] (1)
One has to include also the corresponding CR inactive part of the dielectric function to reproduce the data obtained with non-polarized light. In the above equation,

ε L = ε ∞ - (ε0-ε∞)ω 2 TO (ω+ıγ) 2 -ω 2 TO
includes the high frequency dielectric function ε ∞ and that of the GaAs lattice phonon, ω p is the plasma frequency, ω N P is the cyclotron frequency in the absence of polaron interaction but including (magnetic field dependent) non-parabolicity (NP) effects 16 and η stands for a field independent contribution of the background defects to the width of the CR line. The effective mass m * entering the plasma frequency ω p is calculated with the same NP model. Σ(ω) represents the self energy correction due to interactions between the electron gas and elementary excitations: they will be discussed further but we know from the response function theory that, whatever is the interaction described by Σ(ω), Re(Σ(ω)) and Im(Σ(ω)) should be related by the Kramers-Kröning (KK) relations. The concept of self energy may be discussed because in general, in the absorption process, one should also include the electron-electron interaction: however, in the present case, all data which are analyzed are obtained for magnetic fields higher than 20 T where the filling factor ν < 2 (ν = NsΦ 0 /B and Φ 0 is the flux quantum). In that case 17 , the electronelectron interaction remains weak and negligible at least for lower doped samples.

Before fitting the transmission data, we note that the line-width and intensity of the CR transitions at low and high fields are practically the same. Hence we can assume that there is no loss of carrier density during the magnetic field runs and that all interactions disappear at high fields. Therefore one can fit, for lower and very high fields, the contribution of NP effects and extract the linear dependance of ω N P versus B which will be used in the following analysis.

IV. FITTING THE TRANSMISSION DATA FOR LOWER DOPED SAMPLES

In the fitting procedure, we first neglect the frequency dependence of Σ and are left with two independent fitting parameters ω c = ω N P -Re(Σ) and δ 0 = η + Im(Σ) for each value of the magnetic field (and hence the cor- From these plots we notice that below 35 meV, δ 0 is constant and small being mainly determined by η. Above this energy and approaching hω LO (GaAs), there is a sharp increase of δ 0 observed in sample (c). There are two distinct contributions to δ 0 : one lower field component starts very large but decreases rapidly at higher fields and a second component which starts from almost zero but goes through a maximum at a field B corre- sponding to an energy of about 48 meV. This later contribution is seen in all samples : in accordance with a previous study 13 , we call it "XP". It will be further analyzed below.

Before discussing the main interaction, we notice that two singularities are also observed in Fig. 1 around the TO energies of GaAs and AlAs. When simulating the transmission spectra these singularities are perfectly reproduced, including the over-shoot, by the multi-layer dielectric model without invoking any singularity in δ 0 at the corresponding energies. As an example, we show in Fig. 3, for sample (c), the simulation of the transmission data around hω TO (GaAs) . The observed features around that energy are dielectric artefacts arising from the fact that ω TO is a pole of ω ph . Therefore, for lower doped samples, there is no interaction of the CR line with the TO phonon of GaAs.

We concentrate on the main contribution to the increase of δ 0 . We assign it to the polaronic interaction since it appears with a threshold energy corresponding to the GaAs LO phonon frequency (slab mode). This assignment is consistent with the observation that this interaction is very sensitive to the carrier concentration.

To model this interaction we will used the FHIP model 6 and the subsequent derivation of the conductivity with that model 7 . This derivation predicts, in principle, the emission of polarons at multiple values of hω LO : however, in the spirit of the Feynman polaron model, we restrict our analysis to the resonant emission of one po-laron. Then, for undoped polar material, Im(Σ Pol (ω)) is written as 7 :

Im(Σ Pol (ω)) = C × ω 2 LO ω |A 0 (ω)| 1/2 e -R|A0(ω)| Θ(A 0 (ω))
(2) where A 0 (ω) = ω/ω LO -1 , Θ(x) = 1 for x > 1 or zero otherwise. Expressed in this way, the polaron contribution depends only on two parameters C and R both independent on energy. We have simulated this interaction with the curve, depicted in the upper panels of Fig. 2, with a dashed black line.

One has to simulate the contribution of the "XP" interaction which was observed in all GaAs based samples 13 . In fact, this interaction has also been observed in GaInAs QW 18 and clearly assigned to the contribution of the symmetric interface modes developed in Q2D structures which are active in the infra-red 19 . The wave function of these symmetric interface modes does not vanish at the center of the QW (unlike the anti-symmetric modes) and therefore have a significant overlap with the wave function of the Q2D gas. However the interaction which is also of the Fröhlich type is governed by the Fröhlich constant of the barrier and therefore much less screened than the main polaronic interaction in the QW. Following the model proposed by Mori and Ando 19 , we simulate this interaction with a Lorentzian curve as depicted, in the upper panels of Fig. 2, with a dotted line. As a result, the energy of the maximum of the "XP" interaction, E XP max , is found around 48 meV , an energy inside the RB energy of AlAs (hω TO (AlAs)= 45 meV and hω LO (AlAs)= 50 meV) consistent with the assignment of the "XP" transition. In principle, one should expect, a similar contribution from the interface modes in the RB of GaAs but it is not observed (see Fig. 3) within the experimental errors in sample (c).

With all preceding approximations, one can fit the δ 0 variation using, as a guide, the data points obtained from the first step of our analysis. This fitting provides the total self energy correction Im(Σ(ω)) = Im(Σ P ol (ω)) + Im(Σ XP (ω)) which is displayed as thick red lines in the upper panels of Fig. 2.

Assuming these interactions satisfy the linear response function theory, Im(Σ(ω)) and Re(Σ(ω)) should be related by the Kramers-Kröning (KK) relations. Therefore, as a first constraint on the fitting process, the KK transform of δ 0 (ω) should reproduce the variation of hω c (ω)/B (open squares in the lower panels of Fig. 2). At that level, we compare also the experimental data with the variation of hω N P (B)/B versus hω N P (B) (magenta dashed lines) as obtained at the beginning.

Using this variation, one can refine the whole process: the fitted functions hω c /B(ω) and δ 0 (ω) are then inserted in the multi-layer dielectric model from which one can compute, for each value of B, the TR spectrum and compare it to the corresponding experimental one. With that self-consistent treatment, we obtain an agreement, for all spectra and all samples, which is quite satisfactory (see Fig. 4) and lead us to think that the model has captured the main physical ingredients of the problem, including the quantitative aspects.

V. DISCUSSION OF THE FITTING PARAMETERS FOR THE LOWER DOPED SAMPLES

To treat the electron-phonon interaction in 3D compounds, Feynman 5 introduced the path-integral method which applies the concept of "least-action" to solve the problem. This action describes the time evolution of the Lagrangian at all degrees of interaction. In practise, however, the problem can be solved analytically only if the interaction is harmonic.

The FHIP model uses the path-integral method and mimics the Fröhlich interaction (of Coulombic nature) by an harmonic interaction, with a force constant v 2 -w 2 , v and w being two parameters expressed in reduced units of ω LO .

Expressed, as a function v and w, the two fitting parameters C and R (Eq. 2) used in the preceeding section are not independent: R = (v 2 /w 2 -1)/v and C = (2α F /3) × (v/w) 3 , α F being the Fröhlich constant of the polar material (equal to 0.068 for GaAs). To take into account low dimensionality effects, we replace C by α Q2D × C. Following results from Hai et al. 20 , we have taken α Q2D = 1.2. Within the philosophy of the FHIP model, the parameter w, specific of the phonon, should be the same for all samples and we have adjusted its value in order to reproduce the different R and v values fitted experimentally and shown in table I for w = 0.175±0.005.

In the FHIP model, as already said, the force constant of the harmonic interaction is proportional to v 2 -w 2 entering the numerator of the parameter R. We have plotted, in Fig. 5, the variation of this quantity as a function of the carrier density N s of the QW. This variation clearly demonstrates the screening effects of the "polaronic" interaction. One can fit the data with a linear interpolation (dashed line in Fig. 5) justified by the fact that in Q2D systems the screening effects of the polaronic interaction varies linearly with N s 21 . In this study performed for a QW width of 10 nm, the screening is found to be complete for a critical value Nsc1 of N s of the order of 10 × 10 11 cm -2 : this corresponds to a 3D concentration N 3D1 = 10×10 17 cm -3 . The present FHIP treatment provides a value Nsc = 5.8 ± 0.2 × 10 11 cm -2 corresponding to a 3D concentration N 3D = 4 × 10 17 cm -3 significantly lower than N 3D1 . This extra screening may be the manifestation of recoil effects naturally taken into account by the FHIP model. It turns out that, experimentally, the polaronic interaction around the LO phonon has practically disappeared for sample MA-1490 (sample (d)), see next section) with N s = 6.3 × 10 11 cm -2 consistent with our findings.

In the FHIP model, derived for undoped samples, the action S F of the Fröhlich interaction is mimicked by an harmonic action S H with a strength v 2 -w 2 = √ 8α F /w. Therefore from the limit at N s = 0, see Fig. 5, one can derive the parameters describing the polaronic interaction, with the FHIP model, in GaAs for undoped samples : α F = 0.068, w = 0.175 ± 0.005 and v = 1.065 ± 0.05.

Since the "XP" feature involves the Fröhlich interaction, one should also parameterized it with the FHIP model. However, within the experimental uncertainties, it is difficult to find the energy threshold beyond which it occurs without adding a significant number of broadening parameters. This may reflect the imperfections of the interfaces between the QW and the surrounding superlattices. Therefore, we fit this transition with a Lorentzian curve as proposed by Mori and Ando 19 . However this tells us that E XP max is an upper limit of the interface corresponding mode. This approach allows, nevertheless, to plot the intensity of the fitted Lorentzian versus N s and see that the "XP" contribution should disappear around the same critical value Nsc as that observed for the main polaronic interaction.

VI. EXPERIMENTAL RESULTS FOR HIGHER DOPED SAMPLES

We report now, in Fig. 6 the experimental results obtained for sample (d) with Ns = 6.3 × 10 11 cm -2 . We see that already at B = 24.5 T the intensity of the CR line is recovered showing that the main polaronic interaction has disappeared. However beyond that field some additional interaction remains. To characterize it, we have analyzed it using the standard model (Eq. 1) with Σ(ω)= 0. The resulting fitting parameters hω c /B and δ 0 are displayed in Fig 7.

We note that δ 0 is peaking at an energy "XX", a priori distinct from "XP", because it occurs at an energy below hω TO of AlAs that is outside the RB of this compound. We have performed experiments, not reported here, on other samples with N s = 6.6 × 10 11 cm -2 and 7.1 × 10 11 cm -2 which displayed the same "XX" interac-tion below hω TO of AlAs but with an intensity decreasing with N s . We may question wether the doping of the QW can influence the solutions for the interface modes in such a structure.

The symmetric interface modes (only active in the infra-red range) of a GaAs QW sandwiched between two AlAs barriers are the solutions of the following equation 22 :

ε GaAs (ω)tanh( Qd w 2 ) + ε AlAs (ω) = 0 (3)
where d w is the thickness of the QW and Q is the 2D wave vector of the interface phonon parallel to the interface. The solutions of Eq. 3 do not necessarily exist: they, in particular, depend on the values of the broadening parameters used to simulate the phonon part of the dielectric constant. We have used here, for these parameters, the same values as those adopted to simulate the transmission spectra. For undoped material, the solutions of this equation lie in the upper part of the RB of AlAs and the lower part of the RB of GaAs. We assume, in the following, that Q ∼ 1/l B , l B being the magnetic field length. In the present case, ε GaAs (ω) is magnetic field dependent through the contribution of the cyclotron resonance which is as important as that of the phonon. We have calculated, for all samples, the solutions of Eq. 3 as a function of the magnetic field B: results for sample (d) are displayed in Fig. 8. In this figure, we specify only those solutions (orange dots) which could lead to an extra diffusion of the CR transition: they need to be at an energy lower than the CR energy. Indeed the solutions are not very dependent on the carrier density: they always lie inside the RB of GaAs and that of AlAs and vary slightly with B through the variation of the CR transition . The energies of the "XP" transition observed in the lower doped samples are in a good agreement with the solutions reported in Fig. 8 inside the RB of AlAs. One may questions the validity of the present model because the barriers in our samples are in fact superlattices: we should therefore replace, in Eq.3, ε AlAs (ω) by an effective dielectric constant ε eff (ω) which can be calculated along the lines proposed by Merlin et al. 23 . It turns out that the results are very similar to those found with the simple model: in particular the solutions at high energies always lie inside the RB of AlAs. Therefore, with this approach, we do not have a physical explanation of the origin of the "XX" interaction displayed in Fig. 7. In addition, a closer look at the variation of hω c /B as compared to that of hω N P /B (lower panel of Fig. 7) leads us to think that certain additional effects, hidden in the RB of GaAs, should be present.

We then performed an additional set of experiments on a lift-off sample (MA 1200), referred in the following as sample (e), with a carrier density Ns = 7.4 × 10 11 cm -2 . The corresponding relative transmission spectra are displayed in Fig. 9. (These spectra has been measured with a spectral resolution of 0.5 meV). We observed, as for sample (c), a clear depletion of the CR "intensity" around hω TO of GaAs and AlAs and a smaller one around the energy of the "XX" transition. However, here, the depletion in the RB of GaAs is, clearly, not only due to dielectric artefacts.

In Fig. 10 we plot the experimental results obtained for sample (e) (open black dots): they differ significantly from those displayed in Fig. 3 for sample (c). There is a clear interaction around hω TO of GaAs which almost disappears at hω LO of GaAs. This result is, in fact, not new and has already been reported 12,24 in quantum wells with high electron density. Following these reports, Klimin and Devreese 25 have developed a sophistical model showing that this effect could be due to the resonant magneto-polaron coupling of both types (GaAs and AlAs) of interface phonon modes. Such a coupling is of electrostatic origin and does not imply the Fröhlich interaction. An interesting aspect is that this coupling, as a function of the carrier density, has a threshold below which it becomes relatively weak, explaining why it is not seen for low densities which could be the case for sample (c). In this study the authors focussed their attention on the splitting around hω TO of GaAs without discussing what could happen at higher energies. We think that such a mechanism could be, also, at the origin of the "XX" interaction. In that case, it is interesting to evaluate the relative strength of both interactions. We have tried to quantify the interaction observed in Fig. 10 by simulating the transmission spectra with a simple model for the dielectric function which of course cannot be as simple as that given in Eq. 1. We show in Appendix how one can derive the active part of the dielectric function for a medium where the CR and TOphonon transitions interact with a parameter V TO to be fitted: the results are displayed in Fig. 10 as magenta lines with V TO = 1 ± 0.1 meV. The fit is not perfect but gives an order of magnitude of the interaction which is believed reliable. The simulation of the experimental spectra with the modified model of the dielectric function, given in the appendix, allows to extract here also the imaginary part of the interaction. Results are displayed in Fig. 11.

It is found that δ 0 increases first around hω TO and then around the "XX" interaction similar to that found for sample (d). This treatment gives an idea of the relative strength of both interactions but is not self consistent: in particular, if we set V TO = 0 we find a variation of δ 0 around the "XX" interaction similar to that found here.

VII. CONCLUSION

We have performed far infra-red magneto-optical transmission measurements on a series of n-type GaAs quantum wells, with a width d w of 13 nm, doped at different electron concentrations N s . Below a critical value Nsc of about 6 × 10 11 cm -2 , the dominant features of the spectra are characteristic of the polaronic Fröhlich interaction with the LO phonon mode of GaAs and a similar one, called "XP", with the interface modes of the barrier.

We have been able to extract from the data, the energy dependance of the real and imaginary parts of the selfenergy characteristic of both interactions. Results for the main interaction with hω LO of GaAs were quantitatively compared to the predictions of the FIHP model for different carrier concentrations. It was shown that, below Nsc, the model predicts correctly the carrier dependent screening of the Fröhlich interaction in such structures.

Beyond Nsc, new interactions appear first as a coupling of the CR transition with the TO phonon energy of GaAs and also, presumably, with a mixed mode, called "XX", of both interface phonons. These later results deserve a complete self consistent treatment to be understood.

FIG. 1 .

 1 FIG. 1. (Color on line) Transmission spectra for samples (a), (b) and (c) as a function of the energy for different magnetic field values with steps of 0.5 T. Spectra for samples (a) and (b) are displayed between 24T and 33.5T and spectra for sample (c) between 19T and 33.5T.

  responding value of hω N P ) . The resultant parameters, hω c /B and δ 0 , are displayed (open squares) in Figs. 2, upper and lower panels respectively, as a function of the energy hω N P (B) (in these figures some typical error bars reflect the estimated uncertainties of the fitting procedure).

FIG. 3 .

 3 FIG. 3. (Color on line) Simulation of some experimental relative transmission spectra (open dots) with the calculated ones using the multi-layer dielectric model around the TO energy of GaAs (continuous red lines). Each curve is shifted by 0.5 with a scale enlarged as noted.

FIG. 4 .

 4 FIG. 4. (Color on line) Simulation, for samples (a),(b) and (c) of some experimental relative transmission spectra (open dots) with calculated ones (continuous red lines) using the multi-layer dielectric model where the fitted functions hωc(ω)/B and δ0(ω) have been inserted (see text). Each curve is shifted by 0.5 with a scale enlarged as specified.

  TABLE I. Parameters R and v of the FHIP model for GaAs deduced from the fit of the data, assuming w = 0.175 (hωLO) and α F = 0.068 Sample αQ2D Ns (10 11 cm -2

FIG. 5 .

 5 FIG. 5. (Color on line) Variation of v 2 -w 2 as a function of the carrier density Ns of the three samples (a), (b) and (c) (full squares) . The dashed line is a linear interpolation of data.

FIG. 6 .

 6 FIG. 6. (Color on line) Transmission spectra for sample (d) as a function of the for different magnetic field values.

FIG. 8 .

 8 FIG. 8. (Color on line) Calculated energies, as a function of B, of the interface phonon modes for sample d (orange dots) expected to be active in the polaronic interaction. The red line is the contribution of the non-parabolicity effects ωNP to the CR frequency as compared to the measured values hωc (open blue triangles). ωT1 and ωL1 are the phonon frequencies of GaAs and ωT2 and ωL2 those of AlAs respectively.

FIG. 9 .

 9 FIG. 9. (Color on line) Transmission spectra for sample (e) as a function of the energy for different magnetic field values.

FIG. 10 .

 10 FIG. 10. (Color on line) Transmission spectra for sample (e) inside the RB of GaAs (open black dots). The magenta lines are the results of the fitting of the transmission curves with the model described in the text for an interaction parameter VTO= 1 meV.

FIG. 11

 11 FIG. 11. (Color on line) Variation of the broadening parameters (open dots) δ0 for sample (e), as a function of the energy (hωNP ).

ACKNOWLEDGEMENTS

The LNCMI-CNRS acknowledges the support of the EMFL.

Appendix

In order to include, in a self consistent way, the interaction of the cyclotron transition with that of the TO phonon, one has to express the active part of the dielectric function ε a (Eq.1 ) using the Green function formalism. In the absence of interaction the Green function

0 is written as:

are the two propagators of the problem. Let us define, for the oscillator strengths of the optical transitions, a two dimensional vector

The simplest way to introduce the interaction between both optical transitions is to insert in the off-diagonal terms of Eq.A.1 a finite function that, for simplicity here, is taken as a constant V T O .The perturbed Green function is written as:

from which we can derive the modified active part of the dielectric function ε1 a = ε ∞ ---→ M T G -→ M . This simple model has been used to fit the data reported in Fig. 10


Raul Sena Ferreira

Jeremie Guiochet
email: jeremie.guiochet@laas.fr

Jean Arlat
email: jean.arlat@laas.fr

Helene Waeselynck
email: helene.waeselynck@laas.fr

Benchmarking Safety Monitors for Image Classifiers with Machine Learning

Keywords: machine learning, safety monitoring, benchmark, image classifier, experimental results

High-accurate machine learning (ML) image classifiers cannot guarantee that they will not fail at operation. Thus, their deployment in safety-critical applications such as autonomous vehicles is still an open issue. The use of fault tolerance mechanisms such as safety monitors is a promising direction to keep the system in a safe state despite errors of the ML classifier. As the prediction from the ML is the core information directly impacting safety, many works are focusing on monitoring the ML model itself. Checking the efficiency of such monitors in the context of safety-critical applications is thus a significant challenge. Therefore, this paper aims at establishing a baseline framework for benchmarking monitors for ML image classifiers. Furthermore, we propose a framework covering the entire pipeline, from data generation to evaluation. Our approach measures monitor performance with a broader set of metrics than usually proposed in the literature. Moreover, we benchmark three different monitor approaches in 79 benchmark datasets containing five categories of out-of-distribution data for image classifiers: class novelty, noise, anomalies, distributional shifts, and adversarial attacks. Our results indicate that these monitors are no more accurate than a random monitor. We also release the code of all experiments for reproducibility.

I. INTRODUCTION

Image classifiers based on machine learning (ML) are core components for many safety-critical autonomous applications, like autonomous vehicles. ML models make decisions based on trained past data. A common way to validate them is by analyzing the discrepancy between the predicted values and the ground truth (labels) in a testing activity. If the results are satisfactory, it goes to production. However, even modern ML techniques, such as deep learning, can be wrong in their predictions even with 100% confidence [START_REF] Gal | Dropout as a bayesian approximation: Representing model uncertainty in deep learning[END_REF]. This situation may lead to hazardous situations like the whole behavior of the system may rely on the ML decision. Therefore, it is now urgent to deploy techniques to increase the confidence The research leading to these results has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 812.788 (MSCA-ETN SAS). This publication reflects only the authors' view, exempting the European Union from any liability. Project website: http://etn-sas.eu/. Fig. 1: System under test composed of ML+SM in such ML classifiers. Among techniques coming from the dependability community [START_REF] Avizienis | Basic concepts and taxonomy of dependable and secure computing[END_REF], fault tolerance is a technique that could be applied to deliver a correct service despite the occurrence of errors. Safety monitors (SM) keep the system in a safe state despite hazardous situations [START_REF] Machin | SMOF: A safety monitoring framework for autonomous systems[END_REF].

Potentially, these situations may include errors from the ML model. Many recent works focus on monitors dedicated to the ML model surveillance. They broadly fall in three types of monitors: observation of the inputs of the ML model [START_REF] Liu | Input validation for neural networks via runtime local robustness verification[END_REF], its outputs [START_REF] Hendrycks | A baseline for detecting misclassified and out-of-distribution examples in neural networks[END_REF] or from intermediate layers in case of deep neural networks (DNN) [START_REF] Cheng | Runtime monitoring neuron activation patterns[END_REF], [START_REF] Henzinger | Outside the box: Abstraction-based monitoring of neural networks[END_REF]. However, they are all based on the exploitation of the training data. Therefore, we refer to them as data-based monitors compared to safety monitors based on rules (or safety properties). Similar to uncertainties inherent to the use of ML, the confidence in such SM is an open issue.

Thus, it is essential to estimate how efficient such monitors are and if it is possible to include them in safety monitoring. This estimation should consider all potential situations leading to an error of the ML and should be based on metrics dedicated to measuring the monitor efficiency. Therefore, we focus on a framework for benchmarking such monitors, augmented with an additional primary mechanism to inhibit the decision in error detection. As presented in Figure 1, the complete system under test is composed of an SM that surveil the ML.

Our benchmark adapts and extends current metrics used in the ML community to estimate the SM detection performance at runtime, its impact on the system, and the overhead induced by the use of the monitor. Our main contributions are:

• A new baseline framework for benchmarking SM for ML classifiers. To the best of our knowledge, this is the first work that proposes an initial framework that benchmarks SM for ML components from different perspectives. • A comprehensive benchmark experiment containing different data-based SM implementations, datasets, and results. Our experiments reveal the advantages and drawbacks of the main modern data-based SM approaches for image classifiers built with ML. We perform experiments on five challenging and important types of out-ofdistribution data for image classifiers. This work is organized as follows: in Section II, we provide background on ML classifiers, threats that may affect their results, and current safety monitoring approaches. In Section III, we present an overview of our framework, along with its main objectives. In Section IV we present our experiments, datasets, and results of the application of our framework to three ML monitors. Finally, in Section V, we present our conclusions.

II. BACKGROUND

This work focuses on monitoring ML image classifiers, especially when exposed to out-of-distribution data that can threaten their performance at runtime. In the following subsections, we give a brief explanation of these concepts.

A. Image classifiers with ML

An ML classifier is a software component that uses an ML algorithm for identifying, given an observation, in which class it belongs [START_REF] Alpaydin | Introduction to machine learning[END_REF]. ML classifiers take an input, such as an image or a vector of numerical values, and outputs categorical values (classification) according to a category (class) previously seen during the training process. This component, called the ML model, is usually validated by analyzing the discrepancy between the classified values and the ground truth (labels). If the results are satisfactory, it goes to production. These classifiers are integrated into many perception pipelines of autonomous systems. The prediction of the ML is at the very heart of the safety-critical functions of such systems (e.g., collision avoidance, path planning, and so on).

B. Out-of-distribution data

ML models tend to be biased to the training data [START_REF] Nadeau | Inference for the generalization error[END_REF], resulting in a natural inability of a model to generalize 100% of time even if all available data could be collected. Beyond this fundamental model generalization issue, there is another problem: data incompleteness. That is, rare conditions tend to be underrepresented since the training data represents a subset of all real-world possibilities [START_REF] Shafaei | Uncertainty in machine learning: A safety perspective on autonomous driving[END_REF]. It means that data represents the same target classes from training data, but that has different characteristics, different enough to threaten ML performance. Such data is known as out-of-the-distribution data (OOD). In this paper, we consider five types of OOD data that can threaten an ML model at runtime:

• Novel classes: a scenario where new classes are introduced during runtime. For example, an ML model can be trained to identify dogs but fails to classify a new dog type not present in the training data [START_REF] Perera | Deep transfer learning for multiple class novelty detection[END_REF].

• Adversarial inputs: a situation when ML fails with high confidence due to small input modifications [START_REF] Kurakin | Adversarial examples in the physical world[END_REF]. • Distributional shifts: a condition that decreases the ML performance through time since the training dataset may differ from the real inputs. Such a situation is also known as a concept-drift [START_REF] Ferreira | Amanda: Semi-supervised density-based adaptive model for non-stationary data with extreme verification latency[END_REF]. Examples of distributional shift include a change in class attributes such as dimension, physical characteristics, contrast, brightness, and other pixel-related variations in the image. • Noise: a possibility to receive deteriorated images due to small failures in exteroceptive sensors or unexpected environmental interference. • Anomalies: a severe failure from exteroceptive sensors capable of severely corrupt images. However, contrary to noise, such failures corrupt data so that such images lose semantic value. It means that whatever the ML decision is, it can correspond to an inaccuracy. Examples of such images are black images, images with several shifted pixels, and so on. Next, we detail the current approaches for monitoring ML.

C. From SM to ML monitors

Safety monitoring is a well-known dependability technique already used in embedded systems such as robots or autonomous vehicles. This approach is generally based on a system model or the environment and on properties they should guarantee. We will use rule-based monitors for such approaches (in opposition to data-based monitors presented in the following sub-section). Some techniques for monitoring safety components, consider this component as a blackbox [START_REF] Machin | SMOF: A safety monitoring framework for autonomous systems[END_REF]. It is possible to apply this approach to systems, including ML algorithms. However, as they are considered black-box, the fact that it is an ML-based component has no real impact on the SM design. For example, safety rules can be implemented in an SM to verify if a vehicle can completely stop before reaching an obstacle [START_REF] Ozguner | Systems for safety and autonomous behavior in cars: The darpa grand challenge experience[END_REF] or to avoid a collision [START_REF] Al-Khoury | Safety of machine learning systems in autonomous driving[END_REF]. In both cases, external (exteroceptive) sensors, such as distance sensors, are observed along with internal (proprioceptive) sensors, such as speed, to evaluate if a safety property is violated. It means that it was possible in these situations to have a redundant mechanism to monitor the ML.

However, when ML is in charge of image classification, in most cases, no redundant classification mechanism could be developed to monitor the ML. Indeed, it is usually impossible to use external sensors and external software not based on ML to confirm or invalidate an ML prediction.

One possible direction is then to verify some assertions at the ML levels, for instance, with the use of model assertions [START_REF] Kang | Model assertions for monitoring and improving ML model[END_REF]. This technique is an adaptation of the classical program assertions to monitor and improve ML models. The idea is to verify inputs/outputs that indicate when errors may be occurring in the system. For example, monitoring if an object flickers in and flickers out in the camera indicates a possible failure. Even applying these techniques during design and operation cannot guarantee that a DNN decision is safe. The reason is that, for some corner cases, ML outputs wrong decisions that cannot be verified by inspecting the code logic or the sensor values and lead to hazards [START_REF] Dreossi | Compositional falsification of cyber-physical systems with machine learning components[END_REF].

We can deploy rule-based monitors when redundant observation sources are applied and safety properties are correctly expressed. However, monitoring an ML component for vision is particularly complex to find a redundant source of observation. Moreover, it is not apparent to express a safety property at the level of an ML prediction. For these reasons, a current approach is to develop monitors based on the ML training data: the data-based monitors.

D. Data-based monitors

This approach is usually based on data instead of using a system model or specifications provided by the developers. However, it means that we develop the monitor itself using the same training data as the ML. We propose to classify such monitors in three categories: monitors based on ML inputs, intermediate values, or outputs:

1) Monitoring DNN inputs: An example of this approach is to adversarially train an ML model to reconstruct a noisy image to an image similar to a previous known class [START_REF] Sabokrou | Adversarially learned one-class classifier for novelty detection[END_REF]. It means that a new image arrives at runtime, and this ML model tries to reconstruct this image to a known one from the set of known classes previously seen in the training phase. Suppose there is a considerable difference between the reconstructed image and the actual image. In that case, this actual image is from an unknown class, indicating possible OOD data. A similar idea is tested in [START_REF] Denouden | Improving reconstruction autoencoder out-of-distribution detection with mahalanobis distance[END_REF], in which the authors apply the Mahalanobis distance to improve the autoencoder reconstruction process in order to capture OOD samples better.

Another possible solution based on DNN inputs is to create a radius distance threshold calibrated during the training [START_REF] Liu | Input validation for neural networks via runtime local robustness verification[END_REF]. The idea is to perturb the DNN inputs, observe the correct answer, and determine how considerable the distance is regarding the DNN decisions. The advantage of this approach is that it does not need to inspect the internals of the DNN. The drawback is that it tends to be biased to the training data.

2) Monitoring the DNN intermediate values: An example of this approach is to monitor the neuron patterns observed from the layers of the DNN. Such an approach usually compares the recorded patterns of the DNN activation functions during the training with those during runtime. After the standard training process, a runtime monitor is created by feeding the training dataset to the network to store the neuron on-off activation patterns using binary decision diagrams (BDD) [START_REF] Cheng | Runtime monitoring neuron activation patterns[END_REF] or using a 2D projection [START_REF] Henzinger | Outside the box: Abstraction-based monitoring of neural networks[END_REF]. For example, this projection is an abstraction box built by maximum and minimum activation function values during training. It inspects if the output of an activation function falls inside of this abstraction box during runtime after a new input pass through it. If not, it raises the alarm considering this input as a novel input.

Besides, uncertainty quantification methods can be applied to the intermediate outputs of a DNN [START_REF] Berend | Cats are not fish: Deep learning testing calls for out-of-distribution awareness[END_REF], recognizing activation patterns that do not represent any data from the training dataset. An uncertainty-based supervisor accepts input as a typical sample if its uncertainty is lower than some threshold. Thus, a high threshold leads to many false negatives, and a low threshold leads to many false positives [START_REF] Weiss | Fail-safe execution of deep learning based systems through uncertainty monitoring[END_REF].

3) Monitoring the DNN outputs: For instance, an SM can monitor the values in the last layer of a DNN [START_REF] Hendrycks | A baseline for detecting misclassified and out-of-distribution examples in neural networks[END_REF], verifying the decision's confidence level. However, it is unreliable since DNNs can output a wrong decision even with a high confidence level. For avoiding this problem, Liang et al. proposed enhancing the reliability of out-of-distribution image detection in neural networks (ODIN) [START_REF] Liang | Enhancing the reliability of outof-distribution image detection in neural networks[END_REF] by applying techniques that decrease the DNN confidence values.

Uncertainty quantification methods can also be applied directly to DNN outputs. Some methods that can be used for this purpose are point-prediction networks, MC-dropout, and DNN ensembles. However, for evaluating such methods, it is necessary to apply metrics beyond the traditional combination of false/true positives/negatives, such as S-1 score [START_REF] Weiss | Fail-safe execution of deep learning based systems through uncertainty monitoring[END_REF].

E. Benchmarking SM for ML-based components

Despite several works for monitoring ML at runtime, just a few try to establish better benchmark methodologies. One of these works proposes a less biased evaluation of outof-distribution detectors, called OD-test [START_REF] Shafaei | A less biased evaluation of out-of-distribution sample detectors[END_REF]. Same as our work, the authors present a methodology that divides the data into three parts: training/validation, in-distribution testing set, and out-of-distribution testing set. Henrikson et al. [START_REF] Henriksson | Towards structured evaluation of deep neural network supervisors[END_REF] also proposed a framework for benchmarking DNN monitors, using six OOD datasets and seven metrics.

Nevertheless, our work differs from the two related works presented above in three different aspects. Firstly, we propose a methodology that covers the entire pipeline, from data generation to evaluation. Secondly, we apply metrics beyond the traditional accuracy and positive/negative rates, with additional statistical analysis in the results. Finally, we perform tests using five different types of OOD data across a greater amount of generated datasets. Next, we present our framework.

III. A BASELINE METHODOLOGY FOR BENCHMARKING SM FOR ML CLASSIFIERS

The proposed framework is an adaptation of the FARM [START_REF] Arlat | Fault injection for dependability validation: A methodology and some applications[END_REF] methodology. FARM is a methodology for fault injection for dependability validation. It uses the input domain of a target system as a set of faults F and a set of activations A that specifies the domain to test the target system. The output domain refers to a set of readouts R used for posterior evaluation with a set of metrics M . Our framework is divided into three modules, as illustrated in Figure 2.

After setting some initial parameters such as type of data generation, the amount of data to be tested, which ML models and SM to include, starts with the first module, called the data profile. This module generates benchmark datasets in the next module, called system under test (SUT).

The SUT is responsible for testing the ML and the SM performance, generating the results (readouts) at the end of the process. Readouts are inputs for the evaluation module.

The evaluation module is responsible for analyzing the readouts through several metrics for different components of Fig. 2: A high-level overview of the framework.

the system. It provides the final analysis for each type of readout. Next, we explain each module and its components.

A. Data profile

This is the first module and it is composed of two items:

• Activity: it contains in-distribution (ID) data, that is, instances from a distribution known by the ML.

• Fault template: it contains rules for generating out-ofdistribution (OOD) data presented in Section II.B. The benchmark dataset receives ID and OOD data to test how the ML and the SM behave when exposed to expected and unexpected data. Even though all types of OOD can be part of a unique benchmark dataset, we generated datasets divided by each category of OOD data.

An essential premise is that the generated benchmark dataset is not applied to build or validate the SM or train the ML model. This premise guarantees unbiased experiment results and is more realistic since, in real scenarios, there are no guarantees about which type of data will arrive at runtime.

B. System under test

The system under test (SUT) module receives the benchmark dataset as input and outputs readouts for each component. To validate our framework, we chose ML models that classify images and one SM for each of the three categories mentioned in Section 2. Therefore, this module is composed of two main items: a third-party ML model already trained; and a third-party safety monitor containing a detection mechanism. We call the ML and the SM third-party components since they can be built aside from the framework.

This module simulates a stream of images randomly ordered. We apply datasets with random images since the image datasets available in the literature use such a setting.

The simulation works as a stream of images coming from the benchmark dataset set D that contains images X and labels y, arriving at one of each at a time. The ML model receives this image and makes a ŷ classification. Next, the system triggers the SM, which checks a set of predefined properties. These properties can be the ML's classification ŷ with the associated confidence level, intermediate layers, the input X, or even a combination of two or more of these properties. In the case of this pseudo-code example, the SM inspects the ML model's internal properties (DNN's hidden layer) during the classification, as recently suggested in the literature [START_REF] Cheng | Runtime monitoring neuron activation patterns[END_REF], [START_REF] Henzinger | Outside the box: Abstraction-based monitoring of neural networks[END_REF].

After the inspection, the SM detector mechanism raises a detection alarm or not ( m). For this work, we consider a simple reaction strategy for the SM. If an alarm is raised, it invalidates the ML classification result; otherwise, it accepts the ML classification. Next, we observe whether such intervention produced a desirable outcome for the system or not. We call it overall detection ŝ. A desirable outcome means that canceling an ML output/agreeing with it was beneficial to the SUT. This result is independent of whether the SM-specific detection was correct or not in the OOD detection task. The criteria for considering these two dimensions of the detection (specific and overall) correct or not is discussed in the following subsection. This process continues until the end of the stream. Thus, the readouts are: the ground truth y, the ML classification ŷ, the SM specific detection m, and its overall detection ŝ. They are divided into three different categories:

1) ML readouts: it contains classification (e.g., class number), confidence value from the last layer, and intermediate values from hidden layers. 2) SM readouts: it contains two types of detection: a) specific (e.g., it outputs 1 for OOD detection; 0 otherwise); b) overall (e.g., after OOD detection, it cancels the ML decision (1); 0 otherwise). 3) General readouts: it contains general outputs from all components above, such as processing time and memory. As can be noted, the SM readouts contain results in two dimensions: specific and overall. Considering two dimensions for detection is more realistic and complete than just analyzing the detection rate for the OOD data. The reason is that while the SM tries to detect OOD data, it also can avoid that the ML gives an answer that is different from the ground truth. The opposite is also true. As a result, the SM can incorrectly detect ID data as OOD data, hindering the correct decisions of ML. All possible situations are given in the next Subsection.

C. Evaluation

This module receives the readouts as inputs containing false positives/negatives and true positives/negatives regarding OOD data detection's specific task and the overall task of avoiding an unsafe outcome. This module evaluates two main aspects:

1) The SMs performance: the objective is to assess the SM results regarding the specific task of detecting OOD data. This evaluation's inputs are SM OOD detection (e.g., raises/does not raise the alarm).

2) The overall impact of the SM: the objective is to determine if the SM improves or worsens the overall SUT accuracy. We analyze the SUT using the ML alone (baseline) and ML with the SM. This evaluation's inputs are the overall decisions made by the SUT with ML alone and ML with SM, processing time, and memory. 3) Time and memory overhead induced by SMs: the objective is to investigate the memory and processing time efficiency of these methods during runtime. This module contains two major items: the oracle, and the metrics for ML and SM. The oracle determines positive and negative data and whether a test has passed or failed. Thus, the oracle considers a correct specific detection when the SM correctly detects OOD data. Besides, the oracle considers a correct overall detection for the SUT when the SM correctly avoids a wrong ML classification for ID or OOD data. The oracle takes into consideration the following scenarios to determine whether a readout is correct or incorrect: of avoiding a failure since the SM detection for OOD data also avoided an ML misclassification. * If the ML classification is different from the ground truth: it means a false negative for the overall task of avoiding a failure since the SM detection for OOD data also avoided an ML misclassification. The only exception for the above rules is for novelty class detection. When OOD data arrives in the stream, if the SM correctly detects OOD data, it will always be interpreted as true positive since the SM always correctly cancel the ML classification, independently of the ground truth. Conversely, if the SM does not detect OOD data, it will always be a false negative. If the SM lets the ML deal with a class that it was not trained in before, it can be regarded as unsafe. In order to evaluate these situations, we have selected in the literature a set of metrics that are pertinent for our study. Hence, we apply seven metrics:

• ID
• Matthews correlation coefficient (MCC): it ranges from -1 if (ML or SM always wrong) passing through 0 (ML or SM is accurate as random) to 1 (ML or SM always right). This metric is more reliable than traditional metrics such as accuracy. It yields a high score only if the ML or SM can be assertive in all of the four confusion matrix categories [START_REF] Chicco | The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation[END_REF]. In contrast, accuracy only considers the portion of the right answers. In a scenario in which the number of a class is 80%, the accuracy could yield a score of 80% for an SM that did not detect other classes. results through all datasets. It shows how statistically different are the results between the SMs. Next, we detail each component's choice and its parameters for the three modules, and the benchmark results.

IV. EXPERIMENTS AND RESULTS

A. Data profile

We use three datasets as the activity of our framework:

• GTSRB [START_REF] Sermanet | Traffic sign recognition with multi-scale convolutional networks[END_REF]: German traffic signs with 43 classes, with 39200 instances for training and 12600 for testing.

• BTSC [START_REF] Jain | A novel genetically optimized convolutional neural network for traffic sign recognition: A new benchmark on belgium and chinese traffic sign datasets[END_REF]: Belgium traffic signs with 62 classes, divided into 7000 images for training and testing. • CIFAR-10 [29]: ten general classes (e.g., dog, car ...), with 50000 instances for training and 10000 for testing. Next, we apply a fault template for the five classes of OOD presented in section II-B: novelty, anomaly, distributional shift, noise, and adversarial inputs. For adversarial inputs generation, we apply the fast gradient signed method (FGSM) [START_REF] Dong | Discovering adversarial examples with momentum[END_REF]. For the noise and distributional shift, we generate 19 different transformations with two types of intensity varying from 1 to 5 (e.g., snow (1) = image with a bit of snow; snow (5) = heavy snow) [START_REF] Hendrycks | Benchmarking neural network robustness to common corruptions and perturbations[END_REF]. All transformations were applied over CIFAR-10 and GTSRB datasets. The benchmark datasets are composed of a transformed version of the ID datasets that we previously applied for training the ML algorithm and building the SM. Thus, we use 20% of the ID data for the benchmark dataset without transformation. The entire original dataset is transformed into a specific variation. For obvious reasons, this 20% of ID data was also not applied to the ML training.

For novelty class detection, the fault template applies one dataset as an ID dataset and another dataset with new classes as OOD data, resulting in three benchmark datasets:

• GTSRB as ID data, and BTSC as OOD data: this combination tests whether the SM can distinguish new classes that have similar characteristics to the known ones. • GTSRB as ID data, and CIFAR-10 as OOD data: this combination tests whether the SM can distinguish new classes that are very different from the known ones. • CIFAR-10 as ID data, and GTSRB as OOD data: this combination tests the same as the aforementioned combination. However, since the ID data is different, the ML and the SM are built with different data. Hence, this permutation produces different outcomes. In total, we produced 79 benchmark datasets. More details can be found in the results section of our repository [START_REF] Ferreira | Code repository for the paper: benchmarking safety monitors for image classifiers with machine learning[END_REF].

B. System under test

We simulate a scenario in which the SM has to detect OOD data by checking one RGB-colored image at a time in a randomly ordered stream of images. Once the SM makes a detection, it cancels the ML classification since this classification is potentially wrong. If nothing is detected, the ML classification is accepted.

For the ML model, we use a LeNet [START_REF] Lecun | Lenet-5, convolutional neural networks[END_REF] since it is a simple and traditional convolutional neural network (CNN) algorithm. It contains around 100,000 parameters and 128 neurons in the last hidden layer. For the monitors (noted as SM), we use one method of each of the three different strategies of data-based SM (presented in section II-D) for monitoring a DNN model:

• DNN inputs: adversarially learned one-class classifier for novelty detection (ALOOC) [START_REF] Sabokrou | Adversarially learned one-class classifier for novelty detection[END_REF]. This method learns how to reconstruct each class during the training phase. It receives an image during operation, tries to reconstruct this image to the known class, and analyzes the loss error resulting from this reconstruction. If this loss error surpasses a safe threshold, ALOOC flags it as OOD.

• DNN intermediate values: outside-of-the-box abstraction (OOB) [START_REF] Henzinger | Outside the box: Abstraction-based monitoring of neural networks[END_REF]. This method projects a 2D-box region from the activation function values (RELU) from the hidden layers of the DNN during the training phase. At runtime, it receives an image and projects a point from it built from the outputs of the activation function from the same DNN layer used during the training. If this point falls outside the 2D box, OOB will flag this image as OOD. • DNN outputs: detector of out-of-distribution images in neural networks (ODIN) [START_REF] Liang | Enhancing the reliability of outof-distribution image detection in neural networks[END_REF]. This method learns how to balance the confidence values outputted along with the DNN decision. It uses these values from the last layer of DNN during training and applies a method known as temperature scaling, which will scale the confidence values to a new threshold for considering an image as OOD. During the operation, ODIN verifies if the DNN confidence value over an incoming image respects the threshold. If not, ODIN will flag this image as OOD.

C. Parameter's evaluation and choice

Next, we present the chosen parameters for the SMs. To avoid biased results, all the initial parameters were tuned using only the ID dataset and are summarized in Table I. We explored the original parameters for each SM, and we also found/tested new values/methods not mentioned in their original papers (marked with *). For ALOOC, the authors did not investigate the influence of the optimizers in the original paper. Thus, we tested two different optimizers: ADAM [START_REF] Zhang | Improved adam optimizer for deep neural networks[END_REF] and RMSProp [START_REF] Wichrowska | Learned optimizers that scale and generalize[END_REF]. We also analyzed the best model for each class through different epochs. In general, all the models achieved a better convergence around 200 epochs. Since the threshold value for considering whether a class is considered OOD does not indicate in the original paper, we considered the average reconstruction loss for each class during the training.

For the OOB method, we test the parameter responsible for enlarging the size of boxes (τ ) with the same range of values proposed by the original paper. We applied the same number of clusters as suggested by the original paper (no clusters or three clusters). However, we also tested with a possible optimal number of clusters. The possible optimal number of clusters K for the K-means algorithm contained in the OOB method is chosen through the Elbow analysis [START_REF] Kodinariya | Review on determining number of cluster in k-means clustering[END_REF]. Thus, the best K value was 5 and 17 for CIFAR-10 and GTSRB, respectively. Finally, we also tested three different approaches for the 2D-dimensionality reduction parameter: simple projection (proposed in the original paper), PCA [START_REF] Yang | Two-dimensional pca: a new approach to appearance-based face representation and recognition[END_REF], and ISOMAP [START_REF] Balasubramanian | The isomap algorithm and topological stability[END_REF]. ISOMAP is a popular nonlinear dimensionality reduction method, and PCA is also a popular method, but linear. With these two suggested methods, we can analyze if the choice of the dimensionality reduction methods influences the outcomes.

Regarding ODIN, we set the temperature (1000) and the magnitude parameters (0.0025 and 0.0014, for GTSRB and CIFAR-10, respectively) as suggested by the authors. However, we chose the confidence thresholds for determining OOD data by selecting the lower confidence value outputted from the method when exposed to the training data. In this case, 0.0237 and 0.1007, for GTSRB and CIFAR-10, respectively. We had to assume a confidence value threshold since it would not be possible to use ODIN as an SM without it.

D. Results

1)

SMs performance: we use the positive/negative data and the evaluation metrics as mentioned in Section III.C. Since we measure the detection, there is no necessity to include the measurements for the ML. Best results are written in bold.

Table II shows these results for novelty class: GTSRB as ID dataset and BTSC as OOD dataset; CIFAR-10 as ID and GTSRB as OOD; and GTSRB as ID and CIFAR-10 as OOD. The first benchmark dataset has the challenge of having ID and OOD data with a similar domain, while the other two have very different classes from each other. II: Comparing data-based monitors for novelty class. The MCC results indicate that the SM is sometimes slightly better than a random classifier (MCC ≈ 0). We observe that these methods' weakness is in the high amount of false positives, yielding a borderline MCC performance. According to the results, just ALOOC obtained a false-positive rate of around 50%. However, it had many false negatives, which can be considered worse depending on the scenario. In general, all SM had a poor performance due to the unreliable nature of DNN confidence values, and the high nonlinear nature of activation functions in ODIN and OOB, respectively.

Next, in Table III, we show the same analysis but using CIFAR-10 and GTSRB after being modified through an adversarial attack known as fast gradient sign method (FGSM).

According to the results, for CIFAR-10, the methods achieved negative values for MCC, which indicates performance worse than a random classifier. For GTSRB, the results are slightly better, but the MCC values can be considered Here, we see a surprising but negative outcome: OOB suffers from a lot of false positives. In contrast, ODIN suffers from many false negatives. It means that they always tend to miss a detection or to say that everything is OOD data. ALOOC got a fair MCC value for CIFAR-10 with intense fog and a good MCC value for GTSRB with heavy snow. Another interesting result for ALOOC is that despite the low MCC performance on the other benchmark datasets, it got relatively good micro-F1 results. It highlights that this method tends to have a good amount of false negatives but rarely gives a wrong output when detection is signaled. Again, methods based on inspecting inputs seem to be more effective in detecting changes in the pixel values distribution.

Once we evaluate the methods in all datasets, we investigate how their results statistically differ from each other. The reason is to observe if there is an SM that is better than the others. We applied a Friedman test with Nemenyi posthoc with 95% the confidence level for all methods ranks through every dataset regarding the MCC metric. From that, we build a critical difference diagram illustrated in Figure 3. The result shows that ODIN achieved the best results in more benchmark datasets than the other two SM. However, there is no statistical difference between the SMs. It means that regarding the MCC results, there is no best SM method. Next, we evaluate how much the SM impacts the SUT when it works along with the ML classifier.

2) Overall impact of the SM: As previously explained, the objective is to evaluate how much the SM impacts the SUT when it works along with the ML classifier. Table V shows the overall SM's impact in the SUT when using GTSRB or CIFAR-10 regarding ID data. The results are expressed as MCC values. To evaluate the SM, we also use a percentage of relative change showing how much better/worse it performed compared to the baseline (ML alone). In the GTSRB dataset, the best values for the ALOOC method were obtained when using the ADAM optimizer. In contrast, the OOB method obtained the best results when using 3 clusters, ISOMAP, and an enlargement factor of 0.1. As mentioned earlier, for ODIN, the threshold was set to 0.0237.

For the CIFAR-10 dataset, the best optimizer for ALOOC was ADAM. For OOB, the best parameters were: 5 clusters, ISOMAP, and an enlargement factor of 0.35. For ODIN, we set the threshold to 0.1007.

As can be noted, all SM methods perform worse than the ML alone when exposed to ID data. This result is expected due to the generalization power of ML models. However, a reliable SM is also the one that can perform well when exposed to ID data, avoiding an ML misclassification or simply not raising a false alarm hindering a correct ML classification.

These results show the best SM's performance was obtained by the outside-of-the-box and ODIN, on GTSRB and CIFAR-10, respectively. Results indicate that the SM performed consistently better than a hypothetical random classifier (M CC = 0). However, they also show a significant gap between the performance of the ML alone and the SM for ID data, especially in the GTSRB that has four times more classes than CIFAR-10. This result is interesting since the tested monitors use information from the ML model and build their hypothesis over ID data.

3) Time and memory overhead induced by SMs: We show in Table VI an example of the performance for ALOOC, OOB, and ODIN, respectively, in the three benchmark datasets for novelty class detection. The values are expressed in seconds and represent the average time spent on the tasks of classification and monitoring on a single image. We used a computer with a Processor Intel(R) Core(TM) i7-9850H CPU @ 2.60GHz, and 32GB of memory.

Worth mentioning that we can have slight differences between the prediction time in the three methods since they use different deep learning libraries. ALOOC uses Keras, OOB uses Tensorflow, and ODIN uses PyTorch. Besides, the SUT time also contains the remaining time from other small processes involved in the experiment, so the summing between the prediction and monitoring time is lower than the SUT time. According to the results, ODIN is the fastest method. Except when using OOB with ISOMAP, all three methods needed no more than 0.07 seconds to spot OOD data. Such process time can be considered fast enough to be applied at runtime. Besides, they can be optimized, potentially reducing this time.

However, the monitoring task is always slower than the prediction and can be responsible for 96% of the necessary time for a SUT to do the task. Since memory is also an important constraint, especially on embedded systems, we show in Table VII the memory efficiency of the methods when performing a novelty class detection over the GTSRB dataset. As can be seen, ALOOC needs a considerable amount of memory or disk space due to the necessity of developing one monitor for each class. For instance, even though ALOOC needs just 2.3MB per class, it is necessary almost 100MB to monitor all 43 classes contained in the GTSRB dataset. For OOB, it is also necessary to build one monitor for each class. However, the amount of memory needed is not huge since it uses just some stored arrays to make the boxes. For ODIN, it is not necessary to build one SM for each class. Moreover, since the algorithm needs to be applied during the training phase to collect the thresholds for the confidence values, it can use just a tiny amount of memory to do the inspection.

E. Threats to validity

In order to analyze and mitigate threats to the validity of the results, we present below a summary of arguments for external and internal validation. -Bad parameters choices for the SM: aiming at improving the SM performance, we explored additional parameter values than those explored in their original papers. For instance, we applied different dimensionality reduction methods for the outside-thebox. However, even though different dimensionality reduction methods can bring better results, they also introduce a high cost to time and memory performance. Regarding ALOOC and ODIN, we had to choose criteria to the threshold value for considering whether a class is considered out-of-distribution or not. This way, if one applies different criteria for these methods, they will result in different outcomes. However, such criteria had to be deducted since they are not contained in their original papers. Besides, ODIN used OOD data to calibrate their monitor, what can be considered unrealistic. Hence, in this work, all the parameters were chosen using just ID data. It makes the scenario more realistic and harder, which drastically decreases the SMs performance. -Choice of datasets: even though we chose datasets widely applied in the computer vision literature, the choice of the amount of ID data accessible for building the ML classifier and the SM can influence both. Furthermore, the amount of ID and OOD data in the benchmark datasets also can influence the results. However, we followed traditional ways to divide data (e.g., 80/20 for training and test). Besides, the authors of OOB [START_REF] Henzinger | Outside the box: Abstraction-based monitoring of neural networks[END_REF], and ALOOC [START_REF] Sabokrou | Adversarially learned one-class classifier for novelty detection[END_REF] test their methods considering the same dataset for ID and OOD data, which they test novelty detection by training the ML with 9 classes and omitting one, or training with 8 classes and omitting two, and so on.

However, we in this work we test novelty detection using an entirely new dataset as OOD data along with a part of ID data (ex: CIFAR-10 + GTSRB).

Domain of validity:

The setting CIFAR10-GTSRB constitutes two different datasets that do not have the same validity domain [START_REF] Riccio | Model-based exploration of the frontier of behaviours for deep learning system testing[END_REF] (i.e., GTSRB is not only out-of-distribution but also out of the validity domain). Thus, the task of detecting OOD data should be less complicated in this scenario. However, the SM methods continued to be inefficient.

V. CONCLUSION AND RESEARCH DIRECTIONS In this work, we proposed a framework for benchmarking safety monitors for ML classifiers. We argue that there is a need for a better framework for benchmark SMs based on data by applying more metrics beyond accuracy and AUROC curves. Thus, we proposed a minimal set of measurements that should be considered when benchmarking such solutions.

Besides, we also showed that measuring these SMs is less straightforward than it seems. We presented two dimensions that need to be investigated: the SM's overall impact on the SUT; and the detection performance of the SMs. Our approach allowed insights into three categories of data-based monitors.

Our results indicate at least four general takeaways for the current solutions that are solely based on DNN's data:

• The overall accuracy in the detection tends to be as bad as a random classifier due to the high amount of false positives or false negatives, depending on the scenario. • Such methods tend to negatively affect the system under test since they yield too many false positives interfering with the correct ML decision. • They need a considerable amount of memory or disk space since a monitor can consume up to 100MB. • They are not so fast to perform the detection and have a considerable overhead compared to the ML software. The tested SM were exposed just to novelty detection in their original papers. However, testing these SM to other four types of OOD data was important to show that:

• SM based on DNN inputs: it has the advantage that it does not need to inspect the internals of the DNN. Thus, it can make the monitoring independent of the ML classifier. However, it has the drawback of being biased to the data used during the training. That is, it allows to look at the internal values that led the DNN to decide. However, the drawback is that the activation functions are insufficient to provide all information helpfully learned from the DNN. • SM based on DNN outputs: it has the advantage that it decreases and equilibrates the confidence values from the DNN. However, similarly to approaches based on DNN intermediate values, the drawback is that it relies heavily on the performance of the DNN. A limitation of this work is that we did not apply more and bigger DNN architectures. Such experiments could produce a more robust evaluation. Moreover, we benchmarked just over image datasets, not considering an entire system. Therefore, it is also essential to investigate the performance of the SMs inside a simulator to achieve a complete analysis. Finally, as future work, we also want to add SMs based on uncertainty quantification [START_REF] Berend | Cats are not fish: Deep learning testing calls for out-of-distribution awareness[END_REF], [START_REF] Weiss | Fail-safe execution of deep learning based systems through uncertainty monitoring[END_REF] to our experiments.

The main conclusion is that using such data-based monitors does not provide sufficient confidence for their use in a safetycritical application. Thus, we want to explore how to combine data-based and model-based monitors.

  data arrives in the stream -If the SM detects OOD data: it means a false positive for the specific task of OOD detection. * If the ML classification is equal to the ground truth: it means a false positive for the overall task of avoiding a failure since the SM intervened without necessity. * If the ML classification is different than the ground truth: it means a true positive for the overall task of avoiding a failure since it canceled the ML misclassification for ID data. -If the SM does not detect OOD data: it means a true negative for the specific task of OOD detection. * If the ML classification is equal to the ground truth: it means a true negative for the overall task of avoiding a failure since the ML gave a correct classification even though the SM did not detect OOD data. * If the ML classification is different than the ground truth: it means a false negative for the overall task of avoiding a failure since the ML misclassified the OOD data and the SM did not detect it, which could cancel a wrong ML output. • OOD data arrives in the stream -If the SM detects OOD data: it means a true positive for the specific task of OOD detection. * If the ML classification is equal to the ground truth: it means a false positive for the overall task of avoiding a failure since the ML gave a correct classification but the SM wrongly intervened. * If the ML classification is different than the ground truth: it means a true positive for the overall task of avoiding a failure since the ML gave an incorrect classification and the SM correct canceled the ML decision. -If the SM does not detect OOD data: it means a false negative for the specific task of OOD detection. * If the ML classification is equal to the ground truth: it means a true negative for the overall task

Fig. 3 :

 3 Fig. 3: Critical difference diagram between the three methods.

•

  SM based on DNN intermediate values: it has the advantage of inspecting the ML model as a white box.

TABLE I :

 I SM parameters tested in the experiments.

	SM	Parameter name	Parameter values
	ALOOC Optimizer	ADAM, RMSProp
		Epoch	200
		Loss threshold	average per class*
	OOB	γ	0, 0.1, 0.35
		# of clusters	0, 3, 5*, 17*
		Dimensionality reduction simple, PCA*, ISOMAP*
	ODIN	Temperature	1000
		Magnitude	0.0014, 0.0025
		Confidence threshold	0.0237*, 0.1007*

TABLE III :

 III Comparing data-based monitors for CIFAR-10 and GTSRB datasets with a adversarial attack. F1 for ODIN and ALOOC, in the CIFAR-10 and GTSRB datasets, respectively, the rate of false negatives was high. It means that the ML classifier did not give the correct prediction, and the SM did not detect the attack.Next, in TableIV, we show the results for CIFAR-10 and GTSRB datasets with different types of distributional shift.

				CIFAR-10
	Variation Method	MCC FPR FNR Pr	Re	F1
		ALOOC	-0.23	0.89	0.29	0.28	0.71	0.25
	FGSM	OOB	-0.13	0.92	0.16	0.31	0.84	0.24
		ODIN	0.06	0.14	0.81	0.37	0.19	0.62
				GTSRB
	Variation Method	MCC FPR FNR Pr	Re	F1
		ALOOC	0.19	0.22	0.59	0.44	0.41	0.66
	FGSM	OOB	-0.01	1.0	0.0	0.31	1.0	0.14
		ODIN	0.11	0.92	0.02	0.34	0.98	0.26
	as flawed as a random classifier. Despite the good values
	of micro-

TABLE IV :

 IV Comparing data-based monitors for CIFAR-10 and GTSRB with different types of distributional shift.

				CIFAR-10
	Variation Method	MCC FPR FNR Pr	Re	F1
		ALOOC	0.0	0.0	0.29	1.0	0.71	0.83
	rotated	OOB	0.02	1.0	0.0	0.14	1.0	0.04
		ODIN	-0.1	0.32	0.81	0.09	0.19	0.66
		ALOOC	-0.01	0.42	0.59	0.14	0.41	0.62
	snow (5)	OOB	0.0	1.0	0.0	0.14	1.0	0.04
		ODIN	0.14	0.08	0.81	0.29	0.19	0.8
		ALOOC	0.47	0.03	0.59	0.68	0.41	0.88
	fog (5)	OOB	0.0	1.0	0.0	0.14	1.0	0.04
		ODIN	-0.01	0.21	0.81	0.13	0.19	0.73
				GTSRB
	Variation Method	MCC FPR FNR Pr	Re	F1
		ALOOC	0.0	0.0	0.55	1.0	0.45	0.62
	rotated	OOB	0.09	0.75	0.16	0.21	0.84	0.38
		ODIN	-0.12	1.0	0.02	0.19	0.98	0.06
		ALOOC	0.81	0.0	0.29	1.0	0.71	0.94
	snow (5)	OOB	0.01	0.83	0.16	0.2	0.84	0.29
		ODIN	0.16	0.85	0.02	0.22	0.98	0.28
		ALOOC	-0.28	0.93	0.29	0.16	0.71	0.15
	fog (5)	OOB	0.0	0.84	0.16	0.2	0.84	0.28
		ODIN	0.0	0.98	0.02	0.2	0.98	0.1

TABLE V :

 V MCC values for SUT with/without SM for GTSRB or CIFAR-10 as ID dataset.

	Method	GTSRB CIFAR-10
	ML alone	0.96	0.74
	ML + ALOOC 0.51	0.53
	ML + OOB	0.64	0.61
	ML + ODIN	0.50	0.64

TABLE VI :

 VI Time impact of SM per instance in seconds for novelty class detection.

	Method	ML	SM	SUT
	ALOOC 0.0070 (3.0%) 0.2217 (96.8%) 0.2288
	OOB	0.0021 (3.9%) 0.0529 (96.0%) 0.0551
	ODIN	0.0007 (3.0%) 0.0246 (96.9%) 0.0254

TABLE VII :

 VII Memory size of ML, and SM in MB.

	Method	ML SM
	ALOOC 3.8	98.9
	OOB	4.3	6.4
	ODIN	2	1.5

  All tested SMs were already compared to other methods in their original papers. They obtained the best results during the comparison. Besides, we also chose three different approaches of monitoring (inputs, intermediate values, outputs) that may cover an acceptable range of SM approaches based on data. However, it is worth mentioning that this study is a first benchmark work. Thus, other approaches may be published with better performances in the future. What could have to lead us to the wrong conclusions in our study?

• External validity: • Internal validity: