Garen Avedissian

Jacek Arabski

Jennifer A Wytko

Jean Weiss

Christian Meny
email: christian.meny@ipcms.unistra.fr

Revealing the morphology and the magnetic properties of single buried cobalt-ZnTPP hybrid interfaces by FNR spectroscopy

The deeply buried, yet most important part of any spintronic device is the interface. This is even more interesting and much more complex when soft, light materials like organic molecules are in contact with an inorganic metallic electrode. Hence, exceptional care is required to better understand the phenomena driven by this type of organic/inorganic interfaces. To this end, ferromagnetic nuclear resonance (FNR) spectroscopy studies were performed to investigate the morphology and the magnetic properties of the hybrid organic-inorganic interfaces when zinc tetra-phenyl porphyrin (ZnTPP) molecules are at the vicinity of ferromagnetic metallic cobalt (Co) layers. The FNR experimental results show that when ZnTPP is deposited on top of Co the resulting interface is smoother and sharper compared to the more extended interface obtained when Co is deposited on top of ZnTPP. The shape of the spectra suggests that no chemical bonds take place between the interfacial Co atoms and the ZnTPP molecules and that interactions at the interfaces are governed by weak van der Waals forces. Finally, FNR also showed that the magnetic anisotropy at the Co-ZnTPP hybrid interfaces is reduced compared to the magnetic anisotropy of the Co atoms inside the Co films.

I. Introduction

Carbon-based organic compounds are expected to enhance the performances and functionalities of spintronic devices because of the large spin coherence length and long spin relaxation time expected in molecules. Many interesting physical properties have been observed in metal/organic hybrid systems [1][2][3][4][5]. The interfaces play a big role in these effects and it was even suggested that the properties of devices could be tuned by controlling the morphology of the interfaces. This resulted in the proposed concept of "spinterface" science [6]. However, before controlling and tuning interface morphologies and their physical properties, it is required to correlate existing interface morphologies to the sample properties. The morphology of the interface will depend on many parameters like roughness, atomic diffusion, chemical bonds etc. In this context, zinc tetra-phenyl porphyrin (ZnTPP) molecules have been chosen to build hybrid organic/inorganic heterostructures to study the morphological properties of the hybrid interfaces buried within the heterostructures. Because of the organic nature of one of the layers, conventional characterization techniques are often difficult to use especially when post-growth sample processing is required. In this work a non-invasive ferromagnetic nuclear resonance (FNR) [7,8] spectroscopic method has been developed to investigate the structure and the morphology of buried Co-ZnTPP hybrid interfaces. FNR corresponds to nuclear magnetic resonance (NMR) when it is applied to study ferromagnetic systems. It is also sometimes called zero field NMR or internal field NMR. NMR is widely used in biology or chemistry, but its use to study ferromagnetic nanostructures is much scarcer. Yet, it has been shown that FNR is a very efficient tool to study the structure and morphology of interfaces in thin films, multilayers [9][10][11][12][13][14][15][16][17] and even devices [18]. FNR also provides unique insight into the understanding of the physicochemical properties of assemblies of nano-objects [19][20][21][22][23].

In a previous work, FNR has already been used to probe the continuity of the ZnTPP molecular layers embedded between ferromagnetic electrodes [24]. It was shown that to fully separate the ferromagnetic electrodes a minimum deposited thickness of 15 monolayers (ML) of ZnTPP is required. Making use of this knowledge, the morphology of the Co-ZnTPP hybrid interfaces has been investigated by building a series of ZnTPP/Co(t)/ZnTPP hybrid sandwiches. As already demonstrated for inorganic films, studying a series of samples that are dependent on the Co layer thickness allows determining the extension of the interfaces [7,8,18]. Surprisingly, this method revealed very extended Co-ZnTPP interfaces with unexpected FNR contributions. To understand this phenomenon, an original sample architecture has been implemented to investigate individually the buried bottom ZnTPP/Co interface and the buried top Co/ZnTPP interface. It showed that when ZnTPP is deposited on top of Co, the interface is quite sharp whereas the interface is much wider and involved a much larger amount of Co atoms when Co is deposited on top of the ZnTPP layer. Although asymmetric interfaces have already been evidenced in multilayers through FNR spectra modelling [15], it is the first time that single buried interfaces have been directly investigated by FNR. These investigations also showed that no specific and well-defined environment could be attributed to the interfaces formed by the Co atoms and the ZnTPP organic layer. This suggests that there are no strong chemical bonds between the surface Co atoms and the ZnTPP molecules and that the interactions at the interfaces are governed by weak van der Waals (vdW) forces. Finally, since the local magnetic anisotropy can also be probed by FNR [7,8], it is shown that the Co atoms at the interfaces experience a smaller magnetic anisotropy field than the Co atoms in the centre of the Co films.

II. Materials and methods

Two sets of heterostructures were prepared. First, the Co layers sandwiched by two ZnTPP films:

Si/SiO2 (500 nm)/Cu (25 nm)/Fe (30 nm)/ZnTPP (20 ML)/Co (t in nm)/ZnTPP (20 ML)/Fe (30 nm)/Cr (100 nm), with Co thickness ranging from 2 nm to 6 nm Second, the samples designed for studying the single buried interfaces: Si/SiO2 (500 nm)/Cu (25 nm)/Fe (30 nm)/Co (6 nm)/ZnTPP (20 ML)/Cr (100 nm) and the reversed architecture: Si/SiO2 (500 nm)/Cu (25 nm)/ZnTPP (20 ML)/Co (6 nm)/Fe (30 nm)/Cr (100 nm)

The copper (Cu) seed layers, the chromium (Cr) capping layers as well as the iron (Fe) layers were deposited by direct current (dc) sputtering and under an argon pressure of 4.5 x 10 -4 mbar in a standard sputtering chamber at a rate of 0.6 Å/sec, 0.5 Å/sec and 0.25 Å/sec, respectively. The organic films are deposited in an organic molecular beam epitaxy chamber (OMBE). A total of 20 monolayers (ML) of analytically pure, in-lab synthesized, ZnTPP molecules were sublimed under 2 x 10 -9 mbar of deposition pressure and at a rate of 0.055 monolayer/sec (ML/sec with 1 ML=3.5 Å; monitored with quartz). The Co films were deposited in an ultrahigh vacuum (UHV) chamber by thermal evaporation, with a flux of 0.16 Å/sec under a deposition pressure of 2 x 10 -9 mbar. All deposition chambers are connected under vacuum therefore the samples can be transferred from one chamber to another without breaking the UHV conditions.

In order to check that the Co layers are not contaminated during the deposition of the ZnTPP films two test Co films were prepared. One of the test samples was capped directly after the deposition of the Co film while the second one was introduced in the organic chamber. After ZnTPP started to sublimate, the test Co film was kept in the organic chamber during the same time as the regular samples but the ZnTPP crucible shutter was kept closed. After removal from the organic chamber the sample was capped and its magnetization curve compared the one of the Co film directly capped after the Co deposition without being introduced into the organic chamber. The magnetization of the two samples where very similar and did not show any difference in saturation magnetization. It shows that the sublimation of ZnTPP does not deteriorate the chamber base pressure and does not result in the contamination of the Co film surface.

All FNR measurements were performed in a homemade state of the art FNR set up. Regular NMR spin echo method has been used (pulse length: 3.2 µs, delay: 3 µs). The integrated spinecho intensity was recorded using a broadband pulsed spectrometer with phase-sensitive detection and automated frequency scanning. In FNR, like in all NMR techniques, the degeneracy of the nuclear spins energy levels is lifted by a static magnetic field. The main particularity of FNR is that the static field already exists inside the ferromagnetic samples and has for origin the spontaneous magnetization of the sample. The main contribution to this internal static field is the so called the hyperfine field. Therefore, determining the nuclei resonance conditions is performed by frequency sweeping. The second particularity of FNR is that the amplitude of the radiofrequency (RF) field (the frequency swept field) experienced by the nuclei is enhanced by the magnetic susceptibility of the samples. Therefore, the optimum RF field power required to obtain the maximum FNR signal depends on the magnetic properties of the samples and has to be determined for each sample. The detailed experimental procedure is explained in [7,8]. This procedure allows to get FNR amplitudes that represent the true distribution of the probed atoms (nuclei) versus the RF field frequency. In addition, this experimental procedure also allows measuring the local magnetic stiffness (proportional to the reciprocal of the local susceptibility) in the sample [7,8]. This is represented by a frequency dependent restoring field. This restoring field characterizes the torque experienced by the local magnetic moments when their direction is slightly tilted from their position at rest. Depending on the nature of the samples it can be identified as a local effective anisotropy field, or exchange field, or pinning field… In the samples under investigation in this work the restoring field can be identified as the in plane local magnetic anisotropy field of the samples. The FNR measurements were performed at 1.8 K and the samples' surface area was of the order of 3 cm 2 . All the spectra are recorded from 50 MHz to 320 MHz with 0.6 MHz steps. They are measured typically during 12 hours' accumulation time resulting in the averaging of about 10 6 spectral measurements. As the thickness of the samples is much thinner than the RF penetration length, several micrometers for the considered frequency range, the full stack FNR signal contribution is retrieved. With these experimental conditions the set-up sensitivity is better than 0.1 atomic layer of Co (or 0.02 nm of Co). The only post measurement normalization that is performed is to divide the recorded FNR intensity by the samples' surface area so that any discrepancy in the size of the samples is removed and their respective FNR intensities can be compared directly.

III. Results and discussion

For all samples, the ZnTPP films were chosen to be 20 monolayers thick (≈ 7nm) to ensure the continuity of the molecular layers [24]. It has been shown that this thickness value prevents the migration of metallic atoms through the organic film [24]. However, to ascertain the continuity of the organic films, additional 30 nm-thick Fe layers were added at the top and bottom of the architecture. As shown in [24] (and by the Cu/Co/Fe sample spectrum in black dashed line in figure 1) the contact of Fe with Co would result in high frequency contributions in the FNR spectra [25,26] and would directly reveal any disruption in the organic films.

The FNR spectra for the Cu/Fe/ZnTPP (20ML)/Co (t in nm)/ZnTPP (20 ML)/Fe hybrid sandwich heterostructures are given in Figure 1. No FNR signal was observed above 250 MHz, which confirms that the ZnTPP layers are thick enough to form continuous molecular films. It also shows that no Co-ZnTPP interfacial contribution is found at high frequency. Indeed, chemical bonds between Co and organics can give rise to high frequency contributions as it has been shown in the case of Co atoms bonded to carbon nanotubes [27]. Using the method established for the study of inorganic films and multilayers, the FNR spectra are normalized to the total surface area of the samples [8,18]. With this normalization process, the contributions in the spectra resulting from the ZnTPP/Co/ZnTPP interfaces should show no evolution in their spectral shape once the interfaces are completed. Only the contribution of the bulk part (Co situated far from the interfaces) will increase with the increasing thickness of the Co film. Figure 1 shows that as the Co thickness increases, the position of the main line becomes better defined and reaches a position of 224 MHz close to that expected for Co atoms in a hexagonal closed pack (hcp) crystal structure having its magnetization axis perpendicular to the c-axis of the hcp crystal cell [25] ([0001] growth direction of the Co film with in plane easy magnetisation direction).

Whatever the morphology of the samples (bulk [25,26], films [7,8] or nanoparticles [19]) the main line corresponding to pure Co surrounded by other Co atoms will give no contributions outside the frequencies ranging from 200 MHz to 250 MHz. As mentioned above no contribution has been observed above 250 MHz, therefore only the FNR intensity at low frequency (< 200 MHz) has to be attributed to the Co/ZnTPP and ZnTPP/Co interface contributions. Whereas below about 120 MHz all spectra are superimposed (within experimental errors), between 120 MHz and 200 MHz the spectra intensities continue to increase surprisingly until a Co thickness of 5 nm (the 5 and 6 nm low frequency spectra are superimposed, within experimental error bars). This shows a complex interface behaviour and suggests extended interface regions.

When interface regions are extended, it is not unusual for the interfaces to involve some fraction of Co atoms that are no longer ferromagnetic. These nonmagnetic Co atoms will not be detected by FNR. To probe the presence of non-ferromagnetic Co atoms, the total intensity of the FNR spectra as a function of the deposited Co thickness is represented in Figure 2. As expected, since the spectra intensities are normalized to the samples' surface area, the FNR integral intensity increases linearly with the increase of the deposited Co thickness (Figure 2, black symbol). However, the linear fit (Figure 2, black line) clearly shows that the total FNR intensity does not pass through the origin when extrapolated to zero Co thickness. From the intercept of the linear fit, the amount of Co that is no longer ferromagnetic can be determined. About 1.4 nm of Co are not ferromagnetic at 1.8 K (temperature of measurements) and hence not detected in the FNR spectra.

These dead layers can have several origins. The first one can be a partial oxidation of the Co films. As described in the methods section, it has been checked that the sublimation of the ZnTPP film does not result in the contamination of the Co film surface. The efficiency of the capping layer has also been checked by tracking the magnetic properties of the samples over several months. No ingress of oxygen could be evidenced at this time scale. It has also been reported than when Co is deposited on SiO2 the first layers can get oxidized [28], however in our samples the Co films are separated by a 25 nm thick Cu seed layers. Therefore, oxidation of the Co film can most likely be ruled out. Dead layers can also result from interfacial strains [29] however it looks unlikely that epitaxial strains can be induced on the Co films by the mechanically soft organic films. Dead layers can also result from amorphous phases but as the Co films are deposited at room temperature amorphous phases can only result from alloying with an alien element [10]. Nor Co-Fe nor Co-Zn alloys can produce amorphous phases. Therefore, most likely the observed dead layers result from Co atoms that did diffuse deep into the ZnTPP molecular films when forming the two hybrid interfaces. This has already been suggested in the study of the growth of the ZnTPP films [24]. The plot of the spectra integral intensities for frequencies below 200 MHz (interface range) is also shown in Figure 2. Since the spectra are normalized to the surface area of the samples, these integral intensities should reach a constant value once the interfaces are completed. This is obviously not the case in these samples. Only the integral intensities for the 5 and 6 nm samples become similar. This can be clearly seen from Figure 1 where the 5 and 6 nm low frequency spectra are superimposed (within the FNR experimental error bars). Since the contributions below 200 MHz are out of the Co bulk frequency range it shows that the interfaces continue to build up until 5 nm of Co thickness. However, it is rather surprising that this "building up of interfaces" results in contributions above 120 MHz only. One explanation might be that the top and bottom Co-ZnTPP interfaces are highly asymmetrical. To investigate the asymmetry of the Co interfaces, samples with specific architectures were prepared: Cu/Fe/Co (6 nm)/ZnTPP (20ML) and Cu/ZnTPP (20ML)/Co (6nm)/Fe heterostructures. Such architectures allow to split the contributions of the two buried Co interfaces in two distinct frequency ranges allowing the investigation of single buried interfaces. Indeed, when Co atoms have Fe atoms among their nearest neighbours, the Co resonance frequency increases [24][25][26] and thus the Co/Fe interface contributions will appear at high frequency and the low frequency contributions will originate from the single Co/ZnTPP or ZnTPP/Co interfaces only. The FNR spectra of these two samples are shown in Figure 3. The comparison of the spectra of the top Co/ZnTPP and bottom ZnTPP/Co interfaces shows that below 140 MHz the two interfaces result in identical and superimposable contributions, as observed for the ZnTPP/Co/ZnTPP sandwich samples. However, between 140 MHz and 200 MHz, the two spectra depart significantly. The FNR signal remains weak when the ZnTPP layer is deposited on top of Co while the FNR intensity increases dramatically when Co is deposited on top of the ZnTPP layer. It shows that a much larger amount of Co atoms is involved in the interface region when Co is deposited on top of ZnTPP than when ZnTPP is deposited on top of Co. The slope of the FNR integral intensities versus the deposited Co thickness, Figure 1, corresponds to the FNR experimental intensity resulting from the deposition of 1 nm of Co. Using this value, the amount of Co atoms involved in specific frequency ranges of the spectra can be estimated. From the integral intensity of the low frequency contributions (< 200 MHz) it is therefore possible to estimate the amount of Co atoms involved in the interfaces with ZnTPP. When ZnTPP is deposited on top of Co, the thickness of Co involved in the interface is of the order of 0.4 nm. Considering that a perfectly flat interface would already involve 0.2 nm (one atomic plane) [7,8] of Co, an interface contribution of about 0.4 nm is rather small and is characteristic of a sharp interface. On the contrary, when Co is deposited on top of ZnTPP, the quantitative analysis shows that the interface contribution is much larger: it is of the order of 0.7 nm of Co thickness. This can be understood by considering that the interlayer distance between the organic planes is much larger (0.35 nm-0.4 nm) than that of a metal, therefore a small roughness at the organic layer scale will directly impact a large number of Co atoms. It is thus not surprising to observe that the amount of Co atoms involved in the interface is much larger when Co is deposited on top of ZnTPP than when ZnTPP is deposited on top of Co. It explains why the top interface extends over a large thickness and why in the hybrid sandwich layers, a minimum of 5 nm of Co is required to complete the formation of the interfaces. Compared to other metal/organic systems, when ZnTPP is deposited on top of Co the interface is rather sharp (for example it extends to about 2 nm when C60 is deposited on top of Fe [30]) while it is more extended when Co is deposited on top of ZnTPP.

Considering the shape of the interface FNR spectra, none did show any well-defined, resolved, interfacial environment. Interface spectra consist in unstructured low frequency tails. This suggests that there are no strong interactions between the Co atoms and the organic layer (in other words no chemical ionic/covalent bonds). Indeed, in the case of Co atoms bonded to carbon atoms, well defined FNR lines have been reported [27]. In addition, considering previous works in inorganic films and multilayers, when interfaces are well defined sharp lines are observed (examples are given in [7] and [8]). This is mostly due to the fact that all Co atoms experience the same local environment at the interface. When interfaces are disordered more lines appear and broaden the FNR response. Transposing this in the case of the Co/organic systems if strong chemical bonds exist between Co and the organic molecules they should result in a well-defined local environment for the interfacial Co atoms and therefore in a welldefined resonance line. On the contrary in the case of weak van der Waals interactions the position of the molecules on top of the Co films will be easily influenced by surface defects. Consequently, there will be no specific relationship between the position of the Co surface atoms and the organic molecules. Therefore, the Co surface atoms will experience a large variety of local environments resulting in a large dispersion of their resonance frequencies. Since, even in the case of the sharpest interface (ZnTPP on top of Co) no well-defined interfacial line could be identified, this suggests that there is no well-defined relationship between the position of the surface Co atoms and the ZnTPP molecules. The most likely explanation is that the interfacial interactions are of weak van der Waals type.

FIG. 3:

FNR spectra of Cu/Fe/Co/ZnTPP (black) and Cu/ZnTPP/Co/Fe (red) heterostructures. The low frequency spectral region corresponding to the Co-ZnTPP interfaces is magnified by a factor of 5. Between 140 and 200 MHz, the FNR interface contribution is much larger when Co is deposited on top of ZnTPP (in red) than when ZnTPP is deposited on top of Co (in black).

It has been reported in previous works that organic molecules can have a large impact on the surface anisotropy of the magnetic films. It can result both to interfacial hardening or interfacial softening [31]. This effect can also be investigated by FNR since in addition to the provided structural and chemical information, FNR also allows probing the local anisotropy of the samples. Indeed, as briefly explained in the methods part the procedure used to establish the FNR spectra [7,8] also provides the so-called site dependent restoring field. In the studied samples the local restoring field can be identified as the local magnetic anisotropy field. This

Co-Fe interface 224MHz hcp

Co

information is seldom extracted from the FNR data, but recently it was successfully used to understand the origin of the increase in anisotropy of permanent magnets doped with Co [32]. This approach allows to probe the magnetic anisotropy field of the Co atoms at the interfaces with respect to the magnetic anisotropy of the bulk (the inside) of the Co layers. Figure 4 shows the frequency dependent restoring field obtained on both single Co/ZnTPP and ZnTPP/Co interfaces. The behaviour of the two samples is similar as the largest local magnetic anisotropy is obtained at 224 MHz and corresponds to the Co atoms far from the interfaces, situated inside the Co layers. The low frequency (< 200MHz) restoring field is assigned to the magnetic anisotropy field of the Co/ZnTPP and ZnTPP/Co interfaces and it is typically twice smaller than the anisotropy of the bulk Co atoms. Since the interfaces of the Co layers always show a smaller anisotropy field than the bulk part of the Co layers, it is thus possible to conclude that no interface hardening effect is observed in these layers. Indeed, if interface hardening would take place the interface anisotropy field should be larger than the anisotropy field of the bulk part of the Co layers, what is clearly not the case. It is possible that the hardening effect does exist at a local scale but is annihilated by the interfacial disorder. A similar effect was observed in metallic films and multilayers where the predicted increase of interface anisotropy was only observed for samples showing almost perfectly flat interfaces [8]. Since the Co/ZnTPP interface is much sharper than the ZnTPP/Co interface, the effect of the interface disorder should result in significant differences in their bulk to interface anisotropy field ratios. However, this is not observed experimentally since for both samples this ratio is close to two. The only difference that is observed between the samples is an overall shift of the anisotropy fields. This shows that the samples' average anisotropy is slightly different. The sample with the largest interface disorder has a smaller average anisotropy than the other one. Most likely, the absence of interface hardening is due to the weak van der Waals interactions taking place at the Co/ZnTPP interfaces. These weak interactions combined with some interfacial disorder result in a reduction of the magnetic anisotropy at the interfaces.

IV. Conclusion

Co layers sandwiched between ZnTPP layers as well as single buried Co/ZnTPP and ZnTPP/Co interfaces have been studied in this work. No well-resolved FNR contribution could be attributed to Co atoms situated at the Co-ZnTPP interfaces. This suggests that no strong chemical bonds exist between the Co atoms and the organic layers at the interfaces and that the interfacial interactions are governed by weak van der Waals interactions. In addition, the analyses of the FNR spectra showed that the interfaces are strongly asymmetric. When ZnTPP is deposited on top of Co (Co/ZnTPP), it forms a rather sharp interface. On the contrary, when Co is deposited on top of ZnTPP (ZnTPP/Co) a much larger interfacial zone is formed and the deposition of up to 5 nm of Co is required to complete the interface formation. Finally, the interface magnetic anisotropy field was probed through the FNR restoring field and showed that the magnetic anisotropy field of the interfacial Co atoms is twice smaller than the magnetic anisotropy field of the Co atoms in centre of layers. This is attributed to the weak van der Waals interfacial interactions and/or to the interfacial disorder.

FIG. 1 :

 1 FIG. 1: Full lines: FNR spectra of the Cu/ Fe/ ZnTPP/ Co (t)/ ZnTPP/ Fe samples with t=2 to 6 nm. The signal between 200 MHz and 250 MHz corresponds to the Co atoms situated far from the interfaces and that are surrounded by other Co atoms only. The main line is at 224 MHz close to the one expected for bulk hcp Co. Between 120 MHz and 200 MHz the FNR intensity increases with the increase in Co thickness up to a deposited thickness of 5 nm. Below 200 MHz the 5 and 6 nm spectra overlap. Black dashed line: Spectrum of a Cu/Co/Fe reference sample showing at low frequency the Cu/Co interface and at high frequency the Co/Fe interface [24]. The main line position is 217 MHz since the Co film grows with an fcc structure on top of Cu.

FIG. 2 :

 2 FIG. 2: Black symbols: total FNR intensities of the samples versus the deposited Co thickness. The linear fit (black line) does not pass through the origin, evidencing Co dead layers. Red symbols: integral intensities of the part of the spectra below 200 MHz. The red line is the linear fit of the <200 MHz integral intensities.

FIG. 4 :

 4 FIG. 4:Frequency dependent FNR restoring field. Restoring field can be identified as the in plane local magnetic anisotropy field in the samples. The anisotropy of the interfaces (< 200 MHz) is much smaller than the anisotropy of the bulk part of the Co films (> 200 MHz).

Acknowledgements

This work was funded by the French National Research Agency (ANR) through the Programme d'Investissement d'Avenir under contract ANR-11-LABX-0058_NIE within the Investissement d'Avenir program ANR-10-IDEX-0002-02. We acknowledge Christophe Kieber for technical assistance and the IdEx program of the University of Strasbourg for a scholarship for Garen Avedissian.


Piazzi
email: @inrae.fr>

Keywords: Imports graphics, grDevices, stats, utils Suggests knitr, rmarkdown

Description Add-on to the 'airGR' package which provides the tools to assimilate observed discharges in daily GR hydrological models. The package consists in two functions allowing to perform the assimilation of observed discharges via the Ensemble Kalman filter or the Particle filter as described in Piazzi et al.

airGRdatassim airGRdatassim

Ensemble-Based Data Assimilation with GR Hydrological Models

Description

Add-on to the 'airGR' package which provides the tools to assimilate observed discharges in daily GR hydrological models. The package consists in two functions allowing to perform the assimilation of observed discharges via the Ensemble Kalman filter or the Particle filter as described in Piazzi et al. (2021) <doi:10.1029/2020WR028390>. airGRdatassim currently runs on daily hydrological models (GR4J, GR5J and GR6J, with and without the CemaNeige snow model). The package is developed at INRAE-Antony (Catchment Hydrology research group of the HYCAR Research Unit, France). More information about the efficiency of these data assimilation schemes with GR5J can be found in Piazzi et al. (accepted).

## -Functions and objects

The airGRdatassim package allows users of GR hydrological models to assimilate discharge observations with the aim of improving streamflow simulations. The package has been designed to allow the choice between two sequential ensemble-based data assimilation (DA) techniques, namely the Ensemble Kalman filter (EnKF) and the Particle filter (PF).

The functions are coded in R and both their names and arguments are consistent with the airGR package.

With the aim of providing a user-friendly package, airGRdatassim relies on two main functions:

-CreateInputsPert generates the probabilistic model inputs to perform ensemble-based DA when accounting for the uncertainty in meteorological forcings; -RunModel_DA performs streamflow ensemble simulations with the assimilation of observed discharges through the EnKF or the PF schemes.

Consistently with the airGR package, both structure and class of function arguments are specifically defined to prevent from mis-use and to ensure the flexibility of functions. Advanced users wishing to apply the package to their own models will need to comply with these imposed structures and refer to the package source codes to get all the specification requirements.

## -Models DA schemes are designed to be coupled with GR daily hydrological models, which are implemented in the airGR package. These models can be called within the airGRdatassim package using the following airGR functions (use the command '?airGR' to get the references of the GR models):

-RunModel_GR4J: four-parameter daily lumped hydrological model -RunModel_GR5J: five-parameter daily lumped hydrological model -RunModel_GR6J: six-parameter daily lumped hydrological model -RunModel_CemaNeigeGR4J: combined use of GR4J and CemaNeige -RunModel_CemaNeigeGR5J: combined use of GR5J and CemaNeige -RunModel_CemaNeigeGR6J: combined use of GR6J and CemaNeige ## -How to get started Because airGRdatassim is an airGR-based package, specific airGR functions should be jointly used to ensure the proper use of the airGRdatassim tools. Indeed, before performing the DA-based streamflow simulations, the hydrological model needs to be calibrated through the airGR calibration function. Therefore, the following steps are recommended:

1. refer to the help for Calibration_Michel in the airGR package, run the provided example and then refer to the help for CreateCalibOptions to understand how a model calibration is prepared/made; 2. refer to the help for CreateInputsPert to understand how the probabilistic model inputs are generated, if the uncertainty in meteorological forcings is taken into account; 3. refer to the help for RunModel_DA to understand how to perform the DA-based streamflow simulations; 4. refer to the help for ErrorCrit_NSE and CreateInputsCrit in the airGR package to understand how the computation of an error criterion is prepared/made.

For more information and to get started with the package, you can refer to the vignette: vignette("get_started",package = "airGRdatassim"). ## S3 method for class InputsPert plot(x, which = "all", main = NULL, ColPrecip = "royalblue", ColPotEvap = "green3", ask = prod(par("mfcol")) < length(which) && dev.interactive(), ...) (2008).

Arguments

The random perturbations are provided through a first-order autoregressive model relying on a fractional error parameter equal to 0.65 and a temporal decorrelation length of 1 day for rainfall and 2 days for potential evapotranspiration.

In order to ensure reproducible results, Seed can be set to fix the randomness in the generation of perturbations.

For further details, see the references section.

Nota: The function can be applied when using GR4J, GR5J and GR6J models (i.e. daily model time step), with or withouth the CemaNeige module.

On the graphical outputs:

- "EnKF" discharge assimilation via Ensemble Kalman filter "PF" discharge assimilation via Particle filter "none" discharge assimilation is not performed (i.e. open-loop simulation)

NbMbr

(optional) [numeric] number of ensemble members (minimum of 20 recommanded for the EnKF scheme and of 30 for the PF scheme; by default=NULL: 50 if InputsPert is not set or InputsPert$NbMbr otherwise)

StateEnKF

[character] vector of the names of state variables to be updated via EnKF:

"Prod" level of the production store [mm] "Rout" level of the routing store [mm] "UH1" unit hydrograph 1 levels [mm] (not defined for the GR5J model) "UH2" unit hydrograph 2 levels [mm]

StatePert

[character] vector of the names of state variables to be perturbed via EnKF or PF:

"Prod" level of the production store [mm] "Rout" level of the routing store [ [START_REF] Salamon | Assessing parameter, precipitation, and predictive uncertainty in a distributed hydrological model using sequential data assimilation with the particle filter[END_REF]. It is noteworthy that, when using the EnKF, the perturbation is allowed only for the state variables included in StateEnKF. If the state uncertainty is taken into account, the initial states at the prediction time step t + 1 are the perturbed analysis states resulting from the assimilation and perturbation procedures at time step t.

In order to ensure reproducible results, Seed can be set to fix the randomness in the generation of perturbations.

For further details and guidelines on the choice of the DA technique, see the references section.

Nota: The function can be applied when using GR4J, GR5J and GR6J models (i.e. daily model time step), with or withouth the CemaNeige module (see airGR package).

Value

list runModel_DA function provides a list containing the outputs organised as follows:

$DatesR

[POSIXlt] series of dates (length of IndRun)

$QsimEns

  ., Thirel, G., Perrin, C. and Delaigue, O. (accepted). Sequential data assimilation for streamflow forecasting: assessing the sensitivity to uncertainties and updated variables of a conceptual hydrological model. Water Resources Research, doi: 10.1029/2020WR028390. CreateInputsPert Generation of ensemble model inputs for data assimilation Description Function which perturbs the model inputs to generate probabilistic meteorological forcings required to perform ensemble-based data assimilation. Usage CreateInputsPert(FUN_MOD, DatesR, Precip = NULL, PotEvap = NULL, TempMean = NULL, ZInputs = NULL, HypsoData = NULL, NLayers = 5, NbMbr = 50, Seed = NULL) ## S3 method for class InputsPert x[i]

  solid line: medians of the input values -polygon: minima and maxima of the input values Value InputsPert[list] object of class InputsModel containing the ensembles of perturbed model inputs required to perform data assimilation: $DatesR [POSIXlt] vector of dates $Precip [numeric] matrix (dim(NbTime, NbMbr)) of ensemblist time series of perturbed total precipitation [mm/d] (with N $PotEvap [numeric] matrix (dim(NbTime, NbMbr)) of ensemblist time series of perturbed potential evapotranspiration [mm $NbMbr [integer] atomic vector of number of ensemble members of the InputsModel object InputsModel <-CreateInputsModel(FUN_MOD = RunModel_GR5J, DatesR = BasinObs$DatesR, Precip = BasinObs$P, PotEvap = BasinObs$E) ## run period selection IndRun <-seq(which(format(BasinObs$DatesR, format = "%Y-%m-%d")=="2006-01-01"), which(format(BasinObs$DatesR, format = "%Y-%m-%d")=="2006-01-31")) ## preparation of perturbed meteorological ensemble InputsPert <preview on a subset on one perturbed variable oldpar <-par(mfrow = c(1, 1)) plot(InputsPert[IndRun], which = "PotEvap") par(oldpar) RunModel_DA Run discharge simulations with ensemble-based data assimilation Description Function which performs discharge ensemble simulations with the assimilation of observed discharges through the Ensemble Kalman filter (EnKF) or the Particle filter (PF) schemes. More information about the efficiency of these data assimilation schemes with GR4J can be found in Piazzi et al. (accepted). method for class OutputsModelDA plot(x, Qobs = NULL, main = NULL, ColSim = "orangered", ColObs = par("fg"), ...) Arguments InputsModel [list] object of class InputsModel containing the data required to run the model (see CreateInputsModel for details) InputsPert (optional) [list] object of class InputsModel containing the ensembles of perturbed data required to evaluate the model probabilistic outputs, if the uncertainty in meteorological forcings is taken into account. The variables data proposed in InputsPert will erase the variables data given in InputsModel Qobs (optional) [numeric] time series of observed flow [mm/d] IndRun [numeric] index of period to be used for the model run [-] FUN_MOD [function] daily hydrological model functions (GR4J, GR5J or GR6J; e.g. RunModel_GR5J, RunModel_CemaNeigeGR5J) Param [numeric] vector of model parameters (number of parameters depends on the used hydrological model, from 4 parameters in GR4J up to 10 parameters in GR6J with CemaNeige) DaMethod [character] name of the data assimilation technique:

  [numeric] matrix (dim(NbTime, NbMbr)) of ensemble discharge simulations $EnsStateBkg [numeric] array (dim(NbTime, NbMbr, Nstate)) of ensemble values of background model states (before the fil $EnsStateA [numeric] array (dim(NbTime, NbMbr, Nstate)) of ensemble values of analysis model states (after the filter up $NbTime [integer] atomic vector of length of IndRun $NbMbr [integer] atomic vector of number of ensemble members $NbState [integer] atomic vector of number of of model states On the graphical outputs: -solid line: medians of the input values -polygon: minima and maxima of the input values Author(s) Gaia Piazzi, Olivier Delaigue

  Probabilistic forcing/s is/are generated by stochastically perturbing the meteorological variable/s through a multiplicative stochastic noise, according to the methodology proposed byClark et al.

	FUN_MOD	[function] hydrological model function (e.g. RunModel_GR5J, RunModel_CemaNeigeGR5J)
	DatesR	[POSIXct] vector of dates
	Precip	(optional) [numeric] time series of total precipitation to perturb [mm/d]
	PotEvap	(optional) [numeric] time series of potential evapotranspiration to perturb [mm/d]
	TempMean	(optional) [numeric] time series of mean air temperature [°C] (not perturbed),
		required to create the CemaNeige module inputs
		"all")
	main	(optional) [character] an overall title for the plot (see title)
	ColPrecip, ColPotEvap
		(optional) [character] color to be used for perturbed precipitation and perturbed
		potential evapotransipration (in any format that col2rgb accepts)
	ask	(optional) [logical] if TRUE, the user is asked before each plot, see par(ask = .)
	...	other parameters to be passed through to plotting functions
	Details
	The function generates an ensemble of precipitation or/and potential evapotranspiration time series,
	from data provided as function argument/s. The mean air temperature required to create CemaNeige
	inputs is not perturbed.

ZInputs

(optional)

[numeric]

real giving the mean elevation of the Precip and Temp series (before extrapolation) [m], possibly used to create the CemaNeige module inputs HypsoData (optional) [numeric] vector of 101 reals: min, q01 to q99 and max of catchment elevation distribution [m], if not defined a single elevation is used for Ce-maNeige NLayers (optional) [numeric] integer giving the number of elevation layers requested [-], required to create CemaNeige module inputs, default=5 NbMbr (optional) [numeric] number of ensemble members (minimum of 20 recommanded for the EnKF scheme and of 30 for the PF scheme) Seed (optional) [numeric] seed of random number generator x [InputsPert] containing the vector of dates (POSIXt) and the time series of numeric values list perturbed i [integer] of the indices to subset a time series or [character] names of the elements to extract which (optional) [character] choice of plots (e.g. "Precip" or "PotEvap", default =

  When accounting for the uncertainty in model inputs (see InputsPert), the ensemble discharge simulations are driven by perfect meteorological forecasts, which are generated according to the methodology proposed byClark et al. (2008) (see CreateInputsPert for details).It is possible to enable/disable the update of specific state variables via EnKF by defining/not defining their names in StateEnKF. Please note that StateEnKF is therefore required only when using the EnKF.Both the PF and EnKF can account for uncertainty in model states by perturbing the state variables specified in StatePert

		mm]
	"UH1	unit hydrograph 1 levels [mm] (not defined for the GR5J model)
	"UH2" unit hydrograph 2 levels [mm]
	Seed	(optional) [numeric] seed of random number generator
	x	[OutputsModelDA] containing the vector of dates (POSIXt) and the time series
		of numeric values DA model outputs
	i	[integer] of the indices to subset a time series or [character] names of the ele-
		ments to extract
	main	(optional) [character] an overall title for the plot (see title)
	ColSim, ColObs (optional) [character] color to be used for simulated flow and observed flow (in
		any format that col2rgb accepts)
	...	other parameters to be passed through to plotting functions

Details

Discharge observations are sequentially assimilated at each time step (if DaMethod != "none") via the EnKF or PF schemes. Because of the sequential approach, the analysis states resulting from the assimilation procedure at time step t are used as initial states at the following prediction time step t + 1.


Mathilde Veron

Sophie Rosset

Olivier Galibert

Guillaume Bernard

Evaluate on-the-job learning dialogue systems and a case study for natural language understanding

On-the-job learning consists in continuously learning while being used in production, in an open environment, meaning that the system has to deal on its own with situations and elements never seen before. The kind of systems that seem to be especially adapted to on-the-job learning are dialogue systems, since they can take advantage of their interactions with users to collect feedback to adapt and improve their components over time. Some dialogue systems performing on-the-job learning have been built and evaluated but no general methodology has yet been defined. Thus in this paper, we propose a first general methodology for evaluating on-the-job learning dialogue systems. We also describe a task-oriented dialogue system which improves on-the-job its natural language component through its user interactions. We finally evaluate our system with the described methodology.

Introduction

When deploying a system and making it available to real users, any developer wishes that the system would continue learning on its own when being used. Such a system would be able to continuously improve its components, adapt to unforeseen elements and situations and extend its domain without intervention from developers or domain experts. Especially, developers would avoid manual log analysis and system modifications. This capability is related to Lifelong Learning (LL) [START_REF] Thrun | Lifelong robot learning[END_REF][START_REF] Chen | Lifelong Machine Learning[END_REF] and never-ending learning [START_REF] Carlson | Toward an architecture for never-ending language learning[END_REF][START_REF] Mitchell | Never-ending learning[END_REF]. Recently [START_REF] Liu | Learning on the job: Online lifelong and continual learning[END_REF] explicitly incorporated on-the-job learning in the definition of LL. It consists in continuously learning while being used in production, in an open environment, meaning that the system has to deal on its own with situations and elements never seen before.

The kind of systems that seem to be especially adapted to on-the-job learning are dialogue systems, since they can take advantage of their interactions with users and collect feedback in order to adapt and improve their components over time. Two categories of dialogue systems can be described: task-oriented dialogue systems and conversational dialogue systems. A conversational dialogue system aims at finding the most accurate answer to a user's utterance in an open domain (chit chat). A task-oriented dialogue system aims at fulfilling user's needs relative to a specific domain, like finding information or elements given some criteria (i.e. restaurant hours, a recipe) or performing a specific task (i.e. booking a flight). Such a system is usually composed of different components which can be grouped in the following categories : the language abilities, the strategy and the knowledge. Each of these components could be improved in a lifelong learning scenario. Some dialogue systems performing on-the-job learning have been built and evaluated but no general methodology has been defined yet (see Section 2). Moreover, only few systems making their Natural Language Understanding (NLU) component learn on-the-job have been built, although this step is the first of the dialogue system and is crucial for correctly achieving the task. Thus we describe in this paper the following contributions:

34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada.

• a first attempt at a general methodology to evaluate on-the-job learning dialogue systems.

• a task-oriented dialogue system which improves on-the-job its NLU component by collecting and inferring new training examples thanks to its interactions with users.

• an evaluation framework applying the described methodology on the previous NLU component with user simulation and Knowledge Base (KB) completion simulation to demonstrate the described evaluation methodology.

Related work

Liu (2020) described the three main steps of on-the-job learning under the machine learning paradigm. These three steps can be extended to a system composed of different components (which can rely or not on machine/deep learning algorithms) and thus be defined as following: 1) detect when a new piece of knowledge can be learned, 2) retrieve and identify the new piece of knowledge and 3) adapt the component associated to the new piece of knowledge. As illustrated in this section, the type of the new piece of knowledge and the methods that can be used to perform these three steps depend on the component to improve. Mazumder et al. (2019a) focused on improving the understanding component and built an application independent natural language interface that matches user's commands to patterns (e.g. "draw a X1 circle at X2") associated to a specific action (Natural Language to Natural Language). Their approach relies basically on learning on-the-job new patterns. To detect that a new piece of knowledge can be learned, their system explicitly asks if it correctly understood the user, by only taking advantage of negative examples. The user tells explicitly what was the correct requested action, so that the system can extract a pattern from the initial user's query so that a new paraphrased command can be added to the system. The learning process is thus continuous but under the closed world assumption. To evaluate their different system's versions -including ones that did not learn on-the-job, they collected user's commands and computed the accuracy on the associated system's answers. However this does not allow the comparison over time of different system's states since the data used to make the system learn on-the-job (production) and evaluate it are the same.

A Knowledge Base (KB) can also be improved on-the-job as shown by [START_REF] Mazumder | Lifelong and interactive learning of factual knowledge in dialogues[END_REF].

The authors developed an engine to help dialogue systems improve their factual KB thanks to their interactions with users. When the user asks a question referring to unknown elements (relation or entity), the system asks the user for an example of fact containing the unknown elements. These new facts allow the system to infer other new facts which can help answer the initial question. This way of improving the KB through this inference process is close to the definition of LL. However, the authors decided to focus on the inference engine and directly worked with triple queries instead of natural language, which greatly simplify the detection step and the retrieval of new elements to learn.

To simulate the production environment, they made use of a simulated user which can ask queries with unknown relations and entities. Then they evaluated their system on the sames queries at the end of the production phase, meaning that the evaluation process is not continuous. Moreover they did not compare their system with an initial one that did not learn on-the-job.

On the other hand, [START_REF] Hancock | Learning from dialogue after deployment: Feed yourself, chatbot![END_REF] built a conversational dialogue system which collects new training examples during conversation with its users and which periodically adapts its models thanks to these new training examples (not a continuous on-the-job learning process). They first perform the detection step by inferring the user satisfaction. If the user seems satisfied, the system stores the dialogue as a new training example (imitation). Otherwise it asks the user to give a more appropriate answer and the system's answer is replaced in the dialogue by the answer provided by the user so that it can be used as a new training example. To make their system learn on-the-job, they simulated a production environment on a crowdsourcing platform and evaluated different variants of their system on a separated test dataset at the end on the learning process. Thus they did not evaluate their models in a continuous manner.

Regarding the LL evaluation methodologies, Chen and Liu (2016) described a commonly used evaluation methodology for lifelong machine learning. This methodology does not propose a continuous evaluation either and consists in a general evaluation.

Since there is no consensus about how to evaluate on-the-job learning dialogue systems, we state that a general methodology needs to be defined. Thus we describe in the next section a first attempt at such an evaluation methodology, which takes the continuous aspect into account and clarifies the difference between data used for the production and the evaluation phases. Since no work has been done on improving on-the-job the slot-filling task independently, we describe and give the evaluation results of a task-oriented dialogue system which improves on-the-job this task in the cooking domain in Section 5.

3 General methodologies to evaluate on-the-job learning dialogue systems

We consider a dialogue system that can continuously and autonomously adapt one of its components on one specific task while interacting with users in production (on-the-job learning). The task itself would stay the same over time but the scope or even the domain could evolve. We consider that the system has access to initial data, resulting in the initial state of the system along with an initial performance (e.g. trained model with the initial data). The system is then exposed to unknown situations and elements while being used in production (open environment) and should adapt to them over time, leading to different system states.

The main questions considering such a system are: (1) Does the system forget what it initially learned/knew while being used and while adapting itself -called catastrophic learning in continual learning [START_REF] Parisi | Continual lifelong learning with neural networks: A review[END_REF]? (2) Does the system actually learn/accumulate the new elements that appear while being used, does it improve when adapting itself? (3) Does the system manage to infer additional knowledge from what it learned/accumulated while being used, in order to generalize? In order to be able to truly compare all different system states with each other, the performance of each system state on the specified task has to be evaluated by always using the same test dataset. This dataset is actually composed of three different datasets: test IN IT IAL : data that look like the initial data, in order to evaluate the feature (1); test LEARN : data that contain elements that the system is supposed to learn/accumulate when being used, in order to evaluate the feature (2); test U N KN OW N : data that contain elements that are not in the initial data and that are not going to appear when the system will be used, in order to evaluate the feature (3).

Additionally, the system adaptation has to be autonomous, that means that the system has to adapt without the help of domain experts or developers. Knowing that, the only inputs accepted during the evaluation are interactions with end users, in natural language. Moreover, in a real scenario it is not possible to define beforehand test LEARN and test U N KN OW N . That is why the evaluation is based on user simulation -a common method to evaluate dialogue systems [START_REF] Deriu | Survey on evaluation methods for dialogue systems[END_REF]) -in order to simulate the production phase. Thus, the data used for the simulation have to be similar to test IN IT IAL and test LEARN but be separate from test U N KN OW N .

The system's continuous learning also has to be evaluated. In fact it could be problematic if the user had to correct the system for the same error again and again, waiting for the system to adapt. Moreover, interactions with users and feedback collection should seem as natural as possible for the user. It is necessary if we want the adaptation process to be successful and keep the user willing to help the system to improve. As a consequence, the system performance has to be evaluated after each interaction with a user.

To summarize, the evaluation methodology consists in simulating the user interactions and evaluating after each interaction the current system's performance on test IN IT IAL , test LEARN and test U N KN OW N . We believe this methodology can be adapted to any on-the-job learning dialogue system in any domain.

4 A task-oriented dialogue system improving on-the-job its understanding component

The system that we built is a task-oriented dialogue system in the cooking domain which can continuously and autonomously improve when interacting with users (on-the-job learning). The operation of the system is described in Figure 1.

Description of the task and the model

In this paper, we focus on the improvement of the Natural Language Understanding (NLU) component. NLU is usually composed of two steps: slot-filling and intent detection, but for first experimentation we decided to consider only slot-filling. Slot-filling can be considered as a sequence labelling task which aims at retrieving from a user's utterance a set of concepts. A concept correspond to a tuple (type of concept, mention). For the first experimentation we decided to work on the recipe domain and defined the following 8 types of concepts to detect: recipe type, ingredient, preparation technique, origin, origin adjective, meal, event, other category. For instance, given the following user query "I'd like to prepare chocolate cake for my son's birthday", the concepts to detect are (ingredient, 'chocolate'), (recipe type, 'cake') and (event, 'birthday'). To take negative contexts into account we added the negative variant of each concept (e.g. for "I don't like bananas" the concept to detect is (negative ingredient, 'bananas')).

Slot-filling is often performed using deep neural networks. We decided to use a BiLSTM architecture which is known to achieve good performances on classical slot-filling tasks [START_REF] Béchet | Is ATIS too shallow to go deeper for benchmarking spoken language understanding models?[END_REF]. Before the simulated production phase, we trained a double layered BiLSTM1 with layers of size 128 on train IN IT IAL,T RN (see Section 5.2). The embedding layer is initialized with Word2Vec CBOW [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] vectors pre-trained on Wikipedia [START_REF] Ghannay | What is best for spoken language understanding: small but task-dependant embeddings or huge but out-of-domain embeddings?[END_REF]. Model selection was done by taking the highest F1-score on the development set (train IN IT IAL,DEV , see Section 5.2).

Collecting new pieces of knowledge

To build this on-the-job learning dialogue system, the main idea is to take advantage of interactions with users to collect new training examples to adapt the NLU model. The first step for collecting new pieces of knowledge is to detect that one can be learned. This step is performed by detecting a posteriori that an NLU error occurred [START_REF] Hancock | Learning from dialogue after deployment: Feed yourself, chatbot![END_REF]: first the user addresses the system which answers him/her and tells him/her the concepts it detected (NLU output); if the user continues the conversation, the NLU output is supposed to be correct -principle of progessivity (Albert and Ruiter, 2018); if the system detects that the user is notifying that it misunderstood him/her, the NLU output is considered as incorrect2 .

The next step is the extraction and the identification of the new piece of knowledge, consisting here in labelling the initial user query. If the NLU output is correct, the initial user query is labelled with the concepts detected and stored. If the NLU output is incorrect, the system tries to correct itself. This correction is then proposed to the user: as previously, if the user keeps going, he/she implicitly confirms the correction, if not it means that the new piece of knowledge cannot be extracted. To correct the initial NLU output we make the assumption that the user will rephrase his/her initial query after notifying the system that it misunderstood him/her (e.g. "You misunderstood me, I asked for a cake recipe without eggs"). Note that in practice it is not always true (Hough, 2014, p.29). Under this assumption, the system compares the initial user query and the paraphrase by removing stop words and keeping only the common chunks (e.g. for the initial query "Do you have cake recipes for people allergic to eggs?" and the previous paraphrase, the common chunks are cake and eggs). Then it checks in the KB associated to the dialogue system if an element associated to the chunks exists and tries to extract the type of concept associated. If it manages to extract the type, the initial user's query can be labelled. This step is crucial but depends highly on the quality of the KB and on its capability to be adapted to the concepts to detect.

Since learning on-the-job involves an open world environment, new elements that are not originally in the KB may appear over time. This can impact the performance of the NLU model since it will have to detect new mentions it never saw during the initial training. The KB should thus evolve over time to mitigate this issue. We decided to incorporate this phenomenon in our experiments.

To do it we intentionally removed some elements from the KB and simulate the completion of the KB when the system detects that the user refers to an element missing in the KB. In a real case, we can consider that this completion would be performed by extracting information about the missing element from the web for instance. Moreover, since the labelled training examples collected during interactions with users would be too few to adapt the model, we decided to make the system inferring additional knowledge to augment the number of labelled training examples.

To do it the system uses a basic method based on patterns. For instance, for the user's query "Tonight I have a barbecue, can you suggest me something to prepare?" with the concept (event, 'barbecue'), the pattern "Tonight I have a $event, can you suggest me something to prepare?" can be extracted.

Adapting natural language understanding

Since the continuous learning aspect is important and since a neural network needs time to be adapted, a short term memory which can directly use the knowledge learned from past dialogues has been added to the NLU component. To the best of our knowledge, it is the first time a short term and a long term memory (in our case the adapted model) have been jointly used for learning on-the-job.

The NLU component can be then adapted in two ways: a continuous adaptation with the Short Term Memory (STM) or an incremental adaptation with the adaptation of the NLU model. In this experiment, only new mentions and their associated concept type are considered in the STM. In fact, we consider that a user will expect the system to directly remember new mentions introduced in past dialogues and that he/she will consider that the dialogue system is not learning on-the-job if it is not able to do it. This is less true for patterns. Note that when the model is adapted, the STM is cleared.

After detecting the concepts in the user query thanks to the NLU model, the output is updated through the STM as described in Algorithm 1. The strategy followed has been defined empirically through preliminary experiments. We observed in particular that taking into account the negative context really impacts the NLU update.

To adapt the NLU model, the current model is fine-tuned on the new labelled training examples extracted during interactions with users (train LEARN,n on Figure 1). The adaptation process is triggered if one of the conditions about the number of new mentions, new patterns or new examples is validated. We decided to use replay to prevent catastrophic forgetting [START_REF] Parisi | Continual lifelong learning with neural networks: A review[END_REF] 5 Evaluation framework and results

User simulation

A stated in Section 3, a user simulation is used to simulate the production phase in order to evaluate the system. The simulated user consists here in a ruled-based program. Only the first dialogue utterance is generated (the set of first utterances corresponds to the dataset simulation). The simulated user follows the following scenario: 1) the user asks for a recipe by providing some criteria (i.e. the type of recipe, the ingredient, etc) 2) the system provides an answer and gives the criteria it detected (the concepts) 3) if the criteria detected by the system don't match the ones of the user, he/she tells the system "You misunderstood me. I want a recipe with $concepts." with the concepts in natural language in place of $concepts, otherwise he/she behaves normally 4) if it was wrong the system provides a corrected answer and gives the criteria it corrected 5) if the corrected concepts match the user's criteria he/she behaves normally, otherwise he/she tells goodbye4 . Note here that with this simple user simulation, the first step of on-the-job learning -which consists here in detecting that the system misunderstood the user -becomes trivial.

Data preparation and evaluation

To train, simulate and evaluate our on-the-job learning dialogue system the following datasets are needed: train To evaluate the performance of the NLU component, the F1-score is computed with the conlleval.pl script [START_REF] Kim | Introduction to the CoNLL-2000 shared task chunking[END_REF] at the end of each simulated dialogue after the component's adaptation. To assess if the initial random splits for the data preparation impact the results, we prepared 4 different set of datasets with different random seed values. For each set of datasets an initial NLU model is trained and 2 simulations with the two kinds of replay are conducted.

Results and discussion

Figure 2 shows the evolution of the F1-score during simulation7 . We did not observe significant performance differences when the system adapts its component using Replay on Patterns and Mentions (RPM) or using Replay with Mentions only (RM). On test LEARN both F1-scores increase significantly at the beginning of the simulation suggesting that learning new mentions is in our case more profitable for the system when adapting its component (since simulation is generated randomly there are more new mentions to learn at the beginning of the simulation). However we can see that RM is performing better than RPM (about 2 more points) and that RPM increases progressively until achieving similar F1-score than RM at the end of the simulation. This again suggests that learning new mentions is more profitable since the new mentions will be two times less represented in train LEARN,n for RPM compared to RM. Moreover, the fact that they both seem to reach a plateau shows the limitations of an experimental protocol relying on generated data.

Since the count of fine tunings during the simulation is high (322/327 for RPM/RM), the continuous evolution is not visible on this figure. We computed the F1-score difference between each model fine-tuning to better analyse the impact of the STM. We observed that during the simulation the STM improves the F1-score of maximum 0.43 points (between simulated dialogues 318 and 390 with 16 new mentions) and reduces the F1-score of a maximum of 0.04 points on test LEARN with RPM (mean: 0.01, median: 0).

The We suppose that it comes from the fact that all the new pieces of knowledge available in the simulated dialogues could not be extracted by the system. From the system's log we observe in particular that only 0.14 percent of the user's initial queries could be correctly annotated after the user rephrases his/her query because the system misunderstood him/her. This shows that the KB was not completely adapted for the concepts that have to be detected.

Considering the fact that simple methods have been implemented to perform the three steps of on-the-job learning -as described in Section 4 -we obtain promising results suggesting that using more sophisticated methods could lead to generalisation capabilities demonstrating the possibility of a truly LL task-oriented dialogue system.

Conclusion and future work

In this paper we described a first attempt at a general methodology to evaluate in a continuous manner the capability of a dialogue system to learn on-the-job. To the best of our knowledge it is the first general methodology defined of this kind. This methodology consists basically in simulating the user interactions and evaluating after each interaction the current system's performance on the following test datasets: test IN IT IAL (data similar to initial ones), test LEARN (data that contain elements that the system is supposed to learn during simulation) and test U N KN OW N (data that contain elements that are not in the initial data and that are not going to appear during simulation). We also built a task-oriented dialogue system which can continuously and autonomously improve its understanding component thanks to its interactions with users and evaluated it thanks to user simulation according to the described evaluation methodology. Our methodology enabled us to compare over time the different adaptation methods and identify how they were different, whereas the evaluation protocols used in other works could not have allowed this analysis.

For future work we plan to adapt this experiment to other domains and to add to the evaluation methodology a way to evaluate the system's robustness to noise (e.g. user providing a wrong correction).

Figure 1 :

 1 Figure 1: System operation and evaluation during production (simulated). 1) Training of the initial model on train IN IT IAL , 2) Initialization of the NLU model, 3) Interactions with users, 4) Continuous extraction of new pieces of knowledge (mentions, patterns, training examples) and update of the STM, 5) evaluation on test IN IT IAL , test LEARN , test U N KN OW N and test REAL at the end of each dialogue and 6) If one of the conditions for model adaptation is validated, generation of train LEARN,n from past mentions/patterns and from new mentions/patterns and model fine-tuning.

Figure 2 :

 2 Figure 2: F1-score evolution during simulation on test IN IT IAL , test LEARN , test U N KN OW N and test W EIGHT ED with data prepared with random seed 1 (F 1 W EIGHT ED = F 1 IN IT IAL * 0.2 + F 1 LEARN * 0.4 + F 1 U N KN OW N * 0.4).

  so that train LEARN,n is generated from both patterns and mentions from initial training and past adaptations and from the current new mentions and patterns. For each adaptation the system always generates 1000 training examples and adds the new examples collected during the interactions. For the fine-tuning, a new model model n is initialized with the weight of the previous model model n-1 and trained on train LEARN,n 3 . Two kinds of replay are tested: 1) Replay on Patterns and Mentions (RPM) and 2) Replay on Mentions only (RM). model .mention == cstm.mention and c model .type == cstm.type then 9: add c model to concepts 10: else if c model .mention == cstm.mention and is_negative(c model .type) then

	Algorithm 1 Natural Language Understanding update strategy through Short Term Memory
		Input: user's utterance u
		Output: concepts detected in u
	1: procedure GET_CONCEPTS(u)
	2:	concepts model = get_concepts_f rom_model(u)
	3:	conceptsstm = get_concepts_f rom_stm(u)
	4:	initialize concepts as an empty list of concepts
	5:	if conceptstm then
	6:	for c model in concepts model do
	7:	for cstm in conceptsstm do
	8: if c 11:	add (negative(cstm.type), cstm.mention) to concepts
	12:	else if c model .mention in cstm.mention then
	13:		add cstm to concepts
	14:	else if cstm.mention in c model .mention then
	15:		if c model .mention in KB then
	16:		add c model to concepts
	17:		else
	18:		add cstm to concepts
	19:	if nothing added to concepts then
	20:	add c model to concepts
	21:	else
	22:	concepts = concepts model
	23:	return concepts

  IN IT IAL,T RN (20k), train IN IT IAL,DEV (4k), simulation (20k), test IN IT IAL (1k), test LEARN (1k) and test U N KN OW N (1k). Because we initially had no data for the concept detection task, we decided to generate the data using patterns and mentions. The patterns were

Table 1 :

 1 F1-scores on the test datasets with data prepared with random seed 1. N is the index of the last fine-tuning at the end of the simulation (here N=322/327 for RPM/RM). model LEARN,ST M corresponds to model IN IT IAL adapted only with the STM (no fine-tuning during simulation). model SIM U corresponds to model IN IT IAL fine-tuned on the simulation dataset.

		test IN IT IAL	test LEARN	test U N KN OW N	test W EIGHT ED	test REAL
	model IN IT IAL	98.99	89.33	68.26	82.83	36.18
	model LEARN,ST M	98.50 (-0.49)	91.61 (+2.28)	67.52 (-0.74)	83.36 (+0.53)	-
	model LEARN,RP M,n=N	98.56 (-0.43)	97.48 (+8.15)	71.11 (+2.85)	87.15 (+4.32)	37.16 (+0.98)
	model LEARN,RM,n=N	99.14 (-0.15)	97.63 (+8.30)	70.79 (+2.53)	87.20 (+4.37)	37.70 (+1.52)
	model SIM U	99.94 (+0.95)	99.60 (+10.27)	71.49 (+3.23)	88.42 (+5.59)	41.24 (+5.06)

  Table 1 shows the F1-scores of different models on the test datasets. The results for model LEARN,RP M,n=N and model LEARN,RM,n=N compared to model IN IT IAL show that the NLU component is improving with the simulated interactions thanks to the collection and adaptation methods described in Section 4. Comparison with model LEARN,ST M shows that the STM alone is not sufficient. When comparing with model SIM U we observe similar scores except on test LEARN and test REAL .

Model hyperparameters: Adam optimizer with initial learning rate of 0.001, Word2Vec size 300, CrossEntropy loss.

To perform this first step we used basic regular expressions (e.g. r".*wrong.*", r".*not what I.*").

The parameters are the same than for the initial training except for the epoch size which is set to the length of trainLEARN,n. The same development dataset is also used.

The decision diagram describing the user's and the system's behaviour is is part of supplementary materials.

For the first experiment, we removed from the KB around 40% of the less frequent ingredients so that the unknown mentions in LEARN only consists in unknown ingredient mentions.

collected as part of the ERA-Net CHIST-ERA LIHLITH project.

We do not show the F1-score evolution for the experiments with data prepared with other random seed values since the behaviour is the same.

Acknowledgments

This work has been supported by a CIFRE convention, funded by ANRT (France, convention 2019/0628), and by ERA-Net CHIST-ERA LIHLITH Project, funded by ANR (France, project ANR-17-CHR2-0001-03).