Ian Tancrez

Vickridge

B Xia

J J Ganem

S Steydli

H Tancrez

I Vickridge

RBS and NRA analysis for films with high growth rate prepared by atomic layer deposition

Keywords: Atomic Layer Deposition (ALD), Thin films, multilayer, RBS 1.Introduction

RBS and NRA analysis for films with high growth rate

prepared by atomic layer deposition

We chose first to grow ZnO and TiO2 films because growth of these is well established, with numerous applications, and in addition, the high mass of the cation (compared to the very widely grown Al2O3) substantially simplifies RBS measurements. Films were grown from water and DEZ (ZnO films) or TDMAT (TiO2 films) precursors. Absolute film elemental compositions were determined by RBS, and NRA, and film physical thickness was determined by ellipsometry. The effects of the number of ALD cycles, precursor pulse duration and substrate temperature were investigated. We also grew some films using water highly enriched in 18 O.

Experiment

System description

The home-made ALD system consists of three parts: precursor, oxidant and carrier gas handling system, cross-flow design hot substrate/cold reaction chamber and a pumping system controlled by an in-house Labview control system (as shown in Fig. 1a). The DEZ and TDMAT precursor gases and the water vapour are controlled by fast valves. The flux of N2 carrier gas is maintained by mass flow controller (MFC). A compact manifold system connects several precursor gas bottles to the single cross-flow gas inlet to the chamber. In our cross-flow design, gas enters the chamber through a nozzle nearly parallel to the sample surface, and is evacuated through the opposing face of the chamber by a dry pumping system. Compared with the traditional cross flow gas entrance our system has a small angle with sample surface. Such cross-flow design leads to more homogeneous film growth. The inlet nozzle has a wedge design, which has been shown to improve stability and homogeneity, and to increase deposition efficiency [START_REF] Ritala | Atomic layer deposition, Handbook of Thin Films[END_REF]. The sample holder is insulated from the chamber by a ceramic surround, and heated from the bottom by a resistive element. The sample surface temperature is inferred from the furnace thermocouple, via a careful series of calibration measurements. The passage of carrier gas and precursors in the chamber influences the target and chamber temperatures through convection and conduction. In order to limit transient temperature variations during the ALD sequence, we have developed a temperature control system based on a predictive temperature model. A typical series of precursor pressure pulses vs time includes water vapour pulse and purge time, and precursor pulse and purge time as shown on Fig. 1b.

Experiment description

Si (100) wafers cleaned with acetone followed by absolute ethanol were used as the substrate. Diethlyzinc (DEZ), Tetrakis (dimethylamino)titanium (TDMAT) and water were used as precursors. The pulse time of DEZ and TDMAT was fixed at 15 ms and 200 ms respectively, and the injection time of water was varied from 15ms to 100ms. A long N2 purge time (30s) was applied in order to prevent precursors mixing during deposition. The sample surface temperature was varied from 100℃ to 300℃ and the number of ALD cycles was varied from 5 to 40 for growth of ZnO films and from 5 to150 for growth of TiO2 films. The detailed parameters for deposition ZnO and TiO2 films are summarized in table 1. In this paper, we will present films grown using water of natural isotopic composition, and water highly enriched in deuterium and in 18 scattering angle. The detector solid angle was calibrated with 3% uncertainty using a Bi-implanted reference sample [START_REF] Davies | Calibration of the Harwell series II Bi-implanted RBS standards[END_REF][START_REF] Lemelle | Study of the (010) olivine surface by Rutherford backscattering spectrometry in channeling geometry[END_REF]. Nuclear Reaction Analysis (NRA) was used to measure the 16 O areal densities via the 16 O(d,p1) 17 O nuclear reactions, with respect to a thermal SiO2 film, with 15 atoms/cm 2 as deduced from the film thickness and density determined by ellipsometry [START_REF] Volintiru | In situ spectroscopic ellipsometry growth studies on the Al-doped ZnO films deposited by remote plasma-enhanced metalorganic chemical vapor deposition[END_REF]. An 860 keV deuteron beam of 85 nA in a 2mm beamspot was used, and the protons produced by the nuclear reactions were detected at 150° in a 300mm 2 detector behind a 13 mylar foil which stopped the high flux of elastically scattered deuterons from saturating the detector. 18 O was determined by the 18 O(p,) 15 N nuclear reaction with 85nA of 750 keV protons in a 2mm beamspot, and the alpha particles detected at 150° in a 300 mm 2 detector behind a 13 m mylar film.

[ 16 O]=624.8×10
The physical film thickness obtained from spectroscopic ellipsometry (SE) with a FS-1 Multi-Wavelength Ellipsometer system in the wavelength range of 465-635 nm at an angle of incidence of 65°.

3.Results and discussion

3.1 The growth of ZnO Fig. 2 shows the growth per ALD cycle measured by ellipsometry as a function of the substrate surface temperature with a DEZ precursor pulse time of 15ms and H2O pulse time of 100ms for each cycle. The resulting film thickness ranged from 16 to 60nm. The highest growth rate, of about 14 Å/cycle, occurs in a temperature window between 140 °C and 175°C. At substrate temperatures outside this range the growth rate falls. At lower substrate surface temperatures the DEZ precursors might not have sufficient energy to react with the -OH groups on the surface [START_REF] Kim | Relation between enhancement in growth and thickness-dependent crystallization in ALD TiO2 thin films[END_REF]. At temperatures above 175°C M. Rueter [START_REF] Rueter | The surface reactions of ethyl groups on Si (100) formed via dissociation of adsorbed diethylzinc[END_REF] has proposed that the lowered growth rate is due to surface thermal decomposition of the DEZ. In our system, although the chamber is not deliberately heated thermal radiation from the substrate still increases the temperature of the elements surrounding the substrate plate, including the chamber walls, where the DEZ may be decomposed into Zn and ethyl fragments, inhibiting growth on the substrate. The stable growth rate in the temperature window between 140 and 175℃ suggests that self-limiting ALD growth of ZnO films has been obtained, as has been observed for very similar ALD windows in previous works [START_REF] Guziewicz | ALD grown zinc oxide with controllable electrical properties[END_REF][START_REF] Kim | The properties of plasma-enhanced atomic layer deposition (ALD) ZnO thin films and comparison with thermal ALD[END_REF][START_REF] Janocha | ALD of ZnO using diethylzinc as metal-precursor and oxygen as oxidizing agent[END_REF].

The number of water molecules adsorbed on the surface is related to the pulse time, whilst substrate temperature has a strong influence on the formation of -OH groups formed from the adsorbed water molecules. We therefore investigated the influence of water pulse time on ALD growth within the ALD window. In Fig 3a, we show the number of Zn atoms (determined by RBS) as a function of number of cycles for water pulse times of 15, 50 and 100ms. In all cases film growth is achieved, and after about 10 cycles the growth rate increases as the water pulse time is increased. Clearly, for the shorter water pulse times full coverage of the surface by -OH groups is not achieved. During the first 10 or so cycles, the growth rate is limited by the surface group preparation, as is commonly observed [START_REF] Puurunen | Island growth as a growth mode in atomic layer deposition: A phenomenological model[END_REF][START_REF] Lee | Two-step growth of ZnO films on silicon by atomic layer deposition[END_REF]. Here, the substrate surface is still very smooth as measured by AFM with value about Rq=0.38nm, whereas after the initial cycles the substrate surface can become rougher and facilitate adsorption of water molecules. In Fig 3b, the ratio of Zn to O atoms is plotted as a function of the number of ALD cycles for the 3 water pulse times. The films are all initially deficient in Zn for the first growth cycles, however the ratio of Zn to oxygen increases as water pulse time increases.

This indicates that when the surface is inadequately covered with -OH groups, the reaction of DEZ with the -OH groups is hindered, and it is possible that some of DEZ desorbs from the surface in regions where the -OH coverage is low. Previous study also illustrated that substrate-inhibited growth is caused by a lower number of reactive sites on the substrate than on the ALD-grown material [START_REF] Green | Nucleation and growth of atomic layer deposited HfO 2 gate dielectric layers on chemical oxide (Si-O-H) and thermal oxide (SiO 2 or Si-O-N) underlayers[END_REF][START_REF] Hoyas | Growth and characterization of atomic layer deposited WC 0.7 N 0.3 on polymer films[END_REF]. Since the depth resolution of NRA is unable to distinguish between 16 O in the native oxide of the silicon substrate and that in the overlying ZnO films, we have systematically subtracted the amount of oxygen in the substrate, as measured on an untreated virgin substrate from the same silicon wafer, from the oxygen contents measured for the "substrate plus film" samples.

There remains, however, some uncertainty about the actual number of oxygen atoms in the substrate oxide layer after the first ALD cycles and it is not yet clear to what extent possible underestimation of the substrate oxide film thickness (after the initial ALD cycles) could be responsible for the observed low ZnO ratios for very thin ZnO layers obtained after the first 5 to 15 cycles. Fig. 4 shows the growth rate of TiO2 as a function of the substrate surface temperature with a given TDMAT precursor time of 200 ms and H2O precursor time of 100 ms. At low temperature, the film growth rate is much higher than at high temperature. It has been shown [START_REF] Hudson | <i>Surface Science: An Introduction<[END_REF][START_REF] Norton | Kinetic and mechanistic studies of the thermal decomposition of Ti (N (CH3) 2) 4 during chemical vapor deposition by in situ molecular beam mass spectrometry[END_REF] that TDMAT has an activation energy of just 16 kJ/mol (0.17eV) for adsorption in the low-temperature regime, indicating a thermally activated adsorption surface process. In other words, numerous TDMAT molecules have an energy sufficient for adsorption to the low temperature surface which causes a high growth rate at the low temperature. The growth rate will decrease as the sample surface temperature is increased from 100℃ to 250℃. This behavior is different from many models of ALD processes in the low temperature region. For this case, temperature programmed desorption (TPD) showed the most of the TDMAT desorbed from the surface as the temperature was increased [START_REF] Corneille | Surface spectroscopic studies of the deposition of TiN thin films from tetrakis-(dimethylamido)-titanium and ammonia[END_REF]. Hence, the growth rate of TiO2 films will initially reduce with increasing surface temperature. We note that as the temperature is further increased, the growth rate of TiO2 films increases. The increase in growth rate above 250°C is probably due to chemical vapor deposition (CVD) attributed to the gas phase decomposition of TDMAT. The team of Elam et al. also reported that the decomposition of TDMAT become more important for ALD deposition of TiO2 at high temperature [START_REF] Elam | Surface chemistry and film growth during TiN atomic layer deposition using TDMAT and NH3[END_REF].

Based on the study of influence of substrate temperature on deposition, it was chosen to study the process at low temperature because the growth rate is very high -a clear advantage for commercial application. Hence, the temperature of 100℃ was selected to study the effect of the water pulse duration, which was varied from 15ms to 100ms. In Fig. 5a we observe that for 15ms pulse time, the growth rate of TiO2 films is clearly lower than that for 50ms and 100ms water pulses. The increase of film growth rate with water pulse duration strongly demonstrates that an adequate amount of water needs to be supplied during the ALD process. For the explanation of the increase of growth rate, although deposition of the water and -OH on rutile TiO2 surface is temperatureindependent [START_REF] Abendroth | Atomic layer deposition of TiO2 from tetrakis(dimethylamino)titanium and H2O[END_REF], higher water exposure will likely lead to higher water and -OH coverage, providing more active sites for TDMAT reaction. We also observe, in Fig. 5b, that the Ti/O atomic ratio is highest for the samples grown with 100ms water pulses, further supporting that view that the shorter water pulse times didn't provide enough active sites for TDMAT reaction. We may note that the film thicknesses are not the same under the two different growth orders: the growth of ZnO on TiO2 is accelerated compared with that of ZnO on the Si substrate, whilst the growth of TiO2 on ZnO is retarded with respect to that of TiO2 on the silicon substrate. To our knowledge this has not been reported previously, and we do not have an explanation for this observation, however further work studying the effect of the thickness of the TiO2 layer on the acceleration of the ZnO growth, and the detailed growth kinetics (does the growth rate enhancement reduces as the ZnO film thickness increases?) should throw significant light on the underlying processes. ZnO/TiO2 multilayer films were deposited onto Si (100) substrates by repeating 80 TiO2 ALD cycles and 30 ZnO ALD cycles at 225℃ with the same precursor, water and purge times as for the bilayers described above. The RBS spectrum measured with 2.05 MeV 4 He incident ions is shown in Fig. 7a. The Ti and Zn peaks from the 6 layers are well separated, indicating well-defined TiO2 and ZnO layers, and the spectrum can be fitted well by SimNRA. The layer thicknesses are given in table 2 from top to bottom. The TiO2 film thicknesses are quite uniform and in this case we do not observe the retarding of the TiO2 growth rate on ZnO compared to that of TiO2 on the substrate. We do note, though, that the growth rate of layer 5 (the first ZnO layer, on TiO2) has been accelerated as was observed in the bilayer system. These observations indicate that there may remain some uncontrolled parameters, such as thermal inertia in the substrate holder or surrounding elements, or chemical memory effects in our growth system. Fig. 7b shows the RBS spectrum of a ZnO film grown using water highly enriched in D 218 O. We note here that the actual amount of water consumed for the growth of the film was too small to be observed, and we estimate that a few cm 3 of enriched water will be sufficient for many hundreds of hours of operation.

The presence of both isotopes -18 O and 16 O -is clear from the RBS spectrum but due to high background from the Si substrate and the resulting high uncertainties in estimating the areas under the oxygen signals, areal densities of 16 O and 18 O were measured precisely by NRA under conditions previously described [START_REF] Amsel | Microanalysis by the direct observation of nuclear reactions using a 2 MeV Van de Graaff[END_REF]. Nevertheless, the RBS spectra do show that the two isotopes are present throughout the thickness of the films, and the simulated RBS spectrum using the quantities of 18 O and 16 O derived from NRA (given in the inset) are fully compatible with the measured RBS spectrum.

Although the water used was almost fully enriched in 18 O, the film contains about 30% 16 O. The total amount of 16 O, of 100 × 10 15 atoms/cm 2 , is far greater than that of the substrate native oxide and so must come almost entirely from the growth process, and probably from water adsorbed on the manifold, piping and chamber walls. This indicates that for future isotopic tracing studies of growth mechanisms, special attention will need to be paid to outgassing the entire ALD system between growth sequences using different isotopically distinct water sources. We do note, that ALD films enriched even only to 70% in 18 O will be very useful for the study of annealing processes under standard conditions such as flowing air.

Conclusions

We have successfully developed a home-made ALD system with high growth rates compared with established commercial systems. The system has been successfully demonstrated to grow stoichiometric ZnO and TiO2 films, bilayers and multilayers. The small volume has enabled production of films highly enriched in 18 O, using only very small quantities of the expensive isotopically enriched water source.

Fig. 1 .

 1 Fig. 1. a: Home-made ALD system included deposition chamber; b: the signal of pulse time of water and metal precursors

Fig. 2 .

 2 Fig.2. The variation in growth rate of ZnO films as a function of the substrate surface temperature.The number of ALD cycle was kept at 40 cycles.

Fig. 3 .

 3 Fig.3. a: The variation in growth rate of ZnO films as a function of the ALD cycles at 150℃; b the variation in atomic ratio of Zn/O as a function of the ALD cycles at 150℃

Fig. 4 .

 4 Fig.4. The variation in growth rate of TiO2 films as a function of the substrate surface temperature.The number of ALD cycle was kept at 100 cycles.

Fig. 5 .

 5 Fig.5. a: The variation in growth rate of TiO2 films as a function of the ALD cycles at 100℃; b the variation in atomic ratio of Ti/O as a function of the ALD cycles at 100℃

Fig. 6 .

 6 Fig.6. RBS spectrum and simulation by SimNRA for ZnO-TiO2 double layer films. a: TiO2-first; b: ZnO-first.

Fig. 7 .

 7 Fig.7. a: The RBS spectrum at 2.05MeV simulated by SimNRA. b: RBS spectrum and SimNRA simulation for 18 O enriched ZnO films.

Table 2 .

 2 Table 2 also presents the density of each layer, for the ZnO layer, the density is little lower than the ZnO bulk density with value 5.61g/cm 3 , the density of TiO2 is closed to the rutile density of TiO2 (4.23 g/cm3 ). The thickness of each layer for multilayer film

	Layer	10 15 Atoms/cm 2 Thickness(nm) Density (g/cm 3 )
	1 ZnO	252.0	30.7	5.5
	2 TiO2	150.0	15.6	4.1
	3 ZnO	245.0	30.4	5.4
	4 TiO2	150.0	15.6	4.1
	5 ZnO	312.0	37.7	5.5
	6 TiO2	145.0	15.1	4.0
			Si (100)

Acknowledgement

The ALD system is funded in part by the CEMIP (Centre de Microé lectronique de Paris Ile-de-France). The work has also benefitted from the support of the SAFIR platform of Sorbonne Université , and dedicated internal funding from the Institut des NanoSciences de Paris. We also acknowledge useful discussions with members of the French ALD network RAFALD. B. Xia is funded by the China Scholarship Council for his PhD studies.


Raphaël Duroselle

Md Sahidullah

Denis Jouvet

Irina Illina

Language recognition on unknown conditions: the LORIA-Inria-MULTISPEECH system for AP20-OLR Challenge

Keywords: language recognition, channel mismatch, domain generalization

We describe the LORIA-Inria-MULTISPEECH system submitted to the Oriental Language Recognition AP20-OLR Challenge. This system has been specifically designed to be robust to unknown conditions: channel mismatch (task 1) and noisy conditions (task 3). Three sets of studies have been carried out for elaborating the system: design of multilingual bottleneck features, selection of robust features by evaluating language recognition performance on an unobserved channel, and design of the final models with different loss functions which exploit channel diversity within the training set. Key factors for robustness to unknown conditions are data augmentation techniques, stochastic weight averaging, and regularization of TDNNs with domain robustness loss functions. The final system is the combination of four TDNNs using bottleneck features and one GMM using SDC-MFCC features. Within the AP20-OLR Challenge, it achieves the top performance for tasks 1 and 3 with a Cavg of respectively 0.0239 and 0.0374. This validates the approach for generalization to unknown conditions.

Introduction

Language recognition is the task of predicting the language used in a test speech utterance [START_REF] Ambikairajah | Language identification: a tutorial[END_REF]. State-of-the-art language recognition systems are based on the discriminative training of a deep neural network [START_REF] Snyder | Spoken language recognition using x-vectors[END_REF] with multilingual bottleneck features [START_REF] Fér | Multilingual bottleneck features for language recognition[END_REF]. They rely on automatic training and consequently highly depend on the training dataset. A drop in performance is expected from a domain shift of the test data, such as noise or channel mismatch [START_REF] Mclaren | Analyzing the effect of channel mismatch on the SRI language recognition evaluation 2015 system[END_REF]. Various approaches have been introduced to reduce the impact of domain shift over language recognition systems: design of robust features [START_REF] Plchot | Analysis of BUT-PT submission for NIST LRE 2017[END_REF][START_REF] Abdullah | Cross-domain adaptation of spoken language identification for related languages: the curious case of slavic languages[END_REF], data augmentation of the training set [START_REF] Plchot | Analysis of BUT-PT submission for NIST LRE 2017[END_REF][START_REF] Villalba | End-to-end versus embedding neural networks for language recognition in mismatched conditions[END_REF], feature-based adaptation [START_REF] Castaldo | Compensation of nuisance factors for speaker and language recognition[END_REF] and model-based adaptation of the classifier [START_REF] Abdullah | Cross-domain adaptation of spoken language identification for related languages: the curious case of slavic languages[END_REF][START_REF] Villalba | End-to-end versus embedding neural networks for language recognition in mismatched conditions[END_REF][START_REF] Peng | Adversarial multi-task deep features and unsupervised back-end adaptation for language recognition[END_REF][START_REF] Duroselle | Unsupervised regularization of the embedding extractor for robust language identification[END_REF].

The Oriental Language Recognition (OLR) Challenge has been organized for the last five years with the goal of boosting research on language recognition technology [START_REF] Wang | AP16-OL7: A multilingual database for oriental languages and a language recognition baseline[END_REF][START_REF] Tang | AP17-OLR challenge: Data, plan, and baseline[END_REF][START_REF] Tang | AP18-OLR challenge: Three tasks and their baselines[END_REF][START_REF] Tang | AP19-OLR challenge: Three tasks and their baselines[END_REF]. The AP20-OLR edition [START_REF] Li | AP20-OLR challenge: Three tasks and their baselines[END_REF] focuses on three tasks: cross channel language identification (task 1), dialect identification (task 2) and noisy language identification (task 3). The LORIA-Inria-MULTISPEECH team focused on improving robustness to unknown conditions in order to compete for task 1 (cross-channel identification) and task 3 (noisy conditions). We did not participate in task 2. This paper summarizes our research effort towards the design of a robust language recognition recipe suited for domain generalization.

Three sets of studies have been carried out to design a robust language recognition system. First, we designed a robust frame-level bottleneck features extractor, with a recipe based on an end-to-end speech recognition model. Then, in order to simulate mismatched test conditions, we compared several utterance-level classifiers trained with different recipes, without using training data from the unknown channels, and evaluated their performance on unknown channels. In the paper, we call this step optimization for system robustness. Finally, we designed a system combination strategy. We proposed to train several models on a large dataset including data from unknown channels, and merged their prediction in order to increase robustness. In the following, this step is called the final system design. For each step specific training and testing sets have been used.

The submitted system is the same for task 1 and 3. It is constituted of the fusion of five models: one GMM model and four TDNNs, trained with bottleneck features and different loss functions. We propose a new recipe for training multilingual bottleneck features from a Conformer model [START_REF] Gulati | Conformer: Convolution-augmented transformer for speech recognition[END_REF], artificial data augmentation techniques abiding by the rules of the challenge and stochastic weight averaging [START_REF] Izmailov | Averaging weights leads to wider optima and better generalization[END_REF] to increase generalization of the model. Other key factors that have been investigated are the combination of models with different properties and a duration-dependent calibration [START_REF] Mclaren | Approaches to multi-domain language recognition[END_REF].

Dataset usage

Corpora

The rules of the challenge specify a limited set of corpora, which can be used to develop the systems [START_REF] Li | AP20-OLR challenge: Three tasks and their baselines[END_REF].The training data contains 16 languages. The 10 traditional languages of the OLR challenges are Cantonese (ct-cn), Mandarin (zh-cn), Indonesian (id-id), Japanese (ja-jp), Russian (ru-ru), Korean (kokr), Vietnamese (vi-vn), Kazakh (ka-cn), Tibetan (ti-cn), and Uyghur (uy-cn). Only these languages are provided with transcriptions in the sets AP16-OL7 and AP17-OL3. In addition, recordings for three Chinese dialects are provided: Hokkien, Sichuanese, and Shanghainese. Moreover, data from three non target languages of previous evaluations are provided: Catalan, Greek, and Telugu.

The sets containing recordings from unknown channels are of key interest: AP19-OLR-dev-task2 and AP19-OLR-test-task2. They only contain 6 languages: Japanese, Russian, Vietnamese, Tibetan, Mandarin and Uyghur. In the paper, they are referred to as the development languages. This means that we do not have access to recordings from unknown channels for three of the six languages of task 1: Cantonese, Indonesian and Korean. Consequently, we designed a two-step strategy to use this data from unknown channels first for evaluating performance on unknown channels (optimization for system robustness ) and then to increase the diversity of the training set (final system design).

As mentioned before, the system has been designed in several steps. Train, validation and test datasets used for each step are described in Table 1 and explained below: 2. Optimization for system robustness. In order to replicate the unknown channel evaluation conditions, we design systems using data from mobile channels only. We evaluate them on three datasets: test-2018 (with mobile channel data), dev-2019 and test-2019 (with unknown channel data). To focus on the channel mismatch problem, we only use the 6 development languages.

3.

Final system design. At the final stage of training, once we have selected training recipes robust to channel mismatch, we increase further diversity of the training set by adding data from unknown channels. We train the system with 16 languages. This allows us to evaluate the system for three different sets of languages: development languages, languages of task 1 and languages of task 3. We use two test sets: match (with only mobile channel data) and mismatch (with unknown channel data).

When an original dataset is split into several datasets for our experiments, we use speaker labels (when available) to ensure that all recordings of the same speaker belong to the same set.

Data augmentation

To improve the robustness of the systems to mismatched conditions, we train them with three data augmentation techniques. To abide by the rules of the challenge [START_REF] Li | AP20-OLR challenge: Three tasks and their baselines[END_REF], we do not use any additional speech corpus and only use other files of the training corpus or artificial noise or filters. The three data augmentation methods are: addition of white noise, addition of babble noise (sampled by mixing other files of the training set) and convolution with random artificial band-pass filters.

Frame-level features

We explored two kinds of frame-level features during the challenge: spectral features and bottleneck features (BNF). We also used specific SDC-MFCC features for the GMM model.

Spectral features

Mel-filterbank features and mel-frequency cepstral coefficients (MFCCs) were evaluated. When used without any domain robustness technique, restriction to the telephone bandwidth (300-3800 Hz) was shown useful. No significant gain was achieved by adding pitch features [START_REF] Talkin | A robust algorithm for pitch tracking (RAPT)[END_REF].

The Gaussian mixture model (GMM) system employs shifted delta coefficients (SDC) computed with short-term MFCC features [START_REF] Torres-Carrasquillo | Approaches to language identification using Gaussian mixture models and shifted delta cepstral features[END_REF]. The MFCCs are extracted from speech frames of 20 ms with shift of 50% and 16 filters in mel scale. We process the extracted MFCCs with relative spectral (RASTA) processing for removing slowly varying channel effect. Then we compute SDCs with shift of three frames and a context of seven frames to create 128-dimensional SDC-MFCC features.

For all systems, we applied an energy-based speech activity detector (SAD) to discard the non-speech frames. Finally, we found utterance-level cepstral mean and variance normalization (CMVN) helpful for robustness of the TDNN models.

Bottleneck features

Multilingual bottleneck features [START_REF] Fér | Multilingual bottleneck features for language recognition[END_REF] are very efficient framelevel features for language recognition. Usually, forced alignment between frame-level acoustic features and transcriptions is performed by an automatic speech recognition system for each target language. Then simple neural networks are trained to predict for each frame the assigned phone or triphone. An embedding is extracted from an intermediate bottleneck layer of the system.

In this work, we did not invest resources to develop an acceptable automatic speech recognition system for each target language with the goal of performing forced phone alignment. We simply trained a unique multilingual end-to-end speech recognition system with the connectionist temporal classification (CTC) loss [START_REF] Graves | Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks[END_REF], extending the idea of [START_REF] Ren | Two-stage training for chinese dialect recognition[END_REF][START_REF] Ling | BERTPHONE: Phonetically-aware encoder representations for utterance-level speaker and language recognition[END_REF]. We used the Conformer architecture [START_REF] Gulati | Conformer: Convolution-augmented transformer for speech recognition[END_REF] with an output layer specific to each target language, with 64 mel-filterbank features as input. The ten traditional languages were used for training the Conformer model. A sentence piece model [START_REF] Kudo | SentencePiece: a simple and language independent subword tokenizer and detokenizer for neural text processing[END_REF] was estimated on the training transcriptions to define 2000 target tokens for each language. Finally we extracted frame-level embeddings from a hidden layer of the Conformer model and used them as language recognition features. We used a small Conformer encoder constituted of two blocks with four attention heads.

Neural network training

The utterance-level classification task can be performed by a neural network. The standard time delay neural network (TDNN) architecture for language recognition [START_REF] Snyder | Spoken language recognition using x-vectors[END_REF] was used in this work. First the general training recipe was selected according to experiments with the datasets defined for optimization for system robustness. For this set of experiments, we kept data from unkown channels for evaluation of the system. Then several systems were trained using the datasets defined for the final system design, with regularization loss functions that benefit from training data from unknown channels.

Training recipe

The final training recipe includes:

• the cross-entropy (CE) loss function • stochastic gradient descent with dropout [START_REF] Srivastava | Dropout: a simple way to prevent neural networks from overfitting[END_REF] • specAugment [START_REF] Park | SpecAugment: A simple data augmentation method for automatic speech recognition[END_REF] • the three data augmentation techniques (Subsection 2.2) • stochastic weight average [START_REF] Izmailov | Averaging weights leads to wider optima and better generalization[END_REF]. It consists in averaging parameters of the model along the trajectory of the gradient descent, instead of selecting the parameters corresponding to the best validation loss.

Exploring other loss functions

In the last set of experiments, we used the final system design datasets with training data from unknown channels. Consequently, we trained systems with different loss functions to reduce the domain mismatch:

• additive angular margin softmax (AAM) [START_REF] Deng | Arcface: Additive angular margin loss for deep face recognition[END_REF] as an alternative classification loss function , we used a margin parameter m = 0.1 and a radius s = 40 • regularization of the cross-entropy with maximum mean discrepancy (MMD) between mobile channel and unknown channels [START_REF] Duroselle | Unsupervised regularization of the embedding extractor for robust language identification[END_REF], we compared different weights λ for the regularization loss functions • regularization of the cross-entropy with n-pair loss [START_REF] Duroselle | Metric learning loss functions to reduce domain mismatch in the x-vector space for language recognition[END_REF]

Gaussian mixture model training

As an alternative utterance-level model, we chose the GMMbased model as this gives promising language recognition accuracy for short test utterances [START_REF] Ambikairajah | Language identification: a tutorial[END_REF]. This statistical approach was also traditionally used in NIST language recognition evaluations (LREs) and could be helpful by providing complementary information to the neural network based methods. Languagedependent GMMs are trained using 4096 mixture components and ten iterations of the expectation-maximization (EM) algorithm. During scoring, we compute the log-likelihoods corresponding to each target language model separately.

Fusion and calibration

Standard linear multi-class calibration and score fusion are performed for each task with the FoCal toolkit [START_REF] Brümmer | The FoCal toolkit[END_REF]. This step is performed on the validation set defined for the final system design. The final scores are log-likelihood ratio.

For the selected final combination of systems, we implement a duration-dependent calibration procedure [START_REF] Mclaren | Approaches to multi-domain language recognition[END_REF]. For three ranges of duration (inferior to 2s, between 2 and 4s, superior to 4s), we learn specific fusion and calibration parameters. For test utterances, the duration is estimated thanks to the energy-based SAD module mentioned in Section 3.

Experiments

In this section, we report the main results that lead to the design of the submitted system.

Bottleneck features training

We use the bottleneck features datasets. Models are compared with the average CTC loss for the 10 target languages on the validation set, cf. Table 2. We observe that the combination of cepstral mean and variance normalization (CMVN), specAugment and data augmentation allows to train better speech recognition features. We call these features final BNF, as opposed to the baseline BNF. We use the final BNF in the submitted system. The experiments demonstrate the superiority of the trained bottleneck features over mel-filterbanks and MFCCs. Final BNF are superior to baseline BNF for the challenging dev-2019 set. Finally, stochastic weight averaging is superior to model selection based on the best validation loss. We observed the same behavior for all models. In the following, stochastic weight averaging is applied to all models.

Final system design experiments

To design the final system we train various models with a large dataset containing data from 16 languages, including recordings from unknown channels. We calibrate them on the corresponding validation datasets and evaluate them on two datasets in terms of Cavg [START_REF] Li | AP20-OLR challenge: Three tasks and their baselines[END_REF]: match (with mobile channel data only), mismatch (with some unknown channel recordings). Evaluation is performed for two sets of languages corresponding to task 1 and task 3. Results are presented in Table 4.

For comparison, we also present the performance obtained with a TDNN trained with cross-entropy loss on the optimization for system robustness datasets. All other models are trained with the final system design datasets and achieve better performance. Among the individual models, the best performance on unknown conditions is achieved by the TDNN trained with AAM loss. TDNNs trained with regularization loss functions (MMD and n-pair) do not improve on the test sets but the performance gap between known and unknown channels is reduced. Finally, the GMM model achieves the best performance on matched conditions but is very sensitive to channel mismatch.

Then, starting with the model with best performance, we greedily add models to the final system.We select the combination of five systems: four TDNNs based on BNF trained with additive angular margin softmax, regularization with maximum mean discrepancy with low and high weight values (λ), regularization with n-pair loss and one GMM model.

For known conditions, the best performance of individual systems is achieved by the GMM model (2.53% for task 1). Fusion with the TDNN trained with AAM helps to improve performance (1.54 %) but the addition of three other TDNNs trained with domain robustness objectives does not improve further (1.53%). Conversely, for unknown conditions, the TDNN trained with AAM is better than the GMM model (5.34% instead of 10.82% for task 1). The fusion of both is also helpful and the fusion with three other robust models allows an additional gain in performance (3.67%). Even if the models trained with regularization loss functions achieve a weak individual performance, they are designed to be robust to channel mismatch and are therefore useful on mismatched conditions for fusion with the best models.

Finally, we learn the fusion and calibration parameters for three different duration ranges. At test time, we use the fusion parameters corresponding to the speech duration of the test utterance. Since we had no information about durations of the evaluation utterances, we assumed that this technique was safer, even thought it was not beneficial for our test sets.

Analysis of the submitted system

Once labels of the evaluation sets have been released, we notice that performances on the evaluation test sets of the systems trained with regularization loss functions are consistent with performance on the mismatch set (see Table 4). The two systems trained with the cross-entropy loss achieve a good individual performance, presumably because of matched conditions between the evaluation and training sets. The duration dependent calibration method does not improve performance.

For both tasks we perform an analysis of the errors according to the duration and estimations of signal to noise ratio (SNR) and pitch. Unsurprisingly, we observe that performance improves with duration and SNR. In addition, the submitted system is more efficient for high pitch values (200-250 Hz) than low pitch values (100-150 Hz).

Finally, for both tasks the errors are dominated by a few language pairs, as can be observed on the confusion matrices of Figure 1. A significant improvement in terms of Cavg can be expected by focusing on these specific language pairs.

Conclusion

The Oriental Language Recognition AP20-OLR Challenge was a good opportunity to evaluate the domain generalization ability of training recipes of language recognition systems. In this work, we propose a two-step strategy with the goal of optimally using data from unknown channels during design of the system. First, we design a robust recipe with a training set which does not contain data from unknown channels and keep this data from unknown channels for evaluation on mismatched conditions. Then, we apply this recipe with a larger training set which contains data from unknown channels and use loss functions that enforce invariance between channels, for the few languages for which unknown channel data is available. The main difficulty of this challenge is generalization to unknown conditions: new channel for task 1 and noisy conditions for task 3. The analysis of our systems reveals that their performances on our development sets are consistent with results on the evaluation set for unknown channels.The final performance of the submitted system (which achieved best performance for tasks 1 and 3) validates our combination of robust bottleneck features, data augmentation methods and domain robustness loss functions.

(a) Task 1 (b) Task 3 Figure 1 :

 131 Confusion matrices of the submitted system on the evaluation set (number of utterances). Multiple detections are possible. Language labels are defined in Subsection 2.1. Colored numbers correspond to a high number of errors.

Table 1 :

 1 Usage of datasets for each set of experiments. val. refers to validation.

		bottleneck	optimization	final
	Dataset	features	for system	system
			robustness	design
	AP16-OL7	train & val.	train & val.	train & val.
	AP17-OL3	train & val.	train & val.	train & val.
	AP17-test		train & val.	train & val.
	AP18-test		test-2018	train & val. & match
	AP19-dev-task2		dev-2019	mismatch
	AP19-dev-task3		dev-2019	train & val.
	AP19-test-task1		test-2019	train & val. & mismatch
	AP19-test-task2		test-2019	train & val.
	AP19-test-task3		test-2019	train & val.
	AP20-dialect			train & val.
	1. Training of bottleneck features. We use recordings from
	the ten traditional languages with their transcriptions.

Table 2 :

 2 Automatic speech recognition performance in terms of CTC loss for different training recipesDuring the optimization for system robustness experiments, we train language recognition systems without using data from unknown channels. Performance is measured with the equal error rate (EER) on three test sets: test-2018 (mobile channel), dev-2019 and test-2019 (which contain recordings from unknown channels), with the 6 development languages, cf. Table3. We use a TDNN with the recipe described in Section 4, with different frame-level features and model selection strategies.

	System name	Training recipe	CTC-loss on validation set
	baseline BNF mel-filterbank features	3.94
		+ CMVN	3.69
		+ specAugment	3.52
	final BNF	+ data augmentation	3.02

Table 3 :

 3 Language recognition performance for system robustness experiments. Equal error rate (%) on test sets. All models are TDNNs trained with the cross-entropy loss and data augmentation.

	input features	Model selection	test-2018 dev-2019 test-2019
	mel-filterbanks	best validation loss	11.95	25.48	29.54
	MFCCs	best validation loss	4.75	37.78	24.34
	baseline BNF	best validation loss	3.91	19.11	11.08
	final BNF	best validation loss	3.43	17.24	13.68
	final BNF	stochastic weight average	2.97	16.92	12.78

Table 4 :

 4 Comparison of final systems in terms of Cavg. Performance is measured on the three development sets of the final system design step (validation, match and mismatch) and the evaluation sets (eval). The submitted system corresponds to the last row. It achieves an EER of 2.47% on task 1 and 4.07% on task 3.

			Task 1 (6 languages)		Task 3 (5 languages)
		models	Cavg × 100		Cavg × 100
			val. match mismatch eval	val. match mismatch eval
		pytorch x-vector baseline [15]			13.21		7.15
	individual models	CE, optimization for system robustness datasets 3.94 5.60 CE, system design datasets 2.50 4.69 AAM 3.68 3.14 n-pair 3.22 5.73 MMD λ = 1 3.18 5.39 MMD λ = 100 4.01 5.89 GMM 3.31 2.53	10.30 7.26 5.34 7.25 7.87 7.60 10.82	4.96 3.48 5.91 5.12 2.04 4.34 5.08 3.87 3.20 7.52 2.58 5.14 5.54 2.93 4.99 7.61 3.76 5.24 7.13 3.00 2.69	9.25 7.16 6.25 7.36 8.31 7.29 13.97	4.02 4.94 5.11 5.70 5.58 5.80 9.32
		AAM -GMM	1.37 1.54	4.37	2.82 1.35 1.45	5.73	4.93
	fusion	AAM -GMM -MMD 100 AAM -GMM -MMD 100 -n-pair -MMD 1 0.93 1.53 0.98 1.57 duration-dependent calibration 1.56	3.91 3.67 3.82	2.47 1.00 1.60 2.28 0.97 1.60 2.39 1.61	4.96 4.54 4.60	3.82 3.71 3.74

Acknowledgements

Experiments presented in this paper were carried out using the Grid5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr). This work has been partly funded by the French Direction Générale de l'Armement.


Raphaël Duroselle

Md Sahidullah

Denis Jouvet

Irina Illina

Modeling and training strategies for language recognition systems

Keywords: language recognition, bottleneck features, endto-end speech recognition, multi-task training

Automatic speech recognition is complementary to language recognition. The language recognition systems exploit this complementarity by using frame-level bottleneck features extracted from neural networks trained with a phone recognition task. Recent methods apply frame-level bottleneck features extracted from an end-to-end sequence-to-sequence speech recognition model. In this work, we study an integrated approach of the training of the speech recognition feature extractor and language recognition modules. We show that for both classical phone recognition and end-to-end sequence-to-sequence features, sequential training of the two modules is not the optimal strategy. The feature extractor can be improved by supervision with the language identification loss, either in a fine-tuning step or in a multi-task training framework. Besides, we notice that end-to-end sequence-to-sequence bottleneck features are on par with classical phone recognition bottleneck features without requiring a forced alignment of the signal with target tokens. However, for sequence-to-sequence, the architecture of the model seems to play an important role; the Conformer architectures leads to much better results than the conventional stacked DNNs approach; and can even be trained directly with the LID module in an end-to-end approach.

Introduction

Language recognition is the task of determining the language in a spoken speech utterance [START_REF] Ambikairajah | Language identification: a tutorial[END_REF]. Systems that address this task handle the speech utterances in several steps: they first extract features at the frame level, then they process and aggregate these features at the utterance level, and finally, make a decision.

Standard frame-level features for language recognition are spectral features: filterbanks [START_REF] Cai | On-the-fly data loader and utterance-level aggregation for speaker and language recognition[END_REF], MFCC [START_REF] Martinez | Language recognition in ivectors space[END_REF], SDC-MFCC [START_REF] Torres-Carrasquillo | Approaches to language identification using Gaussian mixture models and shifted delta cepstral features[END_REF] and PLP features [START_REF] Gelly | Spoken language identification using LSTM-based angular proximity[END_REF][START_REF] Ben Zeghiba | Phonotactic language recognition using MLP features[END_REF]. These features can be refined by using a neural network to select relevant information through an auxiliary task, for instance auto reconstruction [START_REF] Ling | BERTPHONE: Phonetically-aware encoder representations for utterance-level speaker and language recognition[END_REF], prosody information prediction [START_REF] Pascual | Learning problem-agnostic speech representations from multiple self-supervised tasks[END_REF], phonotactic information encoding [START_REF] Romero | Exploring transformer-based language recognition using phonotactic information[END_REF] or speech attribute prediction [START_REF] Kukanov | Maximal figure-of-merit framework to detect multi-label phonetic features for spoken language recognition[END_REF]. The most successful approach has been the use of a speech recognition task [START_REF] Fér | Multilingually trained bottleneck features in spoken language recognition[END_REF].

The complementarity of automatic speech recognition (ASR) task to language identification (LID) task is most likely due to the fact that they heavily depend on phonetic information [START_REF] Li | Spoken language recognition: from fundamentals to practice[END_REF]. If both tasks use the same information, relevant features could be more easily discovered with a speech recognition loss which uses a sequence of labels, because providing just the label of the language spoken in the utterance is too coarse for supervision [START_REF] Tang | Phonetic temporal neural model for language identification[END_REF][START_REF] Jiang | Deep bottleneck features for spoken language identification[END_REF].

The most commonly used application of this complementarity is frame-level bottleneck features [START_REF] Jiang | Deep bottleneck features for spoken language identification[END_REF][START_REF] Mclaren | Exploring the role of phonetic bottleneck features for speaker and language recognition[END_REF]. This kind of bottleneck features are used in state-of-the-art language recognition systems [START_REF] Fér | Multilingually trained bottleneck features in spoken language recognition[END_REF][START_REF] Snyder | Spoken language recognition using x-vectors[END_REF]]. An ASR system is trained and used to perform forced alignment between speech utterances and frame-level tokens: morphemes [START_REF] Ma | Language identification with deep bottleneck features[END_REF] or tied phone states [START_REF] Fér | Multilingually trained bottleneck features in spoken language recognition[END_REF][START_REF] Peng | Adversarial multi-task deep features and unsupervised back-end adaptation for language recognition[END_REF]. Then a deep neural network (multilayer perceptron [START_REF] Ma | Language identification with deep bottleneck features[END_REF] or time delay neural network [START_REF] Peng | Adversarial multi-task deep features and unsupervised back-end adaptation for language recognition[END_REF]) is trained to predict the token for each frame (with some temporal context). Finally, framelevel embeddings are extracted from a hidden layer of the network. Those embeddings are called bottleneck features and they are used for the LID task. When several neural networks are trained 'on cascade' to predict the phone labels, they have been called stacked bottleneck features. Training of a language recognition system that uses bottleneck features [START_REF] Fér | Multilingually trained bottleneck features in spoken language recognition[END_REF] requires a two-step strategy. First, the bottleneck feature extractor is trained with an ASR task. Then the language recognition system is trained using those features.

However, this two-step training strategy is far from being optimal. The work in [START_REF] Lozano-Diez | An analysis of the influence of deep neural network (DNN) topology in bottleneck feature based language recognition[END_REF] has empirically shown that the best models in terms of phone state accuracy are not the most relevant feature extractors for language recognition. It even appears that the best ASR performance can be achieved with phone representations independent of the language, which paves the way to language adversarial ASR [START_REF] Adams | Massively multilingual adversarial speech recognition[END_REF]. So far, no method has been investigated to select the best compromise between LID and ASR tasks during the training of a language recognition model. Our work aims to study the optimal use of the ASR task to estimate parameters of a language recognition model.

A straight-forward solution is to include both models into an end-to-end (E2E) architecture. According to [START_REF] Tang | Phonetic temporal neural model for language identification[END_REF], E2E training with the language identification loss is not a successful approach. Also, fine-tuning the ASR encoder by backpropagating the LID loss into the feature extractor does not improve the language recognition performance [START_REF] Tang | Phonetic temporal neural model for language identification[END_REF]. They concluded that it is the phonetic information that helps and not the deeper architecture. Another approach is multi-task training of the whole model with both ASR and LID losses. It has been successfully implemented for one [START_REF] Zhao | Phone-aware multi-task learning and length expanding for shortduration language recognition[END_REF][START_REF] Li | Deep joint learning for language recognition[END_REF] and two [START_REF] Tang | Phonetic temporal neural model for language identification[END_REF] languages for the ASR task, without proposing a principled method to chose the respective weights of both tasks during training.

Recently, the paradigm of bottleneck features training for language recognition has been evolving. The traditional training of bottleneck feature extractors relies on a frame-level forced alignement between spectral features and phone labels [START_REF] Fér | Multilingually trained bottleneck features in spoken language recognition[END_REF]. Consequently, the quality of the bottleneck features depends on the performance of the speech recognition model, which makes inclusion of new languages very expensive. Conversely, several end-to-end automatic speech recognition architectures have been introduced recently [START_REF] Gulati | Conformer: Convolution-augmented transformer for speech recognition[END_REF][START_REF] Amodei | Deep speech 2: End-to-end speech recognition in English and Mandarin[END_REF]. They can be trained with a sequence-to-sequence loss, like connectionist temporal classification (CTC) loss [START_REF] Graves | Towards end-to-end speech recognition with recurrent neural networks[END_REF] and used to extract frame-level embeddings from a hidden layer of the network. With this approach, state-of-the-art language recognition performance has been achieved without defining a frame alignment of phone labels, using only one language for the ASR task [START_REF] Ling | BERTPHONE: Phonetically-aware encoder representations for utterance-level speaker and language recognition[END_REF]. Moreover, for Chinese dialect recognition, the use of the phone forced alignment performed by the acoustic model trained with CTC loss does not improve language recognition performance over the direct use of the sequence-to-sequence bottleneck features [START_REF] Ren | Two-stage training for chinese dialect recognition[END_REF]. Multi-task training of a joint ASR and LID model has been successfuly performed for English and Hindi corpora [START_REF] Punjabi | Joint ASR and language identification using RNN-T: An efficient approach to dynamic language switching[END_REF].

Other approaches to joint ASR and LID training have been proposed. The works in [START_REF] Punjabi | Joint ASR and language identification using RNN-T: An efficient approach to dynamic language switching[END_REF][START_REF] Watanabe | Language independent end-to-end architecture for joint language identification and speech recognition[END_REF] include language labels as additional target tokens of the ASR model, allowing easy language diarization. Also, [START_REF] Liu | A unified system for multilingual speech recognition and language identification[END_REF] introduced a unified system where the ASR model uses x-vectors produced by the LID module and the LID model improves its prediction using ASR confidence scores. In this paper, we do not compare to these systems, but rather focus on the problem of optimally using the ASR task, as an auxiliary task when training a model to yield frame-level features for language recognition.

In this work, we compare different choices of architectures and of training strategies for a language recognition system constituted of a feature extractor and a language identification module. First, we show that end-to-end sequence-tosequence multilingual features trained with the CTC loss are on par with state-of-the-art traditional multilingual bottleneck features trained with phone state prediction [START_REF] Fér | Multilingually trained bottleneck features in spoken language recognition[END_REF]. Then, we show that both traditional bottleneck features (phone prediction) and sequence-to-sequence bottleneck features can be greatly improved by using the language identification loss during training of the feature extractor.

Joint speech and language recognition system

The language recognition system discussed in this work is composed of a frame-level feature extractor and an utterance-level language predictor. We propose to consider it as a whole end-toend architecture. This point of view encompasses state-of-theart language recognition systems [START_REF] Snyder | Spoken language recognition using x-vectors[END_REF] and joint speech and language recognition architectures like the phonetic temporal neural model of [START_REF] Tang | Phonetic temporal neural model for language identification[END_REF]. It allows exploring different system training strategies combining language identification and speech recognition objective.

Framework of the system

A language recognition system operates over a sequence X of varying length T of frame-level features:

X = {x1, x2, . . . xT } (1)
The task consists in predicting a language recognition label y among a fixed set Y. During training of the system, auxiliary speech recognition information can be provided as a target sequence Z of words, phone states or characters, of length L:

Z = {z1, z2, . . . zL} (2)
A joint speech and language recognition architecture is depicted on Figure 1. It is composed of three modules:

• a frame-level feature extractor (F E) that takes as input the sequence X and produces a sequence F of framelevel embeddings of length N ≤ T :

F = {f1, f2, . . . fN } (3)
• an automatic speech recognition (ASR) decoder which takes as input the sequence F and outputs a sequence of speech recognition scores Ẑ Ẑ = {ẑ1, ẑ2, . . . ẑL} (4)

Figure 1: Schematic diagram of the language recognition system. The input frame-level features xt are processed by the feature extractor and the language identification module to produce language identification scores ŷ. The ASR decoder is used for training with the auxiliary speech recognition task.

• a language identification (LID) module which takes as input the sequence F and outputs a vector of language identification scores ŷ

Parameters of the three modules can be trained with two different loss functions:

• a language identification loss LLID(ŷ, y) • an auxiliary automatic speech recognition loss LASR( Ẑ, Z)

Training strategies

Once both the feature extractor and language identification modules have been encompassed into the same model, several training strategies become possible:

• E2E LID: the auxiliary speech recognition information is discarded, and F E and LID are trained together to minimize LLID • 2-step training: first, F E and ASR are trained to minimize the automatic speech recognition loss LASR, then parameters of F E are freezed and parameters of LID are trained to minimize the language identification loss LLID • 2-step then E2E LID: 2-step training is performed, then F E and LID are fine-tuned with the language identification loss • multi-task training: all modules are trained together [START_REF] Punjabi | Joint ASR and language identification using RNN-T: An efficient approach to dynamic language switching[END_REF].

LID is trained to minimize LLID, and ASR to minimize LASR. F E is trained with a multi-task training objective (with weight λ ∈ [0, 1]):

L = λLLID + (1 -λ)LASR (5)

Relation with the classical bottleneck features

Within this framework, the classical bottleneck features approach for language recognition [START_REF] Fér | Multilingually trained bottleneck features in spoken language recognition[END_REF] corresponds to the 2-step training method with the following choices:

• target speech recognition labels Z are phone state labels and a label is provided for each input frame (i.e., L = T )

• the ASR loss LASR is a phone state classification loss for each frame.

• the feature extractor applies identical processing with a DNN for each input frame xt (generally with its context), consequently there is an output frame-level embedding for each input frame (i.e. N = T )

• the ASR decoder applies identical processing with a DNN for each frame-level embedding fn

Experimental setup

In the following, we compare the different training strategies of a language recognition system, with the same corpora and architectures. We also compare classical bottleneck features [START_REF] Fér | Multilingually trained bottleneck features in spoken language recognition[END_REF] with end-to-end sequence-to-sequence bottleneck features.

Model architectures

We use two architectures for the language identification module:

• the vanilla TDNN architecture described in [START_REF] Snyder | Spoken language recognition using x-vectors[END_REF].

• a ResNet architecture with 1D-convolutional layers and a statistical pooling layer identical to the one used in [START_REF] Snyder | Spoken language recognition using x-vectors[END_REF].

The model is composed of a serie of four blocks, constituted of respectively 3, 4, 6, and 3 residual layers and having a growing number of channels: 16, 32, 64, and 128. For the feature extractor and ASR decoder, we investigate two architectural choices. Both produce frame-level embeddings of dimension 80:

• stacked DNNs of the mutlilingual bottleneck feature extractor described in [START_REF] Fér | Multilingually trained bottleneck features in spoken language recognition[END_REF]. For the 2-step training strategy, we use the pretrained models (BUT/Phonexia bottleneck feature extractor) provided by the authors of [START_REF] Silnova | BUT/Phonexia Bottleneck Feature Extractor[END_REF], which take as inputs mel-filterbank features. • Conformer [START_REF] Gulati | Conformer: Convolution-augmented transformer for speech recognition[END_REF], a sequence-to-sequence speech recognition model, with one decoder for each target language. We use a model with 2 Conformer blocks with 4 selfattention heads, 80 channels and a kernel of size 17.

The initial convolutional layer downsamples the input sequence by a factor 4 (i.e. N = T 4 ). We train this model with mel-filterbank input features. We optimized this architecture for feature extraction during the Oriental Language Recognition challenge 2020 [START_REF] Duroselle | Language recognition on unknown conditions: the LORIA-Inria-MULTISPEECH system for AP20-OLR Challenge[END_REF].

Loss functions

In all experiments, the language identification model is trained with the traditional cross-entropy loss as LLID.

For LASR:

• the stacked DNNs of [START_REF] Fér | Multilingually trained bottleneck features in spoken language recognition[END_REF] have been trained with the cross-entropy as a frame-level classification loss function. The target labels were 3096 phoneme states.

• the Conformer is trained with the CTC loss. The target sequence Z is defined by a sentence piece model [START_REF] Kudo | Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing[END_REF] trained with 1000 tokens for each target language.

Corpora

The language recognition performance is evaluated on the corpus NIST LRE 2007 [START_REF]The 2007 NIST language recognition evaluation plan[END_REF]. We perform evaluation for a closedset identification task with the following 14 languages: Ara-bic, Bengali, Chinese, English, Hindustani, Spanish, Farsi, German, Japanese, Korean, Russian, Tamil, Thai, Vietnamese. As a training set, we use a subset of the corpora NIST LRE 2003, 2005[START_REF]The 2007 NIST language recognition evaluation plan[END_REF]train, 2009 (only recordings from telephone channels), Callfriend, and NIST SRE 2008. This training set is the same as in [START_REF] Cai | On-the-fly data loader and utterance-level aggregation for speaker and language recognition[END_REF][START_REF] Ling | BERTPHONE: Phonetically-aware encoder representations for utterance-level speaker and language recognition[END_REF]. We use Babel speech corpus [START_REF]Babel project[END_REF] for the ASR task. Two different sets of languages are used for experiments with the two feature extractor architectures. The BUT/Phonexia pretrained models [START_REF] Fér | Multilingually trained bottleneck features in spoken language recognition[END_REF] have been trained with the following 17 languages: Cantonese, Assamese, Bengali, Pashto, Turkish, Tagalog, Vietnamese, Haitian Creole, Lao, Tamil, Zulu, Kurmanji (Kurdish), Tok Pisin, Cebuano, Kazakh, Telugu, Lithuanian. For our experiments with the Conformer architecture we only use 14 of these languages. We do not use Cantonese, Tagalog and Cebuano.

Training setup

Models are trained with the Adam optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF], with an initial learning rate of 10 -3 , that is halved every time the validation loss does not improve for 3 epochs. Training is performed for 20 epochs for each phase of training, except for training of the ASR model in the 2-step strategy where we use 100 epochs. Stochastic weight averaging [START_REF] Izmailov | Averaging weights leads to wider optima and better generalization[END_REF] is used: the final model is the average of the 20 models of each epoch. The only data augmentation technique is specAugment [START_REF] Park | Specaugment: A simple data augmentation method for automatic speech recognition[END_REF]. Cepstral mean and variance normalization [START_REF] Molau | Feature space normalization in adverse acoustic conditions[END_REF] is applied to the input features.

Language recognition models are trained with balanced minibatches of size 128. Chunks of three seconds are used. For each experiment, at the end of training, two systems are finetuned for longer durations with chunks respectively in ranges of 7-13s and 20-40s. These systems are used for evaluation on test sets of respectively 10s and 30s. For each of the three systems (initial 3s, 10s and 30s), calibration parameters are learned on a held-out validation set extracted from the training set and with the same distribution of durations.

Results and discussion

Performance

For each language recognition and feature extractor model combination, we train a system with the three training strategies: E2E LID, 2-step, and 2-step then E2E LID. Language recognition performance of each method is reported on Table 1 for the three test duration conditions of the NIST LRE2007 corpus. As a baseline system, we train two language recognition models directly with mel-filterbank features, without feature extractor module.

First, we observe that for all architectures, E2E LID training gives acceptable results (contrary to what was observed in [START_REF] Tang | Phonetic temporal neural model for language identification[END_REF]). The additional processing with the feature extractor helps to improve over the baseline system trained with mel-filterbank features. Then we see that 2-step training improves over E2E performance for the traditional stacked DNNs but not for the Conformer. It indicates that the sequence-to-sequence ASR architecture is more appropriate for E2E LID training than the stacked DNNs. For all models, 2-step then E2E LID improves over 2-step training, and it always gives a better performance than E2E LID training for segments of 3s.

The ResNet-1D architecture is consistently better than the TDNN and our best-performing recipe (Resnet-1D with Conformer feature extractor and 2-step then E2E LID strategy) achieves state-of-the-art performance in terms of Cavg on the

Multi-task training

We evaluate the multi-task training strategy with the ResNet-1D and Conformer architectures. Two systems are trained with different choices for the value of the weight λ of the language identification loss. Both systems are trained for 100 epochs (same number of epochs as for training the feature extractor module in the 2-step strategy). Results are reported in Table 1.

A first system is trained with a fixed value of λ = 0.5, which corresponds to an equal weighting of the two loss functions. It achieves similar performance as the 2-step then E2E LID strategy for segments of 3 seconds. In addition, training is performed in a unique step with the multi-task strategy, that makes the training recipe simpler.

In the 2-step then E2E LID strategy, the feature extractor is first trained with the ASR loss and then with the LID objective. It is the most successful strategy. We imitate this approach with a multi-task training strategy where the weight λ is linearly increased from 0 to 1 during training (λ = 0 → 1, in Table 1). This strategy gives the best performance for segments of 3 seconds.

Discussion

Using a Conformer sequence-to-sequence feature extractor, we have successfully trained ASR-based multilingual bottleneck features without explicltly performing forced phone alignment. A similar behavior was observed for a dialect recognition task [START_REF] Ren | Two-stage training for chinese dialect recognition[END_REF]. We compare these features with a very strong baseline: classical multilingual bottleneck features [START_REF] Fér | Multilingually trained bottleneck features in spoken language recognition[END_REF].

An important part of the performance gain with bottleneck features comes from the additional layers of the feature extractor module. E2E LID training of the whole model allows to benefit from the feature extractor and it achieves very good performance with the Conformer model. Pre-training of the encoder with an ASR supervision (2-step training) allows to achieve a good performance, but fine-tuning of the feature-extractor with the language identification loss improves further. Multi-task training of the feature extractor achieves similar results, with a unique step of training. It means that supervision of the feature extractor training with the language identification loss is necessary to achieve the best language recognition performance.

Conclusion

We study the problem of optimally using an automatic speech recognition task to design frame-level features for language recognition. We trained end-to-end sequence-to-sequence multilingual bottleneck features on the Babel corpus and achieved state-of-the-art language recognition performance on the NIST LRE2007 corpus.

We observe that the use of an end-to-end sequence-tosequence speech recognition model trained with the CTC loss achieves similar performance as classical bottleneck features trained with a frame-level phone recognition task, while being easier to train because it does not need a forced frame alignment of target labels. An important part of this gain comes from the architecture of the feature extractor (Conformer instead of stacked DNNs), that can also be successfully trained in an endto-end approach with the language identification module and associated loss.

For both classical bottleneck features and end-to-end sequence-to-sequence ones, a substantial improvement can be obtained by fine-tuning the feature extractor with the language identification loss. A similar performance can be achieved with end-to-end training with a multi-task objective. In the future, we plan to study further the optimal scheduling strategy for the relative weights of the two loss functions in the multi-task objective.

Table 1 :

 1 Language recognition performance on the three test sets of the corpus NIST LRE 2007, corresponding to different durations. For two architectures of the language identificaton module and two architectures of the feature extractor, we explore three training strategies presented in Subsection 2.2. Multi-task training is experimented for the best combination of architectures.

	LID model feature extractor	training strategy	Test -3s (%) EER minDCF Cavg EER minDCF Cavg EER minDCF Cavg Test -10s (%) Test -30s (%)
	None		15.25 11.50 13.76 10.03	6.98	9.02 10.06	6.29	9.03
		E2E LID: FE+LID 11.43 10.03 11.34 5.58	4.85	5.93 3.03	2.28	2.92
	stacked DNNs	2-step: FE then LID	7.93	7.31	8.12 2.43	1.80	2.40 1.49	0.85	1.37
	TDNN	2-step then E2E LID 7.21 6.53 7.38 2.56	1.72 2.28 1.30 0.79	1.61
		E2E LID: FE+LID	7.30	6.05	7.21 2.73	2.28	3.17 1.53	0.80	1.27
	Conformer	2-step: FE then LID	8.01	6.81	8.42 3.08	1.84	2.53 0.91 0.38 0.77
		2-step then E2E LID 6.00 5.48 5.91 2.38 1.63 2.28 0.98	0.52	1.05
	None		13.37 11.58 13.67 7.34	4.79	6.82 7.99	4.49	5.27
		E2E LID: FE+LID 11.92 10.49 11.55 3.73	3.11	3.69 2.13	0.90	1.79
	stacked DNNs	2-step: FE then LID	7.47	6.32	7.21 1.69 1.17 1.69 0.88 0.35 0.62
		2-step then E2E LID 6.24 5.56 6.25 1.97	1.51	2.16 1.44	0.59	1.55
	ResNet-1D	E2E LID: FE+LID	6.51	5.67	6.56 1.63	1.02 1.50 0.70	0.33	0.59
		2-step: FE then LID	9.13	7.71	8.46 2.03	1.41	1.95 1.16	0.57	1.18
	Conformer	2-step then E2E LID 5.76 5.12 5.93 1.48	1.11	1.58 0.70 0.21 0.53
		multi-task λ = 0.5	5.99	4.81	5.91 2.13	1.16	2.03 1.26	0.67	1.31
		multi-task λ = 0 → 1 5.58	4.89	5.78 1.75	1.12	1.79 1.02	0.50	0.71
	three test sets, as compared to the best published results, [7] for
	3s and 10s segments and [2, 39] for 30s utterances.

Acknowledgements

Experiments presented in this paper were carried out using the Grid5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr). This work has been partly funded by the French Direction Générale de l'Armement.