Solène Evain

Ha Nguyen

Hang Le

Marcely Zanon Boito

Salima Mdhaffar

Sina Alisamir

Ziyi Tong

Natalia Tomashenko

Marco Dinarelli

Titouan Parcollet

Alexandre Allauzen

Yannick Estève
email: yannick.esteve@univ-avignon.fr

Benjamin Lecouteux

Solange Rossato

Fabien Ringeval

Didier Schwab

Laurent Besacier

LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech

Keywords: Self-Supervised Representation Learning, ASR, SLU, Speech Translation, Automatic Emotion Recognition

Self-Supervised Learning (SSL) using huge unlabeled data has been successfully explored for image and natural language processing. Recent works also investigated SSL from speech. They were notably successful to improve performance on downstream tasks such as automatic speech recognition (ASR). While these works suggest it is possible to reduce dependence on labeled data for building efficient speech systems, their evaluation was mostly made on ASR and using multiple and heterogeneous experimental settings (most of them for English). This questions the objective comparison of SSL approaches and the evaluation of their impact on building speech systems. In this paper, we propose LeBenchmark: a reproducible framework for assessing SSL from speech. It not only includes ASR (high and low resource) tasks but also spoken language understanding, speech translation and emotion recognition. We also focus on speech technologies in a language different than English: French. SSL models of different sizes are trained from carefully sourced and documented datasets. Experiments show that SSL is beneficial for most but not all tasks which confirms the need for exhaustive and reliable benchmarks to evaluate its real impact. LeBenchmark is shared with the scientific community for reproducible research in SSL from speech.

Introduction

Self-Supervised Learning (SSL) based on huge unlabeled data has been explored successfuly for image processing [START_REF] Bachman | Learning representations by maximizing mutual information across views[END_REF][START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF] and Natural Language Processing (NLP) [START_REF] Devlin | BERT: pre-training of deep bidirectional transformers for language understanding[END_REF]. Recently, pioneering work investigated SSL from speech, and successfully improved performance on downstream tasks such as speech recognition in low-resource scenarios [START_REF] Baevski | Effectiveness of self-supervised pre-training for speech recognition[END_REF][START_REF] Kawakami | Learning robust and multilingual speech representations[END_REF]. One observation that can be made about those recent studies on SSL for speech is that, as common benchmarks are not experimented, comparison of different SSL approaches are difficult to make. In addition, contributions have mostly been done on English, with a few recent studies related to multilingual SSL [START_REF] Conneau | Unsupervised cross-lingual representation learning for speech recognition[END_REF][START_REF] Wang | Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation[END_REF]. We propose to remedy these shortcomings by providing a reproducible benchmark 1 that includes:

1 https://github.com/LeBenchmark/ • a large and heterogeneous collection of French speech utterances (read, prepared, and spontaneous);

• pre-trained SSL models learnt on collections of 1k and 3k hours of French speech;

• assessments on Speech Recognition (ASR), Spoken Language Understanding (SLU), Speech Translation (AST) and Emotion Recognition (AER) in French.

Background

Most deep learning methods highly rely on large quantities of labeled training data. Particularly, current acoustic models require thousands of hours of transcribed speech to achieve stateof-the-art performance. However, this requirement cannot be fullfiled by the majority of the nearly 7,000 languages spoken worldwide. To overcome this, SSL has been recently proposed as an interesting alternative for data representation learning, as it requires less or no annotated data. Such learnt representations have been very successful in vision [START_REF] Bachman | Learning representations by maximizing mutual information across views[END_REF][START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF] and NLP [START_REF] Devlin | BERT: pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Peters | Deep contextualized word representations[END_REF]. Selfsupervised learning from speech consists of resolving pseudotasks, which do not require human annotation, as a pre-training for the real tasks to solve. These pseudo-tasks target predicting the next samples, or solving ordering problems. For instance, Autoregressive Predictive Coding (APC) considers the sequential structure of speech and predicts information about a future frame [START_REF] Chung | An unsupervised autoregressive model for speech representation learning[END_REF][START_REF] Chung | Improved speech representations with multi-target autoregressive predictive coding[END_REF], whereas Contrastive Predictive Coding (CPC) distinguishes a future speech frame from distractor samples [START_REF] Baevski | Effectiveness of self-supervised pre-training for speech recognition[END_REF][START_REF] Schneider | wav2vec: Unsupervised Pre-Training for Speech Recognition[END_REF], which is an easier learning objective compared to APC. Such representations have been shown to improve performance in several speech tasks [START_REF] Chung | Generative pre-training for speech with autoregressive predictive coding[END_REF], while being less sensitive to domain and/or language mismatch [START_REF] Kawakami | Learning robust and multilingual speech representations[END_REF]. It has also been shown that features extracted through a CPC pre-training can be transfered to other languages, with performance being on par or superior to a supervised pre-training [START_REF] Rivière | Unsupervised pretraining transfers well across languages[END_REF].

Gathering a Large and Heterogeneous Speech Collection in French

Recently, large multilingual corpora that include French have been made available, such as MLS [START_REF] Vineel | MLS: A large-scale multilingual dataset for speech research[END_REF] (1,096 h), or voxpopuli [START_REF] Wang | Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation[END_REF] (+4,500 h). However, these are restricted to either read or well-prepared speech, failing to provide diversity in 1).

We detail below the necessary steps for producing the dataset whose statistics are reported in Table 1.

Pre-processing for SSL training Audio recordings were segmented using time stamps from speech transcriptions. We also retrieved, when available, speaker labels and gender information. Following [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF], we removed utterances shorter than 1 s, and longer than 30 s. Finally, when necessary, audio segments were converted to mono PCM files using 16 bits and a sampling frequency of [START_REF] Bänziger | Introducing the Geneva multimodal expression corpus for experimental research on emotion perception[END_REF]

Training SSL Models

LeBenchmark provides four Wav2Vec2.0 models [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF] pretrained on the gathered French data described in Section 3. Following [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF], two different Wav2Vec2.0 architectures (large and base) are coupled with our small (S) and medium (M) corpus to form our set of Wav2Vec2.0 models: W2V2-Fr-S-base, W2V2-Fr-S-large, W2V2-Fr-M-base, W2V2-Fr-M-large. Hyperparameters and architectures for base and large are identical to the ones first introduced in [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF]. All models are trained on four Nvidia Tesla V100 (32GB) until the loss observed on the validation set of the MLS corpus (Section 3) reaches a stable point. Pretrained Wav2Vec2.0 models are shared with the community via HuggingFace2 for further integration with well-known toolkits such as Fairseq [START_REF] Ott | fairseq: A fast, extensible toolkit for sequence modeling[END_REF], SpeechBrain [START_REF] Ravanelli | Speechbrain[END_REF] or Kaldi [START_REF] Povey | The Kaldi speech recognition toolkit[END_REF]. In some downstream experiments mentioned below, we also use two Wav2Vec2.0 (base / large, no finetuning) English models pre-trained on the full LibriSpeech (960 h) corpus by Fairseq, 3 and refer to them as W2V2-En base and W2V2-En large. The XLSR-53-large [START_REF] Conneau | Unsupervised cross-lingual representation learning for speech recognition[END_REF] multilingual model is also used.

Benchmarking our SSL Models

Automatic Speech Recognition

We evaluate the contribution of SSL for ASR using a hybrid DNN-HMM and an end-to-end approach. Datasets The ASR tasks target two different types of corpora: Common Voice [START_REF] Ardila | Common voice: A massively-multilingual speech corpus[END_REF] and ETAPE [START_REF] Gravier | The ETAPE corpus for the evaluation of speech-based TV content processing in the French language[END_REF]. Hybrid DNN-HMM The baseline acoustic models (AM) have been trained on 40-dimensional high-resolution (hires) MFCC features using the Kaldi [START_REF] Povey | The Kaldi speech recognition toolkit[END_REF] toolkit with a state-of-the-art factorized time delay neural network (TDNN-F) [START_REF] Povey | Semi-orthogonal low-rank matrix factorization for deep neural networks[END_REF][START_REF] Peddinti | A time delay neural network architecture for efficient modeling of long temporal contexts[END_REF] on the ETAPE training corpus [START_REF] Gravier | The ETAPE corpus for the evaluation of speech-based TV content processing in the French language[END_REF] only. The model has 12 TDNN-F layers (1,024-dimensional, with projection dimension of 128) and a 3,432-dimensional output layer. It was trained using lattice-free maximum mutual information (LF-MMI) [START_REF] Povey | Purely sequence-trained neural networks for asr based on lattice-free mmi[END_REF] and

Results

The WER results on the ETAPE development and test data sets for the hybrid DNN-HMM models are given in Table 2. Among the models trained on SSL features, two models provide improvement over the baseline AM trained on MFCC features: the model trained on XLSR-53 features (7-8% of relative WER reduction) and the model trained on W2V2-Fr-M-large features (17-20% of relative WER reduction). To our knowledge, this is the first time SSL features are used for hybrid DNN-HMM ASR. Actually, the hybrid DNN-HMM ASR system is much better than its end-to-end counterpart on ETAPE (see next paragraph). This is partly due to the use of speaker adaptation (ivectors) and hand-crafted pronunciation dictionary which might be particularly beneficial to the hybrid DNN-HMM system, compared to end-to-end ASR, for the low resource ETAPE task. Table 3 presents the results achieved with end-to-end ASR on Common Voice 6.1 and on ETAPE datasets. On the ETAPE, filterbank parameters (MFB) got significanly the best results, while on Common Voice, W2V2-Fr-M-large is very close. In all (hybrid and end-to-end) ASR experiments, among the wav2vec models, W2V2-Fr-M-large got the best results.

Spoken Language Understanding

Dataset The MEDIA corpus [START_REF] Bonneau-Maynard | Results of the french evalda-media evaluation campaign for literal understanding[END_REF][START_REF] Dinarelli | A data efficient end-to-end spoken language understanding architecture[END_REF] is used for the French SLU benchmark. The corpus is made up of 12,908 utterances (41.5 h) for training, 1,259 utterances (3.5 h) for development and 3,005 utterances (11.3 h) for test. Model Our end-to-end model has a pyramidal LSTM encoder similar to [START_REF] Chan | Listen, attend and spell: A neural network for large vocabulary conversational speech recognition[END_REF]. The decoder integrates, in addition to the usual attention mechanism for attending the encoder hidden states, an attention mechanism for attending all previous decoder prediction's embeddings, instead of just the previous one [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF]. We use an incremental training strategy similar to [START_REF] Dinarelli | A data efficient end-to-end spoken language understanding architecture[END_REF], by first training an ASR model from scratch which is used to initialize parameters of a SLU model using a simple linear layer as decoder; and then using this simple SLU to initialize parameters of our final SLU model, which uses a LSTM decoder. The model, which is implemented using Fairseq [START_REF] Ott | fairseq: A fast, extensible toolkit for sequence modeling[END_REF], has the same settings as [START_REF] Dinarelli | A data efficient end-to-end spoken language understanding architecture[END_REF] to allow direct and fair comparison.

Results for ASR and SLU obtained with different speech representations are shown in Table 4, and they are given in terms of Word Error Rate (WER) and Concept Error Rate (CER) respectively, which is computed the same way as WER but on concept sequences. The ASR results are included because we use tokenlevel models (ASR) to pre-initialize SLU models. The ⊕ symbol is used for separating Encoder and Decoder names: Kheops is the pyramidal encoder inspired from [START_REF] Chan | Listen, attend and spell: A neural network for large vocabulary conversational speech recognition[END_REF], Basic is the linear decoder, and LSTM is the more advanced LSTM decoder. For ASR, using SSL features as input resulted in an impressive drop in WER, even when using English SSL models. At best, we achieve a WER of 11.77% on the development data with the W2V2-Fr-M-large features. SLU results (SLU decoding in Table 4) follow the same trend. The best performance is obtained again with W2V2-Fr-M-large features, with a CER of 18.54 on the development data. This result improves previous work by almost 5 points (23.39 vs. 18.54), and stands as the new stateof-the-art result using only MEDIA training data for learning SLU models. Better results have been obtained in [START_REF] Caubrière | Curriculum-Based Transfer Learning for an Effective End-to-End Spoken Language Understanding and Domain Portability[END_REF][START_REF] Tomashenko | Recent advances in end-to-end spoken language understanding[END_REF] by using more transcribed and annotated data, in addition to the MEDIA corpus, via transfer learning.

Speech-to-text Translation

Automatic speech-to-text translation (AST) consists in translating a speech utterance in a source language to a text in a target language. In this work, we are interested in translating from French to another language. Datasets We selected subsets having French as the source language in two large multilingual speech corpora: CoVoST-2 [START_REF] Wang | Covost 2 and massively multilingual speech-to-text translation[END_REF] and multilingual TEDx [START_REF] Salesky | The multilingual TEDx corpus for speech recognition and translation[END_REF].

Our benchmark covers translation directions from French to three target languages, English (en), Portugese (pt), and Spanish (es), with following training sizes: 50 h (TEDx/en), 38 h (TEDx/es), 25 h (TEDx/pt), and 180 h (CoVoST2/en).

Features We compared models using 80-dimensional MFB features and SSL representations. In addition to the four French Wav2Vec2.0 models trained in Section 4, we also considered the following off-the-shelf models: English [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF] (W2V2-En-base/large), French [START_REF] Wang | Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation[END_REF] (W2V2-Fr-VP-base/large), and the multilingual model XLSR-53 [START_REF] Conneau | Unsupervised cross-lingual representation learning for speech recognition[END_REF] (XLSR-53-large). For a fair comparison, we did not use additional data augmentation technique nor ASR encoder pre-training in the experiments. Models We trained Transformer [START_REF] Vaswani | Attention is all you need[END_REF] models using the FAIRSEQ S2T toolkit [START_REF] Wang | fairseq S2T: Fast speech-to-text modeling with fairseq[END_REF], and using a small architecture with 12-layers encoder, 6-layers decoder, and hidden dimension D = 256. For models using SSL features, we inserted a block of Linear-ReLU before convolutional layers not only to reduce the number of parameters [START_REF] Nguyen | Investigating self-supervised pre-training for end-to-end speech translation[END_REF], but also because we preliminary observed improved performance with this block.

Results shown in Table 5 highlight the benefit of SSL features only in medium and low-resource settings, namely mTEDx: our W2V2-Fr-M-large produces the best results across all language pairs, except for pt which is too low-resourced to obtain decent BLEU whatever the features used. In the higher-resource scenario (CV2), however, the best-performing SSL features are still 2.65 BLEU point below the MFB ones.

Automatic Emotion Recognition

Automatic emotion recognition aims at detecting human's apparent emotions from sensors such as microphones and cameras. Affective computing has many useful applications in the domain of health, education, art and entertainment. Datasets We used the RECOLA dataset [START_REF]Introducing the RECOLA Multimodal Corpus of Remote Collaborative and Affective Interactions[END_REF], which contains 3.8 h of noise-free recordings of spontaneous interactions between French-speaking subjects solving a collaborative task in Models We used a simple model based on a linear layer mapping features to one emotional dimension, followed by a Linear-Tanh function. The other model is a 1-layer GRU with the hidden layer D = [32, 64], followed by the Linear-Tanh function.

Adam optimiser was used and patience was set to 15 epochs, and the Concordance Correlation Coefficient [START_REF] Li | A Concordance Correlation Coefficient to evaluate reproducibility[END_REF] was used as loss function to train the models as in [START_REF] Weninger | Discriminatively trained Recurrent Neural Networks for continuous dimensional emotion recognition from audio[END_REF][START_REF] Trigeorgis | Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network[END_REF].

Results Best results are obtained by our W2V2-Fr-M-base representation on valence, satisfaction and arousal, c. f. Table 6. With a simpler model, best scores are also achieved on both data sets with the Wav2Vec features, meaning that SSL representations are rich enough to be used with a simple regressor, even for low-quality speech signals (telephone conversations).

Discussion

After training our own SSL models for French, we evaluated them for four speech tasks (ASR, SLU, AST, and AER) using different architectures (shallow and deep architectures, end-toend or not). The learnt SSL models are particularly beneficial for lower resource tasks (SLU, AST/TEDx, AER) or with simpler NN architectures (AER) but they sometimes fail providing a benefit compared to MFB or MFCCs (End-to-end ASR). Finetuning of SSL models could probably help bridging the gap remaining for some tasks, but we used SSL features 'as they are' for this paper. Furthermore, efficient data augmentations techniques for Mel Filterbanks such as SpecAugment were disabled here to provide a comparison with SSL features, so we should highlight that we did not make the best of our Mel Filterbanks. All of these remarks and findings advocate for more exhaustive and reliable evaluations to assess the real impact of SSL for speech systems. We hope that decentralized projects such as LeBenchmark will contribute to this goal.

Table 1 :

 1 Statistics for the speech corpora used to train SSL models according to gender information (male / female / unknown). The small dataset (1k hours) is from MLS only, and the medium dataset (2.9k hours) is from all of them; duration: hour(s):minute(s).

	Corpus	# Utterances	Duration	# Speakers	Mean Utt. Duration	Speech type
	African Accented French [8]	16,402 373 / 102 / 15,927	18:56 -/ -/ 18:56	232 48 / 36 / 148	4 s -/ -/ -	Read
	Att-Hack [9]	36,339 16,564 / 19,775 / 0	27:02 12:07 / 14:54 / 0:00	20 9 / 11 / 0	2.7 s 2.6 s / 2.7 s / -	Acted Emotional
	CaFE [10]	936 468 / 468 / 0	1:09 0:32 / 0:36 / 0:00	12 6 / 6 / 0	4.4 s 4.2 s / 4.7 s / -	Acted Emotional
	CFPP2000* [11] [12]	9853 166 / 1,184 / 8,503	16:26 0:14 / 1:56 / 14:16	49 2 / 4 / 43	6.0 s 5.0 s / 5.0 s / 6.0 s	Spontaneous
	ESLO2 [13], [14]	62,918 30,440 / 32,147 / 331	34:12 17:06 / 16:57 / 0:09	190 68 / 120 / 2	1.9 s 2.0 s / 1.9 s / 1.7 s	Spontaneous
	EPAC** [15]	623,250 465,859 / 157,391 / 0	1,626:02 1,240:10 / 385:52 / 0:00	1,935 -/ -/ -	9 s -/ -/ -	Radio Broadcasts
	GEMEP [16]	1,236 616 / 620 / 0	0:50 0:24 / 0:26 / 0:00	10 5 / 5 / 0	2.5 s 2.4 s / 2.5 s / -	Acted Emotional
	MLS French [17]	263055 124,590 / 138,465 / 0	1,096:43 520:13 / 576:29 / 0:00	178 80 / 98 / 0	15.0 s 15.0 s / 15.0 s / -	Read
	MPF [18], [19]	19,527 5,326 / 4,649 / 9,552	19:06 5:26 / 4:36 / 9:03	114 36 / 29 / 49	3.5 s 3.7 s / 3.6 s / 3.4 s	Spontaneous
	PORTMEDIA (French) [20]	19,627 9,294 / 10,333 / 0	38:59 19:08 / 19:50 / 0:00	193 84 / 109 / 0	7.1 s 7.4 s / 6.9 s / -	Acted telephone dialogue
	TCOF (Adult s) [21]	58,722 10,377 / 14,763 / 33,582	53:59 9:33 / 12:39 / 31:46	749 119 / 162 / 468	3.3 s 3.3 s / 3.1 s / 3.4 s	Spontaneous
	ALL	1,111,865 664,073 / 379,897 / 67,895	2,933:18 1,824:53 / 1034:15 / 74:10	-	-	-
	*version without the CEFC corpus v2.1, 02/2021; **speakers are not uniquely identified.
	the speech samples, such as accented, spontaneous and/or af-
	fective speech. As a consequence, SSL models trained only
	on these corpora may present poor generalisation abilities on

spontaneous or affective speech. In this work, we gathered a large variety of speech corpora in French that cover different accents (MLS, African Accented Speech, CaFE), acted emotions (GEMEP, CaFE, Att-Hack), telephone dialogues (PORT-MEDIA), read (MLS, African Accented French) and spontaneous sentences (CFPP2000, ESLO2, MPF, TCOF), as well as broadcast speech (EPAC). Compared to MLS and Voxpopuli, our dataset is more diverse, carefully sourced and contains detailed metadata (speech type, and speaker gender), which would facilitate future fine-grained analysis of SSL such as training gender/style specific models. Moreover, our dataset has a more realistic representation of speech turns in real life, compared to MLS (see average utterance duration per speaker in Table

  kHz.Final dataset To the best of our knowledge, this is the very first study that explores such a diverse and large ensemble of datasets for SSL training. It includes 2,933 h of speech, from which 1,115 h is read speech, 1,626 h broadcast speech, 124 h spontaneous speech, 38 h acted telephone dialogues, and 29 h acted emotional speech. Regarding gender, we collected 1,824 h of speech from male speakers, 1,034 h from female speakers, and 74 h from unknown gender. This full corpus is refered as medium, and the subset made of MLS only is refered as small.

Table 2 :

 2 ASR results (WER,%) on the ETAPE corpus for hybrid DNN-HMM acoustic models with TDNN-F topology.

	Language Model	ETAPE	ESTER-1.2 + EPAC
	Features	Dev	Test	Dev	Test
	hires MFCC	39.28	40.89	35.60	37.73
	W2V2-Fr-M-large	32.19	33.87	28.53	30.77
	W2V2-En-large	39.93	42.30	36.18	38.75
	XLSR-53-large	36.36	38.19	32.81	35.17
	cross-entropy criteria. Speed and volume perturbations have
	been applied for data augmentation. We used a similar topol-
	ogy to train three other systems with different types of input
	features extracted by W2V2-Fr-M-large, W2V2-En-large [28],
	and XLSR-53-large models. 100-dimensional speaker i-vectors
	were appended to the input features for all the models. Two tri-
	gram LMs were used in evaluation: (1) a larger one with a 82k
	vocabulary and (2) a smaller one trained on ETAPE training
	data only with a 17.5k vocabulary.
	End-to-End Our end-to-end system is defined with the Speech-
	Brain toolkit [30] using an encoder/decoder architecture with
	attention: the encoder is a Convolutionnal Recurrent Deep Neu-
	ral Network CRDNN (VGG + RNN + DNN), and the decoder is
	a joint CTC/Attention LSTM neural network. When used with
	the Wav2Vec2.0 features (same from hybrid DNN-HMM ASR
	experiments), the CNN blocks are removed from the CDRNN

encoder. For end-to-end ASR experiments, the neural network output corresponds to 500 byte pair encoding (BPE) units

[START_REF] Sennrich | Neural machine translation of rare words with subword units[END_REF]

computed on the manual transcriptions of the respective training datasets. No additional language model is used in these experiments, neither data augmentation. For comparison purposes, we also use 80-dimension log Mel filterbank (MFB) features.

Table 3 :

 3 End-to-end ASR results (WER,%) on Common Voice and ETAPE corpora. ( * ) means the training algorithm did not converge to a WER smaller than 100%.

	Corpus	CommonVoice	ETAPE
	Features	Dev	Test	Dev	Test
	MFB	20.19	23.40	54.55	56.17
	W2V2-Fr-M-large	20.23	24.06	55.56	57.04
	W2V2-En-large	34.07	37.29	98.79	99.10
	XLSR-53-large	30.07	32.72	( * )	( * )

Table 6 :

 6 AER results (Concordance Correlation Coefficient of emotion predictions) on the RECOLA and AlloSat test sets. Features We extracted 40-dimensional MFB features that were standardized to zero mean and unit standard deviation according to the training set, and SSL features that were pre-processed by a normalisation layer. Annotations were resampled to match the sampling frequency of the features, which was 100 Hz for MFB and 50 Hz for the Wav2Vec models.

		Corpus	RECOLA	AlloSat
	Model	Feature	Arousal	Valence	Satisfaction
	Linear-Tanh	MFB	0.192	0.075	0.065
	Linear-Tanh	W2V2-Fr-M-base	0.385	0.090	0.193
	Linear-Tanh	W2V2-En-large	0.328	-0.005	0.075
	Linear-Tanh	XLSR-53-large	0.155	0.024	0.093
	GRU-32	MFB	0.654	0.252	0.437
	GRU-32	W2V2-Fr-M-base	0.767	0.376	0.507
	GRU-32	W2V2-En-large	0.629	-0.032	0.334
	GRU-32	XLSR-53-large	0.605	0.320	0.446
	GRU-64	MFB	0.712	0.307	0.400
	GRU-64	W2V2-Fr-M-base	0.760	0.352	0.507
	GRU-64	W2V2-En-large	0.635	0.015	0.367
	GRU-64	XLSR-53-large	0.585	0.280	0.434
	remote condition -training, development and test partitions in-
	clude each one third of the data, and AlloSat [50], a more recent
	corpus containing 37 h of real-life call center conversations in
	French -training: 25.6 h, development: 5.8 h, and test: 6.0 h.
	Both datasets are annotated by several annotators using time-
	continuous dimensions which are averaged to define an emo-
	tion gold-standard: arousal (from passive to active) and valence
	(from negative to positive) for RECOLA, and a dimensional
	axis ranging from frustration to satisfaction for AlloSat.

https://huggingface.co/LeBenchmark

https://github.com/pytorch/fairseq/tree/ master/examples/wav2vec

Acknowledgements

This work was performed using HPC resources from GENCI-IDRIS (Grant 2020-A0091012047). It was also partially supported by MIAI@Grenoble-Alpes (ANR-19-P3IA-0003) and partially funded by the European Commission through the SELMA project under grant number 957017.