Solène Evain

Ha Nguyen

Hang Le

Marcely Zanon Boito

Salima Mdhaffar

Sina Alisamir

Ziyi Tong

Natalia Tomashenko

Marco Dinarelli

Titouan Parcollet

Alexandre Allauzen

Yannick Estève

Benjamin Lecouteux

François Portet

Solange Rossato

Fabien Ringeval

Didier Schwab

Laurent Besacier

Task Agnostic and Task Specific Self-Supervised Learning from Speech with LeBenchmark

Self-Supervised Learning (SSL) has yielded remarkable improvements in many different domains including computer vision, natural language processing and speech processing by leveraging large amounts of unlabeled data. In the specific context of speech, however, and despite promising results, there exists a clear lack of standardization in the evaluation process for comprehensive comparisons of these models. This issue gets even worse with the investigation of SSL approaches for other languages than English. We present LeBenchmark, an open-source and reproducible framework for assessing SSL from French speech data. It includes documented, large-scale and heterogeneous corpora, seven pretrained SSL wav2vec 2.0 models shared with the community, and a clear evaluation protocol made of four downstream tasks along with their scoring scripts: automatic speech recognition, spoken language understanding, automatic speech translation and automatic emotion recognition. For the first time, SSL models are analyzed and compared on the latter domains both from a task-agnostic (i.e. frozen) and task-specific (i.e. fine-tuned w.r.t the downstream task) perspectives. We report state-of-the-art performance on most considered French tasks and provide a readable evaluation set-up for the development of future SSL models for speech processing.

Introduction

Self-Supervised Learning (SSL) based on huge amounts of unlabeled data has been explored successfully for image and natural language processing [START_REF] Bachman | Learning representations by maximizing mutual information across views[END_REF][START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF][START_REF] Devlin | BERT: pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF]. Recently, researchers investigated SSL from speech as well and successfully improved performance on downstream tasks such as speech recognition [START_REF] Baevski | Effectiveness of self-supervised pre-training for speech recognition[END_REF][START_REF] Kawakami | Learning robust and multilingual speech representations[END_REF]. As SSL from speech is a rapidly evolving domain, new models are unfortunately evaluated on different datasets, most of which focus on the English language. In order to carefully assess the progress of speech SSL model-wise and application-wise, common benchmarks are needed. While NLP benchmarking is now widely discussed [START_REF] Ruder | Challenges and Opportunities in NLP Benchmarking[END_REF], multi-task benchmarks are less common in speech despite the fact that the field has a long tradition of evaluation (see for instance long-term NIST and DARPA shared tasks for ASR). We propose to contribute to this by providing a reproducible and multifaceted benchmark for evaluating speech SSL models. By benchmark, and following the definition of [START_REF] Schlangen | Targeting the benchmark: On methodology in current natural language processing research[END_REF], we mean an ensemble of tasks that allow to discriminate learners (i.e. SSL models) based on their ability to perform well on those tasks. We propose an initial set of four main tasks (10 sub-tasks overall), measuring specific speech challenges in French language: Automatic Speech Recognition (ASR), Spoken Language Understanding (SLU), Speech Translation (AST) and Emotion Recognition (AER). This enables to assess the impact of pre-trained speech models that differ along several dimensions: language used for pre-training (French, English, multilingual), amount of raw speech used for SSL pre-training (1k, 3k or 7k hours), model size (base, large). For reproducibility, we also provide pre-trained SSL models learned on a large and heterogeneous collection of speech utterances and believe this is a strong contribution to speech technologies in French. This work extends a preliminary proposal [START_REF] Evain | LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech[END_REF] with a bigger speech corpus for SSL, more SSL models evaluated and shared, as well as experiments comparing task agnostic models (i.e. SSL models trained with pre-training objective on general purpose data) and task specific models (i.e. SSL models obtained after task-adaptive pre-training [START_REF] Gururangan | Don't stop pretraining: Adapt language models to domains and tasks[END_REF] or after fine-tuning for an ASR task). Our website shares models, scripts and results for better transparency and reproducibility of research in speech SSL. 12 Background SSL has been recently proposed as an interesting alternative for data representation learning, as it requires no annotated data. Such learned representations have been very successful in vision [START_REF] Bachman | Learning representations by maximizing mutual information across views[END_REF][START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF] and NLP [START_REF] Devlin | BERT: pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Peters | Deep contextualized word representations[END_REF]. SSL from speech consists of resolving pseudo-tasks, which do not require human annotation, as a pre-training for the real tasks to solve. These pseudo-tasks target predicting the next samples, or solving ordering problems. For instance, Autoregressive Predictive Coding (APC) considers the sequential structure of speech and predicts information about a future frame [START_REF] Chung | An unsupervised autoregressive model for speech representation learning[END_REF][START_REF] Chung | Improved speech representations with multi-target autoregressive predictive coding[END_REF], whereas Contrastive Predictive Coding (CPC) distinguishes a future speech frame from distractor samples [START_REF] Baevski | Effectiveness of self-supervised pre-training for speech recognition[END_REF][START_REF] Schneider | wav2vec: Unsupervised Pre-Training for Speech Recognition[END_REF], which is an easier learning objective compared to APC. Such representations have been shown to improve performance in several speech tasks [START_REF] Chung | Generative pre-training for speech with autoregressive predictive coding[END_REF], while being less sensitive to domain and/or language mismatch [START_REF] Kawakami | Learning robust and multilingual speech representations[END_REF] and being transferable to other languages [START_REF] Riviere | Unsupervised pretraining transfers well across languages[END_REF]. In 2020, a strong speech SSL baseline appeared: the Wav2Vec2.0 model [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF] which relies on the CPC idea of [START_REF] Baevski | Effectiveness of self-supervised pre-training for speech recognition[END_REF][START_REF] Schneider | wav2vec: Unsupervised Pre-Training for Speech Recognition[END_REF] but with discrete speech units that are used as latent representations and fed to a Transformer network to build contextualized representations. Several other bi-directional encoders were also proposed recently: Speech-XLNet [START_REF] Song | Speech-xlnet: Unsupervised acoustic model pretraining for self-attention networks[END_REF], Mockingjay [START_REF] Liu | Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders[END_REF] and [START_REF] Wang | Unsupervised pre-training of bidirectional speech encoders via masked reconstruction[END_REF]. A few recent studies were also related to multilingual SSL models trained on very large multilingual corpora [START_REF] Conneau | Unsupervised cross-lingual representation learning for speech recognition[END_REF][START_REF] Wang | Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation[END_REF].

While there are multiple evaluation benchmarks to assess pretrained models in NLP (see for instance [START_REF] Wang | GLUE: A multi-task benchmark and analysis platform for natural language understanding[END_REF] for English, [START_REF] Le | Flaubert: Unsupervised language model pre-training for french[END_REF] for French and [START_REF] Park | KLUE: korean language understanding evaluation[END_REF] for Korean), we are aware of only two similar initiatives for speech SSL models' evaluation: our own preliminary work [START_REF] Evain | LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech[END_REF] and the Speech processing Universal PERformance Benchmark (SUPERB) [START_REF] Yang | SUPERB: speech processing universal performance benchmark[END_REF] which however targets English language only and does not share pre-trained SSL models as we do.

Gathering a Large and Heterogeneous Speech Collection in French

Recently, large multilingual corpora that include French have been made available, such as MLS [START_REF] Vineel Pratap | Mls: A large-scale multilingual dataset for speech research[END_REF] (1,096 h), or voxpopuli [START_REF] Wang | Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation[END_REF] (+4,500 h). However, these are restricted to either read or well-prepared speech, failing to provide diversity in the speech samples, such as accented, spontaneous and/or affective speech. In this work, we gathered a large variety of speech corpora in French that cover different accents (MLS, African Accented Speech, CaFE), acted emotions (GEMEP, CaFE, Att-Hack), telephone dialogues (PORTMEDIA), read (MLS, African Accented French, MaSS) and spontaneous sentences (CFPP2000, ESLO2, MPF, TCOF, NCCFr), broadcast speech (EPAC) and professional speech (Voxpopuli). Compared to MLS and Voxpopuli, our dataset is more diverse, carefully sourced and contains detailed metadata (speech type, and speaker gender). Moreover, it has a more realistic representation of speech turns in real life, compared to MLS and VoxPopuli (see average utterance duration in Table 1). Statistics are reported in Table 1.

Pre-processing for SSL training: Recordings were segmented using time stamps from transcriptions. We retrieved, when available, speaker labels and gender information. Following [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF], we removed utterances shorter than 1 s, and longer than 30 s. When possible, overlapping speech sentences were also removed. When necessary, audio segments were converted to mono PCM 16 bits, 16 kHz.

Small dataset (≈ 1k hours) is only composed of the MLS corpus for comparison with Wav2Vec2.0 [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF] which uses only read English speech. It is also gender balanced.

Training and Sharing SSL Models

LeBenchmark provides seven Wav2Vec2.0 models [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF] pretrained on the gathered French data described in Section 3. Following [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF], two different Wav2Vec2.0 architectures (large and base) are coupled with our small (1K), medium (3K) and large (7K) corpus to form our set of Wav2Vec2.0 models: W2V2-Fr-1K-base, W2V2-Fr-1K-large, W2V2-Fr-3K-base, W2V2-Fr-3K-large, W2V2-Fr-7K-base, W2V2-Fr-7K-large. We also provide a specific model (W2V2-Fr-2.7K-base) trained on a subset of our medium set only containing MLS and EPAC (2.7K hours of audio) to enable further investigation on the impact of spontaneous speech on SSL representations.

Hyperparameters and architectures for base2 and large3 are identical to the ones first introduced in [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF]. W2V2-Fr-1K, W2V2-Fr-3K, W2V2-Fr-2.7K and W2V2-Fr-7K are trained respectively for 200K, 500K, 500K and 500K updates on 4, 32, 32 and 64 Nvidia Tesla V100 (32GB), with one update corresponding to a call to the .backward() function in PyTorch. Detailed summary of the hyperparameters used to train our SSL models can be found in Table 2. In practice, training is stopped at a round number of updates once the loss observed on the development set of the MLS corpus reaches a stable point (learning curves are given in Appendix A.1).

Pre-trained Wav2Vec2.0 models are shared with the community via HuggingFace4 for further integration with well-known toolkits such as SpeechBrain [START_REF] Ravanelli | Speechbrain[END_REF], Fairseq [START_REF] Ott | fairseq: A fast, extensible toolkit for sequence modeling[END_REF] or Kaldi [START_REF] Povey | The kaldi speech recognition toolkit[END_REF].

Pre-existing Wav2Vec2.0 models obtained from Fairseq5 are also considered in downstream experiments. First, XLSR-53-large is used as a comparison to multilingual models. Then, W2V2-En-base and W2V2-En-large (LS960) are used to assess English representations from LibriSpeech.6

Benchmarking SSL Models

We benchmark SSL models on four different tasks (ASR, SLU, AST and AER) chosen with respect to following criteria: (a) diversity of problems: regression (AER), sequence labelling (SLU) and conditional natural language generation (ASR, AST), (b) diversity of information extracted: transcript (ASR), semantics (SLU), translation (AST) and paralinguistics (AER), and (c) diversity of annotated resources available for downstream task: large (ASR), medium (SLU, AST), small (AER).

As our goal is to evaluate the impact of SSL for the best baselines for each task addressed, we have a different architecture for each task and it corresponds to the best baseline performance we could obtain using MFCC/FBANK features. As a different architecture/approach is used for each task, we evaluate the different SSL models as feature extractors for these tasks. These 'SSL extractors' are either 'task agnostic' or 'task specific' (SSL models fine-tuned on the task data) as further explained below.

Automatic Speech Recognition (ASR)

SSL for ASR is evaluated using both hybrid DNN-HMM and end-to-end approaches. In addition to the source code used to make these ASR experiments (training + decoding), LeBenchmark provides a normalization script for French output text derived from the one applied during the official French ESTER and ETAPE evaluation campaigns [START_REF] Gravier | The etape corpus for the evaluation of speech-based tv content processing in the french language[END_REF] and a unique script to compute the Word Error Rate (WER) from ASR output.

Datasets

The ASR tasks target two different types of corpora: Common Voice [START_REF] Ardila | Common voice: A massively-multilingual speech corpus[END_REF] and ETAPE [START_REF] Gravier | The etape corpus for the evaluation of speech-based tv content processing in the french language[END_REF].

Common Voice is a very large crowd-sourced corpus (477 h) of read speech in French with transcripts -train: 428 h, dev: 24 h, and test: 25 h, while ETAPE is a smaller (36 h) but more challenging corpus composed of diverse French TV broadcast programs -train: 22 h, dev: 7 h, and test: 7 h.

Hybrid DNN-HMM The acoustic models (AM) are trained on 40-dimensional high-resolution (hires) MFCC features or SSL features using the Kaldi toolkit [START_REF] Povey | The kaldi speech recognition toolkit[END_REF] with a state-of-the-art factorized time delay neural network (TDNN-F) architecture [START_REF] Povey | Semi-orthogonal low-rank matrix factorization for deep neural networks[END_REF][START_REF] Peddinti | A time delay neural network architecture for efficient modeling of long temporal contexts[END_REF] on the ETAPE training corpus [START_REF] Gravier | The etape corpus for the evaluation of speech-based tv content processing in the french language[END_REF] only. More details about the AM architecture are given in Appendix A.2.1. Two trigram LMs were used in End-to-End Our end-to-end (e2e) systems are implemented with SpeechBrain toolkit [START_REF] Ravanelli | Speechbrain[END_REF]. The baseline e2e system is fed by 80-dimension log Mel filterbank (MFB) features and based on an encoder/decoder architecture with attention. When used with a SSL pre-trained Wav2Vec2.0 model, the e2e system simply adds an additional hidden layer and an output layer on top of Wav2Vec2.0 architecture. Details are given in Appendix A.2.2.

Results

The WER results on the ETAPE development and test data sets for the hybrid DNN-HMM models are given in Table 3. Among the models trained on SSL features (Table 3, (a)) 6 models provide improvement over the baseline AM trained on MFCC features: XLSR-53, Fr-2.7k-base, Fr-3k-base, Fr-3k-large, Fr-7k-base, and Fr-7k-large. The best SSL features are the ones from the Fr-7k models and they clearly outperform the multilingual XLSR-53-large. In the case of task-specific pre-training, 7 we were not able to significantly improve the best results compared to task-agnostic pre-training. This is probably due to the fact that the obtained representations are not very different in both cases. These results can be compared to the ones obtained by the best ASR system during the official ETAPE shared task: by using 511h of training data (external training data were allowed), their ASR system got a word error rate of 23.6% [START_REF] Bougares | Lium asr system for etape french evaluation campaign: experiments on system combination using open-source recognizers[END_REF], while in the experiments presented in this paper, only 22h of ETAPE training data were used. In the next paragraph, we investigate e2e fine-tuning of the models using transcribed speech. creating an open-domain SLU application, many works focus on specific domains. We focus on the hotel information and reservation domain provided within the French corpus MEDIA [START_REF] Bonneau-Maynard | Results of the french evalda-media evaluation campaign for literal understanding[END_REF][START_REF] Quarteroni | What's in an ontology for spoken language understanding[END_REF]. This corpus is made of 1 250 human-machine dialogues acquired with a Wizard-of-Oz approach, where 250 users followed 5 different reservation scenarios. Spoken data were manually transcribed and annotated with domain concepts, following a rich ontology. The official corpus split is made up of 12,908 utterances (41.5 h) for training, 1,259 utterances (3.5 h) for development and 3,005 utterances (11.3 h) for test. We note that, while all turns have been manually transcribed and can be used to train ASR models, only user turns have been annotated with concepts and can be used to train SLU models. This results in only 41.5 hours of speech training data for ASR models, and only 16.8 hours for SLU models.

Experiments. All our models are based on LSTM [START_REF] Hochreiter | Long short-term memory[END_REF] seq2seq with attention [START_REF] Bahdanau | Neural Machine Translation by Jointly Learning to Align and Translate[END_REF]. Model details and training strategy are described in Appendix A.2.3. We use a total of 3 bidirectional LSTM layers of size 256 stacked in a pyramidal fashion in our encoder and the LSTM decoder has 2 layers of size 256. In addition to using spectrogram features and features from task agnostic SSL models, we also use features from task specific models (SLU on MEDIA). Two types of task-specific pre-training are performed: self-supervised which consists in resuming the SSL model training using the MEDIA training data and minimizing the Wav2Vec 2.0 loss ('(b) self-supervised on MEDIA' in the table, also called task-adaptive pre-training in [START_REF] Gururangan | Don't stop pretraining: Adapt language models to domains and tasks[END_REF]); and ASR supervised ((c) fine-tuned for ASR on MEDIA in the table) which consists in fine-tuning the full SSL model for a supervised downstream task with a CTC loss minimization objective [START_REF] Graves | Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks[END_REF]. In this work we chose to fine-tune models with respect to the ASR task on MEDIA (not the SLU one) to see how it compares to self-supervised fine-tuning. We leave fine-tuning with respect to SLU for future work.

Results for SLU obtained with different speech representations are shown in Table 5. They are given in terms of Concept Error Rate (CER), computed the same way as Word Error Rate (WER) but on concept sequences. CER are accompanied by standard deviations (in gray), computed with the bootstrap method of [START_REF] Bisani | Bootstrap estimates for confidence intervals in ASR performance evaluation[END_REF]. We provide ASR results in supplementary material (table 10). We first note that our spectrogram baseline obtains a substantial improvement over the one in [START_REF] Evain | LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech[END_REF]. Such gain is due to the slightly different settings and model architecture described in the Appendix. Using SSL model features as input resulted in an impressive drop in CER, even when using English SSL models (CER from 31.10 to 20.84 on the test set with the base model). At best, among task-agnostic pre-trained models, we achieve a CER of 15.95 on the test data with Fr-3K-large features. Surprisingly, using features from the model trained with 7k hours of speech (Fr-7K-large), results are worse on both dev and test. In contrast, the 7k-model led to the best results in terms of ASR evaluation (see Table 10 in the Appendix). We performed task-specific pre-training only with the most effective SSL models: French 3k and 7k models and multi-lingual XLSR-53-large. The best overall pre-trained model is the 7k-model fine-tuned for ASR on MEDIA, though results are close to those obtained with features from the 3k-model (13.97 vs. 13.78). Indeed, significance tests in table 11 in the Appendix confirm that these two models are equivalent and they are significantly better than all the others. This shows that pre-trained SSL speech models can be specialized using task specific pre-training with either self-supervised learning on raw speech (block (b) in the table), or fine-tuning on raw speech and associated transcripts (block (c) in the table), the latter being slightly better than the former.

Automatic Speech-to-text Translation (AST)

Automatic speech-to-text translation (AST) consists in translating a speech utterance in a source language to a text in a target language. In this work, we are interested in translating directly from French speech to text in another language.

Dataset We selected subsets having French as the source in the multilingual TEDx dataset [START_REF] Salesky | The multilingual tedx corpus for speech recognition and translation[END_REF]. Our benchmark covers translation directions from French to three target languages: English (en), Spanish (es), and Portuguese (pt), with following training sizes 50 h (en), 38 h (es), and 25 h (pt).

Experiments Our baselines are models using 80-dimensional MFB features. For learned representations derived from SSL models, we focused on the feature extraction approach where features are extracted from either task-agnostic or task-specific pre-training. Task-agnostic pre-training refers to the direct use of SSL models as feature extractors whereas task-specific method consists in one additional phase where the SSL models are further trained on the in-domain task data, with (supervised fine-tuned) or without (self-supervised fine-tuned) labels. We performed supervised fine-tuning with speech transcriptions as labels and leave supervised fine-tuning with AST data for future work. In the task-specific scenario, we only considered three SSL models: two best French SSL models (Fr-3K-large and Fr-7K-large) and one best non-French SSL model (XLSR-53-large). Since the French speech is overlapped between the language pairs, we selected the pair having the most speech data (fr-en) to perform task-specific pre-training and used the obtained models to extract features for the remaining pairs (fr-es and fr-pt). For a fair comparison, we did not use additional data augmentation technique nor ASR encoder pre-training in the experiments. We refer to Appendix A.2.4 for details on the model architecture and implementation.

Results Table 6 displays the results of AST experiments. One can observe that SSL features, whether task-agnostic or task-specific and whether being pre-trained on English, French, or multilingual data, outperform the baselines using MFB features by a large margin (except for the task-agnostic multilingual model XLSR-53 on the two pairs fr-es and fr-pt, which are in very low-resource settings). Among the three groups using SSL features (task-agnostic pre-training, task-specific self-supervised, and task-specific fine-tuned for ASR), the ASR fine-tuning approach (c) yields the best results. We observe considerable improvements from task-specific self-supervised (b) to task-specific finetuned (c) (+6.19, +8.50, +8.53 on average for en, es, and pt, respectively) while the benefits of using self-supervised fine-tuning compared to task-agnostic pre-training are only marginal or even slightly negative. The substantial gains when using supervised fine-tuning approach (even with a somehow indirect signal which is transcripts for the AST downstream task) shows that giving more signals of the task-specific data to the SSL models is helpful. In particular, in the case of task-specific self-supervised fine-tuning (b), we further trained the SSL models for more steps on the raw task-specific data whereas in ASR fine-tuned scenario (c), we used raw data plus the transcripts to guide the SSL models. Focusing on task-agnostic block (a), we see that French SSL models clearly outperform those pre-trained on English and multilingual data. Multilingual XLSR-53 model surpasses the English models on fr-en, yet all of them fail to generate meaningful translations on fr-es and fr-pt where little training data is available. Comparing across different French SSL model sizes (base vs. large), the large architecture yields considerable improvement (nearly 3 to 6 BLEU points) over its base counterpart. When looking into the French SSL models with different amounts of pre-training data (1K, 2.7K, 3K, and 7K), we observe large gains for the base architecture from using 1K to using 2.7K or more pre-training data. There is, however, no significant difference between base models using 2.7K, 3K, and 7K data. Using 7K data even hurts the performance on the pair fr-pt. On the other hand, for the large network, using more data consistently improves the performance on all language pairs. Finally, moving on to task-specific models, Fr-7K-large is the best-performing model (or being on par with the best one) in each group. Noticeably, there is a huge improvement when using the ASR fine-tuning approach (c) for the multilingual XLSR-53 model. The method considerably boosts the performance of the multilingual model (compared to using it directly or further pre-training it on the task data) and makes it even on par with the best French SSL models.

Automatic Emotion Recognition (AER)

Automatic Emotion Recognition (AER) research mostly relies on detecting either different emotion categories such as happiness or sadness, or different emotion dimensions such as arousal and valence. Here, we use sequence-to-sequence models on continuous dimensions of emotion.

Datasets We use RECOLA [START_REF] Ringeval | Introducing the recola multimodal corpus of remote collaborative and affective interactions[END_REF] and AlloSat [START_REF] Macary | Allosat: A new call center french corpus for satisfaction and frustration analysis[END_REF] datasets as in [START_REF] Evain | LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech[END_REF]. RECOLA is a well-known corpus for benchmarking emotion recognition systems, which contains recordings of spontaneous interactions between French-speaking subjects in lab environments. AlloSat is a more recent dataset that contains real-life call center conversations in French. Both datasets are time-continuously annotated by several annotators. The different annotations are averaged to define an emotional dimension gold-standard: arousal (from passive to active) and valence (from negative to positive) for RECOLA with a sampling rate of 25 Hz, and a dimensional axis ranging from frustration to satisfaction for AlloSat with a sampling rate of 4 Hz.

Experiments In addition to using SSL features, we extracted 40-dimensional MFB features normalized to have zero mean and unit standard deviation over the training set. We used simple regression models similar to the ones presented in [START_REF] Evain | LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech[END_REF]. The LinTh model only consists of a linear layer followed by a tangent hyperbolic function and the GRU models are 1-layer GRU with the hidden layer D = [START_REF] Eshkol-Taravella | Un grand corpus oral "disponible" : le corpus d'Orléans 1968-2012[END_REF][START_REF] Lawrence | A concordance correlation coefficient to evaluate reproducibility[END_REF], followed by the LinTh layer. Evaluation metric is Concordance Correlation Coefficient [START_REF] Lawrence | A concordance correlation coefficient to evaluate reproducibility[END_REF] between model predictions and human annotations, as in [START_REF] Weninger | Discriminatively trained Recurrent Neural Networks for continuous dimensional emotion recognition from audio[END_REF][START_REF] Trigeorgis | Adieu features? end-to-end speech emotion recognition using a deep convolutional recurrent network[END_REF].

Results are presented in Table 7. One noticeable result is that, while MFB features cannot reach acceptable performance with the simple LinTanh model, SSL features achieve much better results. As the models get more complex (GRU-32 and GRU-64), the advantage of using SSL features compared to MFB features is less clear. This shows the effectiveness of providing higher level representations (SSL) for AER only when a less complex model (LinTanh) is used. One interesting finding is the ability of the Fr-2.7k-base feature to reach close to best results for most cases even though this SSL model has only been trained on non-emotional speech (Fr-2.7K-base is trained on a subset of our medium set where spontaneous and emotional speech were removed and only read speech was left). Also, since these models are not always better than MFB features when using a more complex model, might show that even though SSL models are able to reach higher level information than MFB, they struggle to extract information related to emotion. We should however highlight the fact that pre-training of 3k models involved less than 1% emotional data (cf. Table 13). Moreover, Fr-1k models, which also only use read speech (but using less data), perform mostly better than Fr-3k and Fr-7k models, which were trained on data containing spontaneous and emotional speech. This shows that by using more data to train SSL models, if mostly non-emotional, we cannot expect better results for the task of emotion recognition. We also observe large variations of performance from one SSL model to another, probably because AER is a very low resource task in this setting. It is thus difficult to conclude on the effectiveness of our SSL models trained on French data compared to the ones trained on multi-lingual or English data. Finally, task-specific pre-training attempts (not reported here) were also made on RECOLA with Fr-3k models but in both self-supervised and ASR based fine-tuning scenarios models did not converge. Further investigations are needed in order to better understand this behavior.

Discussion

On societal and environmental impacts. As an increasing number of NLP papers discussed the potential biases and harms of pre-trained language models and call for more careful design of datasets [START_REF] Rogers | Changing the world by changing the data[END_REF], we set up our large speech corpus with the objective of limiting those in the shared SSL models. First, our speech dataset is carefully documented with relevant metadata (see Table 1) so that it is feasible to analyze the diversity of existing speech sources in terms of social contexts represented (gender, accent, style). As far as gender balance is concerned, we did not manage to have an exact parity in SSL data (our 1k and 3k models have 52% and 38% of female speakers respectively; bigger 7k model do not have enough gender metadata to allow a correct evaluation of gender balance) but we believe the corpus is diverse enough as it was observed that ASR systems, for instance, are overall robust to a certain degree of gender imbalance in the training data [START_REF] Garnerin | Investigating the impact of gender representation in ASR training data: a case study on librispeech[END_REF] (and our gender analysis for ASR confirms this). Also it is worth mentioning that one corpus in our dataset (TCOF) may contain offensive speech but we believe this is not a problem as we only distribute the SSL models (not the signal). License information is also displayed for all sub-corpora (see Table 1). As environmental impact has been highlighted for NLP recently [START_REF] Strubell | Energy and policy considerations for deep learning in NLP[END_REF], we used for training SSL models the CNRS Jean Zay supercomputer 9 which is a low carbon data center situated in a low carbon area (France). In particular, and following the carbon footprint methodology given in [START_REF] Parcollet | The Energy and Carbon Footprint of Training End-to-End Speech Recognizers[END_REF], we estimate that 270kg of CO2 was emitted to train our largest 7K model. In comparison, GP T -3 may emit 10 Tons of CO2 while being trained in France (i.e. lower carbon rate than the USA) [START_REF] Anthony | Carbontracker: Tracking and predicting the carbon footprint of training deep learning models[END_REF]. Sharing our seven models mitigates this impact by alleviating multiple training from the community.

LeBenchmark We have set up a website 10 for LeBenchmark with the aim to: (a) link to the pretrained models and scripts to reproduce experiments presented in this paper, (b) keep track, through a Leaderboard, of future papers and results that would use our evaluation framework, and (c) support contributions for other languages in order to grow LeBenchmark dynamically.

Takeaways After training our own SSL models for French, we evaluated them on 4 speech tasks (ASR, SLU, AST, and AER). For all of them SSL models were beneficial with respect to conventional filterbank of MFCC features. Tasks such as SLU improved drastically with SSL. We also observed that low and medium resource tasks (SLU and AST) and simpler neural architectures (AER with LinTh) benefited more from task-agnostic SSL features than high resource tasks (ASR). We verified the impact of the language used for pre-training: French SSL models are better than multilingual or English SSL models for ASR, SLU and AST in French. SSL architecture size also matters as large models obtained the best performance compared to base ones for ASR, SLU and AST. Regarding amount of SSL pre-training data, setting aside AER for which we observe a lot of variability, training on 3k hours is beneficial compared to 1k but jumping further to 7k is less conclusive (i.e. improves ASR and AST only, not SLU). As task-agnostic SSL pre-training already provides strong results, we demonstrated that performance can be further improved using task specific pre-training: adding a few iterations of self-supervised pre-training on task specific data allows to improve SLU and AST performance. If transcribed speech is available, it is even better to fine-tune SSL models for ASR on data of interest and then use the obtained model as feature extractor for a downstream task. This worked well for SLU and AST and is, to our knowledge, the first time such a task-specific pre-training is efficiently applied to non-ASR speech systems. Finally, while some SSL models were beneficial to AER, this task needs more exhaustive and reliable evaluations to assess the real impact of SSL.

Limitations and future work We currently cover only French language but hope that contributions for other languages would follow in order to grow LeBenchmark dynamically. A more fine-grained analysis of the SSL models' performance (beyond single average metric per sub-task) would be also important to fully understand the pros and cons of each SSL model. Finally, as our collection comes with reliable metadata, it should trigger future analysis works on speech SSL such as training gender/style specific models and analyzing speech SSL biases. between model predictions and human annotations, as in [START_REF] Weninger | Discriminatively trained Recurrent Neural Networks for continuous dimensional emotion recognition from audio[END_REF][START_REF] Trigeorgis | Adieu features? end-to-end speech emotion recognition using a deep convolutional recurrent network[END_REF]. Sampling frequencies of different features, which was 100 Hz for MFB and 50 Hz for the Wav2Vec models, are different from the sampling frequencies of the annotations. Thus, during the training, we re-sampled the annotations to match the sampling frequency of the features and for testing, we re-sampled the output of the model to match the target annotation. Reported numbers on the paper are averaged results over three different random seeds.

A.3 Additional Results for ASR

Tables 8 and9 present the WER by gender reached by different ASR systems on the ETAPE test dataset for end-to-end and hybrid ASR respectively. While in this dataset the WER is lower for female speakers than for male speakers for each ASR system, the relative difference between the results obtained on female voice and the ones obtained on male voices is higher with our Fr-3K-large SSL model.

A.4 Additional Results for SLU

Table 10 reports ASR results on the MEDIA corpus. These ASR models have been used to initialize parameters of basic SLU models with a linear decoder.

Table 11 reports significance test results with the bootstrap method of [START_REF] Bisani | Bootstrap estimates for confidence intervals in ASR performance evaluation[END_REF]. As named in the reference paper, the values reported in the table are the Probability of improvement (Poi) of a system B over a system A, they can be interpreted as 1-p-value. In the table system B are "Fr-3K-large SV" and "Fr-7K-large SV", the two best SLU systems in terms of Concept Error Rate (CER). Between the two best systems, "Fr-7K-large SV" seems to be slightly better with a Poi of 0.78 over "Fr-3K-large SV". But a stronger computation intensive significance test [START_REF] Yeh | More accurate tests for the statistical significance of result differences[END_REF][START_REF] Noreen | Computer-Intensive Methods for Testing Hypotheses: An Introduction[END_REF] shows that "Fr-7K-large SV" and "Fr-3K-large SV" are in fact equivalent (p-value of 0.6 in both directions). The same test gave a p-value < 0.01 in all the other cases, conforming that "Fr-7K-large SV" and "Fr-3K-large SV" are indeed the two best models. Recordings of African Accented French speech.

B Details of the corpora used in the paper

Allosat [START_REF] Macary | Allosat: A new call center french corpus for satisfaction and frustration analysis[END_REF] The corpus is composed of real-life call center conversations in French and is continuously annotated in frustration and satisfaction. Att-HACK [START_REF] Le | Att-HACK: An Expressive Speech Database with Social Attitudes[END_REF] This data is acted expressive speech in French, 100 phrases with multiple versions (3 to 5) in four social attitudes : friendly, distant, dominant and seductive. CaFE [START_REF] Gournay | A Canadian French emotional speech dataset[END_REF] The Canadian French Emotional (CaFE) speech dataset contains six different sentences, pronounced by six male and six female actors, in six basic emotions plus one neutral emotion. The six basic emotions are acted in two different intensities. CFPP2000 [START_REF] Branca-Rosoff | Discours sur la ville. Présentation du Corpus de Français parlé Parisien des années 2000[END_REF] Interviews in Paris and its suburb. Files not included in the CEFC corpus v2.1, 02/2021.

All CFPP2000

files not in CEFC corpus v2.1, 02/2021 CommonVoice [START_REF] Ardila | Common voice: A massively-multilingual speech corpus[END_REF] It is a massively-multilingual collection of read sentences.

French: fr_604h_2020-06-22 EPAC [START_REF] Gravier | The etape corpus for the evaluation of speech-based tv content processing in the french language[END_REF] Conversational speech in French broadcast news. Sub-part from the ESTER Evaluation Campaign (ELRA-E0021). ESLO [START_REF] Eshkol-Taravella | Un grand corpus oral "disponible" : le corpus d'Orléans 1968-2012[END_REF] Contains two subcorpora: ESLO1 + ESLO2 (telephone dialogues, public meetings, etc).

ESLO2 ETAPE [START_REF] Estève | The EPAC Corpus: Manual and Automatic Annotations of Conversational Speech in French Broadcast News[END_REF] Consists of French radio and TV data, selected to include mostly non planned speech and a reasonable proportion of multiple speaker data. GEMEP [START_REF] Bänziger | Introducing the Geneva Multimodal Expression Corpus for Experimental Research on Emotion Perception[END_REF] Audio and video recordings featuring 10 actors portraying 18 effective states, with different verbal contents and different modes of expression. MaSS [START_REF] Marcely Zanon Boito | MaSS: A large and clean multilingual corpus of sentence-aligned spoken utterances extracted from the Bible[END_REF] The Multilingual corpus of Sentence-aligned Spoken utterances has eight languages, and it is made of audio books from the new testament of the Bible.

French MEDIA [START_REF] Bonneau-Maynard | Results of the french evalda-media evaluation campaign for literal understanding[END_REF] A corpus simulating a vocal tourist information server by a Wizard of Oz system. MLS [START_REF] Vineel Pratap | Mls: A large-scale multilingual dataset for speech research[END_REF] A large multilingual corpus derived from LibriVox audio books. French MPF [START_REF] Françoise | Les parlers jeunes dans l'île-de-France multiculturelle[END_REF] Open corpus, created to study the evolution of French language, the growing of a vernacular language, and the effects of the contacts with immigration languages on French. mTEDx [START_REF] Salesky | The multilingual tedx corpus for speech recognition and translation[END_REF] The corpus is a collection of audio recordings from TEDx talks in 8 source languages.

fr-en, fr-es, and frpt NCCFr [START_REF] Torreira | The Nijmegen Corpus of Casual French[END_REF] Corpus composed of filmed casual speech conversations between friends. Portmedia [START_REF] Lefèvre | Robustesse et portabilités multilingue et multi-domaines des systèmes de compréhension de la parole : le projet PortMedia[END_REF] Human-machine interaction, using the Wizard of Oz technique. Two sub-corpora: PM_LANG: dialogues about tourism in Italian. PM_DOM: dialogues about festival ticket booking in French.

PM_DOM RECOLA [START_REF] Ringeval | Introducing the recola multimodal corpus of remote collaborative and affective interactions[END_REF] Audio, visual, and physiological recordings of online dyadic interactions between French speaking participants, who were solving a task in collaboration. TCOF [START_REF] Atilf | TCOF : Traitement de corpus oraux en français[END_REF] "Children" sub-corpus : interactions between adults and children (up to 7 years old). "Adults" sub-corpus : interactions between adults.

Adults

Voxpopuli [START_REF] Wang | VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation[END_REF] This data was collected from 2009-2020 European Parliament event recordings.

French unlabeled + French transcribed

35th

  Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks.

Table 1 :

 1 Statistics for the speech corpora used to train SSL models according to gender information (male / female / unknown). The small dataset is from MLS only. Every dataset is composed of the previous one + additional data; duration: hour(s):minute(s).

	CorpusLicense	# Utterances	Duration	# Speakers	Mean Utt. Duration	Speech type
			Small dataset -1K
	MLS FrenchCCBY 4.0 [27]	263,055 124,590 / 138,465 / -	1,096:43 520:13 / 576:29 / -	178 80 / 98 / -	15 s 15 s / 15 s / -	Read
			Medium dataset -3K
	African Accented FrenchApache2.0 [28]	16,402 373 / 102 / 15,927	18:56 -/ -/ 18:56	232 48 / 36 / 148	4 s -/ -/ -	Read
	Att-HackCCBY N CN D [29]	36,339 16,564 / 19,775 / -	27:02 12:07 / 14:54 / -	20 9 / 11 / -	2.7 s 2.6 s / 2.7 s / -	Acted Emotional
	CaFECCNC [30]	936 468 / 468 / -	1:09 0:32 / 0:36 / -	12 6 / 6 / -	4.4 s 4.2 s / 4.7 s / -	Acted Emotional
	CFPP2000CCBY N CSA * [31]	9853 166 / 1,184 / 8,503	16:26 0:14 / 1:56 / 14:16	49 2 / 4 / 43	6 s 5 s / 5 s / 6 s	Spontaneous
	ESLO2NC [32]	62,918 30,440 / 32,147 / 331	34:12 17:06 / 16:57 / 0:09	190 68 / 120 / 2	1.9 s 2 s / 1.9 s / 1.7 s	Spontaneous
	EPAC**NC [33]	623,250 465,859 / 157,391 / -	1,626:02 1,240:10 / 385:52 / -	Unk -/ -/ -	9 s -/ -/ -	Radio Broadcasts
	GEMEPNC [34]	1,236 616 / 620 / -	0:50 0:24 / 0:26 / -	10 5 / 5 / -	2.5 s 2.4 s / 2.5 s / -	Acted Emotional
	MPF [35], [36]	19,527 5,326 / 4,649 / 9,552	19:06 5:26 / 4:36 / 9:03	114 36 / 29 / 49	3.5 s 3.7 s / 3.6 s / 3.4 s	Spontaneous
	PORTMEDIANC	19,627	38:59	193	7.1 s	Acted telephone
	(French) [37]	9,294 / 10,333 / -	19:08 / 19:50 / -	84 / 109 / -	7.4 s / 6.9 s / -	dialogue
	TCOF (Adults) [38]	58,722 10,377 / 14,763 / 33,582	53:59 9:33 / 12:39 / 31:46	749 119 / 162 / 468	3.3 s 3.3 s / 3.1 s / 3.4 s	Spontaneous
	Medium dataset total	1,111,865 664,073 / 379,897 / 67,895	2,933:24 1,824:53 / 1,034:15 / 74:10	-	-	-
			Large dataset -7K
	MaSS [39]	8,219 8,219 / -/ -	19:40 19:40 / -/ -	Unk -/ -/ -	8.6 s 8.6 s / -/ -	Read
	NCCFrNC [40]	29,421 14,570 / 13,922 / 929	26:35 12:44 / 12:59 / 00:50	46 24 / 21 / 1	3 s 3 s / 3 s / 3 s	Spontaneous
	VoxpopuliCC0 [22] Unlabeled	568,338 -/ -/ -	4,532:17 -/ -/ 4,532:17	Unk -/ -/ -	29 s -/ -/ -	Professional speech
	VoxpopuliCC0 [22] transcribed	76.281 -/ -/ -	211:57 -/ -/ 211:57	327 -/ -/ -	10 s -/ -/ -	Professional speech
	Large dataset total***	1,814,242 682,322 / 388,217 / 99,084	7,739:22 1,853:02 / 1,041:07 / 4,845:07	-	-	-
	*Composed of audio files not included in the CEFC corpus v2.1, 02/2021; **speakers are not uniquely identified.; ***Stats of CFPP2000,
	MPF and TCOF have changed a bit due to a change in data extraction; License: CC=Creative Commons; NC=non-commercial; BY=
	Attribution; SA= Share Alike; ND = No Derivative works; CC0 = No Rights Reserved
	Medium dataset (≈ 3k hours) includes 2,933 h of speech, from which 1,115 h is read speech,
	1,626 h broadcast speech, 123 h spontaneous speech, 38 h acted telephone dialogues, and 29 h acted
	emotional speech. Regarding gender, we collected 1,824 h of speech from male speakers, 1,034 h
	from female speakers, and 74 h from unknown gender.
	Large dataset (≈ 7.7k hours) has 4 additional corpora: MaSS, NCCFr and Voxpopuli (unlabeled
	+ transcribed). It includes 7,739 h of speech, from which 1,135 h is read speech, 1,626 h broadcast
	speech, 165 h spontaneous speech, 38 h acted telephone dialogues, 29 h acted emotional speech, and
	4744 h professional speech. Except for NCCFr, no info about gender is given in the added datasets.

Table 2 :

 2 Hyperparameters of our pre-trained SSL models

	Model	Training data Transformer blocks Model dimension Inner dimension Heads Updates
	W2V2-Fr-1K-base	1,096 h	12	768	3,072	8	200K
	W2V2-Fr-1K-large	1,096 h	24	1024	4,096	16	200K
	W2V2-Fr-2.7K-base	2,773 h	12	768	3,072	8	500K
	W2V2-Fr-3K-base	2,933 h	12	768	3,072	8	500K
	W2V2-Fr-3K-large	2,933 h	24	1024	4,096	16	500K
	W2V2-Fr-7K-base	7,739 h	12	768	3,072	8	500K
	W2V2-Fr-7K-large	7,739 h	24	1,024	4,096	16	500K

Table 3 :

 3 ASR results (WER,%) on the ETAPE corpus for hybrid DNN-HMM AM with TDNN-F topology. Gray numbers indicate 95% confidence intervals.8

	Language Model	ETAPE	ESTER-1.2 + EPAC
	Features	Dev	Test	Dev	Test
	hires MFCC	36.89±0.66	38.50±0.71	29.56±0.70	31.93±0.75
		(a) Task-agnostic pre-training
	En-large	37.68±0.71	40.31±0.75	30.51±0.73	33.32±0.79
	XLSR-53-large	34.28±0.69	36.03±0.72	27.01±0.68	29.64±0.77
	Fr-1K-base	38.91±0.72	41.53±0.80	32.26±0.74	35.69±0.82
	Fr-1K-large	38.77±0.71	40.69±0.67	32.29±0.73	34.91±0.79
	Fr-2.7K-base	32.35±0.66 34.43 ±0.72	26.65 ±0.67 29.31±0.74
	Fr-3K-base	31.98±0.66	33.61±0.73	25.83±0.66	27.82±0.74
	Fr-3K-large	31.85±0.64	33.46±0.69	26.54±0.65	28.56±0.72
	Fr-7K-base	31.96±0.67	33.36±0.72	26.03±0.67	27.09±0.76
	Fr-7K-large	28.75±0.62	30.30±0.68	23.62±0.63	25.64±0.70
	(c) Task-specific pre-training (fine-tuned for ASR on ETAPE)
	Fr-2.7K-base	32.34±0.64	34.46±0.73	26.44±0.66	29.11±0.75
	Fr-3K-base	31.89±0.64	33.47±0.71	26.12±0.66	28.03±0.75
	Fr-3K-large	28.82±0.62	30.19±0.67	23.67±0.62	25.22±0.70
	Fr-7K-base	31.70±0.65	33.32±0.73	25.84±0.67	28.24±0.76
	Fr-7K-large	28.84±0.61	30.29±0.66	23.44±0.62	25.36±0.70

evaluation: (1) trained on ESTER-1.2 and EPAC training data (with a 82k vocabulary) and (2) trained on ETAPE training data only (with a smaller 17.5k vocabulary).

Table 4

 4 presents the results achieved with e2e ASR systems on French Common Voice 6.1 and on ETAPE. Before the use of Wav2vec2.0 models for ASR, the baseline MFB-based system (first line) was the state-of-the-art e2e model on CommonVoice/French. Other lines of table present different Wav2vec2.0 models fine-tuned on labeled ASR data from CommonVoice or ETAPE. Wav2vec2.0 base and large models provided by LeBenchmark outperform clearly En-large and XLSR-53-large models. The best model is Fr-3K-large, pretrained on a smaller training dataset than Fr-7K-large, and it provides the best results on all the experiments. We analyze gender performance in Appendix A.3 and show that female WER is systematically lower than male WER for all systems. Even for our Fr-3K SSL models trained with 38% of female speech only, female WER are particularly low.

	5.2 Spoken Language Understanding (SLU)
	Dataset. Spoken Language Understanding (SLU) aims at extracting a semantic representation from a

speech signal in human-computer interaction applications

[START_REF] Mori | Spoken Dialogues with Computers[END_REF][START_REF] Dinarelli | Concept segmentation and labeling for conversational speech[END_REF][START_REF] Dinarelli | The luna spoken dialog system: Beyond utterance classification[END_REF][START_REF] Dinarelli | Spoken Language Understanding: from Spoken Utterances to Semantic Structures[END_REF][START_REF] Thierry Desot | SLU for voice command in smart home: comparison of pipeline and end-to-end approaches[END_REF]

. Given the difficulty of

Table 4 :

 4 End-to-end ASR results (WER%) on Common Voice and ETAPE corpora, with pre-trained wav2vec2.0 models further fine tuned on labeled ASR data.

	Corpus	CommonVoice	ETAPE
	Features	Dev	Test	Dev	Test
	MFB	17.67±0.37 20.59±0.41	54.03±1.33 54.36±1.32
	En-large	12.05±0.23 14.17±0.52	42.14±0.72 44.82±0.74
	XLSR-53-large 16.41±0.27 19.40±0.29	58.55±0.65 61.03±0.70
	Fr-2.7K-base	11.04±0.27 13.09±0.24	26.23±0.78 29.08±0.80
	Fr-3K-base	11.25±0.23 13.22±0.24	26.14±0.70 28.86±0.79
	Fr-3K-large	8.34±0.18	9.75±0.20	23.51±0.68 26.14±0.77
	Fr-7K-base	10.84±0.21 12.88±0.24	25.13±0.68 28.16±0.79
	Fr-7K-large	8.55±0.18	9.94±0.21	24.14±0.70 27.25±0.78

Table 5 :

 5 End-to-end SLU decoding results (Concept Error Rate %) on the MEDIA corpus.

	Features	Dev	Test
	(from [9]) spectrogram 33.63±1.28	34.76±0.83
	spectrogram	29.07±1.31	31.10±0.83
		(a) Task agnostic pre-training
	En-base	22.38±1.24	20.84±0.68
	En-large	23.31±1.31	25.26±0.77
	Fr-1K-base	22.89±1.26	23.27±0.76
	Fr-1K-large	20.10±1.10	20.66±0.72
	Fr-2.7K-base	18.63±1.13	18.42±0.65
	Fr-3K-base	19.44±1.11	18.56±0.67
	Fr-3K-large	15.96±1.02	15.95±0.62
	Fr-7K-base	20.70±1.07	18.86±0.68
	Fr-7K-large	17.25±1.02	16.35±0.66
	XLSR-53-large	18.45±1.15	18.78±0.66
	(b) Task specific pre-training (self-supervised on MEDIA)
	Fr-3K-large	15.93±1.01	14.94±0.60
	Fr-7K-large	15.42±1.03	15.17±0.60
	XLSR-53-large	16.77±1.09	15.56±0.61
	(c) Task specific pre-training (fine-tuned for ASR on MEDIA)
	Fr-3K-large	14.49±1.06	13.97±0.59
	Fr-7K-large	14.58±1.01	13.78±0.58
	XLSR-53-large	16.05±1.05	15.46±0.60

Table 6 :

 6 BLEU on valid and test sets of multilingual TEDx (mTEDx). The highest value in each group (task-agnostic pre-training, task-specific self-supervised, and supervised fine-tuning) is underlined while the best value in each column is highlighted in bold. Gray numbers denote the standard deviation computed using bootstrap re-sampling[START_REF] Koehn | Statistical significance tests for machine translation evaluation[END_REF]. 41±0.51 20.32±0.[START_REF] Bisani | Bootstrap estimates for confidence intervals in ASR performance evaluation[END_REF] 15.14±0.48 21.69±0.58 21.57±0.52 17.43±0.52 XLSR-53-large 21.09±0.54 20.38±0.56 14.56±0.45 20.68±0.53 21.14±0.55 17.21±0.54

	Features		Valid			Test
		en	es	pt	en	es	pt
	MFB	1.15±0.17	0.67±0.15	0.61±0.13	1.10±0.14	0.87±0.12	0.32±0.03
			(a) Task agnostic pre-training
	En-base	5.54±0.27	1.30±0.17	0.54±0.11	5.20±0.28	1.47±0.15	0.38±0.05
	En-large	4.11±0.25	1.67±0.20	0.32±0.03	3.56±0.22	2.29±0.18	0.43±0.05
	Fr-1K-base	9.18±0.36	5.09±0.27	0.39±0.05	8.98±0.36	5.64±0.30	0.49±0.08
	Fr-1K-large	15.31±0.46 13.74±0.43	8.29±0.34 14.46±0.46 14.77±0.46	9.37±0.38
	Fr-2.7K-base	15.09±0.49 13.27±0.43	4.72±0.27 14.69±0.48 14.04±0.43	5.51±0.28
	Fr-3K-base	15.05±0.49 13.19±0.44	4.44±0.29 14.80±0.47 14.27±0.44	4.72±0.25
	Fr-3K-large	17.94±0.51 16.40±0.49	8.64±0.34 18.00±0.51 18.12±0.48	9.55±0.36
	Fr-7K-base	15.13±0.45 12.78±0.40	2.65±0.20 14.50±0.45 13.61±0.44	2.66±0.23
	Fr-7K-large	19.23±0.54 17.59±0.49	9.68±0.37 19.04±0.53 18.24±0.49 10.98±0.41
	XLSR-53-large	7.81±0.33	0.49±0.13	0.43±0.07	6.75±0.29	0.52±0.08	0.36±0.05
		(b) Task specific pre-training (self-supervised on mTEDx)
	Fr-3K-large	18.54±0.53 16.40±0.48	8.81±0.36 18.38±0.52 17.84±0.48 10.57±0.41
	Fr-7K-large	19.65±0.55 17.53±0.47	9.35±0.36 19.36±0.54 18.95±0.53 10.94±0.38
	XLSR-53-large	6.83±0.33	0.54±0.14	0.34±0.03	6.75±0.32	0.34±0.03	0.29±0.03
	(c) Task specific pre-training (fine-tuned for ASR on mTEDx)
	Fr-3K-large	21.09±0.53 19.28±0.53 14.40±0.47 21.34±0.58 21.18±0.52 16.66±0.49
	Fr-7K-large	21.

Table 7 :

 7 Concordance Correlation Coefficient of emotion predictions on the RECOLA and AlloSat test sets.

						Corpus -Task
	Features	RECOLA -Arousal	RECOLA -Valence	AlloSat -Satisfaction
						Model
		LinTh GRU-32 GRU-64 LinTh GRU-32 GRU-64 LinTh GRU-32 GRU-64
	MFB	.139	.655	.649	.107	.373	.421	.121	.611	.612
	En-large	.465	.517	.542	.154	.220	.221	.102	.490	.480
	XLSR-53-large	.237	.661	.669	.005	.322	.200	.242	.578	.582
	Fr-1K-base	.505	.654	.661	.243	.331	.301	.403	.641	.558
	Fr-1K-large	.507	.709	.708	.196	.555	.234	.175	.601	.597
	Fr-2.7K-base	.521	.720	.741	.208	.498	.530	.437	.646	.687
	Fr-3K-base	.474	.700	.686	.183	.388	.228	.356	.732	.740
	Fr-3K-large	.378	.267	.349	.130	.202	.033	.009	.468	.473
	Fr-7K-base	.502	.700	.702	.214	.406	.358	.394	.653	.653
	Fr-7K-large	.310	.203	.078	.020	.214	.068	.007	.510	.474

Table 8 :

 8 WER results by gender on the ETAPE test dataset for end-to-end ASR.

	Features	WER Male WER Female Relative WER difference between
				Male and Female speakers, %
	MFB	60.2	53.6	11.6
	XLSR-53-large	60.1	57.4	4.6
	En-large	44.3	39.3	12.0
	Fr-3K-large	27.5	21.4	24.9

Table 9 :

 9 WER results by gender on the ETAPE test dataset for hybrid ASR. Training was achieved by Adam optimizer with 250 as the maximum number of epochs; it was stopped after 15 epochs if no improvement over the development set was observed. The loss (and evaluation metric) used here is Concordance Correlation Coefficient[START_REF] Lawrence | A concordance correlation coefficient to evaluate reproducibility[END_REF]

	Features	WER Male WER Female Relative WER difference between
				Male and Female speakers, %
	hires MFCC	32.0	20.9	21.0
	XLSR-53-large	33.3	22.6	19.1
	En-large	30.2	19.8	20.8
	Fr-3K-large (task-agnostic pretraining)	29.7	17.7	25.3
	Fr-3K-large (task-specific pretraining)	26.4	15.7	25.4
	A.2.5 AER
	Implementation details

Table 12 :

 12 Corpora/sub-corpora details (at download time). Click on the Corpus name to access its web page.*t=tokens, w=words, h=hours, min=minutes, sent=sentences, d=dialogues

	Corpus (sub-	Identifier	(ISLRN,	Size*	Modality	Dataset use License
	corpus) name	DOI...)
	African	Ac-	SLR57		22 h	speech, written	SSL	Apache 2.0
	cented French
	Allosat				37 h	speech, written	AER	CC
	Att-HACK		SLR88		>300 sent	speech, written	SSL	CC BY-NC-ND
	CaFE		10.5281/zenodo.1478765	1 h	speech, written	SSL	CC-BY-NC-SA
								4.0
	CFPP2000				20 h	speech, written	SSL	CC BY-NC-SA
	(CEFC com-						3.0
	plement)
	CommonVoice			604 h	speech, written	ASR	CC 0
	fr_604h_2020-
	06-22
	EPAC		483-703-007-740-8	1677 h	speech, written	SSL	ELRA NC
	ESLO				>400 h	speech, written	SSL	CC BY-NC-SA
	(ESLO2)							4.0
	ETAPE		425-777-374-455-4	30 h	speech, written	ASR	ELRA NC
	GEMEP				0.9 h	speech	SSL	academic only,
								NC
	MaSS				≈ 20 h	speech, written	SSL	MIT License
	MEDIA		699-856-029-354-6	1,258 d	speech, written	SLU	ELRA NC
	MLS (French)			1,096 h	speech, written	SSL	CC BY 4.0
	MPF				78 h	speech, written	SSL	CC BY-NC-SA
								4.0
	mTEDx (fr-*)	SLR100		25h -50h	speech, written	AST	CC BY-NC-ND
								4.0
	NCCFr				35 h	Multimedia, written	SSL	academic only,
								NC
	Portmedia		135-793-959-390-8	40.5 h	speech, written	SSL	ELRA NC
	(PM_DOM)
	RECOLA				9.5 h	Multimedia, written	AER	End User License
								Agreement
	TCOF				146 h	speech, written	SSL	CC BY-NC-SA
	Voxpopuli un-			≈ 4.5k h	speech	SSL	CC0
	labeled
	Voxpopuli tran-			≈ 215 h	speech, written	SSL	CC0
	scribed

Table 13 :

 13 Corpora description.

	Corpus name	Description	Used subcorpus
			(if existing)
	African
	Accented
	French[28]

http://lebenchmark.com

https://github.com/pytorch/fairseq/blob/main/examples/wav2vec/config/ pretraining/wav2vec2_base_librispeech.yaml

https://github.com/pytorch/fairseq/blob/main/examples/wav2vec/config/ pretraining/wav2vec2_large_librivox.yaml

https://huggingface.co/LeBenchmark

https://github.com/pytorch/fairseq/tree/master/examples/wav2vec

For the sake of conciseness, we remove the prefix W2V2in all our results table.

Since two types of task-specific pre-training will be provided for SLU and AST, for ASR we only experimented with fine-tuning SSL models for ASR on ETAPE and then using them as feature extractors.

[START_REF] Schlangen | Targeting the benchmark: On methodology in current natural language processing research[END_REF] Error margins corresponding

to 95% confidence intervals were computed using bootstrap re-sampling as proposed in[START_REF] Bisani | Bootstrap estimates for confidence intervals in ASR performance evaluation[END_REF].

Acknowledgements

This work benefited from the 'Grand Challenge Jean Zay' program and was also partially supported by MIAI@Grenoble-Alpes (ANR-19-P3IA-0003). This paper was also partially funded by the European Commission through the SELMA project under grant number 957017.

Appendix and Supplementary Material

A Appendix

A.1 Wav2Vec2.0 training behavior In this Appendix, we report the losses on the development set of the MLS corpus obtained for our different models. Models are stopped at 500.000 steps due to the lack of improvement observed when trained for longer. The acoustic models (AM) have been trained trained on 40-dimensional high-resolution (hires) MFCC features or SSL feature using the Kaldi toolkit with a state-of-the-art factorized time delay neural network (TDNN-F) architecture [START_REF] Povey | Semi-orthogonal low-rank matrix factorization for deep neural networks[END_REF][START_REF] Peddinti | A time delay neural network architecture for efficient modeling of long temporal contexts[END_REF] on the ETAPE training corpus [START_REF] Gravier | The etape corpus for the evaluation of speech-based tv content processing in the french language[END_REF] only. For experiments in this paper, the dimensions of SSL features extracted by 'small' and 'large' models are 768 and 1024 respectively. The models have 12 TDNN-F layers (1,024-dimensional, with projection dimension of 128) and a 3,432-dimensional output layer. 100-dimensional speaker i-vectors were appended to the input features for all the models.

Training Strategy. The acoustic model was trained using lattice-free maximum mutual information (LF-MMI) [START_REF] Povey | Purely sequence-trained neural networks for asr based on lattice-free mmi[END_REF] and cross-entropy criteria. Speed and volume perturbation have been applied for data augmentation [START_REF] Ko | Audio augmentation for speech recognition[END_REF]. We used a similar topology to train all systems with different types of input features.

A.2.2 ASR: End-to-End models

Model The end-to-end system fed by 80-dimension log Mel filterbank (MFB) features is based on an encoder/decoder architecture with attention: the encoder is a Convolutional Recurrent Deep Neural Network (CRDNN: VGG + RNN + DNN), and the decoder is a joint CTC/Attention LSTM neural network. For this ASR system, the neural network output corresponds to 500 byte pair encoding (BPE) units [START_REF] Sennrich | Neural machine translation of rare words with subword units[END_REF] computed on the manual transcriptions of the respective training datasets. This model is used to present baseline results from a system that does not use pretrained models.

When used with a pretrained Wav2Vec2.0 model, the end-to-end model is made of this Wav2Vec2.0 model with an additional hidden layer and an output layer on the top. The hidden layer has the same dimension as the Wav2vec2.0 model output (i.e 768 for base models, 1024 for large ones.) The output layer dimension depends on the number of characters in the training data (78 for CommonVoice, 60 for ETAPE).

Training Strategy No additional language model is used in these experiments, neither data augmentation. To train (with supervision, by exploiting the manual transcriptions) the Wav2Vec2.0-based end-to-end models, two disjoint Adam optimizers are applied: one to handle the Wav2Vec2.0 pretrained weights and another one to update the randomly initialized weights of the hidden and output layers on the top of the model.

A.2.3 SLU Model

The end-to-end SLU model used in this work is similar to the one proposed in previous works [START_REF] Dinarelli | Label-dependency coding in Simple Recurrent Networks for Spoken Language Understanding[END_REF][START_REF] Lugosch | Speech model pre-training for end-to-end spoken language understanding[END_REF][START_REF] Dinarelli | A data efficient end-to-end spoken language understanding architecture[END_REF][START_REF] Evain | LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech[END_REF]. In particular we use a similar speech encoder employing a pyramidal hierarchy of RNN layers like [START_REF] Chan | Listen, attend and spell: A neural network for large vocabulary conversational speech recognition[END_REF][START_REF] Evain | LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech[END_REF]. The decoder has been also improved integrating two attention mechanisms: one as usual for attending the encoder's hidden states; the other for attending all previous decoder prediction's embeddings, instead of the previous prediction only like in the original LSTM-based encoder-decoder models [START_REF] Bahdanau | Neural Machine Translation by Jointly Learning to Align and Translate[END_REF]. Our model is implemented using the Fairseq library [START_REF] Ott | fairseq: A fast, extensible toolkit for sequence modeling[END_REF].

Training Strategy We use a similar incremental training strategy as [START_REF] Dinarelli | A data efficient end-to-end spoken language understanding architecture[END_REF]. In particular we train first only the encoder of our model for decoding tokens as an ASR model. In order to do so we add a linear layer on top of the encoder which maps the hidden states into the output dictionary size. We use the same dictionary for all symbols in our system. We can thus use the token-level model to initialize parameters of an equivalent model which performs SLU by decoding all together tokens, concepts and their boundaries. For instance, given a sequence of N tokens w j+1 , ..., w j+N instantiating a concept C i in a sentence S = w 1 , ..., w M , we use special boundary markers boc and eoc (for start and end of concept) for each concept, modifying the original sequence into S = w 1 , ..., soc, w j+1 , ..., w j+N , C i , eoc, ..., w M . This output format has already been used in previous work [START_REF] Ghannay | End-to-end named entity and semantic concept extraction from speech[END_REF][START_REF] Caubrière | Curriculum-based transfer learning for an effective end-to-end spoken language understanding and domain portability[END_REF], and it is needed for extracting concept values (or attribute values) together with concepts (or attribute names), as described in [START_REF] Dinarelli | Concept segmentation and labeling for conversational speech[END_REF][START_REF] Dinarelli | Spoken Language Understanding: from Spoken Utterances to Semantic Structures[END_REF]. In this work we focus on concept extraction only, we leave the concept value extraction phase for future work. The SLU model trained for predicting the output described above has the same decoder as the token-level model used for initializing its parameters, that is just a linear layer. This first SLU model is used for initializing the parameters of our final SLU model, which has the same encoder, but uses the LSTM decoder described in the previous paragraph. Our training strategy can thus be summarized in the following 3 steps, where the model trained at step i is initialized with parameters of the model trained at step i -1: (1) Encoder+Linear decoder (ASR), (2) Encoder+Linear decoder (SLU), (3) Encoder+LSTM decoder (SLU).

Implementation details All models are learned with an Adam optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF], initial learning rate 5e -5 which is shrinked by a factor of 0.98 at each training epoch, and batches of size 10 for the first 2 training steps (linear decoder), 5 for the last step (LSTM decoder). Models learn to minimize the CTC loss [START_REF] Graves | Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks[END_REF], and we keep the models showing the best error rate on the development data. When learning the final SLU models with a LSTM decoder, we start training with a small warm-up learning rate which is increased linearly up to the initial learning rate during the first 2 epochs. We use this strategy, together with regularization, to avoid catastrophic forgetting [START_REF] Kemker | Measuring catastrophic forgetting in neural networks[END_REF], as these model's encoders are initialized with a model already trained to perform SLU, as mentioned in the previous paragraph. At the decoding phase we average the scores of the 5 best checkpoints on development data.

A.2.4 AST

Model We used a small Transformer [START_REF] Vaswani | Attention is all you need[END_REF] architecture having 6 layers of encoder, 3 layers of decoder, and hidden dimension D = 256 in all experiments. Following previous work [START_REF] Nguyen | Investigating self-supervised pre-training for end-to-end speech translation[END_REF][START_REF] Evain | LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech[END_REF], we inserted a block of Linear-ReLU before convolutional layers in the speech encoder for parameter efficiency and model performance reasons.

Implementation details

Our experiments are performed using the FAIRSEQ S2T toolkit [START_REF] Wang | fairseq S2T: Fast speech-to-text modeling with fairseq[END_REF]. For text pre-processing, we normalize the punctuation and build 1K unigram vocabularies using Sentencepiece [START_REF] Kudo | Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing[END_REF] without pre-tokenization. Following common practice [START_REF] Wang | fairseq S2T: Fast speech-to-text modeling with fairseq[END_REF][START_REF] Inaguma | ESPnet-ST: All-in-one speech translation toolkit[END_REF], utterances having more than 3000 frames are removed for GPU efficiency. All AST models are trained for 500 epochs using the Adam optimizer [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF] in which the learning rate is linearly increased for the first 10K warm-up steps then decreased proportionally to the inverse square root of the step counter. The learning rate for all experiments is set to 2 × 10 -3 . We averaged the last 10 checkpoints and used beam search with a beam size of 5 for decoding. The reported results are detokenized case-sensitive BLEU computed using sacreBLEU [START_REF] Post | A call for clarity in reporting bleu scores[END_REF]. As far as task specific pre-training is concerned, for self-supervised fine-tuning (b) we continued training the SSL models on the task data from the last optimizer state for an additional 20K steps. For ASR supervised fine-tuning (c), we used the same hyper-parameters setup as proposed in the original wav2vec 2.0 paper for fine-tuning large models on 100 h of labeled data. We then used the best checkpoints (fine-tuned on the pair fr-en) to extract features, which are the inputs for the downstream AST models. Table 11: Significance tests with the bootstrap method [START_REF] Bisani | Bootstrap estimates for confidence intervals in ASR performance evaluation[END_REF] between the two best SLU models, in terms of CER, with respect to all the other models. SS and SV in the table mean Self-supervised (block (b)) and Supervised (block (c)) pre-training approaches, respectively.

Significance tests (Probability of improvement [START_REF] Bisani | Bootstrap estimates for confidence intervals in ASR performance evaluation[END_REF]) The link to all datasets used in the paper is provided Table 12. Yes An explanation of any data that were excluded, description of any pre-processing step.

The pre-processing steps are summarized in section 3. Yes An explanation of how samples were allocated for training / validation / testing.

For all Task, we use the standard partitioning as provided in the datasets. Further information can be found in appendix A.2.

Yes

The range of hyper-parameters considered, method to select the best hyper-parameter configuration, and specification of all hyper-parameters used to generate results.

In appendix A.2, the reader can find which hyper-parameters were considered and how their value has beeb chosen. The main reference is the provided github code.

Yes

The exact number of evaluations runs. Yes A description of how experiments were run. Yes A clear definition of the specific measure or statistics used to report results.

Standard, sufficiently well-known, evaluation metrics (WER, CER, BLEU, and CCC) were used.

Yes

Clearly defined error bars. Given that several different tasks were used with different measure, we used different methods to assess the robustness of the results (confidence interval, significance test). These are described in their respective task subsection. Yes A description of results with central tendency (e.g. mean) and variation (e.g. stddev).

Most result tables provide the global score as well as its variation either in the form of stddev or CI. Yes A description of the computing infrastructure used. The supercomputer used is mentioned with an hyperlink providing all the necessary details.


Gaëlle Guigon
email: gaelle.guigon@imt-nord-europe.fr

Mathieu Vermeulen

Mathieu Muratet

Thibault Carron

Towards an integration of the multi-role dimension in the design of learning games: a review of the literature

Keywords: Learning Game, Multi-role, Multiplayer, Serious game, TEL Design

Serious Games (SG) have become popular and their efficiency has been proven in numerous domains throughout the years. You can find them for a plethora of subjects, in bountiful forms and consequently, to answer various needs. They are particularly efficient to reproduce concrete situations and to train on specific objectives. However, most of them are only designed for one player. One explanation might be their design that seems indeed more complex for multiplayer SG. This paper focuses on a certain type of multiplayer SG: those based on multirole. Thus, this paper aims at drawing a state of the art of multi-role SG and their design, as a first step to propose a design model of multi-role SG.

Introduction

Years ago, serious games emerged, took shape and have been developed ever since, taking various forms and fulfilling countless goals. Their definition can be quite complex so here we are choosing the one stated by Michael and Chen [START_REF] Michael | Serious Games: Games That Educate, Train, and Inform[END_REF]: "games whose first purpose was not mere entertainment." As technology facilitates their creation for designers, the number of SG is constantly growing as well as their popularity in the world of education. A special approach of SG, role playing, is particularly interesting as Cheng et al. [START_REF] Cheng | Design Ideas for Issue-Situation-Based Board Game Involving Multirole Scenarios[END_REF] said:"Role playing allows participants to attain knowledge in a different manner and improve their approach to problem solving in a real-world context." It is thus necessary to define a role, as Warrin et al. did [START_REF] Warin | Multi-Role Project (MRP): A New Project-Based Learning Method for STEM[END_REF]: "in a game, the set of objectives, behaviors, rights, and duties assigned to a person or a group of persons is called a role. A person may play several roles, or inversely, a role may involve several people."

We could also add the notion of competences, objects, and powers as roles' attributes. Here we are assuming that multi-role games are multi-users, Calandra et al. [START_REF] Calandra | A Multi-role, Multi-user, Multi-technology Virtual Reality-based Road Tunnel Fire Simulator for Training Purposes[END_REF] precised that "Multi-user refers to the possibility, for different users in different places, to play the available roles all together and at the same time in a shared experience." So it is really important to consider this in the scenario, so that nobody stays inactive during the game. Everyone should have his/her proper objectives within the game and should not be completely dependent on the actions of other players. This complicates even more the conception of the game. We know it can be difficult to design SG in solo mode [START_REF] Marfisi-Schottman | Tools and Methods for Efficiently Designing Serious Games[END_REF][START_REF] Marne | Model and Authoring Tool to Help Teachers Adapt Serious Games to their Educational Contexts[END_REF], and naturally the same assessment seems established concerning multiplayer games [START_REF] Oliveri | Designing an IoTfocused, Multiplayer Serious Game for Industry 4.0 Innovation[END_REF]. Therefore, we can suppose that multi-role games are even trickier to conceive. The research question is: how can we design training scenarios that allow several different roles to be implemented simultaneously? To that end, the present paper aims at drawing a state of the art of multi-role SG. To begin with, the method with which the present state of the art was set up will be introduced. Then, we will explain the categorization of the selected papers. That will allow us to analyse the results in order to exhibit some tracks to answer the research question.

Method

Selection and source. The papers selected for this state of the art are dealing with multi-role serious games or their design. The goal was to find out if some papers concerning the design of multi-role SG already exist. This research was focused on the websites of those editors and academic research databases: ScienceDirect, IEEE, ACM, Scopus, Web of Science and Springer Link. The keywords used on their search engines were: "multi-role" (with and without hyphen) AND "game" and no restriction of date. Here are the results that led to the remaining papers in the first part of Table 1. Science Direct: the access was given via institution and the papers selected are open access and full text access. Only one result was found with the keywords "multirole game" in the title, the abstract or the keywords. No result was found with the keywords "'multi-role' game". The quotation marks were necessary to avoid all the other words beginning with 'multi-'. IEEE: (access via institution) Four results were found. Yet, even if the words "multi-role" and "games" were present in the abstract, two papers didn't deal with a game or its conception, so they were deleted from the selection. Scopus: (access via institution) the search was based within the title, the abstract or the keywords. The search with open access only resulted in five papers. But as before, three papers had no link with multi-role games so they are not included in the analysis. Web of Science: the search provided five results in open access (four of them were already found previously). ACM: focused on the title, keywords and abstract. Only one result was available. Springer Link: without the "preview only content" we found four results that were restricted to the articles containing the word "game" within the abstract. This filter resulted in two papers. Unfortunately, their content was not in relation with the subject of this survey so they were not selected either.

Classification framework. In this survey, we are looking for papers presenting a model or a tool to create multi-role SG. The aim is to find a generic model or tool, if possible, easily understandable by non IT-professionals. Thus, the papers will be sorted in three categories: 1) the theme (to see if it is restricted to a sector or if it is generic), 2) the type (e.g. a game, a model or a tool) and 3) the public (to find if it is usable by anyone). [START_REF] Cheng | Design Ideas for Issue-Situation-Based Board Game Involving Multirole Scenarios[END_REF] Water Game General public Li et al. [START_REF] Li | Study on the Strategy of Playing Doudizhu Game Based on Multirole Modeling[END_REF] Doudizhu (Poker like) Card-playing framework CNN Specialists Warin et al. [START_REF] Warin | Multi-Role Project (MRP): A New Project-Based Learning Method for STEM[END_REF] Information System Project Game Students Argenton et al. [START_REF] Argenton | Multiplayer Serious Games and User Experience: A Comparison Between Paper-Based and Digital Gaming Experience[END_REF] Gliding Game General public Arif et al. [START_REF] Arif | A Blockchain-Based Multiplayer Transaction For Tourism Serious Game[END_REF] Transactions activities Game General public Babu et al. [START_REF] Babu | Collaborative Game Based Learning of Post-Disaster Management: Serious Game on Incident Management Frameworks for Post Disaster Management[END_REF] Post disaster management Game General public Bierre [START_REF] Bierre | Implementing a game design course as a multiplayer game[END_REF] Game design Game Students Burton [START_REF] Burton | The Irrigation Management Game: A role playing exercise for training in irrigation management[END_REF] Irrigation Game Professionals Lee et al. [START_REF] Lee | Cooperation begins: Encouraging critical thinking skills through cooperative reciprocity using a mobile learning game[END_REF] Manufacturing industry consultancy Game Students Schrier [START_REF] Schrier | Using augmented reality games to teach 21st century skills[END_REF] 21st century skills Game Students Schulz et al. [START_REF] Schulz | Scenario-based Serious Game to Teach about Healthcare[END_REF] Health and care Game Professionals Sequeira et al. [START_REF] Sequeira | Let's save resources!": A dynamic, collaborative AI for a multiplayer environmental awareness game[END_REF] Environment AI General Public Stroe et al. [START_REF] Stroe | FireEscape: a Gamified Coordinative Aproach to Multiplayer Fire-Safety Training[END_REF] Fire-safety Game Students

As this search led to few papers, it was extended to multiplayer SG in those same scientific research tools. Indeed, the results on the second part of Table 1 appeared with the keywords: "multiplayer" or "collaborative" or "competitive" or "cooperative" and "serious game" and deal also with multi-role games.

Analysis

First of all, based on this research, we can see that the term"multi-role" (with or without the hyphen) is not often used in SG. Furthermore, even within multiplayer games, they are scarce. Among the seven initial articles, we can see that only one is generic, but is only usable by IT-professionals. All the other papers are dealing with a specific theme and are not supposed to be adapted for another theme. Concerning the extended search, all the results are games (only one paper presents an AI for a game) but the entire selection is oriented to a particular theme and none is generic. It means that there is a genuine need to define a model, a generic tool or at least to be advised on the creation of multi-role SG. Among these papers, only a few explains how the games were designed, how the scenarios were thought through, at least not for non IT-professionals. Usually, the games were presented to explain their benefits and their special features but there is no mention of a specific game engine allowing them to create their game. Unity was mentioned in [START_REF] Calandra | A Multi-role, Multi-user, Multi-technology Virtual Reality-based Road Tunnel Fire Simulator for Training Purposes[END_REF], authors affirm that the possibilities are infinite with this game engine, but it is not easy to handle for teachers, instructional designer or non professionals. Furthermore, the aim is not to create a video game, but to structure a game, to be guided during the scenario creation process and to define the paths, objectives and particularities of each role. We can note that multirole games can be in a physical environment: they are not device-dependent, as several games prove it here [START_REF] Bierre | Implementing a game design course as a multiplayer game[END_REF][START_REF] Burton | The Irrigation Management Game: A role playing exercise for training in irrigation management[END_REF][START_REF] Cheng | Design Ideas for Issue-Situation-Based Board Game Involving Multirole Scenarios[END_REF][START_REF] Li | Study on the Strategy of Playing Doudizhu Game Based on Multirole Modeling[END_REF][START_REF] Warin | Multi-Role Project (MRP): A New Project-Based Learning Method for STEM[END_REF]. The paper presented by Agustin [START_REF] Agustin | Knowledge Representation and Inference Engine Model of SAPS Gaming Concept[END_REF] is the closest to what we are looking for: they propose "a conceptual graph model to represent the concept of gaming" and a "building block of the engine to run and control the game". Even if it seems flexible and quite generic, it appears to be only usable by IT-professionals.

Discussion

Existing tools. As stated before, we found no paper tackling the generic design of multi-role SG. However, a few approaches are interesting and could meet this need. Apart from the results found, a few tools should be mentioned as they bring interesting aspects to answer our questioning: Marne [START_REF] Marne | MoPPLiq: A Model for Pedagogical Adaptation of Serious Game Scenarios[END_REF] created a generic tool, MoPPLiq, to design game scenarios. MoPPLiq has not especially been created for multi-role games, but the method used to conceive a scenario could be adapted to meet this need because it is used to model pedagogical scenarios, with objectives to achieve and conditions to go to different steps. One strategy can be used to reach our goal: we can separate the roles within the same scenario. For instance, Marty [START_REF] Marty | Observation of Collaborative Activities in a Game-Based Learning Platform[END_REF] used a multiplayer game where the players have the same scenario, but are paced differently in order to avoid that all the students progress simultaneously in the game. Another strategy supporting the same scenario is to offer side quests to the players to regulate the flow [START_REF] Bodin | Specifying Collaborative Tools in Game-Based Learning Environments: Clues from the trenches[END_REF]. Next, the tool MAGES [START_REF] Guigon | A Creation Tool for Serious Puzzle Games[END_REF], based on the models SEGAM [START_REF] Guigon | A Model to Design Learning Escape Games: SEGAM[END_REF] and DISC [START_REF] Vermeulen | Teachers at the Heart of the Learning Games Design: The DISC Model[END_REF], could be adjusted for multi-role SG since it is made for multiplayer games. It covers the whole organization part, the pedagogical part and the structure of the game. It would be interesting to adapt it because it has already been made to be generic, so usable in any discipline. Thus, pedagogy can be adapted because the learning objectives can be approached at any moment of the scenario, if they are consistent with the previously defined prerequisites. The tool is structured like a form so it seems easy for non programmers to create a pedagogically coherent scenario. It is also planned to create a database to save several projects on it and to share them.

To go further. Another aspect to consider is management of resources in multi-role games. To encourage cooperation, everyone should not access the same elements at the same time. We could go further and define that only one or a few roles can use a specific resource. It would be convenient if the tool could handle it directly. The question of the persistence of items is also to take into account: do they disappear after a player took them? Definitely or temporarily? All these parameters should be adjustable according to the game objectives. Thus, many criteria are in our sights concerning the design of a multi-role SG authoring tool (e.g. pedagogy, distribution of roles, interactions between players, scenarios, genericity etc.).

Conclusion

To conclude, this state of the art showed that, to our knowledge, there is no model or generic tool created to help designing multi-role SG and usable for non IT-professionals. However, gathering experiences and methods of the papers seen here can give clues to get there. So, creating a model and the reified tool could potentially be very helpful in the process of creating them -in particular for the promotion of cooperative SG which could fit perfectly for multi-role games. In future works, we plan to establish a model based on the relevant concepts seen in this article. This model will lead to the creation of a tool which will eventually enable its users to create games, taking into account the obstacles reviewed in this state of the art. They could be tested in a context of differential learning throughout an asymmetrical gameplay, for instance the reproduction of a scenario between a doctor and a patient or in the civil engineering field, being able to play each actor involved in BIM (Building Information Modeling) process.

Table 1 .

 1 Categorization of multi-role serious games

	Paper	Theme	Type	Target
	Agustin et al. [1]	Generic	Conceptual Graph model IT-professional
	Baturin et al. [5]	Situational analysis	Situational modeling	General public
	Breitner et Smith [8] IT	Software	IT students
	Calandra et al. [10] Fire safety	Game	General public
	Cheng et al.