Lucas Maison

Marcely Zanon Boito

Yannick Estève

Promises and Limitations of Self-supervised Learning for Automatic Speech Processing

Keywords: automatic speech recognition, speech translation, self-supervised learning, speech processing, security

Self-supervised learning (SSL) has recently been successfully introduced as a training strategy for Transformerbased neural models. Thanks to this approach, these models are now able to construct speech representations by using only audio data, without any manual labels (i.e. no supervision). Once trained, they can be leveraged for training competitive endto-end models for speech processing with smaller amounts of annotated data. Moreover, when the available annotated data is plenty, automatic speech recognition (ASR) and translation (AST) systems based on these SSL models are now the new state of the art. In this work, we are interested in their application in challenging settings that are relevant for security. We measure the robustness of a French-based SSL model to African accent, and we present some promising but limited results for speech translation without the use of transcriptions.

I. INTRODUCTION

Speech recognition has been dominated by data-driven approaches for almost four decades. From the 80s until a few years ago, automatic speech recognition (ASR) systems were based on the use of three kinds of knowledge. The first one was captured by the acoustic models, usually based on a Hidden Markov Model (HMM) combined to a Gaussian Mixture Model (GMM) or more recently to a Deep Neural Network (DNN). The acoustic models were designed to compute the likelihood of the presence of a phoneme (the speech unit that discriminates a word in a language) according to the audio signal. The second knowledge was represented by a pronunciation dictionary, in order to map a sequence of phonemes to one or several words. The last knowledge was captured by a language model, in order to compute the probability to observe a sequence of words in a language.

At that time, acoustic and language models were mainly statistical models, and that is why we say that such approaches are data-driven. To get such models accurate and robust, a large amount of training data is needed, following the "there is no better data than more data" paradigm. During the last decade, DNNs for both acoustic and language models have replaced the GMMs, and neural end-to-end approaches eliminated the HMMs, but the need for data remained, at least, the same.

Among this data, the most important modality is having speech audio data with its manual transcription. This paired audio/text data is necessary to train ASR systems. It is also a very costly data that can be rare for many languages.

Self-supervised Learning (SSL) has been recently proposed as an interesting alternative for data representation learning. Proven useful learned representations can be found both in vision [START_REF] Bachman | Learning representations by maximizing mutual information across views[END_REF], [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF] and in NLP [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF], [START_REF] Peters | Deep contextualized word representations[END_REF]. The attractiveness of SSL in general, and SSL from speech in particular, is that it can leverage huge amounts of unannotated data, which is cheaper than the audio/text data used by classical systems. This leveraging can be done by resolving pseudo-tasks, which do not require human annotation, as pre-training a feature extractor, which is then used to extract useful speech representations for the real (downstream) tasks. The two most commonly used approaches for SSL from speech are Autoregressive Predictive Coding (APC) and Contrastive Predictive Coding (CPC). The former's pseudo-task is considering the sequential structure of speech, and predicting information about a future frame [START_REF] Chung | An unsupervised autoregressive model for speech representation learning[END_REF], [START_REF] Chung | Improved speech representations with multi-target autoregressive predictive coding[END_REF], whereas the latter's consists of distinguishing a future speech frame from distractor samples [START_REF] Oord | Representation learning with contrastive predictive coding[END_REF]- [START_REF] Schneider | wav2vec: Unsupervised pre-training for speech recognition[END_REF] which is an easier learning objective compared to APC. These representations have been proven to improve the performance in several speech tasks [START_REF] Chung | Generative pre-training for speech with autoregressive predictive coding[END_REF], while being less sensitive to domain and/or language mismatch [START_REF] Kawakami | Learning robust and multilingual speech representations[END_REF] and being transferable to other languages [START_REF] Riviere | Unsupervised pretraining transfers well across languages[END_REF].

SSL opens new perspectives to build and deploy ASR for low-resource languages, or low-resource domains. Such an approach speeds up the creation of a new ASR system, and reduces its cost, since a significantly smaller amount of annotated data is necessary to get competitive results in comparison to the previous state of the art.

Speech recognition can be involved in many tasks for security purposes. It is also the case for speech translation, on which SSL is also useful, especially for neural end-to-end architectures.

This paper discusses current limitations of the wav2vec 2.0 models, focusing on two applications relevant for security: automatic speech recognition (ASR) and automatic speech translation (AST). It is organized as follows: Section II presents a high-level summarization of the technology behind these SSL models. Section III discusses their application to ASR, especially to process accented French. Section IV discusses their application to AST in an extreme low resource scenario. Section V presents our final remarks. The wav2vec 2.0 proposed by [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF] is an extension of [START_REF] Baevski | Effectiveness of self-supervised pre-training for speech recognition[END_REF], [START_REF] Schneider | wav2vec: Unsupervised pre-training for speech recognition[END_REF], [START_REF] Baevski | vq-wav2vec: Self-supervised learning of discrete speech representations[END_REF]. Depicted in Figure 1, it consists of a multilayer convolutional feature encoder g enc : X → Z, which transforms raw input audio x into latent speech representations z = {z 1 , z 2 , ..., z T } for T time-steps. These latent features are then fed into a Transformer g : Z → C for building contextualized representations c = {c 1 , c 2 , ..., c T } that capture the information of the whole sequence.

The wav2vec 2.0 also performs discretization on the output of the feature encoder z t to q t by using a quantization module Z → Q. The model's Transformer network learns contextualized representations directly from continuous speech representations (z) via time-step masking and a contrastive task (CPC) which identifies the true quantized latent audio representation in a set of distractors for each masked time step. This consequently allows [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF] to train wav2vec 2.0 in an end-to-end fashion, in which all its components are trained jointly toward minimizing an objective (Equations 1, 2, 3).

L = L m + αL d (1)
L m = -log exp(sim(c t , q t )/κ) q∼Qt exp(sim(c t , q)/κ) (2)
L d = 1 GV G g=1 -H(p g ) = 1 GV G g=1 V v=1 pg,v log pg,v (3)
In Equation 1, the training objective is defined as the sum of two components:

• Contrastive Loss L m which is defined in Equation 2.
Particularly, given c t centered over the masked time step t, the model is trained to contrast the true quantized latent speech representation q t from K quantized latent distractors q ∈ Q t uniformly sampled from other masked time steps of the same utterance. sim(a, b) = a ⊤ b/||a||||b|| is the cosine similarity between context representations c t and quantized latent speech representations q t .

• Diversity Loss L d which is defined in Equation 3. It helps to increase the use of the quantized codebook representations, encouraging the model to equally use all the V entries in each of G codebooks by maximizing the entropy of the averaged softmax distribution over the codebook entries for each codebook pg across a batch of utterances. In Equation 1, L d is scaled by α, which is a tunable hyperparameter. Masking: time-step masking mentioned earlier is done by randomly sampling without replacement a certain proportion p of all time steps to be starting indices and then mask the subsequent M consecutive time steps for every sampled index. Note that spans may overlap, and inputs to the quantization module are not masked.

Fine-tuning: the wav2vec 2.0 framework also allows finetuning the pre-trained model directly on ASR (or AST, or speech classification) labeled data by stacking a linear projection layer initialized randomly on top of the context network. Fine-tuned models are optimized by minimizing a Connectionist Temporal Classification (CTC) loss. It has been shown [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF] that fine-tuning even on only 10 minutes of labeled training data (48 recordings of 12.5 seconds on average) helps achieve a respective Word Error Rate (WER) of 4.8% and 8.2% on the test-clean and test-other sets of the Librispeech corpus (read speech).

III. ASR APPLICATION: FROM STANDARD TO ACCENTED SPEECH

In recent years, huge progress has been achieved in the domain of automatic speech recognition (ASR). Neural models now reach human performance on the English ASR task: best systems reach a Word Error Rate (WER) of 5.8% and 11% on Switchboard and CallHome datasets respectively, whereas performance of human annotators is estimated to be around 5.9% and 11.9% [START_REF] Xiong | Toward human parity in conversational speech recognition[END_REF].

However, it has been shown that the performance of these models can decrease drastically when they are used in new or non-ideal conditions. For example, a noisy environment or the accent of a speaker can both impact the quality of the transcription [START_REF] Zhu | A noise-robust self-supervised pre-training model based speech representation learning for automatic speech recognition[END_REF], [START_REF] Fukuda | Data Augmentation Improves Recognition of Foreign Accented Speech[END_REF]. It is important to study the robustness of systems to such conditions, because these are likely to be encountered in real settings. Furthermore, support of accented speech is mandatory if one aspire to build an inclusive, general-purpose ASR system. It could also be of interest for security or intelligence agencies wanting to support a large spectrum of accents. In this section, we focus on the case of accented speech in French, presenting results for ASR models trained on standard and accented French speech.

A. Models and datasets

Focusing on the French language, the LeBenchmark initiative [START_REF] Evain | LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech[END_REF], [START_REF] Evain | Task agnostic and task specific self-supervised learning from speech with LeBenchmark[END_REF] is a proeminent work in the area of SSL benchmarking. It provides evaluation recipes for four downstream tasks (ASR, AST, automatic emotion recognition, spoken language understanding) alongside with wav2vec 2.0 models Fig. 2. ASR results (WER, the lower the better) over the two test sets for models fine-tuned on CommonVoice. of various sizes, and pre-trained using different amounts of speech audio.

For our ASR models, we use the following models from the LeBenchmark: LB-1K-base/large, LB-2.7K-base, LB-3Kbase/large, and LB-7K-base/large, which were pre-trained on respectively 1,096, 2,773, 2,933 and 7,739 hours of French audio [START_REF] Evain | Task agnostic and task specific self-supervised learning from speech with LeBenchmark[END_REF]. The "base" refers to the standard model architecture from [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF] that has 95 millions parameters, while the "large" refers to their larger architecture that presents greater capacity (317 millions parameters).

In addition to these French models, two multilingual models were tested. The first one, Niger-Mali [START_REF] Boito | ON-TRAC consortium systems for the IWSLT 2022 dialect and low-resource speech translation tasks[END_REF], is a base model pre-trained on 641 h of speech in 5 languages, including 111 h of accented French. The second one, XLSR-53 [START_REF] Conneau | Unsupervised cross-lingual representation learning for speech recognition[END_REF], is a large model pre-trained on 56k hours of speech in 53 languages, including 1,429 h of French.

Each pre-trained wav2vec 2.0 model acts as a speech encoder, which is optimized for the ASR task together with an additional feed-forward network. This head network consists of three linear layers with 768 or 1,024 neurons for a base or large model, respectively. Each linear layer is followed by batch normalization and a Leaky ReLU [START_REF] Maas | Rectifier nonlinearities improve neural network acoustic models[END_REF] activation function. We use dropout with p = 0.15 between each linear layer. At last, a final linear layer projects the output into token space, and log-softmax is applied to obtain probabilities of each token. We use individual characters as tokens. Note that we do not apply any language model besides our end-to-end model.

We employ the SpeechBrain [START_REF] Ravanelli | SpeechBrain: A generalpurpose speech toolkit[END_REF] toolkit for all our experiments. All models are fine-tuned during 50 epochs using the CTC loss, and Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] and Adadelta [START_REF] Zeiler | Adadelta: An adaptive learning rate method[END_REF] optimizers are used to update the weights, one for the wav2vec 2.0 model and one for the additional top layers.

We use two different datasets in this study. The first one is the French subset of CommonVoice (CV) 3.0 [START_REF] Ardila | Common voice: A massively-multilingual speech corpus[END_REF] that Fig. 3. ASR results (WER, the lower the better) over the two test sets for models fine-tuned on African Accented French. comprises 56 h of recordings. It represents our reference dataset of unaccented speech. The second one is the African Accented French (AAF) dataset [START_REF]African accented french[END_REF]. It is composed of 13 h of speech 1 . Speakers are from Cameroon, Chad, Congo, Gabon and Niger, and they speak French with a strong African accent. It should be noted that this latter dataset was used as part of the pre-training data for the following LeBenchmark models: LB-3K-base/large, LB-7K-base/large.

B. Fine-tuning ASR models: from non-accented to accented speech

To assess the robustness of the pre-trained wav2vec 2.0 models with respect to accent variability, we fine-tune each model on the train split of CV. Then, we evaluate the resulting models on both the test split of CV and AAF. Results are shown on Figure 2. The trend we observe is that, the more speech data we use for pre-training, the best the model performs, meaning that it better specialized its speech representations.

We also notice that, thanks to their increased capacity, "large" models perform much better than "base" ones. Moreover, multilingual models perform rather badly when compared with French models trained with similar amounts of speech. The best model (LB-7K-large) obtains a WER of 9.37% on CV, which is comparable to the best scores reported in [START_REF] Evain | Task agnostic and task specific self-supervised learning from speech with LeBenchmark[END_REF] on the same dataset (CV). However, this same model scores 20.47% on AAF. Moreover, all the tested models follow a similar trend, with a WER that doubles on AAF compared to CV. This means that these models are likely to make twice as many transcription errors when used by non-native speakers rather than native speakers.

In order to improve the robustness of the models on accented speech, we restart the experiment and fine-tune each pre-Fig. 4. ASR results (WER, the lower the better) over the two test sets for models fine-tuned on a mixed dataset (CV+AAF). trained model on the train split of AAF, and evaluate the resulting models on the same test sets as before. We can see on Figure 3 that doing so greatly reduce the WER on AAF (-75% on average), with our best model now scoring 5.72%. However, this large improvement comes at the cost of a similar performance degradation on CV (+282% on average). This demonstrates that fine-tuning directly on accented speech is beneficial if we desire to transcribe a particular accent, but should not be done if the goal is to build an all-purpose system.

Finally, we created a mixed dataset of accented and native speech by taking the full AAF training set and an equal amount of speech from the CV training set. We use this new dataset to fine-tune the models. Results of evaluation are shown on Figure 4. We can see that fine-tuning on this mixed dataset allows the models to reach good performance on both accented and non-accented speech. Our best model scores 12.38% and 5.47% on CV and AAF respectively. Compared to the models fine-tuned on CV only, these models reach much lower WER on AAF (-75% on average), while only suffering mild performance degradation on CV (+32% on average).

C. Discussion of results

It may seem surprising that the "Niger-Mali" model does not obtain a good performance despite being pre-trained on a large amount of accented French. We believe that the most important factor contributing to low WER is the amount of French audio in the pre-training dataset. The Niger-Mali model is the one with the lowest quantity of French seen during pre-training (111 h), thus explaining its poor score.

The presence of accented French in the pre-training dataset may still play an important role: we can see on Figures 2,3, and 4 that LB-3K-base is achieving slightly better results on AAF compared to LB-2.7K-base, the main difference between these two models being the presence of accented speech in the pre-training dataset of the former.

The second multilingual model has been pre-trained on much more data (56k hours), but only 1,429 h of French. Its scores no better than the other large models pre-trained on much smaller (but French only) datasets. This seems to indicate that multilingual models, despite obtaining good performances on a variety of languages, are not suited for recognizing accented speech.

In summary, in this section we illustrated existing limitations of ASR models for transcribing accented speech in French. We experimented with two multilingual wav2vec 2.0 models, and seven French models, comparing the performance of obtained ASR systems in standard and accented French speech. We find that models fine-tuned on native speech only are not robusts to accent variation, but that incorporating accented data in the fine-tuning dataset greatly improve robustness.

IV. AST APPLICATION: SSL MODELS FOR LOW-RESOURCE END-TO-END AST

Traditionally, the speech translation task is defined in a cascaded fashion: the speech is first transcribed by an ASR model, and then a text-to-text machine translation (MT) module produces the final translation in the target language. The limitations of this approach for AST includes the error propagation between the ASR and MT modules, the omission of speech cues that could disambiguate the information given to the MT module, and the need for both a considerable amount of transcribed and translated data.

Going beyond the practical time and money constrains for producing this data in non-mainstream languages in order to train and deploy cascaded systems, and the cost of training the systems themselves, it is also important to be aware that not all languages present a standard written form. Indeed, most of the world's languages are not actively written, even the ones with an official writing system [START_REF] Bird | Bootstrapping the language archive: New prospects for natural language processing in preserving linguistic heritage[END_REF]: these are called oral languages.

This is one reason behind the recent motivation of the speech community to investigate end-to-end approaches for AST [START_REF] Berard | Listen and translate: A proof of concept for end-to-end speech-to-text translation[END_REF], [START_REF] Weiss | Sequenceto-Sequence Models Can Directly Translate Foreign Speech[END_REF]. We define end-to-end AST as a single optimized model that receives as input speech and produces as output textual translations. Optionally, these models can be jointly optimized for producing transcriptions as well, when these are available during training. This joint training was shown to increase translation performance [START_REF] Le | Dualdecoder transformer for joint automatic speech recognition and multilingual speech translation[END_REF], [START_REF] Sperber | Consistent transcription and translation of speech[END_REF].

In this section we shed light on some limitations of SSLbased end-to-end AST models for processing oral-languages, for which the amount of available data is limited. This is relevant in the context of security because a government or an organization might aim to deploy AST models for minority or dialect languages in areas of particular interest. In these cases, the amount of available data is usually limited. Ideal AST systems for security should thus be able to work in lowresource settings.

This section is organized as follows. We first validate our AST architecture by producing results for three language pairs in the mTEDx dataset [START_REF] Salesky | Multilingual tedx corpus for speech recognition and translation[END_REF] (Section IV-A). Fig. 5. AST results (BLEU scores, the higher the better) over the test set for the three language pairs, and using the base and large wav2vec 2.0 models.

These languages have decreasing amounts of available parallel data: French-English (48 h), French-Spanish (35 h), French-Portuguese (23 h).

Having defined these mid-to-low-resource baselines, in Section IV-B, we explore the case of the Tamasheq dataset presented in this year's IWSLT campaign [START_REF] Anastasopoulos | FINDINGS OF THE IWSLT 2022 EVALUATION CAMPAIGN[END_REF]. The challenge is producing translation, without available transcription, having only 17 hours of speech in Tamasheq aligned to French translations. In this case, we illustrate how general purpose SSL models fail to produce exploitable representations. Lastly, in Section IV-C we summarize our findings on the use of SSL models for low-resource end-to-end AST.

A. AST in mid-to-low-resource settings

The presented end-to-end AST models are similar to the end-to-end ASR model architecture presented in Section III. They are implemented on SpeechBrain [START_REF] Ravanelli | SpeechBrain: A generalpurpose speech toolkit[END_REF], being made of a wav2vec 2.0 as a foundation block, followed by a linear projection, and a Transformer Decoder [START_REF] Vaswani | Attention is all you need[END_REF]. The weights for the wav2vec 2.0 speech encoder block are initialized from the pre-trained SSL models available in the LeBenchmark model collection [START_REF] Evain | LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech[END_REF], [START_REF] Evain | Task agnostic and task specific self-supervised learning from speech with LeBenchmark[END_REF]. 2 The model is trained on the negative log likelihood loss, and two different instances of the Adam optimizer manage the weight updates: one dedicated to the wav2vec 2.0 block, the other one to the following layers.

As aforementioned, we train mid-to-low-resource baselines using the mTEDx dataset. We use as source the French language (speech), and as target languages (text): English (en), Spanish (es) and Portuguese (pt). The resulting language pairs share validation and test sets, but they vary on the amount of available training data. This information (duration), together with the number of speakers (# spk), is presented in Table I. 2 Available at https://huggingface.co/LeBenchmark Figure 5 presents the AST results3 using the three language pairs, and three different wav2vec 2.0 models: LB-1K, LB-3K and LB-7K. These models differ on the amount of training data used during SSL pre-training, with 1K corresponding to approximately 1,000 hours of speech. For each model, we experiment with both base and large architecture sizes. 4Looking at the results using base wav2vec 2.0 models (Figure 5, darker bars), we notice that AST models trained in all language pairs benefit from having SSL wav2vec 2.0 models trained using more data: using LB-7K-base and LB-3K-base as foundation blocks seem to be clearly superior compared to AST models that used LB-1K-base. We however, do not observe a very clear distinction between LB-7K-base and LB-3K-base, which might be due to the wav2vec 2.0 model reaching the limits of its own capacity [START_REF] Evain | Task agnostic and task specific self-supervised learning from speech with LeBenchmark[END_REF].

Focusing on the large models (Figure 5, brighter bars), we notice that the trend is not the same for all languages. For English, it seems to still exist some benefit on employing these larger models, compared to their base counterparts: BLEU scores are higher or equivalent. For the other two languages, we notice that performance using large models is inferior to the one reached by their base counterparts.

For these languages, we have less training examples compared to English: for Spanish we have only 35 h, and for Portuguese only 23 h. We thus believe that this discrepancy in performance between base and large models for these languages might be related to the amount of available data, since for large wav2vec 2.0 models we have an additional 221.2 million trainable parameters. 5 The lack of available data might result on these extra parameters not being properly fine-tuned, thus resulting in the observed deterioration in performance. We believe that the fine-tuning of base models might be more realistic in settings of data scarcity, as the overhead caused by the extra parameters in large architectures seem to be excessive for models working with less than 50 hours of speech.

Finally, it is also important to highlight that the results obtained for our baselines in this work are considerably higher compared to results obtained with AST models trained without SSL models as a foundation block. In [START_REF] Salesky | Multilingual tedx corpus for speech recognition and translation[END_REF], and for the same dataset, they reach BLEU scores of 8.9, 10.6 and 7.9 for English, Spanish and Portuguese respectively.

Summarizing, in this section we presented end-to-end AST models in mid-to-low-resource settings. For the language pair with the most available speech data (fr-en) we observe benefits on having large pre-trained SSL models, and we reach acceptable BLEU scores compared to the literature [START_REF] Ansari | Findings of the iwslt 2020 evaluation campaign[END_REF], [START_REF] Anastasopoulos | FINDINGS OF THE IWSLT 2021 EVALUATION CAMPAIGN[END_REF]. This finding however does not hold as we reduce the amount of trainable data: even 35 h in Spanish seems not to be enough to fully fine-tune a large wav2vec 2.0 architecture, and results for large models drag behind the results for base models.

The challenge of low-resource AST is illustrated with the clear performance drop between our three setups: fr-en reaches higher performance than fr-es (data reduction of 13 h), and the latter outperforms fr-pt models (data reduction of 12 h compared to Spanish, 25 h compared to English). This highlights the existence of a minimal amount of data needed in order to make the training of end-to-end AST architectures based on SSL models exploitable.

B. Use case: AST for Tamasheq

We now present our experiments for the Tamasheq-French dataset in the context of the IWSLT 2022 low-resource speech translation track. The dataset contains 17 h of speech in the Tamasheq language, which corresponds to 5,829 utterances translated to French [START_REF] Boito | Speech resources in the tamasheq language[END_REF]. Additional audio data was also made available through the Niger-Mali audio collection: 224 h in Tamasheq and 417 h in geographically close languages (French from Niger, Fulfulde, Hausa, and Zarma). 6 For all this data, the speech style is radio broadcasting, and the dataset presents no transcription.

We start by training AST models that use wav2vec 2.0 models simply as feature extractors: the output of these SSL architectures replaces commonly used mel filterbank (MFB). This was shown to result in a considerable performance boost using the mTEDx dataset in [START_REF] Evain | LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech[END_REF], and we choose this approach since our results from the previous session hint that end-toend fine-tuning for the AST task requires more data than the available 17 h.

In these settings, we compare two general purpose wav2vec 2.0 models -the multilingual XLSR-53 [START_REF] Conneau | Unsupervised cross-lingual representation learning for speech recognition[END_REF] and the French LB-7K-large [START_REF] Evain | Task agnostic and task specific self-supervised learning from speech with LeBenchmark[END_REF] -against two smaller base models trained in the target language: Tamasheq-only, trained on 243 h of Tamasheq, and Niger-Mali, trained on the totality of the Niger-Mali audio collection (641 h).

Our AST models that use wav2vec 2.0 models as feature extractors are very close to the recipe for low-resource ST from wav2vec 2.0 features described in [START_REF] Evain | Task agnostic and task specific self-supervised learning from speech with LeBenchmark[END_REF]. We use the fairseq s2t toolkit [START_REF] Wang | fairseq s2t: Fast speech-to-text modeling with fairseq[END_REF] for training an end-to-end AST Transformer model [START_REF] Vaswani | Attention is all you need[END_REF] with 4 heads, dimensionality of 256, inner projection of 1,024, 6 encoder and 3 encoder layers. The Transformer is preceded by a 1D convolutional layer (k=5, stride=2) for down-projecting the wav2vec 2.0 large (1,024) or base (768) features into the Transformer input dimensionality. These models are trained for 500 epochs using the Adam optimizer with 10k warm-up steps. For decoding, we use beam search with a beam size of 5. We generate a 1k unigram vocabulary for the French text using Sentencepiece [START_REF] Kudo | Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing[END_REF], with no pre-tokenization. Lastly, we include baseline results that replace wav2vec 2.0 features by 80-dimensional MFB features. In this setting, the CNN preceding the transformer encoder is identical from the one in [START_REF] Evain | Task agnostic and task specific self-supervised learning from speech with LeBenchmark[END_REF].

AST results using the four wav2vec 2.0 models as feature extractors are presented in Table II. For each model other than Tamasheq-only, we investigate fine-tuning on a task-agnostic 6 https://demo-lia.univ-avignon.fr/studios-tamani-kalangou/ fashion for approximately 20,000 updates on all available Tamasheq speech (243 h). This fine-tuning is supposed to reduce performance issues related to domain shift, however, in our setting we do not notice a significant performance gap between fine-tuned models and their pre-trained counterparts.

Regarding the overall very low AST performance, 7 it is notable that the results obtained by using general purpose wav2vec 2.0 models are not very different from the baseline results (MFB). This was also observed in the literature: it seems wav2vec 2.0 models tend to perform poorly as feature extractors in low-resource settings, compared to MFB [START_REF] Berrebbi | Combining spectral and self-supervised features for low resource speech recognition and translation[END_REF]. Finally, although performance is poor regardless of the feature extractor, the wav2vec 2.0 models trained on target data seem to output superior features for Tamasheq-French AST. This is despite the fact they are trained with considerably smaller quantities of data compared to the large and general purpose models.

Based on this finding, the best AST results we reported on [START_REF] Boito | ON-TRAC consortium systems for the IWSLT 2022 dialect and low-resource speech translation tasks[END_REF] used these smaller wav2vec 2.0 models on an end-toend fashion. However, as expected by our mid-to-low-resource baseline results from last section, there are not enough training hours to successfully fine-tune an entire wav2vec 2.0-based AST architecture for Tamasheq-French. 8We circumvent this by exploring the representation from intermediate layers: previous work [START_REF] Pasad | Layer-wise analysis of a self-supervised speech representation model[END_REF] has shown that the middle layers inside the Transformer Encoder inside the wav2vec 2.0 architecture contain a higher abstraction level with respect to the speech signal, being more useful for endto-end ASR fine-tuning compared to the last layers. Inspired by that finding, we experimented pruning the last layers of the wav2vec 2.0 model, which reduced the amount of trainable parameters. Our final model, and the best result for the IWSLT 2022 low-resource task, was an end-to-end wav2vec 2.0-based AST model using the Tamasheq-only model. It comprised only 7 Transformer layers (out of 12) on its wav2vec 2.0 foundation block, and it achieved a BLEU4 of 6.0.

C. SSL models for low-resource end-to-end AST

Throughout this section, we illustrated how the performance of wav2vec 2.0-based AST models drops in settings of data scarcity. One important aspect of our results is the complete lack of transcription we impose to our experimental setup: by doing this we reduce the final translation scores, but we are able to assess performance for situations where this information is not available (e.g. the processing of oral dialects).

The main finding of our AST experiments is the overall under-performance of off-the-shelf wav2vec 2.0 large models in low-resource settings, even after fine-tuning them on target data. We also notice that the wav2vec 2.0 base models we trained with considerable less data were more effective in these same settings. We believe this happens because these taskspecific SSL models better inform downstream tasks such as AST, since there is no domain shift in the data representation. This hints that massive multi-purpose wav2vec 2.0 models might not be the adequate solution for low-resource speech-totext approaches, and that instead, smaller and better informed SSL blocks, trained on target data and/or domain, should be favored.

V. FINAL REMARKS

This paper presented some premises and limitations of wav2vec 2.0 models. These are promising because they allow us to build ASR and AST models with less data than the previous approaches: this is an important issue since in-domain manually labeled data is very rare. This offers a new mean of action in order to address new languages.

Regarding our ASR experiments, we have seen that in order to improve recognition of accented speech, it is necessary to include accented speech in the fine-tuning data. Fine-tuning on both accented and non-accented speech seems to be a promising method for building general-purpose systems. In this work we used an equal amount of accented and native speech in our mixed dataset. The variation of the amount of accented speech in the fine-tuning dataset, the inclusion of additional accents (Swiss French, Quebec French), and the use of data augmentation to increase artificially the amount of accented speech are left as future work.

Regarding our AST experiments, we illustrated how building speech translation models without the existence of transcriptions is a challenging topic, and that models based on off-the-shelf wav2vec 2.0 models fail performance-wise in low-resource settings. Future research will focus on increasing performance in challenging settings: techniques such as speech augmentation, the production of dummy transcriptions [START_REF] Boito | ON-TRAC consortium systems for the IWSLT 2022 dialect and low-resource speech translation tasks[END_REF], and multilingual pre-training and adaptation are promising topics.

The LIA will continue to study these approaches with the goal of making them highly accurate for real-world data.

Fig. 1 .

 1 Fig.1. Illustration of the wav2vec 2.0 framework (left), which jointly learns contextualized speech representations and an inventory of discretized speech units during pre-training on unlabeled data. The fine-tuning step (right) can be applied to different tasks on labeled data. Illustration adapted from[START_REF] Babu | Xls-r: Self-supervised cross-lingual speech representation learning at scale[END_REF]

TABLE I STASTISTICS

 I FOR THE MTEDX FR-{EN,ES,PT} DATASET.

	en			train es		pt		valid		test
	# spk duration	# spk duration # spk	duration	# spk duration	# spk	duration
	250	45:04	196	32:30	112	20:01	12	1:38	10	1:33

The original dataset is larger, but we excluded portions containing annotation errors.

BLEU4 scores computed using sacreBLEU[START_REF] Post | A call for clarity in reporting BLEU scores[END_REF].

There are approximately 221.2 million extra parameters in the large architecture.

In this work we do not explore partially freezing the wav2vec 2.0 blocks.

High-resource end-to-end speech translation BLEU4 scores range from 19 to 30 in the last editions of IWSLT[START_REF] Ansari | Findings of the iwslt 2020 evaluation campaign[END_REF],[START_REF] Anastasopoulos | FINDINGS OF THE IWSLT 2021 EVALUATION CAMPAIGN[END_REF].

The best BLEU score for an end-to-end wav2vec 2.0 AST model was of 2.34, by using the Tamasheq-only wav2vec 2.0 model.

ACKNOWLEDGMENT

This work used HPC resources from GENCI-IDRIS (grants 2020-A0111012991, 2021-AD011013317 and 2021-AD011013331). It was also funded by the European Commission through the SELMA project under grant number 957017.


Jules Bourcier
email: jules.bourcier@preligens.com

Thomas Floquet
email: thomas.floquet@minesparis.psl.eu

Gohar Dashyan
email: gohar.dashyan@preligens.com

Tugdual Ceillier

Karteek Alahari

Jocelyn Chanussot

Jules Bourcier Preligens

Self-Supervised Pretraining on Satellite Imagery: A Case Study on Label-Efficient Vehicle Detection

Keywords: deep learning, computer vision, remote sensing, self-supervised learning, object detection, land use classification, label-efficient learning

niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

I. INTRODUCTION

Very high resolution (VHR) satellite imagery is one of the key data from which geospatial intelligence can be gathered. It is an essential tool to detect and identify a wide range of objects, on very large areas and on a very frequent basis. Recently, we have seen the multiplication of available sensors, which has led to a large increase in the volume of data available. This makes it very challenging for human analysts to exploit these data without resorting to automatic solutions. Deep learning techniques today have been highly effective to perform such tasks. However, training those models requires very large labeled datasets. Annotating objects of interest in VHR images can prove to be very costly, being both difficult and time-consuming, and requiring fine domain expertise. In specific contexts such as in geospatial intelligence, the targets can be intrinsically rare, difficult to localize and to identify accurately. This makes the acquisition of thousands of examples impractical, as is typically required for classic supervised deep learning methods to generalize. Consequently, a major challenge is the development of label-efficient approaches, i.e., models that learn with few annotated examples.

To reduce the number of training samples for difficult vision tasks such as object detection, transfer learning of pretrained neural networks is used extensively. The idea is to reuse a network trained upstream on a large, diverse source dataset. Ima-geNet [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF] has become the de facto standard for pretraining: due to its large-scale and genericity, ImageNet-pretrained models show to be adaptable beyond their source domain, including remote sensing imagery [START_REF] Neumann | Indomain representation learning for remote sensing[END_REF]. Nonetheless, the domain gap between ImageNet and remote sensing domains brings questions about the limitations of this transfer when there are very few samples on the task at hand, e.g., the detection of rare observables from satellite images. To fit the distributions of downstream tasks with maximum efficiency, one would ideally use generic in-domain representations, obtained by pretraining on large amounts of remote sensing data. This is infeasible in the remote sensing domain due to the difficulty of curating and labeling these data at the scale of ImageNet. However, imaging satellites provide an ever-growing amount of unlabeled data, which makes it highly relevant for learning visual representations in an unsupervised way. Self-supervised learning (SSL) has recently emerged as an effective paradigm for learning representations on unlabeled data. It uses unlabeled data as a supervision signal, by solving a pretext task on these input data, in order to learn semantic representations. A model trained in a self-supervised fashion can then be transferred using the same methods as a network pretrained via a upstream supervised task. In the last two years, SSL has shown impressive results that closed the gap or even outperformed supervised learning for multiple benchmarks [START_REF] Caron | Unsupervised learning of visual features by contrasting cluster assignments[END_REF], [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF], [START_REF] Grill | Bootstrap your own latenta new approach to self-supervised learning[END_REF], [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF]. Recently, SSL has been applied in the remote sensing domain to exploit readily-available unlabeled data, and was shown to reduce or even close the gap with transfer from ImageNet [START_REF] Ayush | Geography-aware selfsupervised learning[END_REF], [START_REF] Mañas | Seasonal contrast: Unsupervised pretraining from uncurated remote sensing data[END_REF], [START_REF] Zheng | Self-supervised pretraining and controlled augmentation improve rare wildlife recognition in uav images[END_REF]. Nonetheless, the capacity of these methods to generalize from few labels has not been explored on the important problem of object detection in VHR satellite images.

In this paper, we explore in-domain selfsupervised representation learning for the task of object detection on VHR optical satellite imagery. We use the large land use classification dataset Functional Map of the World (fMoW) [START_REF] Christie | Functional map of the world[END_REF] to pretrain representations using the unsupervised framework of MoCo [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF]. We then investigate the transferability on a difficult real-world task of fine-grained vehicle detection on proprietary data, which is designed to be representative of an operational use case of strategic site surveillance. Our contributions are:

• We apply a method based on MoCo with temporal positives [START_REF] Ayush | Geography-aware selfsupervised learning[END_REF] to learn self-supervised representations of remote sensing images, that we improve using (i) additional augmentations for rotational invariance; (ii) a fixed loss function that removes the false temporal negatives in the learning process. • We investigate the benefit of in-domain selfsupervised pretraining as a function of the annotation effort, using different budgets of annotated instances for detecting vehicles. • We show that our method is better than or at least competitive with supervised ImageNet pretraining, despite using no upstream labels and 3× less upstream data. Furthermore, our in-domain SSL model is more label-efficient than ImageNet: when using very limited annotations budgets (≃20 images totalling ≃12k observables), we outperform ImageNet pretraining by 4 points AP on vehicle detection and 0.5 point mAP on joint detection and classification.

II. RELATED WORK

A. Self-supervised representation learning SSL methods use unlabeled data to learn representations that are transferable to downstream tasks (e.g. image classification or object detection) for which annotated data samples are insufficient. In recent years, these methods have been successfully applied to computer vision with impressive results that closed the gap or even outperformed supervised representation learning on ImageNet, on multiple benchmarks including classification, segmentation, and object detection [START_REF] Caron | Unsupervised learning of visual features by contrasting cluster assignments[END_REF], [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF], [START_REF] Grill | Bootstrap your own latenta new approach to self-supervised learning[END_REF], [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF]. They commonly rely on a pretraining phase, where a neural network, a representation encoder, is trained to solve a pretext task, for which generating labels does not require any effort or human involvement. Solving the pretext task is done only for the true purpose of learning good data representations that allow for efficient training on a downstream task of genuine interest.

B. Contrastive learning

Contrastive learning has recently become the most competitive unsupervised representation learning framework, with approaches such as MoCo [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF], SimCLR [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF], and SwAV [START_REF] Caron | Unsupervised learning of visual features by contrasting cluster assignments[END_REF]. Contrastive methods work by attracting embeddings of pairs of samples known to be semantically similar (positive pairs) while simultaneously repelling pairs of unlike samples (negative pairs). The most common way to define similarity is to use the instance discrimination pretext task [START_REF] Dosovitskiy | Discriminative unsupervised feature learning with convolutional neural networks[END_REF], [START_REF] Wu | Unsupervised feature learning via non-parametric instance discrimination[END_REF], in which positives are generated as random data augmentations on the same image, and negatives are simply generated from different images. Thanks to this pretext task, the encoder learns similar representations for several views of the same object instance in an image and distant representations for different instances. Momentum Contrast (MoCo) [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF] is a strong contrastive method that implements a dynamic dictionary with a queue and a moving-averaged encoder, which enables building a large and consistent dictionary on-the-fly (see section III-A for more details). In this paper, we adopt a recent geography-aware rework of this approach made by [START_REF] Ayush | Geography-aware selfsupervised learning[END_REF].

C. Representation learning in remote sensing

Building on that success in computer vision, SSL has recently been applied to remote sensing and was shown to reduce or even close the gap with transfer from ImageNet. [START_REF] Jean | Tile2vec: Unsupervised representation learning for spatially distributed data[END_REF] first made use of contrastive learning for remote sensing representation learning and [START_REF] Kang | Deep unsupervised embedding for remotely sensed images based on spatially augmented momentum contrast[END_REF] also apply a spatial augmentation criteria on top of MoCo [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF]. These works exploit relevant prior knowledge about the remote sensing domain: the assumption that images that are geographically close should be semantically more similar than distant images. Another way of making the learning procedure geographyaware is to exploit the image time series that one can get from a given geographic area thanks to the frequent revisit of satellites. This approach is adopted by [START_REF] Ayush | Geography-aware selfsupervised learning[END_REF], that use spatially aligned images over time to construct temporal positive pairs. With their geography-aware representations learned on the fMoW dataset [START_REF] Christie | Functional map of the world[END_REF], they improve significantly on classification, segmentation and object detection downstream tasks. However, they do not study label efficiency. In this paper, we apply this method, called MoCoTP, to an operational use-case. We study the label efficiency and bring small extensions to this model, that further improve its performance. In the same vein, [START_REF] Mañas | Seasonal contrast: Unsupervised pretraining from uncurated remote sensing data[END_REF] present a pipeline for selfsupervised pretraining on uncurated remote sensing data and propose a method that learns representations that are simultaneously variant and invariant to temporal changes. They significantly outperform ImageNet pretraining on classification from few labels on medium resolution images (10m). One can also exploit the multispectral and multisensor nature of remote sensing. [START_REF] Stojnic | Self-supervised learning of remote sensing scene representations using contrastive multiview coding[END_REF] split multispectral images into two different subsets of channels and use them as augmented (positive) views. [START_REF] Swope | Representation learning for remote sensing: An unsupervised sensor fusion approach[END_REF] extend this to multiple sensors, taking subsets of the combination of all bands. Regarding the domain of pretraining data in remote sensing, [START_REF] Neumann | Indomain representation learning for remote sensing[END_REF] show that the relatedness of the pretraining distribution to the downstream task can improve performances in low-labeled settings. However, they only study supervised pretraining and classification tasks, and show that transfer performance is very dependent on the labeling and data curation quality in the pretraining dataset. Therefore this leaves unresolved the problem of obtaining generic representations with less dependence on labels and this is where SSL can help.

III. METHOD

In this section, we detail our approach to explore the applicability of SSL to vehicle detection and classification on optical satellite imagery. The overall procedure is described in Fig. 1. We first pretrain a ResNet-50 backbone on the fMoW dataset [START_REF] Christie | Functional map of the world[END_REF] in an unsupervised way with a recent SSL method, MoCoTP [START_REF] Ayush | Geography-aware selfsupervised learning[END_REF]. See section III-A for details on the approach and section IV-A2 for implementation details.

We use the weights of this pretrained backbone on two downstream tasks: (i) fMoW image recog- nition task: we inject the weights in a classifier, and perform linear probing and finetuning. See section IV-A for more details. (ii) Vehicle detection and classification on Preligens proprietary data: we inject the weights in a RetinaNet detector [START_REF] Lin | Focal loss for dense object detection[END_REF], and finetune the entire model. See section IV-B2 for implementation details.

A. Self-supervised learning with MoCo and Temporal Positives

We employ the MoCo [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF] framework for contrastive SSL. The base method we use is from the improved variant MoCo-V2 [START_REF] Chen | Improved baselines with momentum contrastive learning[END_REF]. MoCo learns to match an input query q to a key k + (representing the encoded views of the same sample) among a set of negative keys k -, using the instance discrimination pretext task [START_REF] Wu | Unsupervised feature learning via non-parametric instance discrimination[END_REF]. It uses a deep encoder (e.g. a ResNet [START_REF] He | Deep residual learning for image recognition[END_REF]) to map input image queries and keys to a vector representation space. Negative keys are extracted with a moving average network (momentum encoder) to maintain consistent representations during training, and are drawn from a memory queue. We refer readers to [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF] for details on this. MoCo uses the popular choice of InfoNCE [START_REF] Oord | Representation learning with contrastive predictive coding[END_REF] for the contrastive loss:

L(q, k + ) = -log e (q•k + /τ ) e (q•k + /τ ) + k -e (q•k -/τ ) (1)
where τ is a temperature scaling parameter.

On top of MoCo, we adopt the extension to temporal views proposed in [START_REF] Ayush | Geography-aware selfsupervised learning[END_REF], MoCo with Tem-poral Positives (MoCoTP). It extends the instance discrimination pretext task to use spatially aligned images from different times as positives. Maximizing similarity between temporal views can provide richer semantic information that extracts persistent scene features over time [START_REF] Ayush | Geography-aware selfsupervised learning[END_REF]. The same random augmentations as in MoCo-V2 are also applied on the temporal samples.

Improvements to MoCoTP. Compared to Mo-CoTP, we adopt the following modifications from [START_REF] Bourcier | Evaluating the label efficiency of contrastive selfsupervised learning for multi-resolution satellite imagery[END_REF], to improve the framework of [START_REF] Ayush | Geography-aware selfsupervised learning[END_REF]: (i) In addition to the geometric and color perturbations of MoCo-V2, we apply random vertical flips and rotations by multiples of 90°. Since the data augmentation scheme plays a leading role in contrastive learning [START_REF] Tian | What makes for good views for contrastive learning?[END_REF], we aim to learn representations more suited to overhead images thanks to rotational invariance. (ii) [START_REF] Ayush | Geography-aware selfsupervised learning[END_REF] introduces temporal positives as a drop-in replacement for q and k + in (1). However, this can introduce false negatives. Indeed, at each iteration of training, it may happen that the set of negatives k -contains temporal views for samples of the current mini-batch of queries. Such false negatives will cause an incorrect repulsion between the embeddings of similar samples. To avoid the false negatives to interfere with the learning objective, we simply mask out the logits q•k -in the InfoNCE loss in (1) for every k -that happens to be a temporal view of q.

After being trained on the pretext task for a given number of iterations, the query encoder is extracted and can be transferred to downstream tasks.

B. Transfer to dowstream task

Following the MoCoTP pretraining on fMoW, we transfer the obtained weights on two downstream tasks: fMoW image recognition task and a realworld use-case, vehicle detection and classification on Preligens proprietary data. We refer to the downstream training initialized with SSL weights learned on the fMoW dataset as fMoW-MoCoTP init. For each downstream task, we compare fMoW-MoCoTP init with two baselines:

• IN-sup init: the backbone has been pretrained on ImageNet in a supervised way;

• Random init: the backbone is initialized randomly (i.e., no pretraining).

IV. EXPERIMENTAL SETUP

A. Pretraining and evaluation on fMoW 1) Dataset: For the sake of learning semantic representations in remote sensing, we adopt the fMoW dataset [START_REF] Christie | Functional map of the world[END_REF], following [START_REF] Ayush | Geography-aware selfsupervised learning[END_REF]. fMoW is a public dataset of VHR imagery from Maxar Earth observation satellites, is large-scale with 363,571 training images, and covers 207 countries. It provides images from same locations over time. We apply the self-supervised MoCoTP method for pretraining on fMoW training set using these available temporal views. fMoW also includes ground-truths labels for functional land use classification with 62 diverse categories with a long-tailed distribution. We do not use those labels for self-supervised pretraining, but use them downstream to evaluate representations learned directly for the classification of the images seen during pretraining. Following [START_REF] Christie | Functional map of the world[END_REF] and [START_REF] Ayush | Geography-aware selfsupervised learning[END_REF], we use the fMoW-RGB products for our experiments, which provides 3-bands imagery at 0.5m ground resolution. Preprocessing is applied identically to [START_REF] Christie | Functional map of the world[END_REF] to resize input images to 224×224 pixels.

Ensuing the pretraining stage, the model is evaluated on the land use classification task with the two protocols of linear probing (training a linear classifier on top of frozen features from the pretrained encoder) or finetuning (updating all parameters of the network). To study the label efficiency of the learned representations, supervised evaluation is performed on 1%, 10%, and 100% of the labeled training data. For 1% and 10% labels, we subsample by preserving the base class distribution. The evaluation metric is the F1-score averaged over classes. Testing is performed on the fMoW validation set for comparison with [START_REF] Ayush | Geography-aware selfsupervised learning[END_REF], which consists of 53,041 images.

2) Implementation details: In all our experiments, we use MoCoTP with ResNet-50 for the query and key encoders. We utilize the same hyperparameters as in [START_REF] Bourcier | Evaluating the label efficiency of contrastive selfsupervised learning for multi-resolution satellite imagery[END_REF]. For self-supervised pretraining, we use a learning rate of 3e-2 with a cosine schedule, batch size of 256, dictionary queue size of 65536, temperature scaling of 0.2, SGD optimizer with a momentum of 0.9, weight decay of 1e-4. Augmentations are the same as in MoCo-V2, plus a vertical flip with 0.5 probability and a 90°rotation with 0.75 probability. Pretraining lasts 200 epochs. For linear probing, we use a learning rate of 1, no weight decay, and only random resized cropping for the augmentations. For finetuning, we use a learning rate of 3e-4 for ResNet weights and 1 for the final classification layer, weight decay of 1e-4, and the same augmentations used for pretraining. We compare self-supervised pretraining against Random init and IN-sup init, under the different label regimes. Models are trained with cross-entropy loss and evaluated on epoch with the highest top-1 accuracy on the validation set.

B. Transfer to vehicle instance detection 1) Dataset:

We describe here the Preligens proprietary datasets used for the transfer to object detection. Statistics of our datasets are reported in Table I.

We call 'S' our base dataset. It consists of 204 Maxar WorldView-3 satellite images at 0.3m resolution, and approximately 150k vehicles. To study the label efficiency, we subsample this base dataset S into smaller datasets, 'XS' and 'XXS', targeting respectively 50% and 10% of the observables present in S. We ensure that XXS is included in XS, so that our datasets follow a "Matriochka" structure, which simulates the incremental nature of annotation efforts. The sampling strategy is such that the class distribution is well preserved. To perform variance experiments on our results, we run the sampling thrice and get three different variants of the XS and XXS datasets. We proceed similarly for the S training set by selecting other satellite images that match the class distribution of the initial S dataset. We keep the same validation and testing sets throughout the experiments, and make sure that the geographical sites of the images are distinct between train and test splits. The training and validation raster images are divided into tiles of 512x512 pixels with an overlap of 128 pixels. We take all positive tiles (i.e., tiles containing at least one instance) and some negative tiles, with a positive/negative ratio of 5 and 3 for training and validation sets respectively: this setting allows to focus training efforts on positive tiles while keeping a fair amount of negative tiles. The ground truth labels are non-oriented bounding boxes of target observables with their class label. Our classification problem is composed of 8 vehicle categories: Civilian, Military, Armored, Launcher, Ground Support Equipment (GSE), Electronics, Heavy Equipment (HE), and Lifting Equipment (LE). The classes are naturally heavily imbalanced: the first three classes are very dominant, covering ∼96.5% of the vehicles in our datasets, while the last five classes are very under-represented.

2) Implementation details:

The detection model. The object detection model used is a RetinaNet [START_REF] Lin | Focal loss for dense object detection[END_REF], with a ResNet-50-FPN backbone [START_REF] Lin | Feature pyramid networks for object detection[END_REF] with pyramid levels P 2 to P 6 . The backbone is initialized with the pretrained weights, while the detector specific modules are initialized randomly. We finetune the RetinaNet model endto-end, and use the focal loss objective for the classification [START_REF] Lin | Focal loss for dense object detection[END_REF].

Hyperparameters. We select a learning rate of 1e-4, an Adam optimizer and a batch size of 8, and no weight decay. As data augmentations, we use 90°rotations and flips, as well as CLAHE. We train until convergence and reduce the learning rate by 2 on plateau of the validation loss. Then, the epoch selected for the evaluation of the model is the one achieving the best F1-score calculated on the validation set, with a fixed detection score threshold of 0.15.

Evaluation. The F1-score is computed from the precision-recall curves obtained by varying the detection threshold from 0.15 to 0.9. In the following, we refer to the results on the task of vehicle detection (regardless of the class) as level-1 results, and on the task of joint detection and classification as level-2 results. The IoU threshold matching the predictions with the ground truths is set to 0.0, which is operationally relevant for observable counting purposes. In addition to the level-1 F1-score, we also compute the level-1 average precision (AP) and level-2 mean average precision (mAP) which are commonly used metrics to evaluate detection models.

V. RESULTS

A. fMoW classification

Table II shows the results of linear probing and finetuning on the 62-class land use classification task of fMoW. With 100% labels, we see that our improved reproduction of MoCoTP increases performance by 4.36 pts compared to [START_REF] Ayush | Geography-aware selfsupervised learning[END_REF] in linear probing and 1.62 pts in finetuning. This shows that the use of the rotation augmentations and the correction of false negatives in the loss function is helpful, especially for linear probing, which closes the gap with finetuning completely. As a note, the sole additional augmentations also improve the baselines Random init and IN-sup init of [START_REF] Ayush | Geography-aware selfsupervised learning[END_REF] by 1.29 pts and 0.68 pts respectively. Moreover, MoCoTP shows impressive label efficiency: in the semi-supervised settings of 1% and 10% labels, we see that it gives respectively 96% and 87% of the performance of the network trained with 100% labels. Also, it surpasses IN-sup init by large margins, with e.g. +20.57 pts on 1% finetune. These results indicate that MoCoTP is very efficient at learning semantic features from the upstream dataset. Therefore, this is encouraging in order to transfer to a downstream operational task where labeled data are scarce.

B. Transfer to vehicle detection

1) Label efficiency: Table III and Fig. 2 show the results on vehicle detection. The F1-score with fMoW-MoCoTP init is always higher than with IN-sup init or Random init. fMoW-MoCoTP init achieves an F1-score of 65.1% with only 12k observables on dataset XXS. On dataset XS, with 50% less examples than on dataset S, fMoW-MoCoTP init is only 3.8 pts below the score obtained on dataset S. Moreover, the smaller the dataset, the larger the gap between fMoW-MoCoTP init and IN-sup init or Random init: on dataset S, fMoW-MoCoTP init's F1-score is 5.20 pts better than Random init on average, and also 0.40 pts better than IN-sup init, whereas on the XXS dataset, fMoW-MoCoTP init's F1-score is 39 pts better than Random init, and 3.7 pts better than IN-sup init. These results show that self-supervised in-domain pretraining can be competitive with supervised pretraining on ImageNet, and even give better results in low-label regimes. Moreover, Fig. 3 shows an example of predictions for models finetuned on an XS dataset, that illustrates qualitative improvements of fMoW-MoCoTP against IN-sup init on both false positive and false negative detections.

2) Dominant vs. rare classes: Table IV shows the APs in level-2. fMoW-MoCoTP init achieves significantly higher results than Random init on these six classes. Also, fMoW-MoCoTP init achieves higher AP than IN-sup init on the three dominant classes (Civilian, Military and Armored), that account for ∼96.5% of the vehicles in our datasets. However, IN-sup init outperforms fMoW-MoCoTP init on the very rare Launcher, Electronics and HE classes, that cover ∼2.2% of the vehicles in our datasets. Since mAP gives equal importance to all classes, this translates into much closer values for mAP scores than level-1 AP scores between fMoW-MoCoTP init and IN-sup init methods, as one can see in Fig. 2. This might suggest that fMoW-MoCoTP init is lazier and mainly focuses on the dominant classes. The fMoW dataset used for pretraining contains a long-tailed distribution of semantic categories. One could hypothesize that this leads to representations being more skewed towards over-represented visual concepts than ImageNet, as the latter contains a balanced set of categories, and that such bias may also negatively impact the transfer to under-represented classes downstream. However, further work is needed to provide ground for this hypothesis.

VI. CONCLUSION

In this work, we explored the added value of in-domain SSL for a real-world defense-related remote sensing application: vehicle detection and classification on VHR optical satellite imagery. Considering this downstream task, we compared in-domain pretraining on the fMoW dataset with the traditional supervised ImageNet pretraining. We showed that self-supervised pretraining on fMoW is either competitive with or better than supervised ImageNet pretraining, despite using no upstream labels and 3× less upstream data. To study label efficiency, experiments were performed for different downstream dataset sizes, thereby mimicking different annotation budgets. We showed that in-domain SSL pretraining leads to better label efficiency than supervised pretraining on ImageNet. This result is particularly suitable for defense industry use cases, where labeling data is challenging. Further work could include additional studies such as increasing the amount of in-domain pretraining data, using a vehicle dataset that is more balanced in terms of classes (or deprived of its dominant classes), extending the range of sizes for the downstream dataset, and experimenting other SSL methods, including methods designed specifically for dense downstream tasks like detection [START_REF] Hénaff | Efficient visual pretraining with contrastive detection[END_REF], [START_REF] Wang | Dense contrastive learning for self-supervised visual pre-training[END_REF], [START_REF] Xie | Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning[END_REF].

FromFig. 1 .

 1 Fig. 1. Schematic outline of our method. The top block represents the pretraining phase. The bottom block represents the pretrained weights that are injected into the downstream task model.

Fig. 2 .

 2 Fig. 2. Graphic view of the results of TableIII.

1 IN 3 .

 13 Example of cherry-picked predictions. Here the models have been finetuned on an XS dataset. White regions highlights errors specific to the top model, in solid and dotted style for false negatives and false positives respectively.

TABLE I DATA

 I STATISTICS FOR VEHICLE TRAINING, VALIDATION AND TESTING SETS. FOR EACH OF THE XXS, XS AND S TRAINING SETS, THE REPORTED NUMBERS ARE THE MEAN NUMBER OF OBSERVABLES BETWEEN THE DIFFERENT SAMPLED SETS.

	Dataset	Images	Pos. tiles	Neg. tiles Vehicles Civilian Military	Armored GSE	Launcher Electronics	HE	LE
	XXS	19	597	113	11,833		6,668	3,215	1,516	154		158	64	33	23
	XS	108	3,152	599	58,189	32,937	14,719	8,466	571		732	385	237 139
	S	204	6,438	1,231	115,617	66,504	29,332	16,148	820	1364	698 432 319
	Val	63	2,526	4,178	53,204	31,339	11,919	8,464	607		412	270 130 101
	Test	88	-	-	32,550	19,923	7,542	3,872	361		334	237 184	97
								TABLE II
	RESULTS ON FMOW CLASSIFICATION (F1-SCORE IN %). FOR 1% AND 10% LABELS, THE VALUES ARE 'MEAN (SD)' ACROSS 3
	REPLICATES WITH VARYING TRAINING SAMPLES. * DENOTE OUR IMPROVED REPRODUCTIONS AS DETAILED IN SECTION III-A
					1% labels				10% labels		100% labels
				Linear		Finetune			Linear	Finetune	Linear Finetune
	Random init [1]		-			-			-	-		-	64.71
	IN-sup init [1]		-			-			-	-		-	64.72
	fMoW-MoCoTP init [1]	-			-			-	-		64.53	67.34
	Random init *		-		19.29 (1.65)			-	51.87 (0.50)	-	65.39
	IN-sup init *	32.41 (0.17)	39.43 (1.53)	43.86 (0.07)	57.32 (0.07)	50.25	66.01
	fMoW-MoCoTP init *	60.05 (0.11) 60.00 (0.43)	66.15 (0.11) 66.35 (0.75)	68.89	68.96
								TABLE III
	RESULTS OF EACH METHOD ON EACH DATASET FOR VEHICLE DETECTION (%). THE VALUES ARE 'MEAN (SD)' ACROSS 3
				REPLICATES WITH VARYING TRAINING SAMPLES.
	Metric			F1							AP			mAP
	Training set	XXS	XS		S		XXS		XS	S		XXS	XS	S
	Random init	26.0 (1.9)	55.1 (2.2)	74.7 (1.4)	12.3 (1.9)	46.9 (2.4)	75.3 (3.4)	2.2 (0.3)	8.4 (0.6)	16.3 (0.5)
	IN-sup init		61.4 (0.5)	75.1 (0.8)	79.5 (0.6)	56.1 (0.9)	75.6 (0.5)	80.9 (0.9)	9.3 (0.4)	14.9 (0.6) 19.7 (1.1)
	fMoW-MoCoTP init	65.1 (0.8) 76.1 (0.4) 79.9 (0.3)	60.1 (1.6)	77.3 (0.1) 81.6 (0.5)	9.7 (0.2) 14.4 (0.4)	19.3 (1.0)

TABLE IV AP

 IV PER CLASS (%). RESULTS ON GSE AND LE ARE OMITTED BECAUSE THEY ARE CLOSE TO ZERO DUE TO THE POOR NUMBER OF EXAMPLES IN THE DATASETS. THE VALUES ARE 'MEAN (SD)' ACROSS 3 REPLICATES WITH VARYING TRAINING SAMPLES.

	Dominant classes		Civilian			Military			Armored
	Training set	XXS	XS	S	XXS	XS	S	XXS	XS	S
	Random init	14.4 (1.8)	48.1 (1.2)	75.4 (4.5)	3.1 (1.5)	14.8 (0.6)	29.1 (3.2)	0.0 (0.0)	3.9 (3.5)	14.9 (7.7)
	IN-sup init	54.2 (1.0)	75.4 (0.8)	81.9 (0.7)	17.2 (3.8)	25.0 (2.1)	32.8 (1.9)	0.8 (0.4)	5.7 (0.8)	21.4 (11.4)
	fMoW-MoCoTP init	59.9 (1.0) 79.5 (0.8) 83.6 (1.1)	17.9 (0.9)	25.0 (1.4)	35.4 (3.0)	0.2 (0.1)	6.5 (1.6)	22.4 (11.3)
	Rare classes		Launcher			Electronics			HE
	Training set	XXS	XS	S	XXS	XS	S	XXS	XS	S
	Random init	0.0 (0.0)	0.4 (0.3)	6.0 (2.

Acknowledgments

This work was performed using HPC resources from GENCI-IDRIS (Grant 2021-AD011013097). Karteek Alahari was supported in part by the ANR grant AVENUE (ANR-18-CE23-0011).