Thibault Douzon

Stefan Duffner

Christophe Garcia

Jérémy Espinas

Keywords: Business Documents, Document Understanding, Information Extraction, Pre-Training, BIESO Decoding, Transformer

Transformer-based Language Models are widely used in Natural Language Processing related tasks. Thanks to their pre-training, they have been successfully adapted to Information Extraction in business documents. However, most pre-training tasks proposed in the literature for business documents are too generic and not sufficient to learn more complex structures. In this paper, we use LayoutLM, a language model pre-trained on a collection of business documents, and introduce two new pre-training tasks that further improve its capacity to extract relevant information. The first is aimed at better understanding the complex layout of documents, and the second focuses on numeric values and their order of magnitude. These tasks force the model to learn bettercontextualized representations of the scanned documents. We further introduce a new post-processing algorithm to decode BIESO tags in Information Extraction that performs better with complex entities. Our method significantly improves extraction performance on both public (from 93.88 to 95.50 F1 score) and private (from 84.35 to 84.84 F1 score) datasets composed of expense receipts, invoices, and purchase orders.

Introduction

Business documents are paper-sized files containing useful information about interactions between companies. They may take the form of invoices, purchase orders, various reports, and agreements. The exact layout of a document depends on the issuer, but the contained information is conventionally structured. For example invoices and purchase orders share the same header, table, footer structure that almost all issuers have adopted. Because such documents trace every transaction made by companies, they are the key to business process automation. With the emergence of modern resource planning systems, accurate Information Extraction (IE) has become one of the core problems of Document Intelligence.

Initially, information extraction was done by human operators, but software solutions have been developed since the early days of document analysis to tackle the problem. Their intent was to ease the work of human operators with hardcoded extraction rules. Unfortunately, these rules needed to be adapted for each and every layout of documents. This limitation has led to the rise of Machine Learning (ML) models for automatic document IE.

First ML approaches relied on Optical Character Recognition (OCR) systems to provide the textual content of the document. This transition from image to text allowed for standard Natural Language Processing (NLP) methods to be applied by adopting a Sequence Labeling problem. The amount of labeled data necessary to train accurate NLP models has always been a problem. Business documents are inherently private which strongly limits the quantity of publicly available data. Thus, only companies selling business process automation software are able to collect larger amounts of such data. Moreover, they often rely on their customer to implicitly label the documents.

Most recent proposals often include a pre-training step, where the model is trained on a pretext task. Those pretext tasks are self-supervised problems that teach the model many useful "skills" for manipulating the data. Usually, these tasks are as broad as possible, teaching the model common sense about language, grammar, and global structure.

In this work, we focus on LayoutLM [START_REF] Xu | LayoutLM: Pre-training of Text and Layout for Document Image Understanding[END_REF], a pre-trained Transformer that is specialized in business documents. As shown in Fig. 1, it reuses the same Transformer layer with multi-head attention with the addition of a 2D positional encoding. Its larger version achieved state-of-the-art performance in both document classification and information extraction. However, the required hardware to train it can be repelling. In this paper, we propose new pre-training tasks specific to business documents that will provide additional skills to the model. We also propose a new decoding post-processing algorithm that prevents many errors made by the model due to ambiguities. Combined, our contributions3 allow for the base LayoutLM model to perform on par with the large version.

Related Work

Information Extraction

Rule-based approaches [START_REF] Li | Regular Expression Learning for Information Extraction[END_REF] have been supplanted by Deep Learning models in the last decade. Document IE first capitalized on the state of the art in Named Entity Recognition for NLP [START_REF] Lample | Neural Architectures for Named Entity Recognition[END_REF]. Recurrent Neural Networks with Long-Short Term Memories were first used to encode documents at a word level [16; 23], allowing a simple classifier to predict each word's associated label. Instead of a softmax and cross-entropy loss, a Conditional Random Field [START_REF] Sutton | An Introduction to Conditional Random Fields[END_REF] model has been used in addition to BIESO tags. Other architectures have also been proposed to better adapt to the specificity of the document. For example, graphs [13; 12; 33; 3] and convolutions over a grid [7; 1; 11] constrained the model based on the words' positional information. Because most architectures relied on textual representations, they benefited from pre-trained word embeddings like Word2Vec [START_REF] Mikolov | Efficient Estimation of Word Representations in Vector Space[END_REF] or GloVe [START_REF] Pennington | Glove: Global Vectors for Word Representation[END_REF].

With the emergence of Transformers [START_REF] Vaswani | Attention Is All You Need[END_REF] and text encoders like BERT [START_REF] Devlin | BERT: Pretraining of Deep Bidirectional Transformers for Language Understanding[END_REF], attention-based document analysis models [31; 5; 30] evolved quickly, which resulted in a large improvement of state-of-the-art performance. In line with [START_REF] Katti | Chargrid: Towards Understanding 2D Documents[END_REF] which included both textual and visual representations, multi-modal Transformers [30; 11] superseded conventional textual models.

In parallel to the rise of Transformers, end-to-end IE models tried to reduce the labeling cost. First using RNNs with attention layers [15; 22], then shifting to Transformers [START_REF] Powalski | Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer[END_REF]. Adopting at the same time the Question Answering [START_REF] Gardner | Question Answering is a Format; When is it Useful?[END_REF] (QA) format, instead of the usual Sequence Labeling, provided more flexibility on the predicted labels.

Pre-Training

Semi-supervised training and pre-trained models were popularised in NLP with enriched word embeddings [18; 14; 17]. With the emergence of Transformers, large pre-trained models have been proposed [START_REF] Wolf | HuggingFace's Transformers: State-of-the-art Natural Language Processing[END_REF]. Thanks to their pre-training, they can efficiently adapt to various tasks [28; 27] and data types. In general, these models are pre-trained on large unlabeled datasets in a self-supervised manner. This self-supervision removes parts of the burden of data labeling [START_REF] Sage | Data-Efficient Information Extraction from Documents with Pre-trained Language Models[END_REF] and leverages the huge quantities of available data.

A wide variety of pre-training tasks have been proposed. General-purpose tasks aiming at learning the language and grammar were used first. Autoregressive tasks [START_REF] Radford | Improving Language Understanding by Generative Pre-Training[END_REF] and Masked Language Modeling [START_REF] Devlin | BERT: Pretraining of Deep Bidirectional Transformers for Language Understanding[END_REF] are still frequently used in new pre-trained models as they have proven to be effective in most situations. In addition to incremental improvements [21; 32], some new pre-training tasks were designed to align representations of multi-modal inputs [30; 19].

Models

Architecture

We used the well-established LayoutLM architecture [START_REF] Xu | LayoutLM: Pre-training of Text and Layout for Document Image Understanding[END_REF] which itself is based on BERT Transformer [START_REF] Devlin | BERT: Pretraining of Deep Bidirectional Transformers for Language Understanding[END_REF]. More specifically, we chose the base model4 with 12 layers and 512 dimensions for token embeddings. This model is computationally much more efficient compared to the larger version while still giving very good performance.

Transformer models work on tokens that are in between characters and words. LayoutLM and BERT both use the WordPiece algorithm. We use the same tokenizer as LayoutLM in order to compare our performance with the base Lay-outLM model. It uses a vocabulary size of 30000, and we limit the sequence length to 512 tokens, including the special tokens [CLS] and [SEP]. This limitation due to GPU memory consumption of self-attention operations often forces us to cut documents in multiple pieces of 512 tokens and process them separately.

Contrary to RNNs, all positions in the sequence are equivalent in a Transformer model. To provide information about position inside the sequence, a linear positional encoding [START_REF] Vaswani | Attention Is All You Need[END_REF] is added for each token. Then LayoutLM adapted this positional encoding to a 2D version that can represent the positions of words on a page.

For both pre-training tasks and fine-tuning, we use a simple dense layer to map each token's final internal representation to the dimension of the prediction space. A softmax layer is applied to produce the final model confidence scores.

For training, the cross-entropy loss is used on the model confidence scores.

ConfOpt Post-Processing

We model the Information Extraction task as sequence tagging on tokens. Predictions are done at the token level and then aggregated by a lightweight postprocessing step to give the model's final prediction. In all experiments, we use BIESO tagging. That is, each field to extract is composed of a sequence of target tags of the following types: B for the beginning of the entity, I for inside, E for its end, or otherwise S for a single token entity. O is used for any token that is outside any target label. BIESO is widely used in IE as it provides structure to the target sequence that helps the model.

Instead of the trivial post-processing which consists of simply following the maximum confidence of the model, we decided to decode a model's prediction by solving a basic optimization problem. We will refer to this method as Con-fOpt in the remaining of the paper. The predicted sequence for a target label is the sequence that maximizes model confidence over the whole input sequence. There is a constraint to decode a prediction: it must match the following regular pattern: (BI*E) | S where * denotes zero or many occurrences and | denotes an alternative.

This optimisation problem can be solved with a dynamic programming approach. The model's predictions for one target label can represented as a 4 × N dimensional matrix where N is the sequence length and 4 comes from the 4 tags B,I,E,S. By noting C T,0 the model's confidence in T tag at position 0 and P T,i the best prediction confidence ending at token i with tag T, the objective is to determine S = max 0≤i<N T ∈{E,S} P T,i where

P B,i = C B,i ; P I,i = C I,i + max P B,i-1 P I,i-1 P S,i = C S,i ; P E,i = C E,i + max P B,i-1 P I,i-1
One drawback of this post-processing is dealing with no prediction and nonunique predictions. It can be solved with an empirically determined threshold below which no predictions are made. Though in this paper this is not further studied because fields are mandatory in a document and always unique.

Pre-training

Transformer models provide great performance when first pre-trained on pretext tasks on very large unlabelled datasets. This pre-training is most of the time done in a self-supervised manner in order to avoid the labeling cost. LayoutLM uses Masked Visual-Language Modeling [START_REF] Xu | LayoutLM: Pre-training of Text and Layout for Document Image Understanding[END_REF] which is adapted from BERT's Masked Language Modeling [START_REF] Devlin | BERT: Pretraining of Deep Bidirectional Transformers for Language Understanding[END_REF]. It teaches the model how text and documents are formed at a token level. In practice, at each training step, 15% of the tokens are randomly chosen and replaced by either a [MASK] token, a random token, or not replaced at all. The model tries to guess which token is the most probable right replacement at those positions.

For all pre-training tasks when a document is too long to be processed at once, we randomly select a continuous span of words of maximum size and provide it to the model instead. We expect the model to learn useful features on various parts of documents thanks to the long training. For very short documents, the input is padded to the maximum size.

We introduce two new specific pre-training tasks in addition to Masked Visual-Language Modeling (MVLM). The first one, Numeric Ordering teaches the model how to compare and order numbers. The second one, Layout Inclusion focuses on words in the 2D plane and their relative positioning. We chose to avoid regression tasks, even though their implementation would have been simpler. For example, simply removing the 2D positioning of some tokens, and asking the model to predict tokens' position is an alternative to what we propose. But this does not behave well for a token that could appear either at the top or the bottom of the document: the model would learn its mean position -the middle -where the token would never appear. In the following, we will describe the two pre-training tasks in detail. Fig. 2: A pre-training example with Numeric Ordering task. A random token containing a number is selected, then the target is to predict whether other numbers are smaller or bigger. Some random noise can be added by masking tokens' textual or spatial representations. Only a small part of the document's input is represented in this illustration.

Numeric Ordering Task

Numeric Ordering (NO) focuses on numeric figures in the document and their relative values. Contrary to MLM which only relies on self-supervised data, NO relies on a handcrafted number parser to find and parse all numbers that appear in a document. Because business documents are mostly made of decimal numbers written with digits, we ignore those written out in words. The numeric value of each token is determined by parsing beforehand each word in the document, looking for numbers and ignoring irrelevant characters.

As shown in Fig. 2, the model must predict for every numeric figure in the document if its parsed value is smaller, equal or greater than a randomly selected number among the document. The loss is only computed on tokens starting a new word, but tokens continuing a word are important to determine the value represented by a word.

We want the model not only to reason on the textual features, but also on the spatial context surrounding each figure in the document. Therefore, we randomly mask the textual representations of 15% of the numbers in the document and replace them with the [MASK] token as shown in Fig. 2. For the same reason, we also mask the spatial encoding of 15% of the numbers and make sure both text and position are not masked at the same time. All masked positions are replaced with (1000, 1000, 1000, 1000).

Layout Inclusion Task

We introduce another pre-training task focusing on the 2D positional encoding, which we called Layout Inclusion (LI). Its purpose is to provide a better understanding of document layouts and complex structures. In fact, most business documents, including invoices, purchase orders, and expense receipts, contain tables where the meaning of tokens is mostly driven by their position relative to headers.

As shown in Fig. 3, Layout Inclusion is formatted like a question answering prompt: a question followed by the content of the document. The question is simply a special token [LAYOUT] positioned at random coordinates (x 1 , y 1 , x 2 , y 2 ). The model must then classify every token in the document into 2 groups: either inside or outside of the question token. More precisely, the target answer is whether the middle point of a document token is inside or outside the rectangle described by the coordinates of the question.

Again, the objective is for the model to not only reason on the 2D positions of tokens but also use their textual embedding. In order to force the model to use both representations, we randomly replace 15% of documents token positions with (1000, 1000, 1000, 1000). In case of a random position replacement, the target value is still computed based on the real position of the token, and the model must make its prediction based on the token's text and the neighboring tokens using the classical 1D positional encoding.

Datasets

We used 2 different collections of documents to build 3 datasets for training and evaluation as described in the following. They all contain business documents: invoices and purchase orders for the private collection and expense receipts for the public one. The largest dataset used for pre-training isn't labeled, document samples with their target fields for the others datasets are shown in Fig. 4. Fig. 4: A document sample for each training dataset annotated with the expected predictions. For BDC-PO, we replaced the document with a fictive one due to privacy reasons.

Business Documents Collection

The Business Documents Collection (BDC) is a large private dataset composed of 100k invoices and 300k purchase orders. Those real documents were submitted and processed on a commercial document automation solution in the last 3 years. It contains English-only documents divided into 70000 different issuers. All documents sharing the same issuer usually use the same information template. Therefore, we limited the maximum number of documents of the same issuer to 50. It is important to keep the number of similar layouts in the collection low and the variety of examples high. We used this collection for pre-training language models on business documents that are closer to our final objective than RVL-CDIP [START_REF] Lewis | Building a test collection for complex document information processing[END_REF]. Textual and positional information have been extracted using a commercial OCR system. It achieves excellent accuracy on properly scanned documents and provides accurate word positions. We also use the provided read order to determine the order of tokens when feeding the network. This order determines the 1D positional encoding given to each token that complements the 2D positional encoding.

Because we only used this collection for pre-training models on self-supervised tasks, most documents do not have extraction labeling. Only a subset composed of purchase orders is labeled for the IE task.

Business Documents Collection -Purchase Orders

We selected a subset of the Business Documents Collection to build a labeled dataset of English purchase orders called BDC-PO. It contains almost 9000 different issuers split into training, validation, and test set. In order to not introduce bias for models pre-trained on the BDC, we removed from BDC all documents emitted by a supplier contained in the test set. This means that document layouts contained in the test set have never been seen before by the model at pre-training or training time.

Long purchase orders are rare but can sometimes be longer than 20 pages. If we wanted to train models and make predictions on such documents, we would have to evaluate the model on dozens of inputs for one document. Instead, we chose to limit documents to one page and crop the remaining. It only concerns roughly 25% of the dataset and sometimes impacts the prediction because labels are missing from the input.

The extraction task consists of 3 fields: document number, delivery date, and total amount. Those fields were chosen because they are mandatory for most customers and thus are well labeled at the word level by the end-user. We controlled the labeling quality at the issuer level and rejected from the dataset some issuers with undesirable labeling practices.

ICDAR 2019 -Scanned Receipts

We also trained and evaluated our model on the public Scanned Receipts OCR and Information Extraction [START_REF] Huang | ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction[END_REF] (SROIE) dataset that was published for ICDAR 2019. We focus on the third task which consists in extracting information from the documents. SROIE contains Malaysian receipts split into 626 train and 347 test documents. Unfortunately, we do not have control over the composition of the test set, and most of the test layouts also occur in the training set.

We used the OCR text provided with the dataset instead of using our own OCR system. As others have pointed out [START_REF] Xu | LayoutLM: Pre-training of Text and Layout for Document Image Understanding[END_REF], it contains numerous little errors that negatively affect the final performance. For a fair comparison with the leaderboard, we manually fixed them such that the expected string appears in the input, at least. These fixes mostly concern addresses and company names. It almost exclusively involves fixing errors related to white-spaces and commas.

Experiments

All experiments were performed on a single machine equipped with two Nvidia RTX A6000 with 48Go of video memory each. This allowed us to boost the batch size up to 32 per device on a base transformer model. To further increase the batch size, we also aggregated 6 batches together before propagating the gradient for a total batch size of 192. We used the Adam optimizer with a learning rate of 1e -5 and 200 linear warm-up steps as it improved our model's convergence. We used 1500 training steps for SROIE and 3000 steps for BDC-PO. Finally, we ran each fine-tuning 10 times in each setup to get a precise idea of the performance of the models and the variability of the results. For the different pre-training scenarios, we performed only two runs and the best model was kept.

Post-Processing

This first set of experiments aims at comparing the post-processing used to decode the sequence produced by the model. We want to determine whether our proposed ConfOpt algorithm is competitive with other decoding methods. We decided to use the LayoutLM base model and compare the proposed ConfOpt against two other decoding algorithms as shown in Table 1.

We named Ad-Hoc the basic decoding using the label with maximal confidence for each token. When decoding with this method, a B tag starts a new entity, a I tag continues the previous entity, a E closes the previous entity, and a S tag produces a new entity and closes it right away. Ad-Hoc and ConfOpt use the same model weights in this experiment as they do not introduce any trainable parameters.

The second decoding algorithm uses a Conditional Random Field (CRF) [25; 8] that processes LayoutLM's predictions. In this particular case, we did not use the classical cross-entropy loss but the score provided by the CRF layer. Because the CRF required specific training and did not optimize the same loss, its weights are different from the two other post-processing methods. We evaluated these algorithms on both SROIE and BDC-PO. The results in Table 1 show a tiny improvement using a CRF instead of the Ad-Hoc postprocessing (0.13 and 0.05 F1 points) but those differences are always within one standard deviation range. We would need more evidence to conclude on the effect of adding a CRF layer for the post-processing.

On both datasets, using ConfOpt significantly increases performance (1.06 and 0.22 F1 points) compared to the Ad-Hoc post-processing, even though the model is strictly identical. In light of these results, we decided to use the ConfOpt for the next experiment.

Business Document-Specific Pre-training

We conducted another set of experiments in order to study the effects of the new business data-specific pre-training tasks on the model performance. At the same time, we controlled the performance gap obtained by pre-training with the basic MVLM task on the same new dataset. Both comparisons are insightful to decide whether it is useful to pre-train on clients' data and/or with data-specific pre-training tasks.

For the pre-training part, we always initialize the model's weights with the base version [START_REF] Xu | LayoutLM: Pre-training of Text and Layout for Document Image Understanding[END_REF]. We pre-train models for 20 epochs on 80% of BDC. When using multiple pre-training tasks at the same time, we chose to provide batches of single tasks to the model. Gradient aggregation over multiple batches helps smoothing the update between different tasks. We pre-trained 2 models on the BDC, one with MVLM only and another with MVLM+NO+LI. We evaluated each pre-trained model on both datasets, the results are available in Table 2 for BDC-PO and Table 3 for SROIE . Each cell contains the means of 10 runs with different seeds and the standard deviation is provided for the F1 score. There are a few interesting things to notice.

The first important remark is the importance of the pre-training dataset. Pre-training on BDC significantly improves performance on both SROIE and BDC-PO, even though the pretext training task is the same as what was used for LayoutLM. BDC is more homogeneous and focuses on invoices and purchase orders. Contrary to our expectations, we observe a greater improvement on SROIE than on BDC-PO (0.24 vs 0.2 F1 points). But the overall improvement by using BDC can be explained because RVL-CDIP contains a broader panel of document types and is not specialized like BDC. Even though BDC does not contain expense receipts, its global structure is similar to invoices.

Next, we can compare the pre-training tasks. Introducing Numeric Ordering (NO) and Layout Inclusion (LI) tasks also improves the performance over the previously pre-trained model. We observe a 0.32 F1 point improvement on SROIE but only 0.07 on BDC-PO. We suspect the small improvement introduced by the new tasks can be explained because most useful skills to process purchase orders were learned by pre-training on such documents. The new pre-training tasks help more for generalizing on new types of documents.

We also can look at the results on a field per-field basis. We observe that using BDC over RVL-CDIP improved the recognition of all fields except for the dates in BDC-PO. If introducing new training tasks did not improve all fields, we notice that some fields were greatly enhanced like the total amount in SROIE (1.41 F1 points difference). We expected to observe a greater improvement in the total field with the new pre-training tasks. But it does not seem to improve performance much on BDC-PO's total.

Finally it is interesting to compare on Table 3 our results with the published scores of LayoutLM and LayoutLMv2. Our pre-trained model with NO and LI tasks performs better than LayoutLM large which contains 3 times more parameters. However, LayoutLMv2 -which uses both textual and visual information -performance level is still unreachable for a textual-only model.

Conclusion

In this work, we showed significant improvements are accessible without introducing more trainable parameters and computational complexity. Only using the base transformer architecture, we achieved a performance that is comparable to the large version which contains 3 times more parameters. Pre-trained models can be further specialized through in-domain datasets and specific pretext training tasks. We demonstrated that by introducing a new collection of business documents and training tasks focusing on documents' layout and number understanding. We showed that performance improvements can be imputed to both pre-training tasks (Numeric Ordering and Layout Inclusion) and new pre-training dataset.

In the future, we will investigate on IE as a Question Answering problem. It has already been proposed in the past [START_REF] Gardner | Question Answering is a Format; When is it Useful?[END_REF] as an alternative to Sequence Labeling when fine-tuning models. It should improve the model's generalization capabilities and enable few-shot learning. But nowadays all models are pre-trained, and we would like to study the impact on generalization of a QA-only pre-training.

Fig. 1 :

 1 Fig. 1: LayoutLM architecture. Token embeddings are enriched with 1D positional encoding and 2D spatial encoding specific to this architecture. The number of blocks N varies from 12, for the base model, to 24, for the large one.

Fig. 3 :

 3 Fig. 3: A pre-training example with Layout Ordering task. Coordinates of the purple rectangle are drawn uniformly. Random noise is added by masking the 2D position of some tokens. Only a small part of the document is represented.

  (a) Receipt from SROIE. (b) Purchase Order from BDC-PO.

Table 1 :

 1 Performance comparison on SROIE and BDC-PO between multiple post-processing algorithm. Score is computed on the exact match between the prediction and the target string.

		Fine Tuning (F1 score)
	Post Processing	SROIE	BDC-PO
	Ad-Hoc	93.88 ± 0.59 84.35 ± 0.12
	CRF	94.01 ± 0.55 84.40 ± 0.16
	ConfOpt	94.94 ± 0.38 84.57 ± 0.10

Table 2 :

 2 Model performance when fine-tuning on BDC-PO

	Architecture	Pre Training			Accuracy per field
		Task(s)	Dataset	F1 Score Company Address Total	Date
	LayoutLM base *	MVLM	RVL-CDIP 94.94 ± 0.38	92.91	90.81	89.25 99.48
	LayoutLM base *	MVLM	BDC	95.18 ± 0.23 93.72	91.00	89.48 99.68
	LayoutLM base *	MVLM+NO+LI	BDC	95.50 ± 0.22 93.60	91.41 90.89 99.57
	LayoutLM base [31]	MVLM	RVL-CDIP	94.38
	LayoutLM large [31]	MVLM	RVL-CDIP	95.24
	LayoutLMv2 large [30] MVLM+TIA+TIM RVL-CDIP	97.81

Table 3 :

 3 Model performance when fine-tuning on SROIE. Models name ending with a * are our contribution. The second part contains published scores of the original LayoutLM and LayoutLMv2 as a comparison.

Code available here: https://github.com/thibaultdouzon/business-document-pre-training

Pre-trained weights available here: https://huggingface.co/microsoft/layoutlm-base-uncased

Improving Information Extraction on Business Documents with Specific Pre-Training Tasks


Benjamin Camus
email: benjamin.camus@scalian.com

Scalian Ds

Corentin Le Barbu
email: corentin.lebarbu@scalian.com

Eric Monteux
email: eric.monteux@scalian.com

Robust SAR ATR on MSTAR with Deep Learning Models trained on Full Synthetic MOCEM data

Keywords: SAR, radar, ATR, MSTAR, SAMPLE, deep learning, classification, simulation, synthetic dataset, MOCEM

the promising potential of Deep Learning for Automatic Target Recognition (ATR) on Synthetic Aperture Radar (SAR) images vanishes when considering the complexity of collecting training datasets measurements. Simulation can overcome this issue by producing synthetic training datasets. However, because of the limited representativeness of simulation, models trained in a classical way with synthetic images have limited generalization abilities when dealing with real measurement at test time. Previous works identified a set of equally promising deep-learning algorithms to tackle this issue. However, these approaches have been evaluated in a very favorable scenario with a synthetic training dataset that overfits the ground truth of the measured test data. In this work, we study the ATR problem outside of this ideal condition, which is unlikely to occur in real operational contexts. Our contribution is threefold. (1) Using the MOCEM simulator (developed by SCALIAN DS for the French MoD/DGA), we produce a synthetic MSTAR training dataset that differs significantly from the real measurements. (2) We experimentally demonstrate the limits of the state-of-the-art. (3) We show that domain randomization techniques and adversarial training can be combined to overcome this issue. We demonstrate that this approach is more robust than the state-of-the-art, with an accuracy of 75 %, while having a limited impact on computing performance during training.

I. INTRODUCTION

THE Automatic Target Recognition (ATR) task on Synthetic Aperture Radar (SAR) images consists of designing a classification model to identify objects such as vehicles, captured by a radar. SAR images are (mathematically) complex data characterized by a large dynamic range, an important level of noise (clutter), and a limited resolution. Moreover, they capture multiple and heterogeneous electro-magnetic (EM) effects. Because of these factors, ATR on SAR images is a non-trivial task [START_REF] Novak | An efficient multi-target SAR ATR algorithm[END_REF].

Deep Learning algorithms have demonstrated an important potential to tackle this challenge [START_REF] Housseini | Deep Learning for target recognition from SAR images[END_REF][START_REF] Li | DeepSAR-Net: Deep convolutional neural networks for SAR target recognition[END_REF][START_REF] Morgan | Deep convolutional neural networks for ATR from SAR imagery[END_REF]. However, to be efficient these techniques require large training datasets. The problem is then that collecting and labelling a sufficient amount of SAR images is very expensive and in fact impossible in a defense context because, for instance, the vehicles to classify (e.g. tanks, ships) and / or the radar system (deployed on a moving platform like an aircraft or a satellite) may not be available. Moreover, the measurements may contains bias that may fool the models (e.g., classification based on environment features like bushes, instead of targets features) [START_REF] Belloni | Deep learning and featured-based classification techniques for radar imagery[END_REF].

At the opposite, SAR simulators require only 3D CAD models of the vehicles, materials description and the radar system parameters to generate synthetic images. Besides, a full and strict control of the simulated environment reduces measurement bias, and automates the image labelling process.

For all these reasons, it is interesting to train ATR models using simulated SAR images instead of real measurements. However, because simulations rely on assumptions and real world simplifications (i.e. a model), synthetic data have a limited representativeness. Thus, the synthetic training distribution and the real measurement test distribution differ. This corresponds to the well-known dataset-shift problem [START_REF] Quionero-Candela | Dataset Shift in Machine Learning[END_REF]: ATR models trained on simulated data have a limited generalization capability to real measurements at test time.

Previous state-of-the-art works [START_REF] Lewis | Adversarial training on SAR images[END_REF], especially [START_REF] Inkawhich | Bridging a Gap in SAR-ATR: Training on Fully Synthetic and Testing on Measured Data[END_REF], identified a set of equally promising deep-learning algorithms to tackle this issue. These studies are crucial because they showed that it is theoretically possible to train ATR models with synthetic data and compete with models trained directly on real measurements.

However, the evaluation of these works relies on the MSTAR/SAMPLE [10,[START_REF] Lewis | A SAR dataset for ATR development: the Synthetic and Measured Paired Labeled Experiment (SAMPLE)[END_REF] datasets with simulated data that reproduce as close as possible the real test measurement. As we show in this paper, the problem is that the vehicles, their configurations (e.g. articulated parts orientations), their equipment, and their positions on the images are almost identical between the synthetic training dataset and the real measurements test dataset, due to a perfect knowledge of ground truth by the authors. That is why, in this work we study the ATR problem outside of this very favorable scenario, unlikely to occur in a real operational context. Our contribution is threefold:

1. Using the MOCEM simulator, we produce a new synthetic training dataset for MSTAR that differs significantly from the real measurements. Thus, these simulated data do not suffer from the aforementioned limitations of the SAMPLE dataset. 2. We experimentally demonstrate the limits of the state-of-art approaches using our synthetic data. We show here that, among all the approaches identified in [START_REF] Inkawhich | Bridging a Gap in SAR-ATR: Training on Fully Synthetic and Testing on Measured Data[END_REF], the Adversarial Training (AT) [START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF] is the only one that offers acceptable results outside of the ideal conditions of the SAMPLE data. Still, the models accuracy significantly decrease of almost 35 %.

3. We show that intensive domain randomization [START_REF] Tobin | Domain randomization for transfering deep neural networks from simulation to the real world[END_REF] performed through data augmentation techniques can be combined with AT to overcome this issue and increases the accuracy up to 75 %. The rest of the paper is as follows. Section II, details the related works and the available SAR datasets. Section III presents our synthetic MOCEM dataset. In Section IV, we demonstrate the limits of the state of the art. Section V presents our approach. Finally, in Section VI we detail our results on the MSTAR dataset and compare our approach to the other algorithms of the state of the art.

II. STATE OF THE ART

A. Available datasets

The MSTAR (Moving and Stationary Target Acquisition and Recognition) public dataset [10] comprises SAR measurements of fifteen different targets taken at different depression and azimuth angles by an airborne radar. Following the standard ATR evaluation procedure with MSTAR [START_REF] Ross | Standard SAR ATR evaluation experiments using the MSTAR public release data set[END_REF], the 3671 images collected at a depression angle of 17° constitute the training set whereas the 3203 images with a depression angle of 15° serve to test the models. These data concerns ten classes of vehicles (labelled 2S1, BMP2, BDRM2, BTR60, BTR70, D7, T62, T72, ZIL131 and ZSU23-4), measured almost at each azimuth degree from 0° to 360°. Three variants of the vehicles are available for two classes: the BMP2 and the T72. The vehicles equipment and configuration (e.g. side skirts) may differ from a variant to another (see Figure 1). It is worth noting that the azimuth angles are almost all identical between training and test sets. Moreover, these two sets comprise exactly the same vehicles on same configuration, and they are located at the same positions in the environment. Thus, the model may exploit background information to classify the targets, as the background remains the same between training and test [START_REF] Belloni | Deep learning and featured-based classification techniques for radar imagery[END_REF]. The SAMPLE (Synthetic and Measured Paired Labeled Experiment) public dataset [START_REF] Lewis | A SAR dataset for ATR development: the Synthetic and Measured Paired Labeled Experiment (SAMPLE)[END_REF] comprises pairs of real SAR measurements and simulated images. This dataset is smaller than MSTAR with only 806 synth-real measurements pairs for training (at depression angles of 14°, 15° and 16°) and 539 pairs for testing (at depression angles of 17°). Like MSTAR, SAMPLE comprises ten target classes labelled 2S1, BMP2, BTR70, M1, M2, M35, M60, M548, T72, and ZSU23-4. One can note that five classes are common with MSTAR. In fact, the real measurements of these five classes correspond to the MSTAR ones (but restricted to azimuth range of SAMPLE). However, SAMPLE data does not provide any variant for any class. The SAMPLE dataset suffers from several drawbacks. First, the images azimuth angles range only from 10° to 80° for both training and test datasets. It particularly excludes the cardinal directions that may be the more challenging angles. Secondly, this angular sector may not be representative of the challenges met when classifying images at a full 360° extent. In addition, the SAMPLE authors have deployed considerable efforts to make the synthetic data as close as possible to real measurements, using detailed ground truth information. Thus, the SAMPLE simulations rely on the CAD models of the actual vehicles used during the MSTAR data collection. Furthermore, the authors configured and enriched the CAD models with detailed based on numerous notes and photos of the MSTAR measurements campaign. This very favorable scenario is unlikely to occur in an actual operational context where the measured vehicles may differ significantly from the CAD models used during training. Finally, the SAMPLE authors performed circular shifts on the measured images to center the targets exactly like in the synthetic images (see an example in Figure 2). This alignment is unrealistic because it requires a pair of real and synthetic images of the same target captured at the same angle. Hence, it assumes to know the target class a priori, which is not compatible with the ATR problem. It is important to note that absolute target centering (i.e. without pairs of images) is not trivial on SAR images because the vehicles are only partially visible. Thus, the synthetic and measured targets are difficult to align without paired data. To sum up, MSTAR and SAMPLE are the most complete ATR datasets publicly available in the state-of-the-art. They constitute key contributions to the ATR problem. However, they comprise several biases that may skew experimental results. These two datasets should thus be used with caution.

B. Related works

Ødegaard et al. got mixed results when training an offthe-shelf deep-learning algorithm directly using simulated SAR data [START_REF] Ødegaard | Classification of ships using real and simulated data in a convolutional neural network[END_REF]. Using a transfer learning strategy, Malmgren-Hansen et al. pre-trained an ATR classifier with a large amount of synthetic data before training the model with a smaller set of measured images [START_REF] Malmgren-Hansen | Improving SAR Automatic Target Recognition Models With Transfer Learning From Simulated Data[END_REF]. The drawback of this approach is that it still requires measured data for training, and in many cases, measured images of the target of interest will not be available.

Several works focus on learning a transport function to refine synthetic data by adding features peculiar to measured images. The goal is to transport the synthetic distribution on the measured one to fix the dataset-shift issue. The refined synthetic dataset can then be used to train a regular classifier. Cha et al. trained a residual network to refine synthetic data for ATR [START_REF] Cha | Improving Sar Automatic Target Recognition Using Simulated Images Under Deep Residual Refinements[END_REF]. However, their classifier only achieve an accuracy of 55 % at test time. Lewis et al. [START_REF] Lewis | Generative adversarial networks for SAR image realism[END_REF] and Camus et al. [START_REF] Camus | Refining Simulated SAR images with conditional GAN to train ATR Algorithms[END_REF] trained a GAN to refine synthetic SAR images, with promising results of almost 95 %. However, all these refining approaches require real measurements that may be impossible to obtain. Moreover, Camus et al. demonstrated that GAN are not able to refine new classes never seen during training [START_REF] Camus | Refining Simulated SAR images with conditional GAN to train ATR Algorithms[END_REF].

Inkawhich et al. trained ATR classifiers with the public synthetic data of SAMPLE by combining several algorithms of the literature designed to improve the generalization of deep-learning models [START_REF] Inkawhich | Bridging a Gap in SAR-ATR: Training on Fully Synthetic and Testing on Measured Data[END_REF]. They evaluate all their models on the SAMPLE measured images. The authors compared three architectures: the network of [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF], a ResNet18 [START_REF] He | Deep residual learning for image recognition[END_REF], and a Wide-ResNet18 [START_REF] Zagoruyko | Wide residual networks[END_REF]. The techniques they studied are the following. Gaussian noise is a form of data augmentation that consists of adding random noise to the training images at runtime [START_REF] Shorten | A survey on image data augmentation for deep learning[END_REF]. With dropout [START_REF] Srivastava | Dropout: A simple way to prevent neural networks from overfitting[END_REF], a random noise is injected during training in the hidden units of the network to erase a given ratio of the detected features. Label smoothing [START_REF] Möller | When does label smoothing help[END_REF] prevents models overconfidence and overfitting by modifying the one-hot labels of the training data so that the correct class probability become smaller than one. Mixup [START_REF] Zhang | Mixup: Beyond empirical risk minimization[END_REF] randomly performs convex combinations of pairs of training data and labels. Thus, it promotes linear model behaviors between the training samples. Using a cosine loss [START_REF] Barz | Deep learning on small datasets without pretraining using cosine loss[END_REF] instead of the usual cross-entropy, the model is trained to maximize the cosine similarity between its output and the true labels. AT [START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF] consists of perturbing the training data on a per-pixel basis in order to maximize the model cost (using the gradient information of the network). Thus, the model becomes more robust to adversarial attacks. Finally, the bagging technique [START_REF] Goodfellow | Deep Learning[END_REF] averages the prediction of several models that were independently trained. For further details on these strategies, we invite the reader to refer to [START_REF] Inkawhich | Bridging a Gap in SAR-ATR: Training on Fully Synthetic and Testing on Measured Data[END_REF]. With a ResNet18 and several combinations of these techniques (bagging, Gaussian noise and dropout combined either with label-smoothing, mixup, cosine loss or AT), the authors found accuracies of almost 95 %. This work is groundbreaking because it demonstrates the possibility to train ATR models with full synthetic SAR images instead of real measurements. However, the authors used the SAMPLE dataset that contains several flaws, as stated on the previous section. Therefore, the proposed approaches may not be applicable in an actual operational scenario. That is why in the following section we confront the Inkawhich et al. approaches to other, more representative simulated data.

III. SYNTHETIC DATASET PRODUCTION WITH MOCEM

A. Modeling and simulation

To generate our synthetic database for the MSTAR test data, we use the MOCEM software, which is a CAD-based SAR imaging simulator developed by SCALIAN DS for the French MoD (DGA) for 20 years [START_REF] Cochin | Moceman 'all in one' tool to simulate sar image[END_REF][START_REF] Cochin | Radar simulation of ship at sea using mocem v4 and comparison to acquisitions[END_REF]. To be as representative as possible of an operational scenario, we consider three ground-truth fidelity levels in our simulation.

1. What is known with certainty, and should be therefore accurately simulated: the radar sensor 2. What is partially known, and should therefore only be approximated in the simulation: the target signatures using the appropriate CAD models. 3. What is unknown and should thus not be faithfully modeled: the environment of the targets.

We take off-the-shelf CAD models available on Internet. We use one CAD model per MSTAR class. Thus, we do not simulate the variant vehicles for the T72 and the BMP2 classes. We manually simplify the CAD models meshes (when appropriate) to speed-up the simulation time, and we associate electro-magnetic generic materials (i.e. with welldefined reflectivity, roughness and dielectric constant) to the different facets of the model. We configure MOCEM to simulate the transfer function of the MSTAR sensor (i.e. with similar range/cross-range sampling and resolution, thermal noise level, and Taylor window function) (see Figure 3). We run parametric simulations for the 16°, 17° and 18° incidence angles. For each incidence, we generate images at every 0.5° azimuth for the full [0°, 360°[ range. Thus, we do not consider exactly the same azimuth angles than MSTAR, and our training dataset has different incidence angles than the 15° MSTAR test data. The whole simulation takes two weeks to complete using two standard personal computers.

B. Qualitative evaluation of our dataset

Except for few optical photos per vehicles, we do not know the ground truth of the MSTAR measurements. It is therefore very unlikely that our off-the-shelf CAD models faithfully represent the actual vehicles used during the MSTAR measurement campaign. Moreover, an analysis of the MSTAR photo and SAR images reveals the following differences with ground truth (note that, at the opposite of classical optical images, a small difference on the vehicle geometry may have an important impact on the radar signature of the target). The rear spoilers of the BTR60 are too high on the CAD model. Therefore, we observe dihedral effects on our synthetic data that are not present on the measured images. The D7 blade lies on the ground on the MSTAR photo whereas it does not in our simulation (see Figure 4). Moreover, the dimensions of the CAD model are 25 cm too large, 30 cm too long, and 4 cm too high, according to the D7 user's manual [START_REF]Direct Support and General Support Maintenance Manual (Caterpillar Model D7F)[END_REF]. The fuel barrel of the T62 is too high on our CAD models, and the turret shape does not match the MSTAR vehicle one. Depending on the T72 variant, the CAD turret and the gun orientations are different from ground truth (see Figure 5); the fuel barrel is missing and the equipment are not the same. The CAD model of the ZIL131 is 15 cm too large, 40 cm too long and 6 cm too high. Moreover, the axletree, the rear bumper, and the Front winch are missing in the CAD model. Finally, the turret orientation of the ZSU23-4 differs between the simulation and the measurements (see Figure 6). The log attached at the back of the ZSU23 is also mistakenly associated with a metal EM material.

The partial knowledge of ground truth and the lack of representativeness of CAD models (especially for equipment and articulated parts configurations) are a guaranty to ensure that our synthetic dataset differs significantly from the real measurements. Hence, it is more representative of the ATR challenges than the SAMPLE data.

IV. DEMONSTRATION OF THE STATE OF THE ART LIMITS

A. Demonstration using our MOCEM dataset 1) Datasets: To study the limits of the state of the art, we apply the algorithms identified by Inkawhich et al. [START_REF] Inkawhich | Bridging a Gap in SAR-ATR: Training on Fully Synthetic and Testing on Measured Data[END_REF] on our MOCEM synthetic data. We evaluate our ATR models with the MSTAR test data (at a depression of 15°). Thus, on the contrary to the SAMPLE dataset used in the original paper, the data ranges on a full 360° azimuth extent, the targets are not pairwise aligned on the synthetic and measured images, and the CAD models of the vehicles does not match the measurements ground truth. We are then closer to a real operational scenario than with the SAMPLE dataset.

2) Methods: We use the ResNet18 architecture that gave the best results in the original paper. We follow an experimental plan similar to Inkawhich et al. First, we evaluate a baseline approach that does not involve any improvement techniques. Then, we measure the accuracy obtained with label smoothing (lblsm), mixup, cosine loss and AT applied separately. We also evaluate the models trained with a combination of dropout and Gaussian noise (gauss). Finally, we use Gaussian noise and dropout with either label smoothing, mixup, cosine loss and AT. As the training algorithms are stochastic, we average the accuracy over 50 different trainings for each experimental configuration.

3) Results: The "centered" column of Table I shows the results of these experiments (note that we also present the results obtained by Inkawhich et al. on the original SAMPLE data in the table). We observe an important gap between the results on the SAMPLE data and the MOCEM/MSTAR ones. With our MOCEM data, AT gives the best results with only 27 % of accuracy. Considering that this technique achieves an accuracy of almost 77 % on the SAMPLE data, we have a gap of 50 % between SAMPLE and the MOCEM/MSTAR data. At the opposite, the best algorithm with SAMPLE is a combination of label smoothing, Gaussian noise and dropout that produces models with an accuracy of 91 %. This configuration only reaches 20 % in our more realistic context (i.e. a decrease of 71 %). It is important to note that we observe a good convergence of all the training algorithms. Thus, these poor results cannot be detected during training, which makes any fine-tuning of the hyper-parameters irrelevant.

The absence of (unrealistic) targets alignment in the MOCEM/MSTAR data might explain (at least partially) these poor performances. Thus, we test if a simple data augmentation can fix this issue by making the models less sensitive to targets centering. At runtime during training, we apply circular shifts on the synthetic images. The x and y pixel offsets of each shift are uniformly sampled on the [-5, 5] integer interval. We run again all our experiments with this data augmentation. The Table I shows our results in the "rnd shift" column. We observe that circular shift indeed increases the model accuracy. However, the results are still low compared to the performances of Inkawhich et al. AT is still the best with an accuracy of almost 55 %. However, it is 22 % lower than with the SAMPLE data.

MOCEM training data algorithms

Hypothesis 1: The algorithms identified by Inkawhich et al. are not adapted to a real operational scenario where the synthetic data does not exactly match the measurements ground truth (like with our MOCEM/MSTAR data).

Hypothesis 2: Our MOCEM data are not accurate enough for training deep-learning algorithms (e.g., our simulations might not take account of some important electromagnetic effects).

We invalidate hypothesis 2 later in Section VI.A using our deep-learning approach. To prove hypothesis 1, we slightly modify the SAMPLE test data to be more compliant with an actual operational scenario, and test the models of Inkawhich et al. on this new dataset.

B. Demonstration using a modified SAMPLE dataset

1) Dataset: First, we used the original not-shifted test measured images of SAMPLE. Thus, the targets are not exactly aligned between the synthetic and the measured images, as expected in a realistic use case. Then, we add the MSTAR images of the variant vehicles that comes from elevation 17° and belongs to the restricted [10°, 80°] azimuth range of SAMPLE. In this way, for the T72 and BMP2 classes, the CAD models of SAMPLE does not match the ground truth of the measured images. It is important to note that these extra MSTAR images in SAMPLE does not skew the results because they originate from the same measurement campaign, and we use the same Quarter Power Magnitude (QPM) LUT [START_REF] Doerry | SAR Image Scaling Dynamic Range Radiometric Calibration and Display[END_REF] on the images. This new dataset is more complex than the original SAMPLE one, but it still corresponds to a very favorable and unlikely scenario because: (1) the azimuth range of [10°, 80°] is too limited, and (2) for 75 % of the synthetic training data, the CAD models still match the ground truth of the measured images.

2) Results: However, as shown on the results of Table II, this is enough to show the limits of the algorithms of the literature. The top accuracy is only of 49 % with the mixup, Gaussian noise, dropout combo. This represent a decrease of around 41 % compared to the original SAMPLE test data. The results are better when we add our random circular shift data augmentation but the top accuracy (from the AT, gauss, dropout combination) only reaches 66 % -i.e. -23 % compared to the original SAMPLE data. These weakened results remain however far above the other works of the literature that train ATR models with full synthetic datasets.

TABLE II AVERAGE ACCURACY OF THE MODELS OF THE LITERATURE TESTED WITH

OUR NEW SAMPLE DATASET.

C. Conclusion on the limits of the state of the art

From this series of experiments, we conclude that the algorithms identified by Inkawhich et al. are not adapted to a real operational scenario where the synthetic data does not exactly match the measurements ground truth. This confirm that the SAMPLE dataset is flawed and should be used with caution. In a consistent way with other works of the literature [START_REF] Inkawhich | Bridging a Gap in SAR-ATR: Training on Fully Synthetic and Testing on Measured Data[END_REF][START_REF] Lewis | Adversarial training on SAR images[END_REF], we note that AT has a strong potential to solve the ATR problem as it gives the best performances. However, this technique is not sufficient by itself. In the following section, we detail how we combine AT with an intensive domain randomization strategy to solve this issue.

V. OUR APPROACH

A. Method

In our approach, we make a different implementation of AT than Inkawhich et al. To attack the training images we use the Fast Gradient Sign Method (FGSM) [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF] instead of the iterative Projected Gradient Descent (PGD) algorithm. Although PGD is theoretically more optimal than FGSM, we find better results with the later method. This may be surprising, but this is consistent with other works in the literature that found better results with FGSM than with PGD for training models to be robust to adversarial attacks [START_REF] Wong | Fast is better than free: Revisiting adversarial training[END_REF]. A positive side effect is that FGSM is much faster than PGD because it only requires one additional forward/back-propagation step instead of 50 steps with the PGD of Inkawhich et al. We also find better results when using a L2 norm to bound the adversarial noise (at a value of two in our experiment) instead of the "infinity norm" used by Inkawhich et al. We combine AT with an intensive domain randomization strategy [START_REF] Tobin | Domain randomization for transfering deep neural networks from simulation to the real world[END_REF]. This technique was originally developed to train robotic systems in a simulated environment. It consists of randomizing the simulated environment parameters to introduce as much variations in the synthetic data as possible. Then, the goal of this heuristic it to stretch the synthetic training distribution so that it eventually cover all the measured test distribution. We adapt this strategy to the SAR ATR problem in the following way. For each training epoch, we create a variant of all our MOCEM dataset by randomly determining for each image separately: the range and cross-range resolution, the level and distribution of the background clutter, the thermal noise of the sensor, and the target position in the images (see Figure 7 for examples). We also select the bright points of the target

SAMPLE training data algorithms

that are greater than half the maximum amplitude, and randomly dropout (i.e. assign zero values on) half of them to introduce variations in the target signatures. Finally, we use the bagging method to average the prediction of ten independent models.

We also perform Test-Time Data Augmentation (TTDA) at inference-time with our ATR models [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]. We create 20 variants of each test image by applying random circular shifts of [-5, 5] pixels range in the x and y dimensions. Then, we average the model predictions on the 20 variants to classify the image. The Figure 8 sums-up our complete training workflow.

B. Architecture, optimizer and hyperparameters

We use a DenseNet121 architecture [START_REF] Huang | Densely connected convolutional networks[END_REF], a SGD (Stochastic Gradient Descent) optimizer with Nesterov momentum [START_REF] Nesterov | A method of solving a convex programming problem with convergence rate o (1/k2)[END_REF][START_REF] Sutskever | On the importance of initialization and momentum in deep learning[END_REF], a weight decay of 10 -4 , and a batch size of 128. The learning rate and the momentum vary during the training according to a "1cycle" policy [START_REF] Smith | Super-convergence: Very fast training of neural networks using large learning rates[END_REF][START_REF] Smith | Cyclical learning rates for training neural networks[END_REF], as shown on Figure 9. We found that AT and the domain randomization slow down the convergence of the training. That is why we train our models for 150 epochs. As shown in Table III, we define a uniform distribution for each of the domain randomization variables. We deliberately choose large ranges for the background clutter (level and distribution) and the target positions because this information cannot be known a priori in an operational context [START_REF] Ulaby | Handbook of radar scattering statistics for terrain[END_REF]. To run our experiments, we implement the domain randomization in a Tensorflow (TF) [START_REF] Abadi | TensorFlow: Large-scale machine learning on heterogeneous systems[END_REF] model that performs data augmentation at runtime during the classifier training.

This TF model processes batches of target signatures and shadow masks generated with MOCEM. From these inputs, it produces batches of augmented SAR images. Thanks to this TF model, we only have to simulate each training image once. It is important to note that we could achieve the same results by simulating each image with MOCEM specifically for each training epoch with different simulation parameters. Nevertheless, our MOCEM/TF implementation is significantly more efficient. With our TF data-augmentation model, we are indeed able to generate more than 300,000 augmented SAR images per minutes on a single NVIDIA GeForce RTX 2060 GPU. Thus, our domain randomization approach is fast enough to train ATR models in a reasonable time.

VI. RESULTS

We evaluate our approach and synthetic data through two different experiments. First, we consider a complete and representative use case with the MSTAR test data. Then, to be able to compare our results with the state of the art on SAMPLE, we consider a more restricted scenario.

A. Evaluating our approach and data with MSTAR 1) Accuracy of our approach: Because our training algorithm is stochastic, we train 50 different models with our MOCEM dataset. Then, we test 1e6 combinations of bagging on the MSTAR test data at 15° (including the BMP2 and T72 variants). The average test accuracy of all the bagging is 75.34 %, with a minimum at 74.43 % and a maximum at 76.24 %. Thus, we conclude that with our MOCEM training dataset, our approach gives stable performances that are 48 % higher than the top algorithm of the literature (based on the "centered" results of Table I). This invalidates hypothesis 2, and shows that our MOCEM simulated data are accurate enough to train deep-learning algorithms. Figure 10 shows the confusion matrix of a bagging. Table IV shows the average accuracies per classes over the 1e6 bagging. We observe that for five classes we have an accuracy greater than 80 %. For the two classes with variants -i.e. BMP2 and T72-we have an accuracy greater than 75 %. This shows that we are robust to theses variants. 2) Ablation study: We perform an ablation study to measure the contribution of the different techniques composing our approach. The results are shown on Table V. We progressively remove each method and measure the average accuracy on the MSTAR data over 50 different trainings. TTDA and bagging have a negligible impact on the results (of respectively +1.10 % and +0.88 %). Using a DenseNet121 instead of a ResNet18 adds 6.66 % of accuracy. This is because our intensive AT and domain randomization methods makes the function to fit more complex, and therefore requires a bigger architecture, with more capacities [START_REF] Xie | Intriguing properties of adversarial training at scale[END_REF]. In accordance with our experiments on SAMPLE, we observe that AT significantly increases accuracy of 16.22 %. When removing domain randomization, we need to choose a background clutter level in our synthetic images. This choice is not trivial because it cannot be known a priori [START_REF] Ulaby | Handbook of radar scattering statistics for terrain[END_REF] and it directly determines the signal-to-noise ratio. We test several levels between -20dB.m²/m² and -5dB.m²/m² with steps of 5dB.m²/m². The worst-case scenario corresponds to a clutter of -20dB.m²/m². Compared to this scenario, domain randomization increases accuracy by 24.95 %. The best-case scenario is with a background clutter of -15 dB.m²/m². This is consistent as it is close to the actual average clutter level of the MSTAR data. Even in this optimal context, domain randomization still brings a gain of 7.42 %. Finally, the random circular shifts increases accuracy of 6.27 % and 19.53 % with respectively the worst and the best background clutter level. To sum-up, AT and domain randomization are two essential components of our approach, as they boost accuracy of about 45 %.

B. Comparing our approach with the state of the art 1) Methods:

We run the following experiment to compare our synthetic MOCEM dataset and our deep learning approach to the state of the art. On one hand, we train the different algorithms of Inkawhich et al. with the synthetic SAMPLE data. On the other hand, we use our approach to train an ATR model with our MSTAR/MOCEM synthetic data. To be fair, we use the same ResNet18 architecture (and optimizer) than Inkawhich et al. instead of our DenseNet121. We also disable TTDA with our approach.

2) Datasets: We test all the models on the same measured dataset without using models bagging. For the approaches to be comparable, we restrict the different datasets to the [10°, 80°] azimuth range of SAMPLE and to the five targets that are common between MSTAR and SAMPLE, namely: 2S1, BMP2, BTR70, T72, and ZSU23-4. To be as representative as possible of an operational context, for testing the models we use our modified SAMPLE measured dataset presented previously in Section IV.B.1 (i.e. without targets centering and with the MSTAR variants for the T72 and the BMP2). It is important to note that this test scenario advantages greatly the state-of-the-art approaches because the CAD models of SAMPLE still (unrealistically) match the measurement ground truth for 60 % of the images. To be fair, we restrict our MOCEM dataset to have the same number of images than SAMPLE (i.e. 386 images) with similar depression and azimuth angles (we select the neareast parametric angles of our dataset).

3) Results: Table VI shows the results of this experiment averaged over 50 trainings. We observe that, despite a very unfavorable scenario, our approach has the best results. We are 18 % above the best approach of the literature. When we add a simple random circular shift of [-5, 5] pixel range to the algorithms of Inkawhich et al., we are still 9 % above the top results. With the baseline cases, (i.e. only a ResNet18 without any improvement technique) we observe that the MOCEM and the SAMPLE synthetic datasets give similar results. This may indicate that the two datasets have similar representativeness for SAR ATR. Interestingly, our approach can benefits from additional training images: if we use the all the 2215 images of our MOCEM dataset that belongs to the [10°, 80°] azimuth range, our approach reaches an accuracy of 78.82 % -i.e. +14.74 % compared to a training with only the 386 images that correspond to the SAMPLE angles. training ATR models directly with the MSTAR measurements [START_REF] Morgan | Deep convolutional neural networks for ATR from SAR imagery[END_REF]. However, as discussed in Section II.A, for this baseline result the models are trained and tested on measured images of exactly the same vehicles, with the same configurations and backgrounds. We run the following experiment to estimate the performance achieved outside of this utopian scenario.

2) Method and dataset: We reproduce the Morgan's experiment, with the same algorithm and the MSTAR train and test data of the ten classes. However, we only keep one variant in the training data for the BMP2 and the T72 classes. Then we measure the accuracy of the model on the different variants for these two classes. This way, for two classes the models are tested and trained on different vehicles of the same classes, like in an actual operational scenario. As the training process is stochastic, we average the results over 100 trainings.

3) Results: Table VII and VIII shows the results of this experiment. We observe significant accuracy drops when we test the models on new variants never seen at training time. For the T72 class, we have an average accuracy of 93.44 % when the test variant is the same than the train variant. When the variant are different between test and train, the average accuracy falls at 71.59 % (i.e. a decrease of 21.85 %). For the BMP2 class, the average accuracy drops from 80.45 % to 67.65 % (i.e. -12.81 %).

We want to stress that this experiment has a limited scope because it concerns only two classes. However, this may show that the results of the MSTAR literature should be considered cautiously.

On the same two classes, our MOCEM approach has an average accuracy of 80.16 % for the BMP2 and 77.22 % for the T72 (as shown previously on Table IV). Thus, despite using a full synthetic training dataset, we achieve better results with our approach than models trained directly on measured data but with different variants (+12.5 % for the BMP2 and +5.63 % for the T72).

VII. CONCLUSION

In this work, we study the problem of training ATR classifiers that transfer to real measurements using synthetic SAR images. We demonstrate that the publicly available datasets of the literature (the MSTAR and SAMPLE data) are limited because they are not fully representative of the challenges met in a real operational context. In particular, their training set unrealistically match the ground truth of their measure test data (e.g., the CAD models of SAMPLE are very close to the measured vehicles, the position of the targets are identical on the images). These dataset remains however essential in the literature to conduct experiments. That is why they should be used cautiously.

Considering this potential skew, we show the limitation of the key set of approaches identified by Inkawhich et al. using the SAMPLE dataset. We show that with slight modifications of the test data (removing the targets alignment and adding variants of vehicles for two classes) the top accuracy decreases of 42 % (from 91 % to 49 %).

Using the MOCEM simulator, we produce a new synthetic training dataset for MSTAR that differs significantly from the real measurements. This dataset is then more representative of the ATR challenges. We show that the approaches of the literature give poor results (i.e. only 54 % of top accuracy) with our synthetic dataset. Thanks to all our experiments, we identified that among all the methods proposed by Inkawhich et al., AT is the most promising. However, we conclude that this approach is not sufficient by itself.

That is why we propose a new approach that combines AT with an intensive SAR-based domain randomization technique. Following this strategy, at each epoch we randomly change the training images range and cross-range resolution, the level and distribution of the background clutter, the thermal noise of the sensor, the target position in the images, and the target signature. We implement this domain randomization in a Tensorflow model that performs data augmentation at runtime during training directly on the GPU. However, we could achieve the same results (but less efficiently) by simulating each images with MOCEM specifically for each training epoch with different simulation parameters.

With this approach and our full synthetic MOCEM training data, we achieve an accuracy of 75 % on the MSTAR test data. These results are 48 % above the top results obtained when training the approaches of the literature with our MOCEM dataset. We also show that with MOCEM data, our solution achieves an accuracy 18 % above the top algorithms of literature trained with the SAMPLE synthetic data. In an actual scenario, our solution might give similar results than models trained directly on measured data with classical algorithms, although further experimentations should be done to confirm this point. This work demonstrates that our approach is relevant and that the MOCEM simulator is accurate enough to generate representative synthetic datasets, able to train deep-learning algorithms.

In future works, we plan to extend our domain randomization strategy to modify other SAR-based simulation parameters. To identify these new parameters, we plan to use explicability tools to understand what features are essential to correctly classify the images. We also want to perform a more advanced ablation study to measure which parameters of the domain randomization are essential.

Figure 1

 1 Figure 1 Comparison between SAR images (top) and photo (bottom) of the three MSTAR variants of the T72.

Figure 2 .

 2 Figure 2. Comparison between an original SAMPLE image (duplicated for comparison purpose) of the T72 and its centered version.

Figure 3 .

 3 Figure 3. CAD model (left) and synthetic image example (right) of the T72.

Figure 4 .

 4 Figure 4. Comparison of our CAD (left) and ground truth (right). The actual blade position is represented in yellow, and the estimated ground truth is in white.

Figure 5 .

 5 Figure 5. Actual turret position in our T72 CAD model (in violet) versus estimated position (in white).

Figure 6 .

 6 Figure 6. Actual gun position in our ZSU23-4 CAD model (in violet) versus estimated position (in white).

Figure 7 .

 7 Figure 7. Example of domain randomization: reference image (top left), background clutter level (top center) and distribution (top right), NeSimga0 level (bottom left), target signature (bottom center), and radar resolution (bottom right).

Figure 8 .

 8 Figure 8. Our complete training workflow.

Figure 9 .

 9 Figure 9. Evolution of the learning rate and momentum in our training according to the "1cycle" policy.

Figure 10 .

 10 Figure 10. Confusion matrix of a bagging of models.

TABLE I AVERAGE

 I ACCURACY OF THE ALGORITHMS IDENTIFIED BY INKAWHICH ET AL. ON OUR MOCEM/MSTAR DATA.This experiment demonstrates that we cannot apply the algorithms identified by Inkawhich et al. on our MOCEM/MSTAR data. We formulate two non-exclusive assumptions to explain this fact:

				original results on
		centered rnd shift	SAMPLE
	baseline	23.52%	43.06%	63.84%
	lblsm	23.95%	44.37%	77.49%
	mixup	21.26%	35.97%	80.10%
	cosine loss	22.41%	43.79%	75.67%
	AT	27.20%	54.48%	76.78%
	gauss, dropout	21.00%	30.01%	88.28%
	lblsm, gauss, dropout	20.34%	31.42%	91.09%
	mixup, gauss, dropout	21.42%	28.57%	90.31%
	cosine loss, gauss, dropout 20.04%	27.70%	89.10%
	AT, gauss, dropout	21.87%	36.94%	89.50%

TABLE III HYPER

 III -PARAMETERS VALUES OF OUR DOMAIN RANDOMIZATION IN OUR EXPERIMENTS.

TABLE IV AVERAGE

 IV ACCURACIES PER CLASSES WITH OUR APPROACH.

TABLE VI COMPARISON

 VI BETWEEN OUR APPROACH AND THE STATE-OF-THE-ART.

		mean	max	min
	2S1	95.89% 96.35% 94.53%
	BMP2	80.16% 82.28% 78.53%
	BRDM2	47.62% 51.09% 44.16%
	BTR60	58.99% 64.10% 54.36%
	BTR70	83.39% 84.18% 81.63%
	D7	81.84% 85.04% 78.10%
	T62	80.17% 82.78% 77.29%
	T72	77.22% 79.55% 74.57%
	ZIL131	65.64% 68.98% 62.41%
	ZSU23	72.53% 74.45% 70.07%
							test accuracies
							training data	algorithms	centered	training with
							training data	random shifts
	clutter level without domain rdn (dB.m²/m²)		-20	-15	baseline	35.85%	49.59%
	techniques				accuracy	lblsm	36.59%	51.68%
	bagging, DenseNet, rnd shift, domain rnd, AT, TTDA bagging, DenseNet, rnd shift, domain rnd, AT DenseNet, rnd shift, domain rnd, AT ResNet, rnd shift, domain rnd, AT ResNet, rnd shift, domain rnd ResNet, rnd shift ResNet	19.26% 23.52% 25.53% 43.06% 75.34% 74.24% 73.36% 50.48% 66.70%	SAMPLE	AT, gauss, dropout cosine_loss, gauss, dropout mixup cosine_loss mixup, gauss, dropout lblsm, gauss, dropout AT gauss, dropout	45.99% 43.58% 32.51% 35.42% 43.83% 44.01% 35.32% 43.37%	55.48% 35.69% 45.68% 45.75% 38.36% 44.73% 54.62% 42.38%
							MOCEM	baseline ours	43.06% 64.08%	36.53%
							C. Comparison with models trained on measured data
							1) Motivation: Our 75 % accuracy on MSTAR may
							seems low compared to the 95 % achieved by Morgan when

TABLE VIII INFLUENCE

 VIII OF THE BMP2 MSTAR VARIANTS (IDENTIFIED BY THEIR SERIAL NUMBER) ON THE ACCURACY.