Ibrahim Souleiman Mahamoud
email: ibrahim.souleimanmahamoud@getyooz.com

Joris Voerman
email: joris.voerman@univ-lr.fr

Mickaël Coustaty
email: mickael.coustaty@univ-lr.fr

Aurélie Joseph
email: aurelie.joseph@getyooz.com

Vincent Poulain D'andecy
email: vincent.poulaindandecy@getyooz.com

Jean-Marc Ogier
email: jean-marc.ogier@univ-lr.fr

Multimodal Attention-Based Learning for Imbalanced Corporate Documents Classification

Keywords: Document classification, Imbalanced classifcation, Multimodal classification, Attention mechanism

The corporate document classification process may rely on the use of textual approach considered separately of image features. On the opposite, some methods only use the visual content of documents while ignoring the semantic information. This semantic corresponds to an important part of corporate documents which make some classes of document impossible to distinguish effectively. The recent state-of-theart deep learning methods propose to combine the textual content and the visual features within a multi-modal approach. In addition, corporate document classification processes offer a particular challenge for deep learning-based systems with an imbalanced corpus. Indeed the neural network performances strongly depend on the corpus used to train the network, and an imbalanced set generally entails bad final system performances. This paper proposes a multi-modal deep convolutional network with an attention model designed to classify a large variety of imbalanced corporate documents. Our proposed approach is compared to several state-of-the-art methods designed for document classification task using the textual content, the visual content and some multi-modal approaches. We obtained higher performances on our two testing datasets with an improvement of 2% on our private dataset and a 3% on the public RVL-CDIP dataset.

Introduction

Companies need to manage each day a large number of documents. Those documents represent the life of the company and can be of a large variety of types, classes and origins. They are generally linked to the administrative part (like invoices, letters, receipts) or to the core activity of the company. Such documents are of primary importance as they generally validate an action or a decision inside and/or outside the company. The management of those documents is a challenge between speed and precision. Indeed, an error could have an heavy cost by causing a wrong action or decision. In this context, precision is then preferred to recall.

Many companies use Digital Mailroom system [START_REF] Schuster | Intellix-end-user trained information extraction for document archiving[END_REF] to automatize document processing and so reduce workload and time required. Those systems entries can be modeled as document streams which define many constraints. One of these constraints is a strong imbalanced representation between classes that could affect any learning methods based on a training set. Even if deep learning models recently offer impressive performance in multiple difficult situations, the document stream classification process remains a challenge. This challenge is linked to the training set used to define the model: a strong imbalance inside the training set reduce neural network performances for low represented classes and they become like noises for greater classes. Moreover, all the classes need to be known when training the system or they will be ignored and classified as a wrong known class.

Document classification processes are traditionally divided into two categories: The image processing and the textual analysis categories. Language processing methods offer good performances for classification of corporate documents where textual content is the main resource (Fig. 1a). But their performances depend on the quality of the text extraction process (generally done by Optical Characters Recognition-OCR). OCR is not perfect and an error on most important words could make the classification impossible. OCR quality has recently greatly increased but remains affected by noises and can not recognize handwritten texts (Fig. 1b). In addition, some documents classes contain few text, like advertisement (Fig. 1c). It is then hard to classify a document with this type of method. This is where the image-based approaches are more suitable. In the same way, image processing classify with efficiency documents classes with a template or with a common structure but many classes are visually very close and could only be distinguished by their semantic or few keywords. With the spread of multimodal approaches in the deep-learning era [START_REF] Srivastava | Multimodal learning with deep Boltzmann machines[END_REF], some methods have been proposed to benefit from both sides. Multimodal or Crossmodal systems propose a solution by making a decision based on a combination of graphical and textual information. The way to combine those two modalities then becomes the new main problem in order to keep the best of the two previous approaches. Recently, cross-modal methods demonstrate that they could be considered as the best ones compared to the state-of-the-art image processing system on RVL-CDIP dataset [START_REF] Bakkali | Visual and textual deep feature fusion for document image classification[END_REF] and offering impressive performances.

In our paper, we want to deepen the evaluation of these new approaches in the field of imbalanced classification of document flows by comparing them with more classical methods of the state-of-the-art. In addition, we propose our multimodal system integrating an attention model per modality designed to force the system to learn the most relevant features even with the least represented classes. The second advantage is its ability to visualize features used by the deep neural network in the decision process, thus reducing the black box effect of our architectures.

Related Work

Few works have been proposed on multimodal analysis and classification of documents, and even fewer on the impact analysis of imbalanced number of document per class during the learning process. We will first provide an overview of articles related to one modality (based on visual or textual features), followed by the multi-modal/cross-modal approaches. The last part will focus on attention-based architectures.

Regarding the visual-based approaches, many image classification methods have been efficiently applied on the document classification task since the RVL-CDIP dataset [START_REF] Harley | Evaluation of deep convolutional nets for document image classification and retrieval[END_REF] have been proposed. They are mainly based on deep convolution architecture pre-trained on ImageNet dataset [START_REF] Russakovsky | 0, et ImageNet large scale visual recognition challenge[END_REF]. Multiple architectures were proposed for this task like the InceptionResNet [START_REF] Szegedy | Inception-v4, inception-resnet and the impact of residual connections on learning[END_REF], NasNet [START_REF] Zoph | Learning transferable architectures for scalable image recognition[END_REF] and VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] for naming the most used ones. In first hand, a majority of those methods isn't especially adapted for document. In another hand, some methods propose to take into account structural aspect of documents and divided them in multiple sections like in [START_REF] Harley | Evaluation of deep convolutional nets for document image classification and retrieval[END_REF].

Regarding text content based neural network, they need firstly a word representation system. Most used systems are currently word embedding like Fast-Text [START_REF] Joulin | Fasttext.zip: compressing text classification models[END_REF], BERT [START_REF] Devlin | Bert: pre-training of deep bidirectional transformers for language understanding[END_REF] has became probably the most interesting word embedding technique and seems to be more efficient that previous methods. Those methods are based on various elder statistical strategy like bag-of-words and cooccurrences matrix. They reinforce them with deep learning method trained on huge corpus of text, with more than one million words, like Wikipedia's articles. For the network himself, current research are concentrated on recurrent strategy to keep sentence sequential information with Recurrent Neural Network (RNN) [START_REF] Graves | Speech recognition with deep recurrent neural networks[END_REF] and bidirectional-RNN [START_REF] Zhou | Text classification improved by integrating bidirectional LSTM with two-dimensional max pooling[END_REF]. Some recent works proposed to combine textual and visual features to take the whole document's content into account. [START_REF] Audebert | Multimodal deep networks for text and image-based document classification[END_REF] uses a MobileNetV2 [START_REF] Sandler | MobileNetV2: inverted residuals and linear bottlenecks[END_REF] architecture combined with a CNN model inspired of [START_REF] Kim | Convolutional neural networks for sentence classification[END_REF] by concatenating the network outputs. [START_REF] Bakkali | Visual and textual deep feature fusion for document image classification[END_REF] introduced another architecture with a NasNet [START_REF] Zoph | Learning transferable architectures for scalable image recognition[END_REF] network for the image part and a bidirectional BERT architecture for the text. The paper tested several combinations between the networks. Even if the performances improved recently, the neural network results interpretation remains difficult. This is moreover becoming more important with the development of multimodal classification process and the need to a better understanding of limitations (which modality is penalising the other one).

Attention is one of the most influential ideas in the Deep Learning community. Even though this mechanism is now used in various problems like image captioning and others, it was initially designed in the context of Neural Machine Translation using Seq2Seq Models [START_REF] Luong | Stanford neural machine translation systems for spoken language domains[END_REF]. One important advantage of those attention models is their ability to visualize the features used by the network to make a decision, and then offer a possible interpretation of errors. This system could also be used to reinforce network performance as in [START_REF] Wigington | Multimodal document image classification[END_REF] where the integration of an attention model into a recurrent LSTM offers better performances for image classification. Another work [START_REF] Górriz | Assessing knee OA severity with CNN attention-based end-to-end architectures[END_REF] proposed an attention model architecture combined with the VGG16 network with several layers at different resolutions to generate more global features. The attention model could force the networks to learn relevant patterns. [START_REF] Jetley | Learn to pay attention[END_REF] propose to use attention model to identify salient image regions at different stages of the CNN. During decision, the system reinforces their importance and therefore suppress irrelevant or confusing information.

Our work is particularly interested in the classification of an imbalanced corpus. This subject is close to two image classification challenges known as zeroshot and one-shot learning. The first introduces a context where a part of classes is unknown during training. The second is close but classes are represented by at least one or few documents. Zero-shot solutions [START_REF] Xian | Zero-shot learning-the good, the bad and the ugly[END_REF] mainly use a transfer learning strategy based on semantic description of image. Our context does not offer access to such descriptions and we do not have any prior knowledge about unknown classes. So those solutions can not be used in our situation. One-shot learning methods offer more interesting option with Bayesian-based techniques like in [START_REF] Lake | Human-level concept learning through probabilistic program induction[END_REF] or adapted deep learning methods like in [START_REF] Santoro | One-shot learning with memory-augmented neural networks[END_REF] and [START_REF] Koch | Siamese neural networks for one-shot image recognition[END_REF] but those methods are not directly compatible with our multi-modal approach. In addition, some methods are designed for imbalanced classification out of those challenge like in [START_REF] Lin | Deep reinforcement learning for imbalanced classification[END_REF]. This propose to apply reinforcement learning with higher rewards and penalties on low represented classes.

The originality of our approach lies therefore in the proposal of a model based on attention models that allow us to have a better performance and a better interpretation of the results. In the next section, we will describe our own method for imbalanced multi-modal documents classification.

Problem Definition

The main objective of our proposal is to provide solutions that improve the predictive accuracy in forecasting tasks involving imbalanced dataset. The task of prediction of document classes in an industrial environment involves input flows s 1 , s 2 , s 3 , ..., s n ∈ S, where s n is the number of samples from classes n in the training set. The number of samples per class is by definition imbalanced due to the industrial context. Some document classes are very recurrent (i.e. there will be thousands of invoices) while others are very rare with a very few number of samples (ten or even 1 document) as displayed in Fig. 2.

The objective of this predictive task is to predict the class of samples S. The overall assumption is that an unknown function correlates the samples and ground truth classes of S, i.e. Cn = f (sn). The goal of the learning process is to provide an approximation of this unknown function whatever the quantity of samples available by class. To better approximate this function f , we must know the are significant intra-class variation and inter-class similarity caused by different structure documents for each client brings a great deal of difficulties to classify (see the Fig. 3).

Fig. 2. Distribution of samples by classes for the private data YOOZ

The hypotheses tested in our experimental evaluation are:

1. A higher inter-class variance in the representation space will reduce the confusion between classes and also better predict weakly represented cases 2. The use of an attention mechanic for textual and visual parts will allow to better focus on the important part of the document content and thus reduce intra-class variation errors

Proposed Approach

To see the relevance of our hypotheses, we inspired by state-of-the-art models to propose a classification architecture with three attention models. The combination of these three attention models has not been studied in the state-of-the-art and moreover on imbalanced data. In a first step we will present the architecture without the attention mechanism and then we discuss each of the attention mechanisms.

Model Without Attention

The initial system is composed of two classifiers, one for the visual part and the other one for the text extracted from the image. We use a bidirectional recurrent LSTM network with bert embedding for the text classification and a VGG16 network pre-trained on ImageNet for the image part. The choice of this architecture is inspired by the methods from the state of art for document and image classification. Here, the idea is to use the best systems from the literature as a baseline.

Each of these models has different input feature (denoted X and Z in the Eq. 1). The features of each sample i are decomposed in x i ∈ R d1 for the textual branch and z i ∈ R d2 for the visual branch. d 1 and d 2 are the features dimension respectively for textual and visual parts. N is the size of the corpus (i.e. 1 ≤ i ≤ N ).

The one-hot vector corresponding the label as y i ∈ R L and L corresponds the total number of classes. The one-hot vector values are defined as 1 for the corresponding class in the groundtruth, and 0 for the others.

For the textual content, we uses the first 150 words and extracted the Camembert features [START_REF] Martin | CamemBERT: a tasty French language model[END_REF], where each word is represented by 768 values. We then have d 1 of size (150,768). The input images are color images of size d 2 = (224,224,3) as an input for the VGG16 network.

X [x 1 , x 2 , ..., x i ] T ∈ R N ×d1 Z = [z 1 , z 2 , ..., z i ] T ∈ R N ×d2 Y = [y 1 , y 2 , .., yi] T ∈ R N ×L (1)

Attention Model

The use of Attention Mechanism aims at focusing the network to the most relevant features related to our task (i.e. document classification). This corresponds to learning a matrix (or a mask) which weights features for each class. We propose to adapt this to each branch of our architecture.

Self-attention Mechanism for Label Fitting. The self-attention mechanism used in this article is inspired by [START_REF] Cheng | Attentive graph neural networks for few-shot learning[END_REF]. They propose a self-attention mechanism on the input image to consider the inherent correlation (attention) between the input features themselves, and then use a graph neural network for the classification task. We use this self-attention on both the image and the text to focus our network on common features from the input. This will allow us to exploit the interclass and intersample correlation at the initial stage. Contrary to our baseline, we use the label Y as input and not only to help error propagation. We calculate the correlation between the input data and the label to predict. To do this, we transform X, Z to X , Z which will be the input of our two classifiers. To achieve that, we follow several steps, the first step is to calculate the sample and label correlation matrices as:

C x = sof tmax(XX T ) C z = sof tmax(ZZ T ) C y = sof tmax(Y Y T ) (2)
Here softmax(•) denotes a softmax operator. The inputs of this function have the same dimension where

XX T , ZZ T , Y Y T ∈ R N ×N
The self-attention module exploits C x , C z , C y . Thus, the next step is to fuse Cx and Cy using trainable 1 × 1 kernels as:

C xy = fusion([Cx, Cy]) ∈ R N ×N C zy = fusion([Cz, Cy]) ∈ R N ×N ( 3
)
where e.g. [Cx, Cy] denotes the attention map concatenation, This fusion function fusion is equivalent e.g. C x y = w 1 Cx + w 2 Cy, where the weighted parameters w 1 , w 2 are learned adaptively. We will use C xy , C zy to update both the visual feature X and the textual feature Z. X = XC xy , Z = ZC zy (4)

A t = f (sof tmax(h f )) h ( 5
)
Here h is the hidden LSTM and h f is after having flatten h, denotes a pointwise product operator, and softmax(•) denotes a row-wise softmax operator for the sample. The f function is a dropout which is intended to prevent overfitting on the data by activating neurons with probability p or kept with probability 1p (p fixed to 0.3 in our experiments). We analyze the output of the softmax to determine the words the models of which gave more weight.

Attention for the Visual Content. This mechanism is inspired by the work done by [START_REF] Górriz | Assessing knee OA severity with CNN attention-based end-to-end architectures[END_REF]. Their original goal was to predict the age of people from fixed flexion knee X-ray images. We use this image attention model as it brings two advantages: it helps understanding which part of the image is the most relevant (like the titles of document for instance); it allows understanding which part the attention model focused on, and then to understand its mistakes (for instance, we can determine the area of zones of interest it concentrates whereas it should not. The attention model embedded in the image classification network uses the output of the 5 th layer of the VGG16 model D. This layer is composed of 128 filters and has a dimension of (112, 112, 128). The last layer is our attention layer A and is of dimension (112, 112, 1). The matrix D is multiplied by D to obtain the matrix D . Finally, the dimension is reduced by applying an average pooling layer on D to generate a vector of size (128). The last step consists in normalizing this attention vector by a pooling layer on A. This normalization is the function g(x1, x2) = x1/x2 and the input of the visual classifier will be the output of this normalization function.

of Modalities

As stated at the beginning of this paper, combining the two previous modalities should allow to take advantage of each other to increase the overall performance of the model. The proposed multimodal architecture is illustrated in Fig. 4. First we apply self-attention mechanism on the inputs for a better separation of features in their representation space. Then we use the updated inputs by the self-attention mechanism to classify textual and visual parts, and each part is reinforced with its own specific attention mechanism. Our final classifier is based on the prediction output of the two classifiers. The following equation explain how we combined the two models:

[Y x , Y z ] ∈ R N 2 ×C (6)
Where [Yx, Yz] is the concatenation of the outputs of the text Y x and image Y z classifier. We noticed that the model had some difficulties to classify the weakly represented classes. We then decided to use a weight on the Cross Entropy loss function presented in Eq. 7. The Cross Entropy [START_REF] Nasr | Cross entropy error function in neural networks: forecasting gasoline demand[END_REF] tends to counterbalance the weight of the under-represented classes towards the over-represented ones (the minority classes are often ignored as they represent only a small part of the total loss). The weight assigned to each class corresponds to the inverse percentage of the examples present in the training set. The smaller the presence of a class, the greater its weight on the loss will be.

Loss CE (i) = -W t i log(P (i)) (7)
5 Experiments

Dataset

To evaluate our multimodal attention-based model, we used two different datasets. The first one named "YOOZ dataset" is a private and already imbalanced set issuing from our customers. In order to assess the relevance of our method, and to have a comparison with the best approaches from the literature, we also use the public RVL-CDIP [START_REF] Harley | Evaluation of deep convolutional nets for document image classification and retrieval[END_REF] dataset. This large public dataset composed of a large variety of document classes equally balanced between them. In order to evaluate our contribution, we propose a protocol to unbalance the RVL-CDIP dataset.

The YOOZ dataset is composed of 15 thousands training documents, 2 thousand for the validation set and 5 thousand for testing. This dataset is composed of 47 classes (Invoice, Quotation, general sales condition, check, etc.). Some classes are very similar from the layout and the content point of view while other classes are easily distinguishable. The Fig. 2 proposes an illustration of the classes distribution of our imbalanced dataset. One can observe that four groups appear (from the most frequent on the left to the least frequent classes on the right).

The second dataset, The RVL-CDIP, is balanced by definition with 20 000 images per class in the training set and 2500 images per class in the validation and test sets. In order to assess the relevance of our proposed approach (a multimodal attention-based classification model for imbalanced dataset), we reduced the number of document for each class. This reduction mimic the configuration of real-life conditions (some very frequent classes opposed to classes with very few samples).

To reproduce this unbalanced dataset, in the first we took the first 90 first thousand dataset( image and text) of RVL CDIP according to the order present in the train.txt in the folder labels. The exact document partition ordered by classes (0 to 15) is the following, for each class we recover a proportion to have an unbalanced distribution The textual content of both datasets were extracted from documents with an OCR. For our private YOOZ dataset, we used ABBYY FineReader OCR, while the OCRed version of the RVL-CDIP documents is provided by their authors (more details are available in [START_REF] Harley | Evaluation of deep convolutional nets for document image classification and retrieval[END_REF]).

Implementation Details

For all our experiments, we used the ADAM optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] with β 1 = 0.9, β 2 = 0.999 and a batch size of 64, an initial learning rate of 10 -3 scaled from 0.1 every 3 epochs without improvement in validation loss and an early stopping after 5 epochs without improvement.

Performance Evaluation and Discussions

The first part of our evaluation protocol consists in comparing the proposed approach with the best approaches from the state-of-the-art. Table 1 presents the precision and recall values of all these approaches on the RVL-CDIP original dataset, its imbalanced proposed version and our internal dataset.

When looking at the textual part of the documents, we can see that even if the work done by [START_REF] Bakkali | Visual and textual deep feature fusion for document image classification[END_REF] used a similar approach than ours, we can observe that we have better performances. This can be explained by the fact that the authors of [START_REF] Bakkali | Visual and textual deep feature fusion for document image classification[END_REF] used a LSTM network with BERT embedding. We can here observe that the proposed attention model allows us to increase the precision by 3 points on the whole dataset.

When focusing on the visual part of the architecture, we compared our proposed attention-based pattern mechanism compared to the state-of-the-art model [29]. This model uses both holistic image and image split by region(right, left, top and bottom). We can here again observe that or complex model have better performance (+1%) with less parameters.

Finally, we present the performances of our multimodal architecture on imbalanced dataset to highlight the complementary effects of the textual and visual modalities. can observe that the increase is much important on the imbalanced RVL-CDIP dataset than on our private one. This can be explained by the fact that classes are mostly distinguishable by their structure in the RVL-CDIP dataset, so the model relies on the image when it does not reach class with the text. The M ultimodality SAtt propose allows us to have real gains on both RVL and YOOZ data. As previously mentioned in the industrial context prefers a model with little bad prediction, for this we took a threshold of 0.99 to filter only the cases where the model has more confidence on its prediction.

The multimodal model without model attention and with attention we almost better classify about 64 documents in dataset YOOZ as illustrated in Fig. 5(a) and (b) with threshold 0.9. When we further analyze the results class by class, we find out that the true contribution of this model lies in the reduction of intra-class errors. This contribution on almost all the classes highlight a better separation in the feature representation space.

Following this idea, we propose in our evaluation to illustrate the impact of our attention mechanism on the inter-class correlation. The proposed multimodal module exploits three attention mechanisms and this combination of attention mechanisms effectively guide the feature representation for each document and was designed to better separate classes in the feature space (thus reducing the confusion in the decision process). We then first propose to visualize the way documents are distributed in the feature space using a t-SNE visualization tool. One can observe in Fig. 5 (second line) that the classes are better seperated after applying our self-attention mechanism, demonstrating the effect of our architecture. In order to illustrate the impact of a better distributed and separated classes in the feature space, we also display the total number of error per class before and after applying our attention mechanisms (first line of Fig. 5). We can notice that this better separation entails a much lower number of errors for each class and then validate our initial assumptions. Finally, the past part of our evaluation process tends to illustrate the impact of this better separation of classes class by class. Table 2 presents the number of documents misclassified despite a high threshold of 0.99. The multimodality reduces the misclassified with threshold compared the separate modalities for almost all classes because it manages to exploit best of each modality. For some classes the contribution of multimodality is very low, which can be explained because one of the modalities is much more confident than the other so multimodality will use the prediction of only one modality (e.g. image) and so they will have almost the same misclassified.

Discussion. The challenge we hope to meet is to combine several modalities so that each one brings its own contribution in order to have better performance for both frequent and non-frequent classes.

Our future research will be much more focused on the training technique of our methods in order to better take into account the imbalanced data. We In this article, we have proposed methods using multimodality and attention patterns on images and text. The use of multimodality is necessary in order to have better performance and take advantage of both the features extracted from the text and the image. The combination of three attention mechanisms allows focusing on the most relevant visual or semantic features to classify documents, and ease the understanding of results and mistakes. Our best proposed multimodal weighted system is able to increase by 2% the global precision compared to state-of-the-art architecture. We can also drastically reduce the number of errors, mainly by reducing the confusion between classes. Even if we obtained good performances with our proposed multimodality approach on both the YOOZ and the RVL-CDIP dataset, many perspectives appear. The use of weighted categorical cross entropy has certainly slightly improved the results on some classes with a low presence in the training dataset, but nevertheless we hope to get higher performance with the use of reinforcement learning to strengthen the answer when the models correctly predicts the less frequent classes. This could be done using the work proposed in [START_REF] Lin | Deep reinforcement learning for imbalanced classification[END_REF] which describes the use of reinforcement learning in the case where the data is out of balance. We then expect improving our performances.

Fig. 1 .

 1 Fig. 1. Three examples of documents that can only be classified based on their textual content (a), graphical content (b) or a combination of them (c)

Fig. 3 .

 3 Fig. 3. The two documents on the left come from two different classes even though their structure and contents are similar. The documents on the right come from the same class and we can see that they have a different structure.

Fig. 4 .

 4 Fig. 4. The proposed multimodal model uses the three attention mechanisms described above. We used a small colored matrix to illustrate how the attention mechanism is used (Figure description see part 4)

  [5673*1, 5662*1, 5630*1, 5735*1, 5613*0.5, 5584*0.5, 5559*0.5, 5622*0.5, 5658*0.1, 5649*0.1, 5593*0.1, 5642*0.1, 5553*0.05, 5639*0.05, 5619*0.05, 5569*0.05]. So we get a total of 33 thousand documents for training. we did the same procedure to unbalance the data for the test and validation data, for each we had 16 thousand documents.

Fig. 5 .

 5 Fig. 5. In this figure we compare the errors between (a) the multimodal model with no attention and (b) the model with attentions (YOOZ dataset). These bars represent the error rate after a 99% threshold has been applied. We note that the multimodal model proposes to increase these cases of errors, which is highly appreciated in the industrial context. T-SNE visualization: (c): The embeddings without correlation, (d): The embedding created after correlation with labels (RVL-CDIP dataset). The numbers represent the class in the figure it's placed in the center of each class.

Table 1 .

 1 The results of the models, the models with Att are the models with attentions either on the text or the image while SAtt is Att added the self attention. BiRNN is the bi-directional RNN model and VGG16 is the image classification model.

	Modality	Data	RVL-CDIP	RVL-CDIP unbalanced YOOZ
	Model		Precision Recall Precision Recall	Precision Recall
	Text	Bert	86%	86%	-	-	-	-
		(Souhail CVPR2020)
		biRNN	88.58% 87.95% 79.7%	68.5%	95.7%	94.3%
		biRN N Att	89.5%	88.1% 80.3%	69.8	96.8%	95.2%%
		biRN N S Att	-	-	80.7%	70.2	97.5%	97.3%%
	Image	VGG16 pretained	91.1%	-	-	-	-	-
		(das2018document)
		VGG16	91.8%	90.4% 87.3%	86.8%	88.1%	83.5%
		V GG16 Att	92.2%	91.3% 88.5%	87.4%	88.7%	83.2%
		V GG16 S Att	-	-	88.8%	87.6%	89.2%	83.4%
	Multimodality Multimodality (aude-	90.6%	-	-	-	-	-
		bert19multimodal)
		Multimodality	-	-	90.8%	88.7%	96.6%	96.2%
		M ultimodality Att	-	-	92.03% 90.06%	97.3%	93.18%
		M ultimodality S Att	-	-	92.90% 91.40%	98.2%	97.5%

Table

  The number document of misclassified with a threshold 99 test data. hw = handwritten, adv = advertisement, s.r = scientific report, s.p=scientific publication spec = specification, fileF = file folder, newAr = news article pre = presentation, ques = questionnaire

	Method	Letter	Form email	hw	adv s.r	s.p	spec
	Data	1217	1261	1289	1167	620 641 640	625
	Text	380	441	220	352	467 380 322	117
	Image	320	412	219	322	167 219 208	78
	Multimodality 310	384	201	317	164 210 204	71
	Method	fileF	newAr Budget Invoice pre ques Resume Memo

Acknowledgment. This research has been funded by the LabCom IDEAS under the grand number ANR-18-LCV3-0008, by the French ANRT agency (CIFRE program) and by the YOOZ company.


Ibrahim Souleiman Mahamoud
email: ibrahim.souleimanmahamoud@getyooz.com

Joris Voerman
email: joris.voerman@univ-lr.fr

Mickaël Coustaty
email: mickael.coustaty@univ-lr.fr

Aurélie Joseph
email: aurelie.joseph@getyooz.com

Vincent Poulain D'andecy
email: vincent.poulaindandecy@getyooz.com

Jean-Marc Ogier
email: jean-marc.ogier@univ-lr.fr

Multimodal Attention-Based Learning for Imbalanced Corporate Documents Classification

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Les entreprises ont besoin de gérer chaque jour une grande quantité de documents. Ces documents représentent le cycle de vie de l'entreprise et sont très variés en termes de classes et d'origine. Ils sont généralement liés à la partie administrative et comptable de l'entreprise (factures, lettres, reçus, ...) ou directement associés au coeur de ses activés. De nombreuses entreprises font appel à des systèmes de "Digital Mailroom" [START_REF] Schuster | Intellix-end-user trained information extraction for document archiving[END_REF]) pour automatiser la gestion des documents. L'entrée de ces systèmes se modélise sous la forme d'un flux de documents définissant plusieurs contraintes : un fort déséquilibre de représentation entre les classes au sein de l'ensemble d'entrainement, un temps de traitements qui doit être très faible pour traiter de grands volumes de documents, ou encore la nécessité de limiter les erreurs qui peuvent engendrer de lourdes conséquences (partage d'une information sensible, absence d'information pour la prise de décision, etc).

La classification de documents est traditionnellement divisée en deux approches : une basée sur l'analyse d'image et l'autre sur l'analyse de texte. Ces deux méthodes ont chacune leurs limites. Les méthodes basées sur l'analyse de texte offrent de bonnes performances lorsque le contenu textuel est très présent cependant il dépend fortement de la qualité de l'extraction (généralement faite par une Reconnaissance Optique de Caractères -OCR) notamment lorsque les documents contiennent des annotations manuscrites. Enfin, certaines classes de documents ne peuvent être décrites par une seule modalité. Par exemple, les publicités ne contiennent que peu de texte, tandis que les approches images ne pourront distinguer deux documents visuellement proches comme un devis et une facture. Il apparaît donc nécessaire de tirer profit des forces de chaque modalité pour extraire un résumé visuel et sémantique pertinent.

Récemment, des méthodes multimodales ont prouvé qu'elles pouvaient égaler, voire dépasser, l'état de l'art en matière d'analyse d'image sur la base de données publique RVL-CDIP [START_REF] Bakkali | Visual and textual deep feature fusion for document image classification[END_REF]). Dans notre papier, nous voulons approfondir l'évaluation de ces nouvelles approches sur le terrain de la classification déséquilibrée de flux de documents en les comparant avec les méthodes plus classiques de l'état de l'art. De plus, nous proposons notre propre système multimodal intégrant un modèle d'attention conçu pour contraindre le système à n'entrainer que des caractéristiques pertinentes. Ce modèle est également utilisé pour faciliter l'interprétation des décisions du réseau profond en mettant en avant les caractéristiques principales calculées pendant l'entrainement et diminuer l'effet boîte noire inhérente à l'apprentissage profond.

État de l'art

Les méthodes de classification de documents peuvent être divisées en deux catégories : visuelles et textuelles. Les approches visuelles utilisent principalement une architecture de réseau convolutif profond pré-entrainé sur la base publique ImageNet [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF]). On compte parmi elles, les architectures InceptionResNet [START_REF] Szegedy | Inception-v4, inception-resnet and the impact of residual connections on learning[END_REF]), NasNet [START_REF] Zoph | Learning transferable architectures for scalable image recognition[END_REF]) et VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]) pour ne citer que celles-ci. Les approches textuelles quant à elles résument, dans un premier temps, le contenu du document via un système de représentation des mots. La stratégie la plus utilisée actuellement est la méthode dite d'encapsulement de mots (ou "word embedding") qui s'appuie sur des réseaux de neurones récurrents bidirectionnels dont les principaux représentants sont Word2Vec [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF], Fast-Text [START_REF] Joulin | Fasttext.zip : Compressing text classification models[END_REF]) ou plus récemment BERT [START_REF] Devlin | Bert : Pre-training of deep bidirectional transformers for language understanding[END_REF]).

Quelques travaux récents proposent enfin de combiner les deux approches en une seule pour prendre en compte l'ensemble du document [START_REF] Sandler | Mobilenetv2 : Inverted residuals and linear bottlenecks[END_REF], [START_REF] Kim | Convolutional neural networks for sentence classification[END_REF]) L'article [START_REF] Bakkali | Visual and textual deep feature fusion for document image classification[END_REF] propose trois techniques pour combiner les caractéristiques extraites à partir du texte et de l'image. Bien que ce travail démontre de très bonnes performances, il n'intègre pas la question du déséquilibre entre les classes lors de l'apprentissage, et de son impact sur les performances, alors que cela est un réel problème pour les entreprises.

Le déséquilibre des classes d'un corpus pose de nombreux problèmes pour les réseaux d'apprentissage profond. Notre modèle apporte une première solution à ce problème en donnant plus de poids aux classes faiblement présentes lors de la rétropropagation via l'utilisation de modèles d'attention. L'idée est d'obliger le modèle à se concentrer sur l'essentiel [START_REF] Jetley | Learn to pay attention[END_REF] afin de mieux séparer les classes. L'un des avantages principaux de ces modèles d'attention [START_REF] Jain | Multimodal document image classification[END_REF], [START_REF] Górriz | Assessing knee oa severity with cnn attention-based end-to-end architectures[END_REF]) est leur capacité à visualiser les caractéristiques utilisées par le réseau neuronal pour prendre leur décision. Cela offre ainsi la possibilité d'interpréter les erreurs sur des modèles utilisant les caractéristiques visuelles ou textuelles. Durant le processus de décision, le système renforce leur importance et supprime les informations hors sujet ou portant à confusion.

Modèle proposé

Le modèle multimodal proposé dans cet article se compose d'un modèle textuel et d'un modèle visuel détaillés avec leurs mécanismes d'attention respectifs. Ces mécanismes permettent de focaliser chaque modalité sur ce qui est important tout en facilitant la compréhension du modèle. Enfin, nous présentons notre approche multimodale qui permet de nous assurer que les informations fusionnées sont pertinentes et les moins redondantes possible. Le modèle d'attention incorporé dans le réseau de classification d'image utilise la sortie de la 5ème couche du modèle VGG16 D (composée de 128 filtres) et de dimension (112,112,128). Plusieurs couches de convolution avec un noyau de taille 1 * 1 sont ensuite empilées avec des filtres de dimensions respectives 64 et 16. La couche d'attention correspond alors à la sortie de ce banc de filtres (A) et est de dimension (112,112,1). La matrice D est multipliée par A pour obtenir la matrice D'. Enfin, une réduction de la dimension est réalisée par l'application d'une couche de type "average pooling" sur D' pour générer un vecteur f de dimension (128). La dernière étape consiste alors à normaliser ce vecteur d'attention par une couche de pooling sur A. Cette normalisation est la fonction g(x1,x2)=x1/x2 et l'entrée du classifier visuel sera la sortie de cette fonction de normalisation.

Modèle d'attention

Approche multimodale -texte et image

Bien que les modèles d'attention permettent d'améliorer les performances intrinsèques de chaque modalité, nous proposons en plus dans cet article, d'étudier la combinaison de cellesci. L'objectif étant de démontrer qu'elles permettent de se compléter et d'améliorer les performances globales en termes de classification de documents. En ce sens, l'architecture multimodale proposée (voir Fig. 1) combine les deux modèles présentés précédemment. De manière pratique, Nous commencons par apprendre l'importance de chaque modalité (c'est à dire, les sorties des classifiers visuels I1 et textuels I2) au travers d'un pondération W . Nous concaténons ensuite la sortie de la multiplication du poids W par les vecteurs I1 et I2 pour l'utiliser dans un classifieur final. Ce dernier vecteur (de dimension 2 * nombres de classes) sera l'entrée de notre classifieur final. Les couches de classification s'appuient sur la même architecture pour le modèle visuel, textuel et multimodal, avec deux couches densément connectées séparées par une couche d'oubli (dropout). Chacun de ces modèles est entraîné séparément dans un premier temps. Puis, ces modèles sont également "fine-tunés" afin d'améliorer leurs performances respectives conjointement.

Le deuxième défi adressé dans cet article concerne le déséquilibre fort entre les classes dans le flux de documents. Cela influe sur les performances globales du système, et touche en particulier les classes faiblement représentées. Afin de prendre en compte le déséquilibre entre les classes dans la phase d'apprentissage du réseau, nous proposons de pondérer ces probabilités au travers d'une d'un vecteur de poids N , comme présenté dans l'équation 1. Le poids attribué à chaque classe est le pourcentage inverse des exemples présents dans l'ensemble d'apprentissage. Ainsi, plus la présence d'une classe est faible, plus son poids durant l'entraînement sera important.

Loss CE (i) = -N t i log(P (i))

(1)

4 Expérimentation

Données

Afin d'évaluer notre modèle multimodal basé sur l'attention, nous avons utilisé deux corpus différents. Le premier est un corpus privé de l'entreprise Yooz et déjà déséquilibré provenant de nos clients. Le corpus YOOZ se compose de 15 000 documents pour l'apprentissage, 2 000 Afin de permettre une comparaison équitable de notre approche avec les modèles de l'état de l'art, nous utilisons également le corpus RVL-CDIP de [START_REF] Harley | Evaluation of deep convolutional nets for document image classification and retrieval[END_REF]. Ce vaste corpus public est composé d'une grande variété de classes de documents équilibrées entre elles. Afin d'évaluer notre contribution, nous proposons un protocole pour le déséquilibrer. RVL-CDIP est un jeu de données équilibré composé de 20 000 images par classe en apprentissage et 2500 images par classe dans les jeux de validation et de test. Afin de valider notre approche, nous avons réduit le nombre de documents pour chaque classe. Nous avons pris 100%, 50%, 10% ou 5% des documents dans chaque classe. Nous avons donc obtenu un total de 33 000 documents pour l'apprentissage et 16 000 pour la validation et les tests.

Pour notre jeu de données privé, nous avons extrait le contenu textuel des documents avec un OCR (ABBYY FineReadear 12), tandis que la version OCR des documents RVL-CDIP est fournie par leurs auteurs [START_REF] Harley | Evaluation of deep convolutional nets for document image classification and retrieval[END_REF]).

Résultats

Le tableau 1 résume les résultats des principales méthodes de l'état de l'art et celles que nous proposons. Nous tenons à rappeler que les autres approches de l'état de l'art ont été entraînées sur l'ensemble des données alors que nos méthodes n'ont été entraînées que sur un ensemble déséquilibré de données. Les auteurs de la première méthode [START_REF] Das | Document image classification with intra-domain transfer learning and stacked generalization of deep convolutional neural networks[END_REF] ont utilisé un réseau de neurones VGG16 et ont obtenu une valeur de performance de 91% en preci-sion,leurs images ont été divisées en quatre zones (en-tête, pied de page et côtés gauche et droit de la page) afin d'avoir des processus de classification spécifiques par zones. Nous pouvons observer que les systèmes que nous avons proposés ont obtenu des performances inférieures sur l'image uniquement. Ceci valide notre hypothèse qu'un ensemble d'entrainement réduit a un impact sur les performances finales (leur réseau VGG16 a été entrainé sur l'ensemble du jeu de données RVL-CDIP contrairement au nôtre).Pour la partie textuelle seule, [START_REF] Bakkali | Visual and textual deep feature fusion for document image classification[END_REF] TAB. 1 -Évaluation des performances sur RVL-CDIP Maintenant que nous avons validé les performances de notre approche proposée sur un ensemble de données publiques, nous présentons dans le tableau 2 les performances obtenues sur notre ensemble de données privée. On peut observer qu'une fois de plus, nous avons obtenu les meilleurs résultats avec l'utilisation d'une approche mutlimodale. De plus, les documents de ce jeu de données se distinguent principalement par le texte. L'utilisation de l'image dans un seul canal ne permet pas dans la plupart des cas de classer correctement le document.

Enfin, nous pouvons conclure que l'approche proposée améliore toujours les résultats même lorsque l'ensemble d'apprentissage est réduit et déséquilibré.

Conclusion

Dans cet article, nous avons proposé des méthodes utilisant la multimodalité et les modèles d'attention sur le visuel et le textuel pour la classification de document. L'utilisation de la multimodalité est nécessaire afin de tirer parti des modèles textuel et de l'image pour renforcer les performances. Nous utilisons ici des modèles d'attention pour améliorer les performances mais

  Apprentissage multimodal basé sur des modèles d'attention pour la classification de documents dans un contexte déséquilibré Ibrahim Souleiman Mahamoud * , Joris voerman * , Mickaël Coustaty * * Aurélie Joseph * , Vincent Poulain d'Andecy * , Jean-Marc Ogier * * 1 Introduction

  3.1.1 Module d'attention textuellePour la partie textuelle nous nous appuyons sur une chaîne classique d'extraction d'information textuelle (suppression des "stop-words", des chiffres, des symboles et enfin une lemmatisation des mots). Ce contenu prétraité est ensuite résumé sous la forme d'un vecteur à l'aide du modèle Bert[START_REF] Devlin | Bert : Pre-training of deep bidirectional transformers for language understanding[END_REF]. Pour cela, nous avons converti les 250 premiers mots de chaque document en un vecteur grâce à une architecture Bert pré-entrainé sur l'anglais et le français[START_REF] Martin | Camembert : a tasty french language model[END_REF] car nos documents sont soit en anglais soit en français. La taille de ce vecteur est de 768. Une fois le texte résumé sous la forme d'un vecteur, nous utilisons une architecture récurrente de type LSTM afin d'apprendre les relations intrinsèques entre les mots du document. Enfin, nous avons placé notre mécanisme d'attention juste après la sortie du réseau récurrent. Le modèle d'attention utilise la sortie de notre modèle LSTM noté h de dimension (250,768). Ce modèle d'attention est composé d'une couche dense suivie d'une couche d'activation softmax. Cela permet d'obtenir un vecteur d'attention t de dimension (250). Le produit scalaire entre le vecteur t et h sera l'entrée du classifieur texte. Nous utilisons le vecteur t pour analyser l'attention des mots.3.1.2 Module d'attention visuelleLe mécanisme d'attention visuel s'inspire des travaux réalisés par[START_REF] Górriz | Assessing knee oa severity with cnn attention-based end-to-end architectures[END_REF]. L'utilisation de ce modèle permet de concentrer le système sur la partie la plus pertinente d'une image (comme les logos, les signatures ou tout autre élément graphique distinctif). Afin de tester cette idée, nous proposons d'utiliser l'architecture VGG16 de[START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] pour extraire les caractéristiques visuelles. Suite aux performances rapportées par Jain et Wigington (2019), nous avons utilisé une approche classique d'apprentissage par transfert, avec un modèle VGG16 pré-entraîné sur le jeu de données d'ImageNet. Nous proposons de coupler ce réseau aux travaux décrit dans Górriz et al. (2019) comme modèle d'attention. Ce modèle propose d'ajouter une couche d'activation de type sigmoïde pour générer un masque d'attention. Ce masque permet ainsi de focaliser le système sur des zones visuelles via une fusion dans la dernière couche de l'architecture VGG16.

FIG. 1 -

 1 FIG. 1 -Modèle proposé utilisant la multimodalité et le modèle d'attention

  a utilisé une approche similaire à la nôtre mais nous pouvons observer que nous avons de meilleures performances. En effet bien qu'ayant la même architecture, l'utilisation de notre modèle d'attention nous permet d'augmenter la précision d'un point.Les meilleurs résultats pour la classification de documents sont obtenus grâce à l'utilisation d'une architecture multimodale, L'approche multimodale proposée dans[START_REF] Audebert | Multimodal deep networks for text and image-based document classification[END_REF] utilisait un réseau VGG16 et un réseau LSTM récurrent avec ELMO pour le contenu textuel. Nous comparons notre travail avec ce modèle et nous pouvons observer que nous avons une performance légèrement supérieure alors que nous avons utilisé un modèle attaché qui a été entrainé sur un ensemble de données plus petit et déséquilibré. Cela met clairement en évidence la robustesse du modèle que nous proposons pour les données déséquilibrées. La multimodalité a beaucoup mieux fonctionné sur le corpus RVL-CDIP car les classes se distinguent surtout par leur structure, de sorte que le modèle s'appuie sur l'image lorsqu'elle n'atteint pas la classe avec le texte.

	Modèles	Precision Rappel
	Modèle proposé utilisant la partie visuelle (VGG16 + Attention)	83.4%	83.2 %
	Partie Visuelle (VGG16_pretained Das et al. (2018))	91.1 %	-
	Partie textuelle (BERT Bakkali et al. (2020))	86%	86%
	Modèle proposé utilisant la partie textuelle (LSTM + attention)	87.1%	86. 9%
	Approche multimodale Audebert et al. (2019)	90.6 %	-
	Modèle multimodal proposé avec attention (LSTM + VGG16 + Attention)	91.1 %	90.7%
	Multimodalité + Attention + Pondération (LSTM + VGG16 + Attention + Weighted loss)	92.3%	92.0%

  Évaluation des performances sur les données YOOZ également pour mieux interpréter les sorties des modèles. Avec l'utilisation de notre système multimodal pondéré avec attention, nous obtenons une augmentation de précision d'environ 2% comparé à de l'état de l'art. Malgré les bonnes performances que nous avons obtenues grâce à la multimodalité, tant sur notre jeu de données que sur RVL-CDIP, il reste encore beaucoup à faire. Nous avons envisagé l'utilisation d'un système d'attention multimodal et des fonctions de coût plus adaptés. D'autres pistes pour mieux extraire les mots les plus pertinents dans un document sont en cours.

	Modèles	Precision	Rappel
	Modèle proposé utilisant la partie visuelle (VGG16 + Attention)	87.9%	87.4%
	Modèle proposé utilisant la partie textuelle (LSTM + attention)	96.2%	95.7%
	Modèle multimodal proposé avec attention (LSTM + VGG16 + Attention)	96.8%	96. 1%
	Multimodalité + Attention + Pondération (LSTM + VGG16 + Attention + Weighted loss)	97.3%	97.18%
	TAB. 2 -

Summary

The corporate documents classification process may rely on the use of image analysis approach considered separately of textual features.

The recent state of art deep learning methods propose to combine those two within a multimodal approach. In addition, corporate documents classification processes offer a particular challenge for deep learning based systems with an imbalanced corpus. This paper presents an evaluation of several state of the art methods designed for document classification task using the textual content, the visual content and some multi-modal approaches. We complete this evaluation with our own method, a multi-modal network with an attention model .This combination offers a performance gain of 1% for our private database and 3% for the public RVL-CDIP database.


Ibrahim Souleiman Mahamoud
email: ibrahim.souleimanmahamoud@univ-lr.fr

Mickaël Coustaty
email: mickael.coustaty@univ-lr.fr

Aurélie Joseph
email: aurelie.joseph@getyooz.com

Vincent Poulain D'andecy
email: vincent.poulaindandecy@getyooz.com

Jean-Marc Ogier
email: jean-marc.ogier@univ-lr.fr

+ Aurélie

QAlayout: Question Answering Layout based on multimodal Attention for visual question answering on corporate Document

Keywords: Visual question answering, Multimodality, Attention mechanism

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Imagine a near future where you will not process any document but your digital clone will do it for you. It will summarize and/or extract the relevant and useful information for you. For some, this future seems close while for others it is a simple illusion. Some companies have already started this transition such as the New Zealand company UneeQ [START_REF] Tomsett | [END_REF], which created an avatar of Albert Einstein that you can interact with on various topics.

If we go back to the present, companies try to provide easy-to-use solutions to extract information from the corporate documents. These corporate documents are varied in both content and form (i.e. invoices, order form, resume, pay slip etc). Thus the customer processing are evolving everyday, and if they today only focus on the extraction of information from invoice, tomorrow they will ask for more and more intelligent process of documents (i.e for instance the automatic processing of collaborators resume, or the automatic linking of file folders). We would therefore like to have a method to extract information with few data and able to adapt to another type of document while taking into account the changes in layout and content to be extracted.

To answer these problems, various state-of-the-art papers have tried to provide a solution. Some few-shot learning [START_REF] Wang | Generalizing from a few examples: A survey on few-shot learning[END_REF] methods were trained only a corpus composed of a hundred of documents. These methods have all shown their weakness when used with a large dataset. Some other methods, based on deep-learning techniques such as Lambert [START_REF] Garncarek | Lambert: Layout-aware language modeling for information extraction[END_REF] or LayoutLM [START_REF] Xu | Layoutlm: Pre-training of text and layout for document image understanding[END_REF],have appeared recently. Their performances were impressive but with the drawback that these methods require a lot of data to converge. A very large dataset of annotated documents is not always available and in order to deal with this limitation, some recent works proposed to use incremental methods [START_REF] Wang | Incremental learning from scratch for task-oriented dialogue systems[END_REF] which can evolve over time.

We propose in this paper a method to process different types of documents and can also extract the different information by the customer for a question answering model. The usefulness of visual question answering on corporate documents unlike predefined extraction is to allow to extract more general information that can be adapted to a new corpus of data.

In general when we read a paper whether it is scientific or other, some questions come to mind. For example for our paper, if you try to understand our contribution you quickly read the abstract or propose method to have an answer. This natural ability to focus on a part of the information from questions is innate to us we look at what is essential to solve a problem. It is by being inspired by this that the mechanism of attention has seen the day several papers such as [START_REF] Vaswani | Attention is all you need[END_REF] are sold the merit of mechanism of attention. The attention mechanism is used in several situations, some to optimize the performance of their model, others to be able to interpret the model and thus brings some element of answer to the behavior of this black box. The visual question answering is known to be very popular in the community because many problems can be solved with it, from simple questions on images of natural scenes [START_REF] Goyal | Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering[END_REF] to medical assistance [START_REF] Lin | Medical visual question answering: A survey[END_REF] to help specialties to better understand some images thanks to the strength of accumulating a large amount of information to synthesize and then to keep it in its memory of artificial neural networks.

The visual questions answering approaches on corporate documents are minimal and the data also concerns this subject. In this article we will describe our method based on the Qanet architecture [START_REF] Yu | Qanet: Combining local convolution with global self-attention for reading comprehension[END_REF]. The key motivation behind the design of our model is as follows: The convolution layer helps to capture the local structure of the image and text. The co-attention layer allows to have a global relationship between the inputs in order to define the positive or negative impact of one in relation to the other. We have evaluated our model on the Squad [START_REF] Rajpurkar | Know what you don't know: Unanswerable questions for squad[END_REF] and Docvqa [START_REF] Mathew | Docvqa: A dataset for vqa on document images[END_REF] datasets. The choice of these two datasets is that one is only textual, it will allow us to test and optimize the textual part of our model while the Docvqa is a corpus of documents of different type, it will allow us to test if our model will manage to use well this multimodal corpus.

The contributions of this paper are summarized as follows:

-We have made available a corpus of Visual question answering data VQA-CD for corporate documents, this corpus is to our knowledge the first in the state-of-the-art. We have annotated 3 thousand questions from ∼ 693 documents.

-We propose a multi-modal co-attention model for visual question answering. This template will use the visual and textual content features of this document and the layout features for each word. This model learns the best way to use cross-modality to predict the answer of a question.We use this self-attention to focus our network on common features from the input. This will allow us to exploit the context and query correlation at the initial stage.

Related work

The approaches proposed for VQA are generally distinguished in three categories. Some use a single modality like Qanet [START_REF] Yu | Qanet: Combining local convolution with global self-attention for reading comprehension[END_REF]. Although these methods show good performances by using some transformer-based model [START_REF] Vaswani | Attention is all you need[END_REF], they remain less effective when the multimodal understanding is necessary.

The second category of methods proposes to rely on multimodal architectures in order to be able to deal with the visual and the textual content at the same time [START_REF] Wu | Multi-modal answer validation for knowledge-based vqa[END_REF]. These methods have then be designed to include a second modality in the proposed architecture. Even if only these models require better performances, this can only be done by using large dataset and they require to re-train the model each time new kind of input is used. This relies on the fact that the document layout may vary in a significant way and this is not taken into account. For example, if you train a model on a dataset where the addresses are generally located at the top of documents, this one will have difficulties to find them elsewhere (addresses could be located at the bottom, or in the middle of new documents and dataset). Such kind of models then need to be re-trained on new dataset which should come with its annotations.

To overcome this problem, some recent models like LayoutLM [START_REF] Xu | Layoutlm: Pre-training of text and layout for document image understanding[END_REF] have been introduced to be able to add a third modality representing the layout information of documents. The most recent and popular model from this third category includes the famous LayoutLM [START_REF] Xu | Layoutlm: Pre-training of text and layout for document image understanding[END_REF], LamBERT [START_REF] Garncarek | Lambert: Layout-aware language modeling for information extraction[END_REF] and ViBERTgrid [START_REF] Lin | Vibertgrid: A jointly trained multi-modal 2d document representation for key information extraction from documents[END_REF]. LAMBERT [START_REF] Garncarek | Lambert: Layout-aware language modeling for information extraction[END_REF] proposed a model based on the Transformer encoder architecture RoBERTa [START_REF] Liu | Roberta: A robustly optimized bert pretraining approach[END_REF]. The main contribution of this paper was to propose a generalpurpose language model that views text not simply as a sequence of words, but as a collection of tokens on a two-dimensional page by applying relative attention bias. In their industrial context the use of text, bounding box and therefore not image allows to eliminate an important performance factor in industrial systems. They have conducted several evaluations on several public datasets The Kleister NDA and Kleister Charity, SROIE and CORD. A deep experimental study allows comparing it to other state-of-the-art methods such as their baseline Roberta, LayoutLM [START_REF] Xu | Layoutlm: Pre-training of text and layout for document image understanding[END_REF], and LayoutLMv2 [START_REF] Xu | Layoutlmv2: Multi-modal pre-training for visuallyrich document understanding[END_REF]. Although lambert is a good method, it does not take into account the whole image and the correlation that can exist between the layout information and the textual content. ViBERTgrid [START_REF] Lin | Vibertgrid: A jointly trained multi-modal 2d document representation for key information extraction from documents[END_REF] proposed a new multi-modal network by combining the best of BERTgrid and CNN to generate a more powerful grid-based document representation. It simultaneously encode the textual, layout and visual information of a document in a 2D feature map.

In order to go one step further, and to reduce the gap between users' need and the documents, we propose to integrate a question answering approach in this kind of architecture. To the best of our knowledge, and as discussed before, different 2D representation of documents have been proposed in the litterature but none of them integrate a question answering in the process and no works have proposed to inclide an attention mechanism mixing the question and the 2D representation.

Problem Definition

The main objective of our proposal is to provide solutions for the automatic comprehension of documents that requires both a visual analysis and a semantic understanding of their content. The visual analysis remains essential to capture some contextual information. In parallel, the document layout is necessary to extract the correct word in the document (like when a human distinguish two similar tokens based on their visual context). The other hand, automatic comprehension means being able to provide an answer to a question about the content of the document. Starting from a visual context with D documents, we can define the document image I = i 1 , i 2 , .., i d , its associated bounding boxes (one for each word) B = b 1 , b 2 , .., b m and its textual information based on the extracted text (using an OCR) T = t 1 , t 2 , .., t m with m being the number of words from the document. We also propose to define the query sentence (with k words) Q=q 1 , q 2 , ..; q k which then correspond to the resquest that a user could submit to the system (what he/she is looking for). The questions are related to the corporate document field (i.e. What is the total amount of this invoice? What is this type of document? ) and the answer varies according to the question they can be categorical, numerical or textual.

I ∈ R d , B ∈ R d×m , T ∈ R d×m andQ ∈ R d×k (1)
The questions refer to the document's content. The answers are therefore information extracted from the text present in the document. The objective of our proposed prediction task is then to predict the present beginning and ending word of the answer present in the text. The general assumption is that an unknown function correlates the samples of the questions with the prediction of the beginning word s1 and the end word s2 , i.e. (s1, s2) = f (Q). The goal of the learning process is to provide an approximation of this unknown function. To better approximate this function f , we need to know the correlation between the question and the answer from the visual and textual features and from the similarity between the questions.

Proposed approach

In order to set up our proposed model, we got inspired by state-of-the-art models such as Qanet [START_REF] Yu | Qanet: Combining local convolution with global self-attention for reading comprehension[END_REF] and LayoutLM [START_REF] Xu | Layoutlm: Pre-training of text and layout for document image understanding[END_REF] to propose an architecture based on textual and visual attention models. As we tend to address industrial tasks, where the time needed to process each document must remain low, we only use convolutional and self-attention mechanisms, discarding recurrent neural network (known as slower architectures as they can not process the input tokens in a parallel way).

Another limitation from the most recent and relevant state-of-the-art models relies on their large number of parameters (i.e. 300M parameters for Lay-outLMV2). Setting up a model mixing visual content, the semantic, the layout and the question would then require more than 600 million data to converge. We would like to remember that in industrial context, no large annotated dataset could be available, and they would then require a huge effort of annotation to link question and content. We then propose to develop a model able to learn with few data (e.g. hundreds or thousands of annotated documents) while maintaining the execution time as low as possible (generally, companies may not devote more than a second to each document).

Several studies have been conducted to know if we could establish a correspondance between the number of training data and the number of model parameters. The paper [START_REF] Zhang | Understanding deep learning requires rethinking generalization[END_REF] conducted several tests on hypothesis related to the generalization capacity of neural networks. The authors demonstrated that "Theoretically, a simple two-layer neural network with 2n + d parameters is capable of perfectly fitting any dataset of n samples of dimension d". We can then observe that state-of-the-art approaches have much more than 2n+d parameters and one of our hypothesis is to propose a model with much less parameters (less than 10 Millions). In practice, finding the optimal number of parameters is not an easy matter. It depends on the diversity and number of data available in the training set and on the selected architecture. Models with many parameters are likely to overfit while they have the advantage of being able to model much more complicated knowledge (like some latent relationship between the data). Methods with a few numbers of parameters (i.e. less than 20M) are not much studied in the state-of-the-art especially applied on the corporate document. Our second contribution is then to propose a model having a maximum number of 8 millions parameters and which having almost similar results to the state-of-the-art method thanks to our proposed attention mechanisms. In a first step, we will present the global architecture. Then we will describe the used encoder architecture, and finally we will discuss the proposed co-attention. The probality of the beginning and the end of the answers corresponding the label as y i ∈ R 2×m .The beginning and the end correspond to the first and last words that the answer will contain, which is a selection from the words present in the document. The symbole m corresponds the total number of word in the textual context. The textual content of a document is different, we can have a document with ten words while others exceed 500 words. So that our entries are of the same dimension we keep the prepocessing method described in Qanet to keep one size for all entries. For the semantic embedding of each document, we chose to keep the 400 first words. If the number of word is lower than this, we add some padding tags. We justify this choice of 400 words as we observed that the the average numbers of words in corporate documents is around 300. This parameters could be obviously modified to other context, but we use this value even for larger datasets like DocVQA. In a similar way, we chose to limite the question size to the 50 first words. Once the text has been extracted, we use the classical BERT embedding features [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF], where each word is represented by 768 (d 1 ) values. We then obtain a total size of (400,768) for the textual context, and a total size of (50,768) for questions.

For the bounding boxes, we used the outlines of the word from the original image and we normalized them to the width and height of the image. We then obtained a two dimensions vector d 2 composed of their cartesian coordinates (x max ,y max ,x min ,y min ).

The last part of the documents input relies on the visual content. To this end, we resized document images to a dimension vector d 3 = (224, 224). We then use a VGG16 convolutionnel network to embbed the visual content to obtain a (512,7,7) feature dimension vector. This vector is then provided as an input to our co-attention model.

I = [i 1 , i 2 , ..., i n ] ∈ R D×d3 T = [[t 1 , t 2 , ..., t m ], ..] ∈ R D×m×d1 B = [[b 1 , b 2 , ..., b m ], ..] ∈ R D×m×d2 Q = [[q 1 , q 2 , ..., q k ], ..] ∈ R D×k×d1 (2)
As described in the figure 1 ,it's following end-to-end operations:

-In a first step, the input described in the equation goes through models such as Bert and Vgg16 to extract the relevant embedding for our model. -Then all the input text tensors are passed to the embedding encoding layer which is a single encoding block with 4 conv layers. 8 attention heads are used in the auto-attention module which is the same for all encoding blocks of the model. -Finally we use self-attention to transform our input into new input, allowing us to exploit the correlation between question and context (e.g. textual context, bounding box and image) at the initial stage.

The self-attention mechanism used in this paper is inspired by [START_REF] Cheng | Attentive graph neural networks for few-shot learning[END_REF]. They propose a self-attention mechanism on the input image to consider the inherent correlation (attention) between the input features themselves, and then use a graph neural network for the classification task.

Self-attention allows us to transform T, B, Q, I into T ′ , B ′ , Q ′ , I ′ which will be the inputs to our co-attention model. To do that, we follow several steps, the first one consists in computing the correlation matrices between the sample and the label.

C t = sof tmax(T T T ) C i = sof tmax(II T ) C b = sof tmax(BB T ) C q = sof tmax(QQ T ) (3)
Here softmax(•) denotes a softmax operator.The inputs of this function have all the same dimension where BB T , QQ T , II T andT T T ∈ R N ×N

The self-attention module exploits C t , C i , C b , C q (see eq 3).

C m = f usion([C t , C i , C b , C q ]) ∈ R N ×N (4)
where [C t , C i , C b , C q ] denotes the attention map concatenation. This fusion function f usion is equivalent e.g

C m = w 1 C t + w 2 C i + w 3 C b + w 4 C q ,
where the weighted parameters w 1 , w 2 ,w 3 and w 4 are learned adaptively and N is batch size .

T ′ = T C m I ′ = IC m B ′ = BC m Q ′ = QC m (5)
These modified inputs as indicated in the equation 5 will be reused in the coattention part of the model. In the following sections we will detail the encoder part and the co-attention part.

Encoder

As presented in figure 2, the encoder is composed of several convolution layers and a attention layer. Unlike the traditional convolution layer, we use depthwise separable convolution [START_REF] Chollet | Xception: Deep learning with depthwise separable convolutions[END_REF]. In this paper [START_REF] Chollet | Xception: Deep learning with depthwise separable convolutions[END_REF] depthwise describes that this convolution layer is memory efficient and has better generalization. That help us because this model will be used in a production system and it should be light and fast. For this work, we chose to set up the kernel size to 7, the number of filters to d = 128 and the number of convolutionnal layers within a block to 3. The output of this convolutionnal model is then transfered to an attention layer.

Fig. 2. One Encoder

We adopt the multi-headed attention mechanism defined in [START_REF] Vaswani | Attention is all you need[END_REF] which, for each position in the input, called a query, computes a weighted sum of all positions, or keys, in the input based on the similarity between the query and the key, as measured by the system. the scalar product.

As one can see in the figure 2, the model involves residual connections, layer normalizations and dropouts too. For each input x and a given operation f , the f is defined as f = (layernorm(x)) + x [START_REF] Ba | Layer normalization[END_REF] of identity at the input and output of each block repeat. This block is repeated 7 times. The proposed "Co-attention" step proposed in our work is inspired by the attention flow layer from the BIDAF architecture [START_REF] Seo | Bidirectional attention flow for machine comprehension[END_REF] (see Fig. 3. It calculates attention in two directions. Context-query attention tells us what query words are the most relevant to each context word like descripted in eq6 and 7. The input 1 (i 1 ) represents the textual context, the boundind box of each word is input 3(i 3 ) and input 2 (i 2 ) represents the question finally input 4 (i 4 )the whole box and encoded context and query respectively. Given that the context length is j and query length is m, a similarity matrix is calculated first. The similarity matrix captures the similarity between each pair of context and query words. It is denoted by S and is a n-by-m matrix. The similarity matrix is calculated as,

Co-Attention

S = f (i 1 , i 2 ) ( 6
)
where f is a trilinear similarity function defined as,

f (i 1 , i 2 , i 3 , i 4 ) = W 0 [i 1 ; i 2 ; i 1 • i 2 , i 3 • i 4 ] (7)
5 Experiments

Dataset

Proposed dataset VQA-CD is a new public dataset containing 3000 questions extracted ∼ 693 documents from RVL-CDIP [START_REF] Harley | Evaluation of deep convolutional nets for document image classification and retrieval[END_REF]. To the best of our knowledge, no public visual question answering dataset exists for corporate documents. We then decided to annotate and to share our work with the scientific community. This dataset is based on the public RVL-CDIP [START_REF] Harley | Evaluation of deep convolutional nets for document image classification and retrieval[END_REF] dataset. This document extracted from RVL-CDIP [START_REF] Harley | Evaluation of deep convolutional nets for document image classification and retrieval[END_REF] invoice class does not only contain invoices but also other types of document such as purchase order. The documents found in VQA-CD dataset contains some documents are well structured while others contain some semi-structured or raw text. This heterogenity of content mimics the industrial context.

When we annotated, we were careful to make sure that each question could be repeated on other documents in order to have a balanced corpus. This balanced corpus is necessary because the inherent understanding of language forces the model to focus on the question rather than the image. For example, if the question is "How much is the ..." and the prediction of the model is a number even if this number does not correspond to the expected one, the model assigns a high probability to it, the same for other type of question ("who is ...", "is it ...?", etc) If for example the question is "What is the total amount" and this question is found in two images I and I'. If the answer related to two images differs then the model will be forced to learn the visual feature to distinguish it.

In order to ensure a kind of compatibility with the DocVQA dataset, we use the same organization of the dataset. We separated our questions for each document randomly. We then divided our 3000 questions into train, test and validation. We took 50% for train and 25% for validation and test.

SQuAD dataset Stanford Question [START_REF] Rajpurkar | Know what you don't know: Unanswerable questions for squad[END_REF] Answering Dataset is a reading comprehension dataset consisting of questions posed by crowdworkers on a set of Wikipedia articles. The answer to each question is a segment of text. This dataset consists of 100,000 questions. The corpus content is only text and no layout or image information is available.

DocVQA dataset is to our knowledge the most complete dataset in both content and number of samples. The dataset consists of 50,000 questions defined on 12,000 document images. In the figure 4 you will notice the most recurrent words in the questions and also in the answers.So we can see our questions are often linked to a date or an amount inside in the document.

Implementation details

The training phase of our model was run with the ADAM optimizer with β1 = 0.8, β2 = 0.999 and a batch size of 64, an initial learning rate of 10-3 scaled from 0.1 every 3 epochs without improvement in validation loss and an early stopping after 5 epochs without improvement.

Performance evaluation

To evaluate the performance of our models we will use three metrics (ANLS,F1-SCORE,EM). Average normalized Levenshtein (anls) measure the distance between two string (q k , p k ) see the equation 8. where Q is the total number of questions. Each question can have 3 answers and k words , q k the ground truth answers and p k is the prediction of model. N L(q k , p k ) is the Normalized Levenshtein distance between ground truth and the prediction . Then a threshold τ = 0.5 to filter NL values larger than τ by returning a score of 0.

AN LS = max

1..3 s(q k , p k ) s(q k , p k ) = (1 -N L(q k , p k ) if N L(q k , p k ) < τ 0 if N L(q k , p k ) >= τ (8) precision = 1 * same word tail(p k ) recall = 1 * same word tail(q k ) f 1 = 2 * precision * recall (precision + recall) (9)
The metric F1-SCORE is descripted in 9 . Where same word count the number of similar words between GT and the prediction.

In the table 1, we have the results on the corpus squad containing only text and the corpus Docvqa containing image and text.

First we tried to compare the performance of the text-only part using the context text and the question only (QAlayout(Only Text)) with Bert [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF].

When we compare our QAlayout(Only Text) to the state-of-the-art Bert [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF] model on the squad corpus we get better results as described in the section. These good performances also add up to a much faster training time. The different attention mechanisms that we have detailed in section Global description have largely contributed to these results. QAlayout(Only Text) despite its good performance has limitations. These limitations in a text-only corpus may be due to not understanding the syntactic structure. For example if the question is "What is the name of the Bungie Inc. founder who is also a university graduate?" and the context contains the following words: "In the arts and entertainment, minimalist composer Philip Glass, dancer, choreographer and leader in the field of dance anthropology Katherine Dunham, Bungie founder and developer of the Halo video game series Alex Seropian, ..." our model predict "Katherine Dunham" while the correct prediction is "Alex Seropian". Although Katherine is close to the word founder this does not grant her the status of founder. QAlayout(Only Text) also has other limitations related to the question related to the document layout. For example in the result table, we notice that the QAlayout(Only Text) is good when the question is related to free Text and has trouble with question related to Figure or Form.

The QAlayout method using the image and the characteristic bounding box in addition to the QAlayout(Only Text) is that allowed to provide answers to the question related to visual structure of the text. The performance has greatly improved on the question related to figure/diagram or form.

In the table 3 we also get the performance of our model on the VQA-CD dataset.

In our VQA for corporate documents, we have either ∼ 80% of the answers with one word and ∼ 17% of contain only two words. Unlike to the other VQA task where the answer size is longer. These one-word answers will certainly result in a kiss on the F1-SCORE. You will notice that the multimodal model is better than the QAlayout(Only Text) model 5% in all scores . For the VQA-CD corpus we have managed to compute several scores because we have the GT.

The metrics ANLS, F1-SCORE and EM have a value of 100% each if the correspondance between the prediction and GT are totally similar. Their difference comes if the prediction is different from the GT in this case for the em score will have 0% (i.e. either we have all or nothing for this score). For example if the model predicts 1200 and the GT is 100 for the metric ANLS we would have 1 -transformation cost so NL is 1-0.25=0.75 and as the threshold is equal to 0.5 (i.e the same as the paper [START_REF] Biten | Icdar 2019 competition on scene text visual question answering[END_REF] ) this result it will be taken into account in the final results. In a document this two numbers (1200,100) can be two different amounts or just a case where the ocr can't extract the character 2 in the image. This score ANLS remains an approximate value that can help in some cases to limit the impact of ocr errors. Finally for the metric F1-SCORE as it is based on the token, we have calculated two tokens one at the word level and another at the character level (i.e. in table 3 it is the token at the word level). The F1-SCORE that we obtained at character level is ∼ 60%. It remains clearly higher than the token words. Nevertheless this F1-SCORE metric is not adapted in our task VQA for corporate document because either we take a token at word level and as the majority of our tokens are one word we end up with a very low score or we take at character level and therefore the ANLS score would be more accurate.

Although our model has a good performance, in most cases it confuses the GT amount with another amount or extracts only a part of this answer. The other errors are often OCR errors as the images VQA-CD are old with low-resolution Also the limitations of QAlayout are also numerous. Sometimes, these limitations are due to a bad understanding of the visual part (i.e difficulty to correlate the elements inside the image). Also the multimodality help us on some cases its performance is not yet the desired one.

Conclusion and future work

Visual question answering is a task that requires a good understanding of both visual and textual information by correlating this information with the question. We propose a fast and accurate end-to-end method QAlayout that uses visual information the whole image document or layout as well as textual information. Our QAlayout method uses an attention mechanism to take into account the inherent correlation between the question and its visual or textual context at the input of the model with self-attention or after with co-attention. Compared to some state-of-the-art models we have much better results while having less parameters (8M). The limit number of parameters corresponds to the expectation of the industrial context which requires a fast training and prediction time while keeping a reasonable performance. We also contributed to annotate a new dataset VQA-CD containing 3000 questions on corporate documents. Despite the good performance of our model, some limitations exist and we will try to provide a solution. For example, we will build a graph system to establish links between words in the textual content or between areas in the visual content.

Fig. 1 .

 1 Fig.1. Description of the QAlayout model. The inputs are the textual context extracted from the image, the visual context is whole image . Also the layout information (i.e bounding box of each word) and finally the question. We will try to predict two probality the beginning and end of the answer from the word present in the textual content.We have a self-attention and co-attention mechanism to learn multimodal comprehension

Fig. 3 .

 3 Fig. 3. Description of the one co-attention

Fig. 4 .

 4 Fig. 4. Word clouds of words in answers (left) and questions (right).

Table 1 .

 1 This table contains the results of the proposed QAlayout model and the results of the state-of-the-art method (LayoutLm,Bert).

	Modality	Data		SQUAD DOCVQA
	Method	Param F1-SCORE ANLS
		Bert	∼ 110M 74.430%	45.57%
	Text only	QAlayout(Only Text) ∼ 1M	82.19%	48.63%
		LayoutLMv2	∼ 426M	-	86.72
	Text + Layout + Image	LayoutLM	∼ 160M	-	68.93
		QAlayout(All inputs) ∼ 8M	-	77.65%

Table 2 .

 2 The result for the different categories of question-answer in the Dovqa

	Method	Figure/Diagram	Form	Table/List Layout Free text
	Bert	22.33%	52.59%	26.33% 51.13% 77.75%
	QAlayout(Only Text)	18.53%	56.12%	35.85% 53.42% 74.75%
	QAlayout(All inputs)	39.20%	73.21%	86.21% 63.20% 71.96%
	Method	Image/Photo Handwritten Yes/No Others
	Bert	48.59%	35.65%	3.45% 5.778%
	QAlayout(Only Text)	30.10%	37.26%	17.24% 34.50%
	QAlayout(All inputs)	44.66%	62.82%	59.07% 67.49%

Table 3 .

 3 The results VQA-CD dataset with different metrics

	Method	ANLS F1-SCORE Exact
	QAlayout(Only Text) 36.29% 29.16% 25.58%
	QAlayout(All inputs) 42.54 % 35.92% 33.01%

Acknowledgment

This research has been funded by the LabCom IDEAS under the grand number ANR-18-LCV3-0008 , by the French ANRT agency (CIFRE program) and by the YOOZ company.