Romain Karpinski
email: romain.karpinski@yahoo.fr

Abdel Belaïd
email: abdel.belaid@loria.fr

Semi-Synthetic Data Augmentation of Scanned Historical Documents

Keywords: Synthetic image generation, BOI, TOI, Inpainting method

This paper proposes a fully automatic new method for generating semi-synthetic images of historical documents to increase the number of training samples in small datasets. This method extracts and mixes background only images (BOI) with text only images (TOI) issued from two different sources to create semi-synthetic images. The TOIs are extracted with the help of a binary mask obtained by binarizing the image. The BOIs are reconstructed from the original image by replacing TOI pixels using an inpainting method. Finally, a TOI can be efficiently integrated in a BOI using the gradient domain, thus creating a new semi-synthetic image. The idea behind this technique is to automatically obtain documents close to real ones with different backgrounds to highlight the content. Experiments are conducted on the public HisDB dataset which contains few labeled images. We show that the proposed method improves the performance results of a semantic segmentation and baseline extraction task.

I. INTRODUCTION

Nowadays, deep learning methods assert themselves as robust methods of image processing. However, these techniques often require a lot of training data to be precise and reliable. Regarding historical documents, the amount of training data is often low because it requires the images to be analyzed and labeled by a domain expert. This labeling is time consuming therefore costly at all points. Analyzing historical documents is a challenging task since they contain a lot of difficulties. Among those difficulties, we can find heterogeneous layouts and degradations such as stains, ink drop, ink fading or missing regions. Regarding the layout, the arrangement of the lines in images makes their processing difficult.

The method employed can be summarized in three steps: First, the TOI is extracted using the binarized image. Second, all pixels composing the TOI are removed from the original image before applying an inpainting technique. As widely described in [START_REF] Ravi | Image in-painting techniques-a survey and analysis[END_REF], inpainting is a technique inspired from art restorer. It aims to restore missing or degraded parts of an image. It can be used to recover missing regions or remove undesired objects in an image. Finally, a new semi-synthetic image is then obtained by mixing in the gradient domain TOI and BOI from different sources.

In this paper, foreground pixels are referring to ink pixels that were added to the original blank page. A TOI is a collection of foreground pixels. As opposed to foreground pixels, background pixels refers to pixels that were present in the original blank page. Therefore, a BOI is recomposed from all non foreground pixels. The paper is organized as follows. Section II presents methods related to data augmentation and degradation models and their differences with our method. Section III describes in details the proposed method. The experiments performed with their results are described in section IV. Finally, a conclusion in section V provides a synthesis of the work and gives future trends.

II. RELATED WORK

The literature contains few works related to data augmentation, especially for historical documents. Kanungo et al. [START_REF] Kanungo | Global and local document degradation models[END_REF] proposed a global and local degradation model. Their model is based on real perspective distortions appearing during the scanning process. They take into account physical deformations at a global level. Then, a morphological model is used for local distortions. It changes the pixel values depending on two conditional probabilities: the probability that a foreground pixel becomes a background pixel and vice-et-versa.

To generate synthetic handwritten text lines, Varga et al. [START_REF] Varga | Generation of synthetic training data for an hmm-based handwriting recognition system[END_REF] use cosine waves and apply them to text lines images in different ways to obtain synthetic data. The results obtained showed that in the majority of the experiments, an improvement of the recognition rate was observed. Kieu et al. [START_REF] Kieu | A character degradation model for grayscale ancient document images[END_REF] use several degradation models to generate semi-synthetic historical documents. The most common degradation models present in historical documents are employed. The ground truth is generated along the image. The method works in three steps. First, real objects are extracted from document images such as characters, background images, figures, etc. Then, degradation models (opacity, curvature, character pixel modification) are used to add noise to source images. Finally, a end-user defines the parameters for the synthetic data to be generated.

Fischer et al. [START_REF] Fischer | Generation of learning samples for historical handwriting recognition using image degradation[END_REF] propose a method to generate training samples for historical handwriting recognition. Three degradation models are applied on binary images: Kanungo [START_REF] Kanungo | Global and local document degradation models[END_REF], character degradation from [START_REF] Kieu | A character degradation model for grayscale ancient document images[END_REF] and geometric distortion from the evaluation of [START_REF] Liang | Geometric rectification of camera-captured document images[END_REF]. The results show that the best error reduction is obtained by combining the three degradation models. The error reduction of character recognition is reaching 16.53% on Saint Gall and 20.05% on Parzival dataset.

In Kieu et al. [START_REF] Kieu | Semi-synthetic document image generation using texture mapping on scanned 3d document shapes[END_REF], 3D meshes are first extracted from real documents and then a set of 3D shapes is set up. These 3D shapes are used on 2D documents to generate 2D deformed document images. One disadvantage of this method is the acquisition of the 3D shapes set which requires a 3D scanner.

Seuret et al. [START_REF] Seuret | Gradient-domain degradations for improving historical documents images layout analysis[END_REF] propose to integrate real degradation patches in the original images. The method focuses on stains integration which can be summarized in two steps: First, noise patches containing stains are manually extracted from historical document images. Then, stains are pasted onto images in the gradient domain.

We were inspired by the works of [START_REF] Kieu | A character degradation model for grayscale ancient document images[END_REF] and [START_REF] Seuret | Gradient-domain degradations for improving historical documents images layout analysis[END_REF] by combining different image processing techniques such as object detection, inpainting and image edition in the gradient domain.

Recently, Capobianco et al. [START_REF] Capobianco | Docemul: a toolkit to generate structured historical documents[END_REF] aim to generate synthetic documents similar to existing ones. They employ a binarization technique to extract the text lines. Then, the background image is obtained by replacing text line pixels by the mean value of a window of size WxW around each pixel. The image structure is then defined by an XML file which allows to create variable structured images. To reconstruct the final synthetic images, they employ cursive fonts and dictionnaries. However, in some cases, cursive fonts may not be available and the image structure can be hard to capture when there are a lot of variations in the dataset. Similarly, Journet et al. [START_REF] Journet | Doccreator: A new software for creating synthetic ground-truthed document images[END_REF] generate synthetic documents by extracting layout and characters using the Tesseract OCR. Then, the image is inpainted to recover the background image. The synthetic images are built according to the extracted characters, backgrounds and layouts. While this method allows precise reconstructions and combinations, when dealing with historical documents, it can be hard to segment the characters due to the cursive nature of handwriting. Both methods require the intervention of an user or a ground truth while our fully automatic method does not.

Our contribution in this work is the generation of synthetic historical documents similar to existing documents, without any user intervention, and no ground truth required to generate the images.

III. PROPOSED APPROACH

As said before, the data augmentation in our case, is performed in three steps: 1) TOI Extraction, 2) BOI Extraction and 3) TOIs and BOIs Mixture. We will describe them in detail in the rest of the paper.

A. TOI Extraction

In the case of historical documents, it is desired to identify the foreground pixels of the image to isolate the text. The TOI is represented as a binary mask by binarizing the original image. While the binarization is not perfect, it can be considered as an approximation of the TOI. In this case, the quality of the TOI will rely on the binarization quality.

A crucial objective is to obtain a high recall of foreground pixels without getting too much of background pixels. In fact, a high recall is more important than a high precision because remaining foreground pixels will be present in the final image. These foreground pixels will be used during the reconstruction of the background as background pixels, which will introduce errors. For this reason, most of the time, a morphological dilation with a small structuring element is applied to the binary mask to ensure that the all foreground pixels are taken into account.

We visually compared the binarization quality with a binary mask obtained by filling the text regions represented as a polygon. The Figure 1 shows three images: (a) is the original image, (b) and (c) are respectively the binary mask obtained from the ground truth and from the binarization.

The binarization is performed using an adaptive thresholding by a sliding window of size s. The threshold is performed as the weighted mean of neighborhood values where the weights derived from a Gaussian window. A quantity c (shifting constant) is subtracted from the computed weighted mean. The parameters used for binarizing all images are s = 31 and c = 21, were empirically chosen. We can observe that the TOI obtained from a binarization is including more background pixels than the TOI obtained from the ground truth. In addition, not all foreground pixels are obtained by the binarization. This behavior was expected since the ground truth is much more precise than the binarization which considers the image as a whole while the ground truth focuses on specific areas. Once the TOI is computed, it is used to obtain the BOI as described in the next subsection.

B. BOI Extraction

The extraction of background images is done by using an inpainting method designed to remove objects or recover missing or degraded portions of an image.

Inpainting techniques can be classified into two categories. The first one is exemplar based where for a given region to inpaint, one looks for similar regions in available parts of the image.

The second one is diffusion based where the authors use pixels surrounding the region to inpaint to propagate the information successively.

Inpainting methods receive as inputs the image and a binary mask of the regions, to inpaint. We use the inpainting method of Telea [START_REF] Telea | An image inpainting technique based on the fast marching method[END_REF] which is implemented in the computer vision library Opencv [START_REF] Bradski | The opencv library[END_REF]. This method is diffusion based and considers the boundaries of regions to inpaint. For each point p on the boundaries, a small area B (p) of size composed of q points is defined. For small , the first order approximation I q (p) of point p, relatively to each q point, is defined by the Equation 1.

I q (p) = I(q) + I(q)(p -q) (1)
where I(q) is the image and I(q) is the gradient value of point q. Each point p is inpainted using a weighted average of the first order approximation for each q ∈ B (Equation 2).

I(p) = q∈B (p)
w(p, q)I q (p)

q∈B (p) w(p, q) (2)
The weighted function w(p, q) is defined as a product of three characteristics as shown in the Equation 3.

w(p, q) = dir(p, q) × dst(p, q) × lev(p, q) (3)
dir(p, q) is directional which increases the contribution of pixels close to the normal direction. dst(p, q) is the geometric distance which decreases the impact of pixels q farther from p. lev(p, q) is the level set distance which guarantees that pixel close to the boundary contributes more.

Inpainting points must be done in increasing distance order to the boundaries to mimic the way manual inpainting is done. Therefore, the fast marching method, which ensures that the point inpainted is the closest one to the known image, is employed. To inpaint the whole image, the boundary must advance one step toward its center once all its points have been inpainted. This step is repeated until there are no more boundaries to inpaint.

In the context of image documents, we seek to remove all foreground pixels and retrieve missing background pixels.

Here, the image to inpaint is the original image and the binary mask is the one computed for the TOI. The Figure 2 shows BOIs using the ground truth and the binarization. BOI obtained from the ground truth TOI contains less noise because of the precision given by the ground truth. Regarding the BOI issued from the binarization, it contains remaining text pixels which are introducing noise in the BOI. The quality of the BOI relies on two separate aspects: TOI quality and inpainting performances. The TOI quality have a huge impact since it may miss foreground pixels or include too much background pixels. For the latter, the amount of information on the background may not be sufficient to fill properly the area of the TOI. The inpainting technique used is highly responsible for the visual quality of the BOI. The method decides how to fill the regions from the TOI in the original image.

C. TOIs and BOIs Mixture

A generated image, which is semi-synthetic, is obtained by using the mixed seamless cloning as described by Perez et al. [START_REF] Pérez | Poisson image editing[END_REF]. The authors propose several tools for editing images by using generic interpolation based on solving Poisson equations. In short, they offer an efficient solution to obtain an image from its gradient field by guiding the interpolation of the area to be changed. The mixed seamless cloning operates in the gradient domain.

Let I a and I b be two images where I a is the source image and I b the destination image with their gradient field respectively a b . Let Ω be the area to insert the image and v the final gradient field. The gradients are mixed by following the equation 4.

∀x ∈ Ω, v(x) = b , if | b | > | a | a , otherwise (4)
When combining two images, the algorithm needs to be provided with the position of one relatively to the other one. The strategy used to determine this position is to use Minimum Bounding Rectangle (MBR) of the TOIs from both images. This means that we consider the extracted foreground pixels as foreground pixels, therefore the MBR represents the text area. To insert a given TOI in a BOI, we first compute the transformation required to transform the M BR T OI into M BR BOI and apply it to the TOI, i.e. we fit the new text in the area of the original text. Then, the TOI is inserted in the BOI using the mixed seamless cloning. This translation and scaling is necessary if we want to insert the text objects in the areas provided for this purpose in the layout. Ground truth annotations for the synthetic documents can be obtained by applying the same transformation on each region. In Figure 3b and 3c are two images generated with the foreground pixels of Figure 3a. One can observe that background images are different (backgrounds in Figure 3a and Figure 3b are cropped) and therefore the foreground pixels that will be added to them must be carefully placed to fit the original text location. It is easy to obtain the ground truth for the generated data and it does not rely on the annotations structures. All modified objects require that the same transformation, as for inserting the TOI in the BOI, be applied to them.

Given n training images, we can augment the set by mixing every BOI with every TOI which gives at maximum n 2 augmented images. Among those n 2 images, there are n images which are the reconstruction of the original training images. These reconstructions introduce a small amount of noise and will give approximately the same results no matter what the quality of the foreground and the background separation is. This is due to the fact that the foreground pixels will be pasted in the image at the same position they were in the original one. Because we extract information and insert them back, the result can be considered as equivalent to the original. Augmenting the data with this protocol leads to a large amount of images even when the dataset is small. Another strategy is to use backgrounds from image that are not annotated to increase the background variability in augmented images. The resulting synthetic images have the following properties:

• Since TOIs are rescaled before being integrated into a BOI, the generated images offer scale variations of the text.

• Similarly, since the M BR T OI is transformed to match the M BR BOI , the images offer translation variation. • By using different background images, the same text is represented in several different contexts thus highlighting it.

These properties implied by the generation method allow to introduce variabilities in synthetic images using existing images with limited variability.

IV. EXPERIMENTS

A. Dataset

To conduct the experiments, we used the images from the HisDB dataset [START_REF] Simistira | Diva-hisdb: A precisely annotated large dataset of challenging medieval manuscripts[END_REF] subset "cb55". It consists of 20 training images, 10 validation images, 10 public test images and 10 private test images. The reported evaluations are performed on both the public and private test sets. The task selected for the experiments is part of the ICDAR 2017 competition on Layout Analysis for Challenging Medieval Manuscripts [START_REF] Simistira | Icdar2017 competition on layout analysis for challenging medieval manuscripts[END_REF] (Task II) and aims to retrieve the baselines from the main text lines.

The system used to handle this task is divided in two parts:

• First, a neural network is employed to perform the semantic segmentation of the images using a fully convolutionnal neural network [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF]. Semantic segmentation aims to produce class probabilities for each pixel.

In our case, the two classes are either background pixel or baseline pixel.

• Second, a simple post processing step extracts the baselines points from the predicted images.

C. Metrics

The metric used for the semantic segmentation is the F1 measure of pixels per class. Regarding the baseline evaluation, following the protocol of [START_REF] Simistira | Icdar2017 competition on layout analysis for challenging medieval manuscripts[END_REF], we employed the evaluation toolkit as described in [START_REF] Diem | cbad: Icdar2017 competition on baseline detection[END_REF].

D. Neural network description

The neural network used is a U shaped fully convolutional neural network of depth 3, similar to [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF]. A U shaped neural network is composed of an encoder and a symmetrical decoder. It owns skip connections which allow to use the encoded feature maps during the decoding steps. The first convolutionnal layer has 64 filters and this number is multiplied by two each time we go down of one level and divided by two when going up. To avoid memory issues with varying input sizes, we control the images shape by rescaling them arbitrary to the fixed size (720, 560). It receives as input, the normalized image and produces for each pixel p i,j , the probability to belong to one of the two classes.

The training has been performed with the cross-entropy loss function and the Adam optimizer with a fixed learning rate of 1e-5. For all the experiments, the neural network has been trained for 200 epochs. Since classes are very unbalanced, we weighted the loss function according to the frequency of each class.

E. Baseline extraction

Prediction masks given by the previous step can result in over-segmented fragment of lines because of the thin nature of the baselines. The post processing step is performed by using a closing morphological operation to connect components horizontally in order to correct the over-segmentations. Then, a thinning operation is applied on the image to obtain a skeleton. Horizontal run-length are identified and there extreme points extracted. For each connected component, extracted points are used with the least minimum square method, to produce the final baseline.

F. Results

The results of the semantic segmentation can be observed in Table I. When comparing k 0 and k 1 , it is clear that the augmentation method substantially improved the performances of the neural network. For the public test set, the background F1 measure increased of 0.7% and the baseline F1 measure of 17,33%. For the later, this is a relative improvement of 36.95%. Results on the private test are similar to the public one with an improvement of the performance of 14.42% (relative improvement 29.95%) for the baseline class. Now, considering the k 1 and k 3 sets, no further improvement has been noticed. This can be due to the fact that synthetic documents does not introduce more variability or that we reached the neural network capacities to predict baseline pixels.

The results of the baseline extraction algorithm are presented in Table II. The row labeled as Ground truth is here to reflect the quality of the baseline extraction technique. There is only one image where it fails partially due to the smoothing of the extracted baselines which makes them too far from the ground truth ones thus considering them as errors. This shows the theoretical maximum performances when having Qualitative results of generated images are shown in Figure 4 V. CONCLUSION In this paper, we have introduced a new generic method for generating automatically semi-synthetic data from existing images. The foreground pixels are identified and separated from the original image. Then, the BOI is retrieved using an inpainting method and the extracted foreground pixels. Finally, the semi-synthetic image is obtained by integrating TOI from one source image in the BOI of another one. This technique is simple and can easily be implemented. Since generated images contain most of existing background and foreground pixels, they look realistic regarding the other documents of the dataset and can improve the performance of a semantic segmentation task. We plan to extend the data augmentation method by studying the effect of TOI degradation before applying them to the BOI, to see how a system could be further improved. Finally, this automatic technique needs further refinement to be able to use curriculum learning by creating documents with a gradually increased difficulty.

Figure 1 :

 1 Figure 1: Example of binary masks obtained with the two methods. (a) The original image. (b) The mask obtained from the ground truth. (c) The mask obtained from the binarization.

Figure 2 :

 2 Figure 2: Examples of BOIs. (a) The original image. (b) The BOI obtained from ground truth. (c) The BOI obtained from binarization.

Figure 3 :

 3 Figure 3: Example of generated images. (a) The original image. (b-c) Generated from other BOIs and the TOI of (a).

B

  . Generation protocol Let T images be the set of training images and |T images | their number. BOI from the training set are extracted to produce |T images | background images. This has been performed automatically with the use of the binarization technique. Let T synthetic images be a set of size |T images | of synthetic images generated using random TOI from T images and all background images. We produced 3 training sets to perform the experiments: T k exp = T images + k * T synthetic images with k ∈ [0, 1, 3] the number of synthetic sets. Ground truth images are generated by drawing baselines with a thickness of 7 pixels.

Figure 4 :

 4 Figure 4: Example of generated images. (a) The original image. (b) The ground truth image. (c-e) Baseline predictions for respectively k=0, k=1 and k=3.

Table I :

 I Results of the semantic segmentation task for each experimental set. the perfect semantic segmentation masks. We provided to the table the results of the competition as given in[START_REF] Simistira | Icdar2017 competition on layout analysis for challenging medieval manuscripts[END_REF] for comparison. We can clearly see that the data augmentation method is enabling the tested method to reach performances close to the state of the art. This demonstrates that our fully automatic synthetic data augmentation technique is sufficient to improve a system. The gain of performance is approximately equivalent to the semantic segmentation performance. The use of the data augmentation allowed to increase the performance from 81.75% to 96.59% for the public set and from 87.95% to 98.96% for the private set.

		Public			Private
	F1 -background F1 -baseline F1 -background F1 -baseline
	k=0	98.80	46.9		98.76		48.14
	k=1	99.50	64.23		99.43		62.56
	k=3	99.45	64.26		99.42		62.68
		Public			Private
		Recall Precision F1	Recall Precision F1
	Ground truth 98.79	99.33	99.06	100	100	100
	k=0	97.91	70.17	81.75	99.46	78.83	87.95
	k=1	98.25	93.11	95.61	98.7	96.24	97.45
	k=3	98.36	94.89	96.59	96.54	93.64	95.07
	System-8	-	-	-	-	-	98.96
	System-2	-	-	-	-	-	95.97
	System-7	-	-	-	-	-	95.34

Table II :

 II Results of the baseline extraction tasks on cb55.


T Lupinski
email: ts.lupinski@gmail.com

A Belaïd

A Kacem Echi

On the Use of Attention Mechanism in a Seq2Seq based Approach for Off-line Handwritten Digit String Recognition

Keywords: end-to-end trainable system, sequence-tosequence, attention mechanism, handwritten digit string, transfer learning

In this work, we investigate the use of the attention mechanism in deep learning for a better reading of handwritten digit strings in digitized images. The proposed recognition system built upon a CNN (Convolutional Neural Network) and two RNNs (Recurrent Neural Networks), acting as Encoder and Decoder and using the attention mechanism. We used a 1D mechanism for attention location with a "soft" alignment attention which has the peculiarity of having an easily calculable gradient and thus to integrate well with the network. Experimental results on data from ORAND-CAR A, ORAND-CAR B and CVL HDS databases compare favorably to other published methods.

I. INTRODUCTION

The recognition of handwritten digit string is an important subject of research in the field of document analysis and recognition. It is motivated by the wide variety of potential applications, such as ZIP codes, bank checks, etc. The challenge is to recognize numeral strings of unknown length which are not neatly written. Although the recognition of isolated digits is now very successful, the recognition of digit strings remains an ongoing challenge in the field of handwriting recognition. In fact, the recognition of digits is considered easier as there are less possible symbols, but no language model can help in this task. The previously proposed recognition systems, achieving the best results, usually rely on a heavy set of heuristics and over-segmentation and show their limitations when faced with complex cases such as overlapping digits. Most of systems built to recognize unconstrained numerical strings suffer from the segmentation module which reads a string of digits and segments them into isolated characters. The problem is that we do not usually know the number of digits in the string and so the optimal boundary between them is unknown. Such a problem has been dealt with in different ways [START_REF] Graves | Offline handwriting recognition with multidimensional recurrent neural networks[END_REF], [START_REF] Matan | Multidigit recognition using a space displacement neural network[END_REF], [START_REF] Saabni | Recognizing handwritten single digits and digit strings using deep architecture of neural networks[END_REF] and one way to approach it is to see it as a sequence-to-sequence problem with a sequence of image patches as input and a sequence of characters as output [START_REF] Sueiras | Offline continous handwriting recognition using sequence to sequence neural networks[END_REF]. To solve sequenceto-sequence problems with different length from input to output, the key point is to find the alignment between them.

Notice that for digit string recognition, the alignment is not straightforward. The problem of the alignment is generally handled either by posterior or prior probabilities. Based on posterior probabilities, we recourse to stochastic models like HMM (Hidden Markov Model) to extract the alignment from the probabilities outputed for each sub-part by looking for the most probable sequence of labels to fit the output [START_REF] Vogel | HMM-based word alignement in statistical translation[END_REF]. In the same way and more recently, the CTC (Connectionist Temporal Classification) [START_REF] Graves | Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Network[END_REF] has showed successes in many sequence labeling tasks with the strong ability of dealing with the problems where the alignment between the inputs and the target labels is unknown. In the prior probabilities based alignment, the attention mechanism is most often used in sequence-to-sequence models to learn many-to-one alignment [START_REF] Bahdanau | Neural Machine Translation by jointly learning to align and translate[END_REF]. Moreover, it had led to better results than HMM and CTC, for text translation [START_REF] Bahdanau | Neural Machine Translation by jointly learning to align and translate[END_REF], speech recognition [START_REF] Chorowski | Attention-Based Models for Speech Recognition[END_REF] and word recognition [START_REF] Sueiras | Offline continous handwriting recognition using sequence to sequence neural networks[END_REF]. Combining the attention mechanism with HMM or CTC might also lead to improvements [START_REF] Kim | Joint CTCattention based end-to-end speech recognition using multi-task learning[END_REF]. For handwritten digit string recognition, the best results are achieved, using the CTC [START_REF] Zhan | Handwritten digit string recognition by combination of residual network and RNN-CTC[END_REF]. In this work, the objective is to explore the use of the attention-based approach for the specific task of handwritten digit string recognition. To support our opinion, we opted for an Encoder-Decoder paradigm, as in [START_REF] Sutskever | Sequence to Sequence Learning with Neural Networks[END_REF] and proposed a recognition system built upon a CNN (Convolutional Neural Network) and two RNNs (Recurrent Neural Networks) which are powerful neural network architectures used for modeling sequences. Here, the two RNNs act as Encoder and Decoder and use the attention mechanism which enables the system to focus on relevant parts of the input more than the irrelevant parts and consequently to enhance its recognition rate and speed up the processing of large volumes of data. A description of the proposed approach, along with the preliminary experimental results is presented here.

II. PROPOSED APPROACH

The proposed approach is inspired from the sequence-tosequence approach of Sutskever and al. [START_REF] Sutskever | Sequence to Sequence Learning with Neural Networks[END_REF] and motivated by its successful application on handwritten word recognition by Sueiras and al. [START_REF] Sueiras | Offline continous handwriting recognition using sequence to sequence neural networks[END_REF]. As it will be explained later, our system is able to transform a variable-length sequence of pixel columns, extracted from the handwritten digit string image, into a variable-length sequence of digits to form a numerical string. As the typical sequence-sequence architecture consists of an encoder and a decoder RNN, our system is built with on three main components which are a CNN and two RNNs used as Encoder and Decoder respectively. A potential issue with this Encoder and Decoder approach is that the neural network needs to be able to compress all the necessary information of an input sequence into a fixed-length vector. This is very hard to do in practice, and this problem gets worse the longer the input sequence is. Such problem can be solved using the attention mechanism which allows the Decoder to attend to different parts of the source sentence at each step of the output generation. Thus, each Decoder output depends not just on the last decoder state, but on a weighted combination of all the input states. In the next subsections, the focus is on the use of the attention mechanism in our system: the best location, the used alignment, the energy and the context vector computing.

A. Attention Mechanism Location

Two possible locations of attention mechanism in our system:

• Before the RNN Encoder, the attention in made in 2D (see Figure 1 We opted for this sequential attention which is the simplest.

B. Attention Mechanism Alignment

The use of the attention mechanism vary according to the context and there is no predefined model, but one can classify the mechanisms in several categories:

• "Hard" attention: The mechanism uses a predefined function, which can be driven, and which outputs a sub-part of the entries on which to work. • "Soft" attention: The mechanism will calculate probabilities so that an area is interesting and weighted entries according to their interest, the output will be of the same size as the input but with its values weighted to bring out one or more areas.

• Feature-based: The attention mechanism will choose where to focus its attention based on what it looks at by detecting interesting areas.

• Location-based: The attention mechanism will choose where to pay attention depending on the position where it was looking before. Feature-based and Locationbased are not exclusive and a mechanism can well use both. For alignment, we had to learn it, since no rule helps us to give the alignment given images diversity. Here, we used a "soft" attention which has the peculiarity of having an easily calculable gradient and thus to integrate well with the network and to be trained at the same time as the latter contrary to the "hard" attention.

C. Attention Mechanism Energy Computing

To choose which part of the inputs to focus on, the mechanism needs to perform a score for each input, which we call energy. The most common ways to compute such energy are:

• Multiplication between each entry and the context vector:

e i = h i × c t-1 • Weighted multiplication: e i = W × h i × c t-1 • Weighted addition: e i = W × h i + U × c t-1
A neural network of Perceptron type is often used to compute this energy by learning weights. For energy computing, we chose the additive form with tanh: a non linear activation function. Once computed, the energy is normalized with a Softmax function to have weights between 0 and 1.

D. Context Vector Computing

The context vector is the output of the attention mechanism. It is computed by weighting each entry by its associated energy and then summing these vectors. The context vector will be supplied to the Decoder for each time step. The number of time steps and the size of the output vector sequence are predefined. It is up to the Decoder to choose when to stop.

E. Retained Architecture

The global architecture is inspired from [START_REF] He | Deep residual learning for image recognition[END_REF] and summarized in Table I. The proposed system consists of three stages as follows: Figure 2 shows an overview of the proposed system. At the output of the CNN, the feature maps of size (V × I × S) are subdivided into sequences (V × M where M = I × S).

Then, the first LSTM Encoder produces a sequence (V ×K). These sequences are considered by the attention mechanism which selects L sequences of size K. Theses sequences are then provided to the second decoder LSTM. At the end, a sequence of labels (L × A) is produced. Hereafter more details about the architecture components are given.

1) Image input: . We normalized input images in height, in width and in value for channels. As previously mentioned, the images must have the same height, limited here to 32 pixels as a compromise between efficiency and computation costs. We then completed the images with 0 value to make them 256 pixels wide, allowing the use of any batch size what speed up the training. We finally resorted to histogram stretching so that each image has pixel values between 0 and 255, invert them to have "white on black" instead of "black on white" images and adjusted them so that the mean is equal to zero and the standard deviation equal to one. All this is done for convenience purposes.

2) CNN for Feature Extraction: Convolution network is a powerful tool for extracting features. We used a classical sequence of convolutional layers followed by max pooling layers, doubling the number of filters of convolutional layers up to 512 after each max pooling layer. The feature maps obtained after all these layers are flattened into one features map, then divided into a sequence of column features vectors, ordered from "left to right". As in [START_REF] Zhan | Handwritten digit string recognition by combination of residual network and RNN-CTC[END_REF], we used a 2 × 2 pooling size for the first max pooling layer. For the next max pooling layers, we used a 1 × 2 pooling size to down sample just in height dimension, not in width. By this way, we kept the sequence size long enough to have more precision.

3) Encoder-Decoder: The Encoder takes the sequence of features vectors and transforms it into a sequence of encoded vectors. Then, the Decoder sequentially uses an attention mechanism to choose a part of this encoded sequence and decodes one character out of it, until the "End Of String" (EOS) token is outputted by the Decoder. The key difference between the Encoder and the Decoder is the order of the execution: as the attention mechanism needs the previous cell states of the last layer of the Decoder, the computation can only be done step by step instead of layer by layer. This model can handle images of any width as both the CNN and RNN do not have parameters based on the length of the sequence, but the height must be fixed as the RNN needs a fixed-length vector as input for each time step.

4) Attention Mechanism: For the attention mechanism, we proposed recurrent attention to both characteristics and position. At each step of attention, the mechanism receives the output sequence of the Encoder, with the attention vector at the preceding time step, and at the output of the Decoder at the previous time step as shows Figure 3, and outputs a context vector using the following equations. At each time step T , and at each of the positions L in the sequence, the energy is calculated by EQ. ( 1):

e T,L = w ×tanh(W ×C D T -1 +V ×E L +U ×A t-1 +b) (1)
where w, b, W, V are trainable network parameters, C D T -1 is the output of the Decoder, E L is the element in the sequence outputted by the Encoder at position L, A t-1 is the attention vector at time t -1, and b is a bias. A Softmax is calculated (

) 2
The context vector of this time step T is computed by summing as in EQ. (3):

C T = i (A T,L × E L ) (3)
These equations are illustrated in Fig. 4. The diagram shows at the top how the attention mechanism takes into account the encoded sequence and the state of the encoder at time t -1. This state is used to obtain a value for each element of the sequence that will be added to the values of the sequence. This diagram reproduces the attention mechanism of [START_REF] Sueiras | Offline continous handwriting recognition using sequence to sequence neural networks[END_REF]. Adjusting the Perceptron, responsible for the weight of the mechanism, we achieved very interesting results, as displayed in Figure 5 where we see the attention which follows the figures.

III. USED DATASETS

To evaluate the proposed system, we choose the ORAND-CAR A and B and the CVL HDS dataset used in ICFHR 2014 [START_REF] Diem | ICFHR 2014 competition on handwritten digit string recognition in challenging datasets[END_REF]. Classical methods with explicit segmentation achieved good results on this dataset but all sequence-to-sequence based approaches perform poorly, and one of the reason of this could be the lack of variety in the sequence which prevent the systems to generalize enough. To evaluate our system performance on this dataset we had to:

• Separate the test dataset in two sets: the numbers present in the training set and the rest. • Shuffle the training and the test set with the same number of images as before, but with all 26 numbers present in the training set. MNIST: It is the standard for handwritten single digit recognition [START_REF] Lecun | Gradientbased learning applied to document recognition[END_REF]. For simplicity, we adapted it to our problem by concatenating the digits to create digit strings. As MNIST has no information about the identity of each writer, we could not guarantee an homogeneity in writing style inside the string as it could be observed in the ORAND- Height and width re-scaling are the same for each digit in an image but spacing and rotation can vary inside the same image. The regular background is a randomly chosen pattern repeated over all the image. The aleatory background is a Gaussian noise applied to the image before placing the number over it. None is just the same background as MNIST digit: all White or all Black if inverted. Thus, we generated two datasets: MNIST-M1 and MNIST-M2. The difference between M1 and M2 is the number of allowed variations, the generator for M1 is only allowed to vary the number of digits composing each string, with static positioning and no re-scaling or rotation, but the generator for M2 can use all available variations. The training sets are composed of 30000 images generated using only digit image for the MNIST training set, and test sets are composed of 5000 images generated using only digit images for the MNIST testing set.

IV. EXPERIMENTS AND RESULTS

We implemented the attention mechanism as a recurrent cell as it acts in a sequential way and uses the attention vector for the previous time step to compute the attention vector for the current one. As the attention also needs the state of the Decoder which uses the context vector produced by attention, we created one unique recurrent cell combining the attention mechanism and the LSTM cell used as Decoder. We used a computer having a CPU Intel Xeon E5-2620 v4, without GPU and Keras 2.2 with Tensorflow 1.8 back end. Categorical cross entropy was used as loss function during training.

A. Performance Evaluation

We used a hard and soft metric to evaluate the performance of our system. The hard metric being the Word Recognition Rate (WRR), validating only if all labels in the strings are truly recognized. The soft metric is the Character Recognition Rate (CRR) which computes the Edition distance (Levenhstein distance) between the string recognized and the labels to get the Character Error Rate (CER) with CRR = 1 -CER. It seems that the soft metric is more reliable, to compare the results, than the hard one which is not linear. In fact, an improvement of 5% in the soft metric can result in a improvement of 0 to 20% in the hard metrics depending on where the initial precision stand. But, when we compared with other systems, we only used the hard metric for conformity reasons.

B. Network Variations

To find the best architecture for our system, we tried various settings and retained those between parentheses:

• Number of layers inside the CNN: 3, 4, ( 5 II where retained parameters are between parentheses. We made these variations in the same order as they are presented upside, by testing with all the different values, then fixing these parameters with the best values obtained before switching to the next parameters.

C. Transfer Learning

During training, the attention mechanism revealed to be hard to train from scratch and on harder database like ORAND-CAR A or B. The convergence was reached in a sub-optimal way. The attention weights were almost random and the results were not satisfactory. To overcome this problem, we resorted to the transfer learning to not start from scratch and provide the network with a good starting point. Transfer learning is simply initializing the network with weights obtained by training the network on another dataset. We used the generated datasets MNIST-M1 and MNIST-M2 and achieved better results and a good working attention as intended (see Figure 9). As shown in Table III, the proposed system reaches the highest performances on the public datasets, compared to those of systems in ICFHR 2014 competition (indicated by stars) with a recognition rate 84.01%. Notice that the state of the art mentions two other works using RCNN [START_REF] Shi | An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition[END_REF] and ResCNN [START_REF] Zhan | Handwritten digit string recognition by combination of residual network and RNN-CTC[END_REF] that exceed this rate but we had no way to verify it.

V. CONCLUSION

In this work, we postulate that handwritten digit string recognition can be successfully done without any segmentation. For that, we proposed a deep network architecture consisting of one CNN and two RNNs, one acting as an Encoder and the other as a Decoder, by resorting to the attention mechanism. Experiments carried on several datasets, whether on numerical chains created artificially from MNIST or directly on datasets proposed in competitions, demonstrate the contribution of the attention mechanism in improving the recognition performance. The different choices of alignment, calculation of energy and contextualization have been beneficial. We show that the proposed system achieve competitive results, compared to those of ICFHR competition. Additional work is necessary to go beyond the scores displayed in the state of the art that we were able to verify in this work. Future work concerns the improvement in the use of attention mechanism and its training to better the alignment. A more dedicated architecture to the digits chains of digits must also be reflected.

Figure 1 .

 1 Figure 1. Attention mechanism location: (a) Before the RNN Encoder, (b) After the RNN Encoder.

  (a)): This allows to play on the verticality of the image, by visualizing a 2D window of attention. It chooses the zones of the image to look at, move the attention to the right by following the digits one by one. • After the RNN Encoder, the attention in made in 1D (see Figure 1(b)): It selects the parts of the sequence to keep for digit recognition. It singles a portion of the encoded sequence as a potential digit for the Decoder.

Figure 2 .

 2 Figure 2. System overview.

Figure 3 .

 3 Figure 3. Global computation of the attention mechanism: E i is the i th element in the sequence outputted by the Encoder, D j is the j th output of the Decoder.

Figure 4 .

 4 Figure 4. Local computation of the attention mechanism

Figure 5 .

 5 Figure 5. Attention vector.

Figure 6 .

 6 Figure 6. Samples from the ORAND CAR databases A and B.

Figure 7 .

 7 Figure 7. Samples of the CVL HDS dataset.

Figure 8 .•

 8 Figure 8. Samples from the MNIST Multi-digits database 1 and 2.

), 6 , 7 •

 67 Reduction of the sequence length: (2), 4, 8, 16 • Number of layers of the Encoder: 1,(2), 3 • Number of layers of the Decoder : (1), 2 • Normalization Function of Attention : (Softmax), Sharpmax, Smoothmax. During experiments, we used the ORAND-CAR A Dataset with 2000 images per epoch and a batch size of 100. The obtained results are displayed in Table

Figure 9 .

 9 Figure 9. Visualisation of the attention mechanism at each step on one sample from MNIST-Multi1, and one sample from MNIST-Multi2.

Table I NETWORK

 I SETTINGS SUMMARY, FROM TOP TO BOTTOM.

	Type	Configuration
	Input	Input image, 256x32
	Convolution	#maps: 64, k3x3, s1x1, p1x1, act:Relu
	MaxPooling	k2x2,s1x1
	Convolution	#maps: 128, k3x3, s1x1, p1x1, act:Relu
	BatchNormalisation	-
	MaxPooling	k1x2,s1x1
	Convolution	#maps: 256, k3x3, s1x1, p1x1, act:Relu
	BatchNormalisation	-
	MaxPooling	k1x2,s1x1
	Convolution	#maps: 512, k3x3, s1x1, p1x1, act:Relu
	BatchNormalisation	-
	MaxPooling	k1x2,s1x1
	Convolution	#maps: 512, k3x3, s1x1, p1x1, act:Relu
	BatchNormalisation	-
	Flatten	Axis 2 & 3
	LSTM	#hidden units:256
	LSTM	#hidden units:256
	Attention Cell
	LSTM	#hidden units:256
	InnerProduct	#units:11
	Softmax	-

Table II CHARACTER

 II PRECISION RATE OBTAINED FOR EACH CONFIGURATION.

	CNN layers	Character Precision	Execution time for
		Rate	one Epoch
	3	91.53	70s
	4	92.13	90s
	(5)	92.54	-
	6	92.58	-
	7	91.89	-
	MaxPooling	-	-
	width Reduction
	(2)	95.02	-
	4	92.54	-
	8	92.88	-
	16	85.64	-
	Encoder Layers	-	-
	1	91.25	-
	(2)	95.02	-
	3	94.29	-
	Decoder Layers	-	-
	(1)	95.02	-
	2	94.87	-

Table III RESULTS

 III ON ORAND-CAR A, ORAND-CAR B AND CVL HDS.

	Competitor	ORAND-CAR A ORAND-CAR B CVL HDS
	Tebessa I*	0.3705	0.2662	0.5930
	Tebessa II*	0.3972	0.2772	0.6123
	Singapore*	0.5230	0.5930	0.5040
	Pernambuco*	0.7830	0.7543	0.5860
	BeiJing*	0.8073	0.7013	0.8529
	Proposed	0.8401	0.82.97	0.2245

• Features Extraction: the CNN gets out a feature map, from the input image, and divides it into a sequence of vectors columns. The max pooling is repeated here to


Romain Karpinski
email: romain.karpinski@yahoo.fr

Abdel Belaïd
email: abdel.belaid@loria.fr

Semi-supervised learning through adversary networks for baseline detection

Keywords: Semi-supervised learning, Semantic segmentation, ARU-Net, Adversary networks

The aim of this paper is to propose a new strategy adapted to the semantic segmentation of document images in order to extract baselines. Inspired by the work of Gr üning [7], we used a convolutional model with residual layers enriched by an attention mechanism, called ARU-Net, a post-processing for the agglomeration of predictions and a data augmentation to enrich the database. Then, to consolidate the ARU-Net and help explicitly model dependencies between feature maps, we added a module of "Squeeze and Excitation" as proposed by Hu et al. [9]. Finally, to exploit the amount of unrated data available, we used a semi-supervised learning, based on ARU-Net, through the use of adversary networks. This approach has shown some interesting predictive qualities, compared to Gr üning's work, with easier processing and less task-specific error correction. The resulting performance improvement is a success.

I. INTRODUCTION

There are two tasks: detection and extraction of baselines. Baseline detection can be thought of as a semantic segmentation task where every pixel over which baselines pass should be classified as such. The extraction is done from the prediction of the detection and provides the coordinates of the lines. The U-net architecture, as proposed by [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF], is designed to perform semantic segmentation on biomedical image documents. This architecture consists of a contraction path (encoder) that captures the context of the image, and an expansion path (decoder) that, from the encoded feature maps, returns to the original image size.

At least two systems of the literature [START_REF] Fink | Baseline detection in historical documents using convolutional u-nets[END_REF], [START_REF] Grüning | A Two-Stage Method for Text Line Detection in Historical Documents[END_REF] use this architecture for the extraction of baselines but remain either very greedy in place memory or too consuming data or computing time due to the multiplication of layers. In this work, we propose an improvement of this architecture by the addition of a Squeeze and Excitation module (SE) and a semi-supervised automatic learning through the use of adversary networks. We used the same dataset, cBAD, as [START_REF] Fink | Baseline detection in historical documents using convolutional u-nets[END_REF], [START_REF] Grüning | A Two-Stage Method for Text Line Detection in Historical Documents[END_REF] to be able to compare our results.

The paper is organized as follows: In section II, a study of the state-of-the-art baseline detection methods is performed. In section III, we detail the different modules of the system. In Section IV, the experiments are commented. Finally, section V concludes the work and gives some research ideas for the future.

II. RELATED WORK

We focus here on the study of three methods in particular, all of which are recent and two of the three participated in the cBAD competition [START_REF] Diem | Scriptnet: Icdar 2017 competition on baseline detection in archival documents[END_REF].

The first is that of Renton et al. [START_REF] Renton | Handwritten text line segmentation using fully convolutional network[END_REF] detecting X-Heights instead of baselines. To make the labeling of the pixels, they use an architecture inspired from VGG16. This architecture has a series of convolutions followed by a max pooling, with a number of filters increasing after each max pooling, except for the penultimate max pooling. The VGG16 architecture is modified to make it fully convolutional. To do this, the authors remove the max pooling operations and keep the first six layers of convolutions. To compensate for this, they use dilated convolutions to obtain a broader context. The results obtained on the basis of cBAD for the sub-base "simple" are: 75% F-Measure, 66% Precision and 86% Reminder on baselines. This method is interesting because the size of the image does not change. However, this architecture is greedy in memory.

The second method proposed by Fink et al. [START_REF] Fink | Baseline detection in historical documents using convolutional u-nets[END_REF] consists of several steps. At first, a U-net is applied to image patches and predicts a probability for each feature on each patch. They add to U-net a fully connected layer before performing the classification. Once the characteristics of each patch have been extracted, they use a second U-net to perform baseline detection. The idea behind this strategy is to provide the context network to perform labeling and avoid edge effects. Instead of using the cross entropy as a loss function, they use a modified version of the Dice coefficient. This allows them to improve the accuracy of baseline extraction from 76.59% to 81.41%. The extraction is performed from the predictions of the second network using the least squares method. The extracted baselines then go through a post-processing step which aims at removing the erroneous lines and joining the lines being cut into pieces. A weakness of this method is that it uses several networks that make it slower compared to a method with a single network.

The third method described by Grüning et al. [START_REF] Grüning | A Two-Stage Method for Text Line Detection in Historical Documents[END_REF] is a two-step method: at first, one uses an ARU-net (a U-net having residual connections in its blocks and an attention mechanism) to detect the pixels belonging to the baselines. Then, a post-processing is operated to extract the baselines from the predictions. The ARU-net works by using the input image with n extra scales of the input image. These n different scales denoted S n , correspond to the successive sub-samples of the input image by a factor f via the average pooling operation. The input of the post-processing used corresponds to the prediction of the network performing the detection of the baselines and the separation of lines. This mechanism has a cost in terms of time because it is necessary to obtain the predictions of the RU-net for each scale. The presented algorithm includes a heavy post image processing. The prediction is binarized before selecting points of interest. One uses curvature and distance conditions to group the points. The treatment is effective but heavy. It corrects errors made by the system but is task specific.

III. PROPOSED APPROACH

The proposed method is based on a well-reinforced U-Net to which we have added different modules. It uses semi-supervised learning with adversary networks. We will describe in the following all these developments.

A. Squeeze and excitation module

The Squeeze and Excitation (SE) module, as defined by Hu et al. [START_REF] Hu | Squeeze-and-excitation networks[END_REF], is added to the residual connections to help explicitly model the dependencies between the feature maps. At first, the branch of the module SE reduces the size of the characteristic maps to a value by calculating the average of the values for each map; This operation is called Global Average Pooling. Then, two fully connected layers are applied to the maps to model their relationships. In order to obtain a weight, each output of the last layer is activated by a Sigmoid function which gives a value in the interval [0; 1]. This value is then used to weight the values of the second inactivated branch. We verify in experiments the effectiveness of this module and show that we can improve the ARU-net architecture by adding it to the residual connections.

B. Attention network

The attention for each scale is calculated with a simple FCN (Fully Convolutional Network) containing only 4 convolutional layers and a small number of filters whose last layer is the classification layer which gives a single map per scale. This unique map corresponds to the A i maps and it will be used to weight the feature maps. Although this network is too small to perform baseline detection directly; It has a sufficient number of filters and layers to detect the different attentions to be worn for each scale. This attention mechanism will specialize the attention scales to identify a type of feature such as background, text, text contours or noise, depending on the ease of detection that is a function of the scale. Indeed, it is easy to roughly detect the location of lines when the image is small, but it is difficult to accurately predict the location of the baselines. On the other hand, with a high scale, it is difficult to capture the entire context of the baselines because the pixels can be remote and out of the local scope of successive convolutions.

C. Post-processing

The method used to extract baselines from predictions consists of 6 steps:

1) The prediction of baselines by the network is binarized in order to simplify the extraction and to filter the low probabilities. 2) Next, the algorithm Mean Shift [START_REF] Comaniciu | Mean shift: A robust approach toward feature space analysis[END_REF] is used to group the points close to each other.

3) The Density-Based algorithm (DBScan from [START_REF] Ester | A densitybased algorithm for discovering clusters in large spatial databases with noise[END_REF]) is used to group the previously formed groups into larger ones that will represent the final lines. 4) The end group points are plotted on a blank image before extracting the related components. 5) The orientation of each of the related components is calculated to detect whether they are horizontal or vertical for the next step. 6) The baselines are calculated by performing a polynomial regression from the related components.

We have carried out different experiments by varying the parameters of the architectures and we will report those that have produced interesting results compared to the state of the art (see Table I). The optimizer used is RMSprop from [START_REF] Hinton | Rmsprop: Divide the gradient by a running average of its recent magnitude, Neural networks for machine learning[END_REF] and the learning rate is 10 -3 . The experiment L corresponds to the best results obtained and to the best convergence. In this experiment, instead of making a progressive learning which simultaneously increases the difficulty of each transformation, we decide to follow on these transformations once the current configuration is well acquired.

D. Semi-supervised learning

The principle, depicted in Figure 1, is to train the network first on annotated documents and then use the result of this training on non-annotated documents in order to obtain predictions. Then, from these predictions, the goal is to determine which pixels can be trusted. Once the correct areas or pixels are identified, we can use them to drive the network training. The pixel selection can be defined by a set of rules and constraints induced by the task to be performed. For example, in the case of baseline detection, one can rely on the form of predictions to filter out areas of the image that are not clearly predicted as lines. However, the difficulty in this case would be to identify the missing baselines and those that are in excess. Another way to make this selection is to learn to predict the correct areas, of ambiguous zones, that need to be set aside. In order to achieve this semi-automatic learning, we choose to use adversarial networks as defined by [START_REF] Hung | Adversarial learning for semi-supervised semantic segmentation[END_REF].

1) Adversarial neural networks: The adversarial neural networks, more specifically the networks of generative adversarial neurons, propose a new method which makes it possible to estimate a generative model by means of an adversarial training. This method uses two neural networks: a generator and a discriminator D. The generator G must capture the distribution of the truth data while the discriminator D must estimate the probability that a data originates from the truth or it is generated by the generator. D must therefore minimize the probability of making a mistake. G's goal is to maximize the probability of deceiving D into believing that the data generated comes from the truth. This training can be seen as a min-max game with two players: G and D.

Let p data be the truth distribution of the data, p g , the generator distribution and y, the data coming from either the truth or generated by the generator. In order to train the generator, we need a random noise p z (z) that allows the generator to match this noise with the truth distribution, in such a way that G(z) ∈ p data .

Let D be a discriminator that takes an input y and gives as output a value D(y) representing the probability that y comes from the generator in relation to the truth. D is used to differentiate between p data and p g .

In order to allow the generator to capture the difference determined by the discriminator, G is trained by minimizing log(1 -D(G(z)). We can then formulate the problem by determining that the training corresponds to a min-max set defined by the function V (G, D) as shown by the Equation 1. The first part of the equation (before the sign +) aims to minimize the probability of not correctly classifying a data as a given data belonging to the distribution of the truth. The second part (after the sign +) is true for the min part of the min-max problem.

min G max D V (G, D) = E y∼p data (y) [log(D(y)]+ E z∼pz(z) [1 -log(D(G(z))] (1)
Figure 2 shows a schematic example of the training of an adversarial network. More concretely, we use two loss functions L d and L adv in order to train the two networks. The first, L d , is used to oblige the discriminator to differentiate between y∼p data and G(z)∼p z . The second, L adv , is used to drive the generator through the discriminator. Gradients are calculated from the discriminator. However, the weights of the latter are not changed. Then, by back-propagation, the gradients of the discriminator are transmitted to the generator until the input. It is by this mechanism that the generator absorbs the difference between its distribution p g and that of data p data which has been captured by the discriminator. 2) Baseline detection: In the configuration of [START_REF] Hung | Adversarial learning for semi-supervised semantic segmentation[END_REF], the truth distribution p data is given by classes of baseline pixels in images. Also, random noise is no longer a noise nor random because we do not want to generate images from a given space. Actually, the space of p z is already defined in our case and it corresponds to the images. The generator must, from an image, generate the associated prediction. The discriminator always has the same role and must differentiate between a baseline prediction from the generator and a truth image from the p data distribution.

Instead of training the adversarial networks directly, the authors combine the classical learning that drives the generator by measuring the error associated with its prediction in relation to a truth, with the adversarial learning. In addition, the discriminator (an FCN) no longer gives a single value for an entry, but provides a probability of belonging to p data for each pixel of the prediction. The authors determine that this modification is essential to improve the results. The loss function of the generator is therefore defined by the Equation 2 with L ec = H(Y, G(z)), summed cross entropy, L adv , the cross entropy of the adversarial training defined by the Equation 3, and λ adv , the weight given to the adversarial training.

L g = L ec + λ adv L adv (2)
L adv = H(Y, D(G(z))) (3)
In addition, they consider that the probability given by the discriminator, can be used as a confidence measure to detect reliable areas. In this case, the correct areas are those that have the same distribution as p data . This allows us to set up a semi-supervised learning on non-annotated images. A non-annotated image z = S n is labeled by the generator G(z), then the discriminator determines the areas of the image that correspond to the truth distribution D(G(z)). The output of the discriminator is thresholded by a value T d to obtain a binary confidence mask I(D(G(z)) > T d , with I(•) the indicator function. The generator's prediction is also thresholded by T g giving I(G(z)) > T g = Ŷ to obtain the class of each pixel.

L semi = - m i=1 |C| j=1 I(D(G(z)) > T ) • Ŷ × log(G(z)) (4) L g = L ec + λ adv L adv + λ semi L semi (5)
As we saw earlier, we replace the summed cross entropy by the mean cross entropy. We will now study through experiments how the network can be trained with this new architecture and what are the performances according to several scenarios.

IV. EXPERIMENTS

We have experimentally studied how the network can be driven with this new architecture and what are the performances according to several scenarios. This is related in the following paragraphs and the results are reported in the table II.

We started by studying the impact of the number of elements in the training set on the performance of the system. We choose to train the network conventionally with λ adv = 0 and λ semi = 0 using 25% (L2) and 50% (L1) of the training data. As one might expect, performance decreases as the amount of data in the training set also decreases. However, it can be noted that even with half of the data, the results are close to the L0 experiment (100% of training data). This means that the network used has a good ability to generalize with little data.

A. Adversarial training

The next experiment was to use an adversarial training by following the Equation 2. We start by training a network with 25% of the data and using a weight of λ adv = 0.01, as defined in [START_REF] Hung | Adversarial learning for semi-supervised semantic segmentation[END_REF]. The results obtained are not at all satisfactory because the network does not converge. Indeed, the adversarial learning completely blocks all the training because the discriminator can easily make the difference between the truth data and the predictions.

Since the discriminator is good, the generator must be as good as it is to make the task of the discriminator more difficult. However, it is easier to tell the difference between the predictions and the truth than it is to label the historical images. As a result, the generator fails to follow the rate imposed by the discriminator. Even having a weight of λ adv = 0.01 which makes the adversarial loss function small compared to the cross entropy loss function. The simplest solution found by the generator to fool the discriminator is to predict no baseline. Since the predicted image is homogeneous, it is difficult for the discriminator to differentiate from images of truth. So we get a solution that is not satisfactory because the generator is blocked in this configuration and can hardly get out.

Balancing the two networks is a complex task that is very time consuming. We tried to slow down the pace of the discriminator but without success. At each attempt, the discriminator finally takes over the generator which causes a collapse of the system. To overcome this problem, there is a technique known as label diffusion that makes it possible to disseminate class values for images of truth. This technique consists in adding a noise in the probabilities to be predicted. That is, instead of having to predict 0 or 1, the network must predict 0+b or 1-b with b a fairly low value and b < 1 |C| . For example, with b = 0.2, the network must predict for a twoclass classification, the vector [0.8; 0.2] instead of having to predict [1.0; 0.0]. If we draw a different value b for each pixel, we end up with a distribution p data closer to that naturally given by a network. In this case, the task of the discriminator becomes much more complex because it can no longer be based on remarkable values.

In the case of baseline detection images, we apply a Gaussian noise on the baseline pixels, making them more like a prediction. The results obtained with this method are visible in the Table II by the L4 experiment, compared to the systematic failure of the L3 experiment. The authors of [START_REF] Hung | Adversarial learning for semi-supervised semantic segmentation[END_REF] explain that in their case, this diffusion is not necessary and that the network converges correctly. In the case of baselines, the peculiarity of the form of truth necessitates the use of this technique.

The next experiment, L5, aims to determine whether the concatenation of the source image with its prediction or truth helps the discriminator to determine the correct regions and to differentiate between bad or correct labeling. Indeed, the generator could predict a labeling that strongly resembles the truth without being fair. For example, on an image containing no line of text, predicting regular lines would be considered fair by the discriminator because it is an image that can be present in the truth. The only condition for determining the error is to have the information at the origin of the labeling. The discriminator must therefore predict P (y|z) for each pixel instead of P (y) denoted as "DX" in the tables. It can be seen that this addition improves system performance with a gain of +0.8% F-Measure.

The experiment L6 uses a weight λ adv = 0.001 instead of λ adv = 0.01 to see if the adversarial training is degrading or improving performance. We can see that the results obtained with λ adv = 0.01 are the best of the two configurations with a difference of 1.89% F-Measure. The adversarial training allows, compared to the traditional training, to improve the results of 2% at the level of the F-Measure which is a consequent improvement, considering the small number of data.

We conduct further the experiments by training the adversarial network with 100% of the data (experiment L9). The results obtained are the best, with all experiments combined with 94.66% of F-Measure (+0.91% F-Measure compared to the second). This result confirms again the interest of using this training method.

Figure 3(c) shows the prediction obtained with the L9 experiment. It can be noted that the prediction obtained with the L9 experiment is much more precise and clear than that of the L experiment. Figure 3(d) shows the result of the prediction given by [START_REF] Grüning | A Two-Stage Method for Text Line Detection in Historical Documents[END_REF]. We find the same prediction characteristics as with our implementation, that is to say, the prediction is diffuse in some locations. With the adversary training, we do not find diffuse regions as in Figure 3(b) and Figure 3(d). This limits the false detection of baselines and makes them much easier to extract and allows the use of a simple extraction method. We can therefore conclude that with the use of a more efficient system than ours combined with the adversarial training, an increase of the results of the state of the art is expected.

B. Semi-supervised adversarial training

We have conducted an experiment on the semi-supervised part of the learning phase of the system (experiment L10 in Table II). The network learning is defined by the Equation 5with T d = 0.8 and T p = 0.2. In addition, we set λ semi = 0 for the first 20 epochs. As the authors [START_REF] Hung | Adversarial learning for semi-supervised semantic segmentation[END_REF] explain, this avoids using noisy predictions of the discriminator and gives the two networks time to give correct results. The results of the experiment are not very conclusive. Indeed, the performances are degraded (-0.8 %) of F-Measure.

However, it can be noticed that the accuracy obtained is greater with an improvement of 0.98% at the expense of a recall loss of 2.49%. This can be explained by the fact that the network learns from predictions where baselines are completely missing. Increasing the accuracy indicates that the learning is reliable when the lines are detected and the discriminator has validated them. However, when the lines are missing, if the discriminator does not classify their pixels as false, the network will learn to ignore them which will lower the recall. The experiments carried out made it possible to discern several points. First, the SE module improves network performance, which confirms that the ARU-net architecture must be used in combination with this module for better results. Second, cross entropy is not always related to network performance and may not be the best loss function for this task. Indeed, the calculation of the entropy does not take into account the form of the predictions. Nothing prevents the network from predicting a scatter plot for example, while predictions must look like lines. If we could integrate the shape of the elements to predict, then, the performances would be improved.

The networks of opposing neurons make it possible to correct this point by considerably improving the form of the prediction. As the architecture is stabilized, the pursuit of work oriented towards semi-supervised learning is now possible to make use of non-annotated data. The use of a simple post-processing allows, the case of an industrial use, to reduce the time required to treat an image which can result in a saving of time and money for a company.

The method was inspired by [START_REF] Grüning | A Two-Stage Method for Text Line Detection in Historical Documents[END_REF] work but we get good results with a simpler post-processing and better predictions. The addition of the SE module has made a strong contribution to making the network even better, although this has not been explicitly demonstrated for lack of time.

We have no better results than [START_REF] Grüning | A Two-Stage Method for Text Line Detection in Historical Documents[END_REF] in the quantitative sense but more qualitative. The quality of the predictions obtained is characterized by the shape of the predicted lines which makes it possible to apply a simpler post-processing. More efforts should have been put in place to develop a postprocessing to obtain better quantitative results. The problem with the baseline extraction case is that it does not evaluate the performance of the detection of these lines.

V. CONCLUSION

A semi-supervised approach, through the use of adversary networks, has been studied and adapted to the detection of baselines in historical documents. The use of the diffusion of the labels was essential in order to stabilize the architecture. Moreover, we have seen that the concatenation of the source image and its prediction or truth given to the discriminator increases the performances. It also allows the discriminator to differentiate between good and bad labeling based on the original image. In conclusion, the consequent improvement in performance through the use of limited data is a success. On the second point, more experiments are required to obtain a stable learning mechanism on unannotated data. The work done lays the groundwork for further work and describes a method for implementing semi-supervised learning through adversarial networks. We have no better results than [START_REF] Grüning | A Two-Stage Method for Text Line Detection in Historical Documents[END_REF] in the quantitative but more qualitative sense. The quality of the predictions obtained is characterized by the shape of the predicted lines which makes it possible to apply a simpler post-processing. More efforts should have been put in place to develop a post-processing to obtain better quantitative results. The problem with the baseline extraction case is that it does not evaluate the performance of the detection of these lines.

Figure 1 .

 1 Figure 1. Functioning schema of a semi-automatic learning algorithm.

Figure 2 .

 2 Figure 2. Functioning schema of adversarial neural network.

Figure 3 .

 3 Figure 3. Comparison of the truth image of line detection with respect to predictions on the image (a)