Julien Langlois
email: j.langlois@wedo.team

Harold Mouchère

Nicolas Normand

Christian Viard-Gaudin
email: christian.viard-gaudin@univ-nantes.fr

3D Orientation Estimation of Industrial Parts from 2D Images Using Neural Networks

Keywords: Neural networks, 3D pose estimation, 2D images, deep learning, quaternions, geodesic loss, rendered data, data augmentation

In this paper we propose a pose regression method employing a convolutional neural network (CNN) fed with single 2D images to estimate the 3D orientation of a specific industrial part. The network training dataset is generated by rendering pose-views from a textured CAD model to compensate for the lack of real images and their associated position label. Using several lighting conditions and material reflectances increases the robustness of the prediction and allows to anticipate challenging industrial situations. We show that using a geodesic loss function, the network is able to estimate a rendered view pose with a 5 • accuracy while inferring from real images gives visually convincing results suitable for any pose refinement processes.

INTRODUCTION

Pose estimation of objects has recently gained lots of interest in the literature for its wide application possibilities such as robotic grasping for bin picking. Nevertheless, this task remains challenging in the case of industrial parts, usually texture-less and not appropriate for key-points and local descriptors. Moreover, in a non-controlled and unfriendly industrial environment many issues have to be tackled akin to low luminosity or cluttered scenes. To be embedded in a bin picking process, any pose estimation module requires high precision recognition while offering acceptable execution speed not to penalize the following industrial treatments. It is even more arduous when the part needs to be precisely placed on an assembly module afterwards. Estimating the position of any 3D objects in a scene has gained interest in the past years thanks to neural networks and their abilities to extract relevant features for a given task. However, this task remains challenging when handling industrial parts because of their often black and glossy plastic material: in hostile plant conditions this material is known to constrain the prediction capabilities of image-based algorithms. Many works have been conducted with a depth information allowing algorithms to learn spatial features to recognize [START_REF] Bo | Unsupervised feature learning for rgb-d based object recognition[END_REF] or estimate the positions of objects within an image [START_REF] Hodan | Detection and fine 3d pose estimation of texture-less objects in rgb-d images[END_REF] (Shot-ton et al., 2013). In this paper we show that a simple 2D information is enough to predict the rotations of a part given a certain confidence score using a CNN and quaternions. However, the training dataset creation based on real images appears to be a painful process. Thereby, we also propose a dataset generation framework using rendered views from a CAD (Computer-Aided Design) model as a way to offset the lack of real images labeled with the accurate object position [START_REF] Su | Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views[END_REF]. Using an alpha channel (transparency) offers the possibility to add backgrounds and rotations without any image cropping. Scene parameters (lightening, scale, reflectance...) must be decisive for a relevant feature learning to infer from real images thus we choose them according to the part appearances in real views [START_REF] Mitash | A selfsupervised learning system for object detection using physics simulation and multi-view pose estimation[END_REF]. When directly regressing the pose of an object, the Euclidean distance is often employed as a loss function to approximate the distance between quaternions [START_REF] Melekhov | Relative camera pose estimation using convolutional neural networks[END_REF] [START_REF] Kendall | Posenet: A convolutional network for real-time 6-dof camera relocalization[END_REF]) [START_REF] Doumanoglou | Siamese regression networks with ef-ficient mid-level feature extraction for 3d object pose estimation[END_REF]. In this paper we propose a geodesic distance-based loss function using the quaternion properties. Operations of the quaternion algebra are a combination of simple derivable operators which can be used in a gradient-based backpropagation. We achieve to get a network which offers a great pose estimation with a low uncertainty over the data. With a wide variety of scene parameters, the network is able to estimate the pose of the object in a real image with a sufficient accuracy to be fed into a pose refinement process.

RELATED WORK

Local Features Matching

Local features extraction has been widely studied in the literature as it easily gives matching pair opportunities between two views. These features were first constructed from parametric shapes isolated within the image contours [START_REF] Honda | A handling system for randomly placed casting parts using plane fitting technique[END_REF]. The industrial parts need to have specific shape singularities to be properly localized, though. More recently, the well known SIFT local descriptor was employed to get a pose description of an object [START_REF] Lowe | Distinctive image features from scaleinvariant keypoints[END_REF]. In [START_REF] Gordon | What and where: 3d object recognition with accurate pose[END_REF] a matching between the extracted SIFT with the pose description of the object is performed to get a camera pose estimation. Because of the high dimension of the features vector, the SIFT severely impacts the algorithm computation time. Later, the SURF local descriptors, faster to extract, were introduced. However, they appear to be less robust to rotation and image distorsion than SIFT [START_REF] Bay | Surf : Speeded up robust features[END_REF]. To be computed, the local descriptors often rely on the object texture, one element absent from industrial parts. Moreover, they suffer from high luminosity and contrast variations which make them improper to be used in a challenging plant environment. Using the depth channel of RGB-D images, [START_REF] Lee | Adaptive bayesian recognition and pose estimation of 3d industrial objects with optimal feature selection[END_REF] proposes an ICP algorithm fed with 3D SURF features and Closed Loop Boundaries (CLB) to estimate the pose of industrial objects. However, the system can not deal with occlusions likely to happen inside a bin.

Template Matching

To tackle the issue of texture-less objects, template matching methods are employed to assign more complex feature pairs from two different points of view. Prime works built an object descriptor composed of different templates hierarchically extracted from an object point of view and later compared with an input image through a distance transform [START_REF] Gavrila | Multi-feature hierarchical template matching using distance transforms[END_REF].

To handle more degrees of freedom in the camera pose estimation, the recent machine learning techniques are used to learn the object templates and the associated camera pose to infer the position and then refine it using the distance transform. However, the algorithms still need a contour extraction process which is not suitable for low contrasted, noisy or blurred images. In [START_REF] Hinterstoisser | Dominant orientation templates for real-time detection of texture-less objects[END_REF] the discretized gradient directions are used to build templates compared with an object model through an energy function robust to small distorsion and rotation. This method called LINE is yet not suitable for cluttered background as it severely impacts the computed gradient. A similar technique is proposed in [START_REF] Muja | Rein-a fast, robust, scalable recognition infrastructure[END_REF]. The arrival of low-cost RGB-D cameras led the templates to become multimodal. LINEMOD presented in [START_REF] Hinterstoisser | Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes[END_REF] uses a depth canal in the object template among the gradient from LINE to easily remove background side effects. The method is later integrated into Hough forests to improve the occlusion robustness [START_REF] Tejani | Latent-class hough forests for 3d object detection and pose estimation[END_REF]. To deal with more objects inside the database, [START_REF] Hodan | Detection and fine 3d pose estimation of texture-less objects in rgb-d images[END_REF] proposes a sliding window algorithm extracting relevant image areas to build candidate templates. These templates are verified to get a rough 3D pose estimation later refined with a stochastic optimization procedure. The camera pose estimation problem can be solved with template matching but remains constrained to RGB-D information to achieve acceptable results. In this paper we propose to use 2D images without depth information therefore not suitable for this type of matching.

Features Matching from Neural Networks

Neural networks offer the ability to automatically extract relevant features to perform a given task. In a pose estimation problem they use convolutional neural networks to compute object descriptors as a database prior to a kNN algorithm to find the closest camera pose in the set [START_REF] Wohlhart | Learning descriptors for object recognition and 3d pose estimation[END_REF]. The loss function is fed with a triplet formed with one sample from the training dataset and two others respectively close and far from the considered camera position. Forcing the similarity between features and estimated poses for two close points of view is also employed in [START_REF] Doumanoglou | Siamese regression networks with ef-ficient mid-level feature extraction for 3d object pose estimation[END_REF] with a Siamese neural network doing a real pose regression. Although these methods are quite appealing, they still take advantage of RGB-D modalities. In [START_REF] Kendall | Posenet: A convolutional network for real-time 6-dof camera relocalization[END_REF] a deep CNN known as PoseNet based on the GoogLeNet network is built to regress a 6D camera pose from a 2D image. Whereas slightly different from our work for dealing with urban scenes, the work shows the continuous 3D space regression capability of a CNN in an end-to-end manner. Other works are using pre-trained CNN for object classification to estimate a view pose through SVMs. The depth channel of the RGB-D image is converted into an RGB image to be fed into a CNN with relevant 3D features [START_REF] Schwarz | Rgb-d object recognition and pose estimation based on pretrained convolutional neural network features[END_REF].

POSE REGRESSION

In this section we first define and transform the pose regression problem as a camera pose estimation problem using a bounding sphere and quaternions. To generate the pose views later fed into the CNN, a 3D pipeline rendering a CAD model seen under several scene atmosphere conditions is presented. Finally, the convolutional neural network which regresses the pose of the object is described from its architecture to its loss function based on the geodesic distance in the quaternions space.

Problem Definition

An object O is placed in a world scene with its frame position

P O = (x O , y O , z O ) and orientation expressed as the Euler triplet R O = (φ O , θ O , ψ O ).
The objective is to retrieve the object orientation from a point of view given by a fixed top camera. A trivial problem shift is to consider a fixed object with a mobile camera encompassing the scene localized with

R C = (φ C , θ C , ψ C
) lying on a bounding r-radius sphere centered on the object's centroid (Figure 1). Using the convention Z -Y -Z for the Euler triplet allows us to easily map the camera position according to the sphere azimuth and elevation. The third angle ψ C is then the camera plane rotation angle (Figure 2).

X O Z O Y O Y C X C Z C X O Z O Y O Y C Z C X C
X O Y O Z O φ C θ C Z C Y C X C ψ C ψ C Figure 2:
The camera is placed according to the

(φ C , θ C , ψ C ) Euler's triplet using the Z -Y -Z convention.
Any composition of rotations can be written as an axis-angle rotation according to the Euler's rotation theorem. Following this equivalence, to avoid any gimbal locks and improve computation performances, the triplet R C is written as the quaternion q C = (w C , v C ) according to the chosen Euler convention, representing an axis-angle rotation with a vector expressed as an hypercomplex number. Any camera orientation is now independent from the Euler's convention used. Using a telecentric lens on the camera would allow us to estimate the position P O and orientation R O of a part without any perspective effect: two objects similarly orientated but widely separated in the world scene would take the same shape in the top-view camera plane. The side effect is that different Z altitudes would give identical zooms (subject to magnifying errors) thus misleading the frame position estimation (Figure 3). It means that the system still needs to use an entocentric lens to predict the r sphere radius. Knowing the object centroid in the image plan plus the estimated r sphere radius is enough to obtain each object frame position P O among R C from our neural network prediction. The position estimation P O remains out of this paper scope yet so we suppose the radius r is known and we only use entocentric rendering for the orientation estimation R C .

Pose Generation

The industrial parts inside a bin have many possible poses as they can lie on each other. Thereby, creating a training dataset of real images covering all possibilities is tedious and complex. Moreover, every point of view will rely on a specific lightening condition not necessarily consistent with different bin-picking module placements in a factory. In this work a CAD model of the object is available so that a dataset can be generated during a rendering process. Using a virtual dataset has several advantages including the anticipation of different atmosphere conditions, backgrounds, scene cluttering and occlusions happening in a nonfavorable plant environment.

Camera Placement

A CAD model is placed on the center of a 3D scene according to its frame then textured to reproduce the glossy black plastic material of the real part. The camera positions are evenly picked on an r radius sphere surface with a specified number of poses using a golden spiral algorithm [START_REF] Boucher | Points on a sphere[END_REF]. The camera frame is computed so that the Z axis is pointing toward the scene center and the X axis remains horizontal (meaning ψ C = 0). This results in two angles φ C and θ C positioning and a camera plane rotation ψ C . Notice that this third angle can be generated by rotating the result image which can be done on the fly during the training phase thus reducing the dataset size on the hard drive.

Scene Parameters

For each point of view, the lighting conditions as well as the material reflectance are modified. This emphasizes different areas of the part so that the neural network can learn different features from the training dataset (Figure 4). A relevant scene parameter range comes from the study of real images taken under several conditions as shown in Figure 5. However, this might not always look realistic since the real material has no constant reflectance on its surface and it may contains blur or timestamp prints as seen in the topleft image in Figure 5. Finally, to avoid any pixelation the software antialiasing is applied and the image is generated with a size of 64 × 64 pixels. This image resolution is approximately the object bounding box size inside a bin rendered with an initial camera resolution of 1600 × 1200 pixels (which is an industrial standard) at a working distance of 2 meters. The scene background is set as an alpha channel (full transparency) to apply different backgrounds on the fly during the future training phase (Figure 8). The background can be defined as a color or a texture picked within high resolution samples showing material such as wood, plastic or metal as we don't know the nature of the bin in advance. The RGB image with the alpha channel leads to a 4-channels image later flattened to grayscale during the training process.

Network Architecture

Architecture

To estimate the pose of a single part from an RGB image we use a CNN architecture composed of convolutional and fully connected layers. Our network can only deal with a specified object but seen in different lighting conditions and with several backgrounds. In this case, using 2 convolutional layers is enough to extract shape and reflection features. On the other hand, relations between the extracted features and the quaternion are quite complex which basically leads to use at least one massive fully connected (FC) layer. Inspired by the recent work of [START_REF] Wohlhart | Learning descriptors for object recognition and 3d pose estimation[END_REF], an efficient way to bypass such computation time and memory consumption is presented in [START_REF] Doumanoglou | Siamese regression networks with ef-ficient mid-level feature extraction for 3d object pose estimation[END_REF]) using a Siamese architecture forcing via the loss function a similarity between features and pose estimations for similar samples. From these works, we also use a last FC layer described as the descriptor layer but we propose a network which has an increasing number of filters per layer as we handle only one object and thus do not need a sparse description of the part. In the following description, CONV (a, b) stands for a convolution layer with b filters of size a×a, POOL(a) for a max pooling layer of size a × a and FC(a) for a fully connected layer of a hidden neurons. This CNN is built as follows: CONV (11, 96) -POOL(2) -CONV (5, 384) -POOL(2) -FC(512) -FC(64) -FC(4). The network is fed with a 64 × 64 grayscale image. All layers are using a ReLU activation function while the output layer uses a tanh function as the prediction domain is [-1, 1] which is suitable for the quaternion space. The images are normalized prior to the input layer.

Loss Function

An object orientation R O can be described with an Euler's triplet (φ O , θ O , ψ O ). However when using unitary quaternions for rotations, R O can also be described by two quaternions : q O and -q O . Thus to estimate the discrepancy between two poses, the usual L 2 distance is not suitable since L 2 (q O , -q O ) is not null. Even if in the end of the learning process the prediction q P is getting close to the ground truth q T leading the L 2 distance to be small, the network behaviour in the beginning of the learning process is not clear. Moreover, using the L 2 norm constrains the network to predict the exact same vector regardless of the quaternion geometric possibilities. For the quaternions are lying on a 3-sphere, we build a loss computed with the geodesic distance θ between q T and the normalized predicted quaternion q P to tackle this issue [START_REF] Huynh | Metrics for 3d rotations: comparison and analysis[END_REF]. Using the scalar product . , we have:

q T , q P = cos θ 2 = cos(θ) + 1 2 (1)
With ( 1) we define the geodesic loss for the i th example in the dataset as:

L G i = 1 -q T i , q P i 2 (2)
This loss is suitable for a gradient-based backpropagation for it is made with simple derivable operators. With this objective function, the network is able to predict the truth q T but also -q T . To properly use this loss term, the predicted quaternion is manually normalized thus, the network does not tend naturally to predict unitary quaternion (real rotations) when not fully converged. One simple way to force this is to add a β weighted penalty when the prediction's norm differs from 1.

L N i = β(1 -||q P i ||) 2 (3)
The final loss is built with the geodesic error L G i (2) and normalization error L N i (3) mean over the samples i among a L 2 regularization term weighted by λ to prevent any overfitting:

L = 1 n n ∑ i=1 (L G i + L N i ) + λ||w|| 2 2 (4)

EXPERIMENTS

In our experiments we used an industrial black plastic part generated by its CAD model. We first produce the dataset for the neural network according to a certain number of evenly distributed positions on a sphere under several scene parameters. During the training, we process each image to create different backgrounds and render the third angle.

Protocol

The training dataset is generated with 1000 evenly distributed positions on the sphere, 3 material reflectances and 2 orthographic camera lens scales to avoid overfitting leading to a training dataset of 6000 images. With a number of 1000 positions, the average geodesic distance between two close samples is 6.3 • . The validation and test datasets are made with 1000 random samples extracted from 10000 even positions on the sphere set with random light conditions and reflectances for each one.

The training is performed with minibatches containing 100 samples. For each sample, a process can add the third angle ψ C by rotating the image plane and changing the associated quaternion. The quaternion q C modification is however not straightforward as it needs to first be converted into a rotation matrix Rot C then to the corresponding Euler's triplet using the proper convention. Once the third angle of the triplet is modified, the new quaternion is computed and placed into the minibatch sample (Figure 7). A colored or textured background is then applied to each image sample to remove the alpha channel. The image color channels do not provide relevant information as our parts are made of black plastic. Thereby each image is flattened to obtain a grayscale picture.

q C Rot C (φ C , θ C , 0) +(0, 0, ψ C ) q C '
After the image process, the minibatches are normalized and zero-centered.

In a first experiment, we only generate and estimate the two first angles φ C and θ C . Learning the third angle requires a strong variability in the dataset so that the network needs lots of epochs to converge. The third angle ψ C learning starts with widely separated angles within the [-180 • , 180 • [ angle space (a 90 • step). Through the epochs, the step is reduced to finally reach 10 • (Figure 11). We introduce two metrics suitable for the angle estimation error. First, we retrieve the Euler's triplet from the predicted quaternion using the rotation ma- trix. We define the angle error vector E = (E φ , E θ , E ψ ) as the absolute difference between the angles predicted and the ground truth. However, the error under each Euler's axis does not represent how far the estimated pose if from the truth as they can be cumulated. Yet, we build a second metric G obtained with the geodesic distance between the predicted and the ground truth quaternion coming from (1).

G = |cos -1 (2 q P , q T 2 -1)| (5)
The geodesic distance shows the smallest transformation needed to align the prediction with the truth and can be seen as a cumulative Euler angle error for each sample. Thus it is expected to be higher than any components of E.

For the training phase we use a Nesterov momentum backpropagation with an initial learning rate of 0.001 decreased through the epochs and a 0.9 momentum. The λ regularization weight is set to 0.001 as well as β. The implementation is Python-based with a Theano+Lasagne framework. The training dataset is build with a Blender rendering pipeline and learned on a Nvidia Tesla P100 in roughly 10 hours. The parameters giving the best score for the validation dataset are kept for the test phase.

Results

Looking at the results in Table 1 we are able to estimate both φ C and θ C with an average accuracy of 3 • when only the two first angles are learned with a random background. The last estimated angle ψ C can also be retrieved for an unitary quaternion since it can represent any rotation composition. However we see that the mean ψ C value is high because no variability on it has been seen in the training dataset (Figure 9). As expected, the geodesic error is the highest because it represents an Euler angle cumulative error. When the third angle ψ C is learned, we observe that its distribution gets a larger prediction uncertainty than in the two angles scenario but has a lower mean (Figure 10). With a 6 • resolution for φ C and θ C and 10 • for ψ C , the angle error medians are under 3 • which shows that the network is performing the expected regression task and do not tend to classify to the nearest learned angle (Table 2). It is interesting to note that in a two angles learning scenario, the network converges really fast at 100 epochs under a 5 • error whereas the three angles learning scenario needs more than 1000 epochs (Figure 11). Some samples are shown with the associated geodesic error in the Figure 12. The trained network has two max-pooling layers (Figure 6) which are known to be invariant to small rotations. Despite their abilities to reduce the training time and complexity, they tend to constrain the prediction capacities of the algorithm. When estimating the object pose from real images, the task remains challenging for we do not dispose any telecentric dataset. Only a small number of views with limited perspective effects can be fed into the network. With an only rendered-based training, we are still able to estimate the pose of the object with a visually convincing precision (Figure 13). Even if we do not have any ground truth information, the pose of the object inside the real image can be roughly estimated by the mean of a 3D modeler as in Figure 5. With this information, the average Euler's angles error reaches 22 • .

mean median E φ 4.39 • 2.0 • E θ 1.93 • 1.46 • E ψ 9.24 • 1.52 • G 3.3 • 2.8 •

CONCLUSION

In this paper we proposed a convolutional neural network performing a 3D pose regression of a known obmean median E φ 6.46 • 3.0 ject in a scene. We showed that once the network has learned from rendered data, it can infer from rendered images the three Euler's angles with a 5 • accuracy despite the max-pooling layers dropping the third angle estimation performance. Inside a real industrial bin, the parts can lie on each other so that severe occlusions are likely to happen. However, our network does not learn how to handle these configurations. Some works already tackle this issue with a loss function term handling an occlusion factor [START_REF] Doumanoglou | Siamese regression networks with ef-ficient mid-level feature extraction for 3d object pose estimation[END_REF]. In a future work, occlusions are going to be taken into account among an instance segmentation algorithm also based on a convolutional neural network to get an end-toend learning process. When the network is fed with real image containing small perspective effects, the estimated pose is visually convincing however the accuracy exceeds 22

Figure 1 :

 1 Figure 1: Top: The object O placed in the world scene and its CAD frame ( X O , Y O , Z O ) viewed from a fixed top-view camera. Bottom: Finding the O frame is equivalent to a camera frame ( X C , Y C , Z C ) evaluation problem in the scene when the object is fixed.

Figure 3 :

 3 Figure 3: Same scene rendered with different camera lens. Left: entocentric lens rendering different altitude and perspectives. Right: telecentric lens giving the same aspect for every object including scale.

Figure 4 :

 4 Figure 4: Effects of several lightening conditions in the scene and different object material reflectances on the rendered image.

Figure 5 :Figure 6 :

 56 Figure 5: Relevant scene parameters can render realistic views of the part. Left: the real image pose. Right: the rendered approximated pose.

Figure 7 :

 7 Figure 7: Quaternion modification when adding a third angle ψ C .

Figure 8 :

 8 Figure 8: Image processing during the minibatch creation: the third angle ψ C is generated on the fly before the image is flattened to obtain a gray level matrix.

Figure 10 :

 10 Figure 10: Box-plot of the angle estimation errors with three angles generated (φ C , θ C ,ψ C ).

Figure 11 :

 11 Figure 11: Geodesic error of test batch during the two (red) and three (blue) angles training phase. For the three angles training, the third angle steps are reduced at 200, 400, 700 and 900 epochs.

Table 1 :

 1 Pose estimation angle error with a two angles training.

	E φ
	E θ
	E ψ
	G
	0	2	4	6	8	10
			Angle error ( • )
	Figure 9: Box-plot of the angle estimation errors with two
	angles generated (ψ

C , θ C ).

Table 2 :

 2 Pose estimation angle error with a three angles training.

		•
	2.8 • E ψ 6.56 • E θ G 5.14 •	2.26 • 2.31 • 4.61 •

we aim at learning perspective effects by associating any quaternion estimations to the object (x, y) position in the image plane.


Adeline Granet

Emmanuel Morin

Harold Mouch Ère

Solen Quiniou

Christian Viard-Gaudin

Transfer Learning for Handwriting Recognition on Historical Documents

Keywords: Handwriting Recognition, Historical Document, Transfer Learning, Deep Neural Network, Unlabeled Data

In this work, we investigate handwriting recognition on new historical handwritten documents using transfer learning. Establishing a manual ground-truth of a new collection of handwritten documents is time consuming but needed to train and to test recognition systems. We want to implement a recognition system without performing this annotation step. Our research deals with transfer learning from heterogeneous datasets with a ground-truth and sharing common properties with a new dataset that has no ground-truth. The main difficulties of transfer learning lie in changes in the writing style, the vocabulary, and the named entities over centuries and datasets. In our experiment, we show how a CNN-BLSTM-CTC neural network behaves, for the task of transcribing handwritten titles of plays of the Italian Comedy, when trained on combinations of various datasets such as RIMES, Georges Washington, and Los Esposalles. We show that the choice of the training datasets and the merging methods are determinant to the results of the transfer learning task.

INTRODUCTION

Historical documents are more and more digitized to preserve them and to ease their accessibility and diffusion. Thus, information retrieval within historical documents is a real challenge. Moreover, the quantity of images is so large that manual information mining remains a time-consuming task. Over the last decade, historical data have become a principal target for classification [START_REF] Cloppet | ICFHR 2016 competition on the classification of medieval handwritings in latin script[END_REF], line detection [START_REF] Murdock | ICDAR 2015 competition on text line detection in historical documents[END_REF], and keyword spotting [START_REF] Puigcerver | ICDAR 2015 competition on keyword spotting for handwritten documents[END_REF]. Standard end-to-end text recognition systems consist of three steps [START_REF] Fischer | Automatic transcription of handwritten medieval documents[END_REF]: manual labelling of data to create a ground truth; specific pre-processing operations such as denoising documents and segmenting them into blocks, lines, or words; training of a dedicated recognizer using the alignment between text images and manual labels.

In [START_REF] Lladós | On the influence of word representations for handwritten word spotting in historical documents[END_REF], the authors study the problem of handwriting recognition (HWR) without training data for historical documents. Using a non dedicated dataset is ill-advised because there may be several problems caused by significant differences in terms of period and geographical area, which often affect the script style. Finding a training dataset for keyword spotting or handwriting recognition, meeting the desired distinctive characteristics, is a complicated task. However, some studies attempt to use modern data spotting systems for historical documents, as [START_REF] Frinken | Adapting BLSTM neural network based keyword spotting trained on modern data to historical documents[END_REF] which mixes different resources to transcribe another resource based on its target vocabulary: this approach is called Transfer Learning. We are especially interested in Transductive Transfer Learning which focuses on domain adaptation [START_REF] Pan | A survey on transfer learning[END_REF]) by using various source and target data for the same given task. We want to use this method and to push it even further by multiplying the number of annotated data used as sources and by adding parameter transfer.

In this paper, we are studying a new resource (financial records of the Italian Comedy) using a minimum amount of information and without a groundtruth. This prevents the direct use of traditional methods of HWR (see section 2). That is why we want to build a recognition system able to transfer knowledge on unknown data, without annotating more data. To overcome the lack of data, the Transductive Transfer Learning seems to be a good alternative. The chosen recognition system is a BLSTM-CTC system (Bidirectional Long Short-Term Memory and Connectionist Temporal Classification) including a Full Convolution Network, among the current state-of-the-art systems; this avoids abstracting the specific extraction of features. Three datasets are used during the training step of the network, each dataset sharing at least one feature with our Italian Comedy data.

The rest of this paper is organized as follows. We present the state of the art on handwriting recognition systems in Section 2. Then, we describe our HWR system, its structure, and its relevant postpreprocessing steps, in Section 3. The dataset we use is presented in Section 4 and we report our experiments and results in Sections 5 and 6.

STANDARD HWR SYSTEMS

At the end of the 90's, Hidden Markov Models (HMMs) had become a reference due to their ability to learn sequentially. Moreover, they could integrate knowledge in the form of lexicons and language models to better label sequences [START_REF] Bunke | Off-line cursive handwriting recognition using hidden markov models[END_REF]. Rapidly, they were strengthened by neural networks and hybrid neuro-markovian systems further improved the local and global representation of characters [START_REF] Koerich | A hybrid large vocabulary handwritten word recognition system using neural networks with hidden markov models[END_REF].

At the same time, neural networks evolved with new types of neurons, namely recurrent neurons. Contrary to a simple neuron, a recurrent neuron allows a connection to itself. Thus, recurrent neural networks (RNN) store more information from all inputs during training [START_REF] Senior | An offline cursive handwriting recognition system[END_REF]. Then, the Long Short-Term Memory (LSTM) block [START_REF] Hochreiter | Long short-term memory[END_REF] appeared to solve the problem of the vanishing gradient. This block allows the training of the network to converge.

Recently, multi-dimensional neural networks with LSTM (MDLSTM) have outperformed traditional networks. The first one was a bidirectional recurrent network with LSTM (BLSTM); it became a reference for HWR systems [START_REF] Fischer | Automatic transcription of handwritten medieval documents[END_REF]. Nowadays, multidimensional recurrent neural networks won competitions such as [START_REF] Grosicki | ICDAR 2009 handwriting recognition competition[END_REF]. In [START_REF] Graves | Offline handwriting recognition with multidimensional recurrent neural networks[END_REF], the multidimensional part of MDLSTM is made of four parallel layers across each direction on raw images. All the context can thus be used without any restriction.

In HWR, all the previously introduced networks cannot automatically align the input sequence with its labels. So, [START_REF] Graves | Supervised Sequence Labelling with Recurrent Neural Networks[END_REF] presented a connectionist temporal classification (CTC) layer which computes a sequence of labels to avoid the segmentation of the input sequence into characters or words. Another important step in HWR is feature extraction. Although traditional methods such as HOG [START_REF] Terasawa | Slit style HOG feature for document image word spotting[END_REF] have proven their efficiency, new methods integrating convolutional neural networks (CNN) have begun to replace them [START_REF] Suryani | On the benefits of convolutional neural network combinations in offline handwriting recognition[END_REF]. Other fields like Visual Recognition and Description also use methods based on CNN and LSTM [START_REF] Donahue | Long-term recurrent convolutional networks for visual recognition and description[END_REF]. We can distinguish two different approaches: the first one is a simple CNN and includes at least one layer of full connected neurons at the end of the network, and the second one mainly uses convolution and max pooling layers.

Multilingual systems were developed in parallel of MDLSTM. Some use identical configurations on independent training datasets [START_REF] Voigtlaender | Handwriting recognition with large multidimensional long short-term memory recurrent neural networks[END_REF], others dedicate one specific layer for each language and for each task in a recurrent neural network [START_REF] Moysset | The A2iA multi-lingual text recognition system at the second maurdor evaluation[END_REF] but few systems are trained on multilingual datasets at the same time [START_REF] Kozielski | Multilingual off-line handwriting recognition in real-world images[END_REF]. Regarding monolingual or multilingual HWR systems, it is common to use ngram language models at the word or character level, and a dictionary closed on the training set to improve the decoding step. In [START_REF] Oprean | Using the web to create dynamic dictionaries in handwritten out-of-vocabulary word recognition[END_REF], the authors use Wikipedia to create a dynamic dictionary for each word detected as out-of-vocabulary.

FCN-BLSTM-CTC RECOGNITION SYSTEM

In our HWR systems, the concept of Transductive Transfer Learning is performed through the training and the validation of the weights of the neural network on a set of three datasets. Then, the saved weights are used either to perform tests directly on the new dataset or to perform fine-tuning by initializing the new network. Our neural networks are made up of two parts: feature extraction and handwriting recognition.

The feature extraction part is directly integrated in the HWR system with a fully convolutional neural network (FCN). A graphical representation of the two architectures that we are using in our experiments is presented in Figure 1. The neural network takes an input image with a fixed-height of 120 pixels and a variable width denoted t. The common part of both networks is composed of three layers of convolution with a kernel size of 5x5 and same-padding (corresponding to the gray part of Figure 1). Then, the first network, called CNN 32, has 3 layers of convolution with 32 filters, while the second one, called CNN 128, has 3 layers of convolution with a filter number which doubles: 64 and 128 filters. Each of the convolutional layers is followed by several max-pooling layers. Finally, the last shape obtained has dimensions 32x1x t 20 for CNN 32, and 128x1x t 20 for CNN 128. This first part is directly connected to the second part of the network, aka the handwriting recognition system.

The second part of our neural networks respects the initial structure of the BLSTM neural network proposed in [START_REF] Graves | Supervised Sequence Labelling with Recurrent Neural Networks[END_REF]. The network is com- (10 x t/2 x 32) (10 x t/2 x 64)

(5 x t/4 x 32)

(5 x t/4 x 128)

(1 x t/20 x 32)

(1 x t/20 x 128) posed of two hidden layers, forward and backward. Both of them are made up of 100 LSTM blocks. Through the training, these layers are independent, one using the information following the time, and the other from the future to the past. LSTM blocks control the influence of long-term information across the network; it is interesting for long images such as lines of text. Then, the weighted sum of both hidden layers is provided to the output layer built with 75 softmax neurons. This corresponds to 52 lower and upper characters, 10 digits, and some punctuation symbols. At each time step, one output neuron represents one character and an additional neuron acts as a "joker", called "blank" label. Finally, the CTC is applied to the output in order to label the sequence. The provided output can be decoded by several algorithms, such as the Token Passing Algorithm, or the Prefix Search Decoding that can include a language model. In our experiments, the Best Path Decoding is used: at each time step, the most active node is selected, which finally gives the most probable path [START_REF] Graves | Supervised Sequence Labelling with Recurrent Neural Networks[END_REF]. To obtain the final sequence, all consecutive character labels are deleted except the first one, as well as blank labels. With this method, we stay in the framework where we have no prior knowledge on the data.

CASE STUDY: RECORDS OF THE ITALIAN COMEDY

This paper deals with transcription and information extraction issues from historical documents with few or no annotated data. Our research aims at providing a handwriting recognition solution for documents of the Italian Comedy, from the 18 th century. They are provided by the BnF 1 (Bibliothèque nationale de France) as part of the ANR project CIRESFI. This data consists of more than 28,000 pages of financial records covering one century. Several evolutions were noticed within this dataset. The first one is related to the language which switches from Italian (with several dialects) to French. The second one is a change in the structure preserving the quantity of information. For one day, we can find the date, titles of the plays, rev-1 http://gallica.bnf.fr/accueil/ enues, expenses, actor names, and also some notes (as shown on Figure 2) but this layout fluctuates over the decades. Further works are in progress on the detection and segmentation of these fields for each page. In this part of the project, our study focuses on the title field. This can be explained by the large collection of play titles of the Italian Comedy. The title field contains the list of plays that have been performed that day. Sometimes, this list gives more information, such as if it was a premiere, if it was played in special places, or in front of the king's court. An example of such complementary information is shown in Figure 2. The title field explains that this was the first performance of "Sophie ou le mariage caché" ("Sophie or the secret marriage") which was a comedy in three acts, preceded by "Arlequin toujours Arlequin" ("Arlequin always Arlequin"). Sometimes, actor names replace the names of their characters in the title. Thus, our collection of titles can not be considered as a ground truth but as a source of information.

The writing style is also an issue. At the beginning of the century, Italian actors wrote the records themselves. From the mid-century, there was only one writer for thirty years. Furthermore, there are differences as compared to the contemporary writing: special characters, as the long form of 's' (Figure 3 Thanks to a participative annotation website 2 , we were able to collect information on the position of the fields in the pages as well as their transcription. The algorithm of Seam Carving proposed by (Arvanitopoulos and Süsstrunk, 2014) has been applied to segment the blocks into title lines. Finally, we have manually validated the collected transcriptions of each block, each line, as well as the segmented lines themselves. Thus, we collected 971 validated lines with their transcriptions. To create the datasets for training, validation and test, we paid attention to distribute the lines of titles in both French and Italian among the various datasets. Moreover, title lines coming from the same page were not separated.

The aim of our work is not to provide another HWR method. Thus, we use CNN-BLSTM neural networks with CTC to produce knowledge transfer at two levels: from modern data to historical documents based on [START_REF] Frinken | Adapting BLSTM neural network based keyword spotting trained on modern data to historical documents[END_REF] and from at least one language to another one (or more). We want to solve two problems: the first one is related to the language which is mainly in French; the second one is the old style of our data. This is why we use two types of data sources: the first one is data in French and the other one is data from the same period.

EXPERIMENTAL SETUP

Datasets

To apply transductive transfer learning, we need to carefully choose the data used to train our systems. To our knowledge, there are no available annotated data meeting all our criteria: from the 18 th century, in French and Italian, with a closed vocabulary on the Italian Comedy. Thus, each selected dataset has at least one characteristics in common with our data. The descriptions of the datasets are shown in Table 1.

2 http://recital.univ-nantes.fr/ Georges Washington (GW) The GW dataset [START_REF] Fischer | Lexicon-free handwritten word spotting using character HMMs[END_REF] contains 20 pages of letters from George Washington to his associates during the 18 th century. The writing style is very similar from letter to letter. The images are binarized and normalized. To be comparable to the state-of-the-art, the original partition of the dataset in train, validation and test is used in our work. In fact, as four partitions of the data exist, we randomly selected one of these partitions.

Los Esposalles (ESP) The ESP dataset [START_REF] Romero | The ESPOSALLES database: An ancient marriage license corpus for off-line hwr[END_REF] consists of 173 pages of old Spanish marriage records. The whole pages are provided with the segmentations and transcriptions of their words. There is only one writer for these pages. The raw images are used with a normalization of the height of 120 pixels (like the GW images).

RIMES (RM)

The RM dataset [START_REF] Augustin | RIMES evaluation campaign for handwritten mail processing[END_REF] is a French database used in several ICDAR competitions. It is composed of 12,723 pages of administrative letters written by 1,300 volunteers. The gray scale images are used with the same height normalization as the others. The official split for the IC-DAR 2011 competition [START_REF] Grosicki | ICDAR 2011: French handwriting recognition competition[END_REF] provides 12,111 lines (including 11,333 lines for the training set) and 66,979 words (including 51,739 words for the training set). We keep this distribution.

Italian Comedy (CI)

The CI dataset is a French and Italian dataset describing play titles. It is composed of 151 play titles. The images were also standardized at 120 pixels high and in gray scale too.

From all datasets, we have removed characters in the ground truth such as '#', '/' or '$' and replaced them by a "joker" character because they could not appear in the Italian Comedy. The accentuated characters and also the tied letter in the RIMES transcriptions are replaced by their simple form. Thus, we consider that the 'é' character is one form of 'e' like 's' has a long form and a short form in the 18 th century.

System training

We have already detailed the architecture of our system in section 3 as well as some parameters. We carried out our experiments in successive stages. The first experiment allowed us to optimize the architecture of our system by evaluating it on two labeled datasets. With the best parameters obtained, we realized another experiment in order to select the best pairing of the dataset and to test the word recognition task on the Italian Comedy data. These results will be used for the third and final experiment which involves directly testing the transfer learning and the fine-tuning on our Italian Comedy data. In transfer learning, the generalization capability of the classifier must be maximized: the classical technique of early stopping is used in order to select the best network. To avoid overfitting, training continues until the Negative Log-Likelihood (NLL) computed by the CTC is no longer decreasing during 20 epochs on all the validation datasets. One set of weights of the neural network is backed up for each validation dataset only if the NLL drops. The training is realized through all the parts of the network: feature extraction with the FCN, handwriting recognition with the BLSTM, and data labeling with the CTC.

Our first experiments on GW demonstrated that the training is performed more efficiently on the line images when it is gradually done. Hence, all images in the training set are sorted in an ascending order according to their label length. In this way, the training step goes from isolated characters to words, and finally to long lines. First, experiments exclusively run on the word datasets. Then, we extend them to the lines. This allows an increase in the performance of the BLSTM-CTC across all datasets.

To evaluate the performance of our system, we used the recognition rate at the character level (CRR) and at the word level (WRR). The CRR is defined by

CRR = N -(S + D + I) N ,
with N the number of characters in the reference images, S the number of character substitutions, D the number of character deletions, and I the number of character insertions. The WRR is computed similarly on words. These two measures are case sensitive and treat the space as a character into a line. For the decoding step, no dictionary or language models were used on the three labeled datasets. Through our experiments we aims at defining a simple system capable of performing the transfer learning.

Experimental Results

Previous experiments showed that the training phase is more efficient when it is gradually realized. In the learning from scratch on the lines of GW, after 100 epochs, the cost stays high and the recognition rate is about 10.7%. Nonetheless, if the network is fine tuned with the weights from the previous training on the word images, the recognition rate reaches 77.3% in 33 epochs. So, this gradual training has been used for the following experiments.

Optimizing the System Architecture

This part presents the optimization of the system architecture. Indeed, the parameters of each separate part of our system (CNN and BLSTM-CTC) have an impact on the quality of the transcriptions, as well as on the transfer phase. Those experiments aim at evaluating the various parameters that can influence our system, i.e. the CNN size in terms of number of extracted features and number of cells in the hidden LSTM layers.

The GW W dataset is used more often for word spotting [START_REF] Rath | Word spotting for historical documents[END_REF][START_REF] Fischer | Lexicon-free handwritten word spotting using character HMMs[END_REF][START_REF] Frinken | A novel word spotting method based on recurrent neural networks[END_REF] than for HWR or word recognition [START_REF] Lavrenko | Holistic word recognition for handwritten historical documents[END_REF]. The latter uses a HMM with a cross validation and 19 pages to train the system. It achieves a CRR of 56.8% without the out-of-vocabulary words. With only 15 pages to train our system, we achieve a CRR of 28.7%. In order to improve our results and deal with the limited amount of data in GW W , we have paired it with the more extensive dataset RM W , for each experiment.

Table 2 presents the results obtained by four systems trained on the RM W ∪ GW W training set and tested on RM W and GW W test sets. The systems differ from the number of features (32 or 128) and from the number of cells in their two LSTM layers (50 or 100). When the number of extracted features is set to 32, we can see that increasing the number of cells leads to a better rate on the characters. When the number of features is increased to 128, the character recognition rate on GW W evolves from 40.5% to 48.5%. The same observation can be made on RM W , where the character recognition rate increases by 6.6%. However, the increase in the word recognition rate is not as important on the two test sets. Finally, these results allow us to conclude that the best configuration is obtained with 128 feature outputs from the CNN and with 100 LSTM cells, which was originally defined by [START_REF] Graves | Offline handwriting recognition with multidimensional recurrent neural networks[END_REF]. These settings will be kept for the rest of our experiments. We can also conclude that the addition of a large dataset helps to improve the results.

Optimizing the Word Datasets

Now that the architecture of the system has been set up, we have to define the datasets that will be used for training and testing our system. Among the three datasets, two must be dedicated to training and the third one must play the role of our CI data since we only have a limited amount of CI data with a groundtruth. The RIMES dataset must be part of the training dataset because it is the only one French dataset that we have here. Among studies on the RIMES dataset, [START_REF] Pham | Dropout improves recurrent neural networks for handwriting recognition[END_REF] use a deep recurrent neural network composed of MDLSTM and CNN. The authors vary the number of LSTM cells from 30 to 200. They obtain a 84.9% character recognition rate with 50 cells, and a 84.2% character recognition rate with 100 cells. Progressively, our system (presented in Table 3) tends to reach this state-of-the-art. The high recognition rate obtained with RM and ESP on the words push us to select them for the last series of experiments on CI. These first experiments on transfer learning at the word level shows that it is a difficult task.

Experimenting Transfer Learning

In the previous experiments, we were in a phase of training and testing on labeled data. The last experiments deal with the learning transfer process on the line images, especially those of CI. It is interesting to evaluate the impact that can have the addition or not of target data during the training. Table 4 presents the result of three different experiments which include tests on CI. Each of them uses the saved weights from an experiment (indicated by an ID in the train column).

Id Train

Test CRR

E 3 (E 2 ) ∪ RM L ∪ ESP W CI L 10.6 E 4 (E 2 ) ∪ CI L CI L 25.5 E 5 (E 3 ) ∪ CI L CI L 28.7
Our user case corresponds to the first line. With the help of annotated data, we could achieve the results obtained on lines 2 and 3. At first, we add RM L while keeping ESP w and using the saved weights from the last experiment E 2 on the words presented in Table 3. In the experiments, we found out that finetuning is only useful when the resources used to set up the weights are still present because the system forgets. Then, these saved weights are used to perform fine-tuning on CI L . This specialization of the network on the target data during the learning allows to increase the recognition rate of the characters by 15%. Finally, in order to observe the impact of the " space " character during the learning, the weights saved when learning on E 3 are chosen to perform fine-tuning on CI. It shows a 3.2% increase of the character recognition rate with respect to the rate obtained when the weights saved on E 2 were used.

Figures 4 and5 show the curves obtained when fine-tuning the learning on CI L , from RM W ∪ ESP W versus RM L ∪ ESP W . In addition to improving the character recognition rate, we note that the learning is faster, 38 epochs against 27 epochs, when using RM L instead of RM W . Furthermore, the initial validation NLL is twice as low with RM L and it quickly reaches a stability level, when the "space" character was learned before. With the fine-tuning on RM W , the character recognition rate on the validation set grows gradually. We can notice a sharp fall at the beginning of the learning curve: it seems like the system forgets part of what it has learned to better specialize to CI L . With RM L , the character recognition rate curve behaves as the training continues as before the fine-tuning. It is also interesting to note that, here, the breakpoint on the validation NLL is just after the best value reached for the character recognition rate. Furthermore, we can see that, with just one iteration on a small set of target data, the character recognition rate already exceeds the first experiment we had conducted by testing directly on CI L . Thus, a very simple system architecture without any language model helps to successfully achieve the learning transfer.

CONCLUSION

We performed experiments on transductive transfer learning for the task of handwriting recognition on a new historical dataset. Firstly, we optimized the parameters of the system to obtain the simplest and most performing system. Then, we defined the best pairing of datasets to realize the transfer learning. Moreover, it is necessary to pay attention to the balance of the representation of words in a dataset because this has a strong impact on the system. The last experiment with the CI data shows that even if both measures of recognition rate are low, we have a progression.

Our experiments allow us to conclude that HWR systems quickly specialize on learning data. We have found out that the addition of one or more resources makes it possible to improve character recognition. For now, it is still necessary to add a small amount of target data in the learning, to achieve a minimum recognition rate. In the long term, we want to build a system based on capitalization of resources in order to avoid those phases of annotation and manual validation. Our aim remains to be able to carry out transfer learning on data without prior costly knowledge. These results will guide our future experiments. Besides the addition of resources and their balance control, we can explore several orientations. We will focus on solutions which do not require costly ground-truth. For example, using unsupervised training can take advantage of thousands unannotated available pages. This can be done by using an autoencoder for feature extraction layers instead of our FCN. Another low cost resource are dictionaries which can be collected from topic related corpus but this solution implies to add a language model post-processing of the network outputs. Finally we can also set up a deeper network with several LSTM layers as it is often used in the state-of-the-art. However, increasing the system size runs against the results presented here to obtain a good recognition rate on data that are not used during learning. There is still some works to succeed in the recognition of new digitized documents from multilingual and multi-period resources.

Figure 1 :

 1 Figure 1: Graphical representation of the two architectures of the convolutional neural network part used for feature extraction. The blue version is called CNN 32, and the pink one is called CNN 128.

Figure 2 :

 2 Figure 2: Example of a financial daily record for the Italian Comedy with identification fields.

  (a)); evolution of the spelling, like using 'i' or 'j' indistinctly (Figure 3(b)); abbreviations such as "&c." for " etc " or symbols such as '•' (Figures 3(c) and 3(d)).

Figure 3 :

 3 Figure 3: Special characters and abbreviations in the Italian Comedy documents.

Figure 4 :

 4 Figure 4: Fine-tuning of CI L on E 2 : evolution of Log-Likelihood by epoch

Figure 5 :

 5 Figure 5: Fine-tuning of CI L on E 3 evolution of Log-Likelihood by epoch

Table 1 :

 1 The selected datasets. In the upper part, each common point with the Italian Comedy is shown in bold. In the lower part, the data distribution on the training, validation and test sets is given as well as the name of the associated dataset.

		Dataset	Georges Washington Los Esposalles	RIMES	Italian Comedy
		Language	English	Spanish	French	French (and Italian)
		Period	18 th century	18 th century	21 st century	18 th century
	Pixel value	Binarized	Grayscale	Grayscale	Grayscale
	Words	Train Validation Test	2,402 (GW W ) 1,199 1,292	45,102 (ESP W ) 51,739 (RM W ) 5,637 7,464 5,637 7,776	---
	Lines	Train Validation Test	325 (GW L ) 168 163	---	11,333 (RM L ) 1,332 778	582 (CI L ) 195 194

Table 2 :

 2 Results for several systems with the same training set, RM W ∪ GW W .

	Features Cells Test	CRR WRR
	32	50	GW W 40.5 RM W 63.5	22.2 34.8
	32	100	GW W 43.9 RM W 67.6	22.2 39.9
	128	100	GW W 48.5 RM W 70.1	25.5 41.7

Table 3 :

 3 Results obtain on word images for systems with 128 features and 100 cells for the LSTM layers.

	Id	Train	Test	CRR WRR
			GW W	48.5	25.5
	E 1 RM W ∪ GW W	RM W	70.1	41.7
			ESP W 9.0	0.3
			GW W	6.3	0.3
	E 2 RM W ∪ ESP W	RM W	71.1	42.0
			ESP W 91.1	75.9

Table 4 :

 4 Results for systems with 128 features, 100 cells for the LSTM layers and fine-tuned with the saved weights from the different experiences.


Valentin Belissen
email: valentin.belissen@limsi.fr

Annelies Braffort
email: annelies.braffort@limsi.fr

Michèle Gouiffès
email: michele.gouiffes@limsi.fr

Towards Continuous Recognition of Illustrative and Spatial Structures in Sign Language

Keywords:

In recent research, attention has been drawn to recognizing lexical signs in continuous Sign Language (SL) corpora, often artificial. Because SLs are singularly structured by the use of space and iconicity, this focus does not allow for an easy transition towards SL understanding and translation. In this paper, we discuss the necessity and realizability of recognizing higher-level linguistic structures in SL videos, like classifier constructions, using natural corpora such as Dicta-Sign-LSF-v2.

Introduction

Most research on Continuous Sign Language Recognition (CSLR) has focused on the recognition of lexical signs, which are "highly conventionalised signs in both form and meaning [...] consistent across contexts" [START_REF] Johnston | Auslan Corpus Annotation Guidelines[END_REF]. However, SL discourse also makes extensive use of space and illustrative structures -i.e. depicting signs (DSs) -that, conversely, can not be listed in a dictionary.

In this paper, we present experiments for the continuous recognition of DSs on the finely annotated dialogue corpus Dicta-Sign-LSF-v2 [START_REF] Belissen | Dicta-Sign-LSF-v2: Remake of a Continuous French Sign Language Dialogue Corpus and a First Baseline for Automatic Sign Language Processing[END_REF]. In this regard, a compact signer representation and recognition model are detailed here.

The current paradigm and limits

Because the common acceptation of CSLR focuses on lexical signs, popular corpora -which are few -do not include higher-level linguistic annotations. Furthermore, the observed SL is often made of artificial sequences, or interpreted SL -like in RWTH Phoenix Weather [START_REF] Forster | Extensions of the Sign Language Recognition and Translation Corpus RWTH-PHOENIX-Weather[END_REF] -which can be somewhat different from natural SL. In this approach (see for instance [START_REF] Cui | A Deep Neural Framework for Continuous Sign Language Recognition by Iterative Training[END_REF]), the usual metric is the Word Error Rate, that measures the normalized discrepancy between annotated and recognized sequences of signs.

However, this perspective is bound to be limited to very simple SL utterances. Indeed, natural SL exploits some key properties: Simultaneity: a great number of manual and non-manual articulators make it possible to convey information simultaneously, on different time scales. Iconicity and use of space: building on the visual modality, iconicity enables to show while saying. Using the signing space in a visual way to structure discourse is also fundamental, and forms the core of the visual grammar.

A newer acceptation of CSLR, not restricted to the recognition of lexical signs and with more attention to iconicity, is thus needed.

A newer acceptation for CSLR

Natural SL corpora, annotated in detail, are an interesting material to get closer to SL translation. With this objective, we propose that CSLR should include the recognition of elements other than the conventional lexical signs. Dicta-Sign-LSF-v2 [START_REF] Belissen | Dicta-Sign-LSF-v2: Remake of a Continuous French Sign Language Dialogue Corpus and a First Baseline for Automatic Sign Language Processing[END_REF], stemming from [START_REF] Matthes | Dicta-Sign -Building a Multilingual Sign Language Corpus[END_REF], is a French SL (LSF) corpus based on dialogue with very loose elicitation guidelines, thus highly representative of natural SL. With a total length of more than 11 hours, the annotated manual activity, inspired from the convention of [START_REF] Johnston | Auslan Corpus Annotation Guidelines[END_REF], covers lexical signs, as well as DSs -with sub-categories -, pointing signs, buoys and more.

DSs, that amount to about 20% of the annotated frames of this corpus, appear to a major category of non-lexical signs, using the iconic modality. Some-times referred to as classifier construction or illustrative signs, sometimes building on purely lexical signs, they are used to visually describe the location, motion, size, shape or the action of referents. In the following, we focus on the recognition of DSs. In terms of performance metrics the temporal -and possibly spatiallocalization should be accounted for. Frame-wise F1-score, for instance, appears to be a good starting point.

Signer representation and learning model

A compact and generalizable representation of signers in videos is obtained by separately processing the upper body, hands and face.

The OpenPose library [START_REF] Cao | Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields[END_REF] is used to get a reliable 2D estimate on the upper body pose, that is later turned into a 3D estimate by training a similar model as that of [START_REF] Zhao | A Simple, Fast and Highly-Accurate Algorithm to Recover 3D Shape from 2D Landmarks on a Single Image[END_REF]. Another Neural Network (NN) model [START_REF] Bulat | How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)[END_REF] makes it possible to get a 3D estimate on the face pose, and a SL-specific model [START_REF] Koller | Deep Hand: How to Train a CNN on 1 Million Hand Images When Your Data is Continuous and Weakly Labelled[END_REF] yields hand shape probabilities for each hand. Then, meaningful SL features like relative joint positions, speeds and accelerations, angles etc. are derived to form a compact signer representation vector (about 400 features).

A convolutional-recurrent NN (CRNN) can thus be trained to take these features as input, and recognize different kinds of manual unit types, like lexical signs, DSs, pointing signs, etc.

Recognition results

Using the previously described signer representation, we trained our learning model for the recognition of DSs on Dicta-Sign-LSF-v2, in a signer-independent fashion, which is known to make learning more difficult as well as more generalizable. Fig. 1 illustrates the value and effectivity of such a recognition model. Indeed, the selected sequence clearly can not be reduced to its three lexical signs. Two long DSs are used to depict birds, their important number, the location and extent of their flight, the form of their trajectory, etc. The frame-wise F1-score for this particular sequence reaches 86%, while it averages 59% on the whole corpus.

Conclusion and perspectives

In this paper, we have insisted on the central role of iconicity and spatial structure in SL discourse, highlighting the fact that Lexical Sign Recognition is only a part of the CSLR task. Since prevalent SL corpora have intrinsic limits in terms of generalizability and do not include annotations outside lexicon, we felt it was important to point out that richer corpora do exist, with fine temporal annotations.

As a first attempt on the Dicta-Sign-LSF-v2 corpus, we have trained a CRNN to recognize depicting signs. Interesting recognition scores are met, especially when considering the unclear boundary between lexical and depicting signs. Indeed, this frontier is dependent upon the chosen linguistic model, with no clear consensus on the matter [START_REF] Cuxac | La langue des signes française (LSF): les voies de l'iconicité[END_REF]. Using finely annotated datasets, like Dicta-Sign-LSF-v2 or other corpora initially intended for linguistic studies, the relevance of prevalent linguistic descriptions of SLs could be further questioned and our work be extended. Conversely, the usual CSLR setting, with lexical annotations and WER metric prevents one from conducting this type of research. It implicitly uses the hypothesis that SL discourse can be described with sequences of lexical signs, which we have shown is far from sufficient.

Beside more analysis on the performance metric and linguistic model, future work will include further reflection on the ways spatial information can be annotated and included in automatic recognition models. On a long-term basis, we will also reflect on how to go from the detection of important discourse elements like illustrative structures to global SL understanding.

"Fig. 1 :

 1 Fig. 1: Utterance from Dicta-Sign-LSF-v2 [1], with a strong use of space and iconicity (video reference: S7_T2_A10 -duration: 4 seconds). From top to bottom: thumbnails, proposed translation, expert annotation for the manual activity -Lexical signs and Depicting signs, each on three tracks (right handed (RH), two handed (2H), left handed (LH)) -and recognition results for depicting signs (F1-score: 86%).


Peter Eckmann

Nico Riedel

Gerben ter Riet Halil Kilicoglu

Cyril Labbé

Jennifer Byrne

Guillaume Cabanac

Amanda Capes-Davis

Bertrand Favier

Shyam Saladi

Automated screening of COVID-19 preprints: Can we help authors to improve transparency and reproducibility?

niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Automated screening of COVID-19 preprints: Can we help authors to improve transparency & reproducibility?

Peter Eckmann 1 , Nico Riedel 2 , Halil Kilicoglu 4 , Cyril Labbé 5 , Gerben ter Riet 6,7 , Jennifer Byrne 8 , Guillaume Cabanac 9 , Amanda Capes-Davis 8,10 , Bertrand Favier 11 , Shyam Saladi 12 , Peter Grabitz 2,3 , Alexandra Bannach-Brown 2 , Robert Schulz 2,3 , Sarah McCann 2,3 , Rene Bernard 2,3 , Anita Bandrowski 1 , Tracey Weissgerber

Evaluation•

  Goal to automatically evaluate COVID-19 preprints for reproducibility criteria • Each preprint is downloaded, parsed, and analyzed by a set of tools: ○ SciScore screens for rigor criteria defined by NIH and resources used (software tools, cell lines, etc.) ○ ODDPub screens for the presence of open data and code ○ Limitation-recognizer screens for study limitation statements ○ Barzooka screens for bar graphs used for continuous data ○ JetFighter screens for rainbow color maps ○ Seek&Blastn screens for correctly identified nucleotide sequences ○ Trial-identifier screens for and verifies clinical trial numbers Results • Study design features ○ 75% of analyzed preprints are secondary analyses, modeling studies, or cell line studies ○ 20% addressed sex as a biological variable, despite known sex differences in COVID-19 ○ 6.1% used model organisms, mainly mice • Transparency ○ 34.4% included self-acknowledged study limitations ○ 14.3% shared open code ○ 13.6% of preprints shared open data • Data presentation ○ 7.6% used rainbow colormaps, which are not colorblind safe and can create visual artifacts for viewers with normal color vision ○ 7.3% used bar graphs for continuous data, which can lead to misleading figures • Combined, the automated Tweets have been viewed about 380,000 times • Current average of ~1,000 views and ~10 link clicks per day • The account has accumulated a total of ○ 2459 link clicks ○ 98 retweets ○ 42 replies Conclusions • It is feasible to conduct large-scale automated screening of preprints for common quality criteria and provide feedback to study authors and readers before publication • Reports can publicly raise awareness of factors that affect study quality and reproducibility, while helping authors to present their research in a more transparent and reproducible manner.

  La Jolla, CA, USA, 2 Berlin Institute of Health, Germany, 3 Charité, Berlin, Germany, 4 Univ. of Illinois at Urbana-Champaign, IL, USA, 5 Univ. Grenoble Alpes, France, 6 Univ. of Amsterdam, Netherlands, 7 Amsterdam Univ. of Applied Sciences, Netherlands, 8 Univ. of Sydney, Westmead, New South Wales, Australia, 9 Univ. de Toulouse, France, 10 CellBank Australia, 11 Univ. Greonble Alpes, La Tronche, France, 12 California Institute of Technology, CA, USA

				2,3
	1 UC San Diego, Barzooka	JetFighter Images	trial-identifier Preprint	limitation-recognizer Text Study type ODDPub Preprint dataset SciScore classifier
			HTML Report
		hypothes.is		@SciScoreReports

Background

• Preprints have grown in popularity since COVID-19 emerged • Rapid publication is useful during a pandemic, but the lack of peer review has concerned many scientists • Can we evaluate preprints at scale without relying on authors or the knowledge of readers?


CMS plants do not acclimate to higher growth light intensity as the WT. This is shown by a lower quantum yield of CO 2 assimilation under ambient air conditions and lower activation state of photosynthetic enzymes. Currently, the fate of excess chloroplastic electrons in CMS plants grown at 350 µmol m -2 s -1 as compared to the WT is investigated and tend to favour an higher photorespiration in the CMS 350.

CMS leaves showed higher resistance to MV-induced oxidative stress as indicated by lower: chlorophyll bleaching, photoinhibition of PS II and catalase activity. Furthermore, CMS leaves maintained higher rates of total chloroplastic electron transport and CO 2 assimilation after the same time of incubation and the same concentration of MV, suggesting that altered transcript and/or protein contents of some antioxidative enzymes (SOD, APX, GR and CAT) are involved in oxidative stress resistance, by confering a more elevated protection in CMS than in the WT.

Acclimation to growth light intensity

Spectrophotometric assays RubisCO / NADP-MDH / SPS Chl a/b

Are CMS leaves affected in secondary photosynthetic reactions limiting carbon assimilation ?

2

Absolute and relative stimulation of initial Rubisco and SPS activities by high light (350 µmol m -2 s -1 ) as compared to low light growth (80 µmol m -2 s -1 ) in WT and CMS leaves measured either after one hour at growth light intensity or after 3h illumination at 1000 µmol m -2 s -1 PAR.

Are mutant leaves able to acclimate photosynthesis to growth light intensity ?

Investigations of light responses curves of carbon net assimilation in ambiant air. The different colours represent a light growth intensity ranging from 80 (in bright blue) to 350 (in red) µmol m -2 s -1 PAR.

CMS

leaves are unable to acclimate photosynthesis to higher growth light intensity as observed in the WT. Are CMS leaves affected in primary photochemistry which might limit carbon assimilation ?

Acclimation of a

 a Nicotiana sylvestris mitochondrial mutant to growth light intensity. 1 Laboratoire Ecologie Systématique Evolution, UMR 8079, UFR Scientifique d'Orsay, Université Paris XI, bât. 362, 91405 Orsay cedex, France. 2 Laboratoire Mitochondries et Métabolisme, Institut de Biotechnologie des Plantes, Université Paris XI, bât. 630, 91405 Orsay cedex, France. priault.pierrick@ese.u-psud.fr Respiratory chain of CMS II mutant of Nicotiana sylvestris mutant of Nicotiana sylvestris is functionally defect in complex I of the mitochondrial electron transport chain (1) but has a slightly higher dark respiration rate. Furthermore, enzymes of the alternative respiratory pathway (NAD(P)H DH, AOX) are elevated in CMS as compared to the WT (see figure). As mitochondrial function interacts with photosynthetic metabolism, this mutant was used as a tool to investigate the importance of mitochondria in photosynthetic acclimation to light. CO 2 /H 2 O exchanges Chl Fluorescence (Licor 6400-40, Mini PAM, PAM 2000) Priault, P. 1 , Cornic, G. 1 , De Paepe, R. 2 and Streb, Pet al., submitted (2) Stimulation of Rubisco and SPS initial activity by a higher growth light intensity is impaired in CMS. Maximal activities are very similar between CMS 80 and WT 80 and increased in both genotypes in response to higher growth light intensity (data not shown).

9

  Further investigations compared low light grown CMS and WT leaves (LL=80 µmol m -2 s -1 : WT 80, CMS 80) with high light grown leaves (HL=350 µmol m -2 s -1 : WT 350 and CMS 350) only. CMS An (µmol m -2 s -1 ) PAR (µmol m -2 s -1 ) WT An (µmol m -2 s -1 ) PAR (µmol m -2 s -1 ) Is the acclimation response of CMS leaves affected by different redox homeostasis possibly regulating gene expression ? 3 Redox parameters are not affected in CMS as compared to the WT. A: Light response curve of the photochemical quenching (qP) reflecting the redox state of the plastoquinone pool (=1-qP). B: Initial and total activities of NADP-MDH (µmol m -2 s -1 ) measured at growth light intensity after 1 h in the light period and after 3h illumination at 1000 µmol m -2 s -1 PAR and 25 °C. Investigation of qP as redox parameter of the ETR chain and NADP-MDH as redox parameter of the chloroplast stroma. A: relation between quantum yields of PSII and of CO2 assimilation calculated from light responses curves performed under 21% O2 and 380 ppm CO2. B: light response curve of excess electron transport not used for carbon assimilation calculated as ETR -(4*An+Rd) assuming that 4 electrons are necessary to assimilate one molecule of CO2 (WT 80 : black squares ; CMS 80 : black circles ; WT 350 : white squares ; CMS 350 : white circles).

B

  Priault et al., submitted (2) The mismatch of ETR, carbon assimilation and initial activity of Rubisco is further investigated in CMS 350 leaves in order to characterize alternative electron sinks such as : and Catalase activity are less affected by MV treatment in CMS plants than in WT plants independent of growth light intensity. The inhibition of electron transport and carbon net assimilation is delayed in CMS plants as compared to WT plants when incubated at the same concentration of MV. Priault et al., submitted (2)

Acknowledgments : Authors would like to thank Chantal FRESNEAU for SPS assays, Guillaume TCHERKEZ for mass spectroscopy and Graham Noctor for helpful discussion. References : (1) Gutierres et al., 1997 (PNAS : 94, pp. 3436 -3441) ; (2) P. Priault, C. Fresneau, G. Noctor, R. DePaepe, G. Cornic and P. Streb The mitochondrial CMSII mutation of Nicotiana sylvestris impairs acclimation to growth irradiance (submitted to J. Exp. Bot. ).

These results show that mitochondrial function is necessary for acclimation of plants to growth light intensity and for oxidative stress response.