Simon Thomas

Clement Chatelain
email: clement.chatelain@insa-rouen.fr

Laurent Heutte

Thierry Paquet

Yousri Kessentini

Clément Chatelain

C Chatelain

A Deep HMM model for multiple keywords spotting in handwritten documents

Keywords: Handwriting Recognition, Keyword Spotting, Hidden Markov Models, Deep neural network, Hybrid Deep architecture, DNN HMM

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Recently, several systems for spotting words in unconstrained handwritten documents have been developed [START_REF] Cao | Template-free word spotting in low-quality manuscripts[END_REF][START_REF] Choisy | Dynamic handwritten keyword spotting based on the nshp-hmm[END_REF][START_REF] Rodríguez-Serrano | A similarity measure between vector sequences with application to handwritten word image retrieval[END_REF][START_REF] Rodríguez-Serrano | Handwritten word-spotting using hidden markov models and universal vocabularies[END_REF][START_REF] Chatelain | Recognition-based vs syntax-directed models for numerical field extraction in handwritten documents[END_REF][START_REF] Frinken | A novel word spotting method based on recurrent neural networks[END_REF][START_REF] Fischer | Lexiconfree handwritten word spotting using character hmms[END_REF]. Such word spotting systems aim at detecting and recognizing the queried words in document images. They can be applied simply for querying textual handwritten documents, but such systems may also be a component for indexing handwritten document databases by spotting the multiple keywords on which indexation is per-formed, following the popular bag of word indexation scheme [START_REF] Paquet | A categorization system for handwritten documents[END_REF].

Formally, one can distinguish word spotting approaches and keyword spotting approaches. The former aims at detecting a single word in a collection of document images in order to retrieve the subset of document images that contain this specific word. On the contrary, the latter aims at detecting, in a single document, a set of informative words that allow to predict its content or its category using the popular bag of words paradigm for example. These later approaches are used for automatic document categorisation [START_REF] Paquet | A categorization system for handwritten documents[END_REF][START_REF] Vinciarelli | Offline recognition of unconstrained handwritten texts using hmms and statistical langage models[END_REF] for which a reduced lexicon of keywords is needed so as to discriminate between a set of predefined categories of documents. In the literature, most of the word spotting approaches follow the former approach [START_REF] Cao | Template-free word spotting in low-quality manuscripts[END_REF][START_REF] Choisy | Dynamic handwritten keyword spotting based on the nshp-hmm[END_REF][START_REF] Rodríguez-Serrano | A similarity measure between vector sequences with application to handwritten word image retrieval[END_REF][START_REF] Rodríguez-Serrano | Handwritten word-spotting using hidden markov models and universal vocabularies[END_REF][START_REF] Chatelain | Recognition-based vs syntax-directed models for numerical field extraction in handwritten documents[END_REF][START_REF] Frinken | A novel word spotting method based on recurrent neural networks[END_REF][START_REF] Fischer | Lexiconfree handwritten word spotting using character hmms[END_REF], but very few of them are designed for multiple keywords spotting [START_REF] Paquet | A categorization system for handwritten documents[END_REF][START_REF] Thomas | An information extraction model for unconstrained handwritten documents[END_REF]. Even if the two tasks (word spotting vs. keyword spotting) are used in a different context, they rely on the same handwriting detection/recognition methods and both can be used to perform a quick, writer-independent detection and identification of words in document images.

Word spotting approaches proposed in the literature fall into the two following categories. On one hand, image based methods, also known as "query-by-example", operate through the image representation of the words [START_REF] Cao | Template-free word spotting in low-quality manuscripts[END_REF][START_REF] Rath | Features for word spotting in historical manuscripts[END_REF][START_REF] Adamek | Word matching using single closed contours for indexing handwritten historical documents[END_REF][START_REF] Rusiñol | Browsing heterogeneous document collections by a segmentation-free word spotting method[END_REF]. On the other hand, recognition based methods, or "query-by-string" methods, operate with an ASCII representation of the keywords [START_REF] Choisy | Dynamic handwritten keyword spotting based on the nshp-hmm[END_REF][START_REF] Rodríguez-Serrano | Handwritten word-spotting using hidden markov models and universal vocabularies[END_REF][START_REF] Paquet | A categorization system for handwritten documents[END_REF][START_REF] Frinken | A novel word spotting method based on recurrent neural networks[END_REF]. The first approach has the major drawback to be writer-dependent since it is based on a matching distance between the query image and the word images in the document. It is therefore limited to single writer applications and cannot be used for content-based document categorization. On the contrary, recognition based approaches are not limited to a single writer, but this is obtained at the expense of a more complex matching process, derived from conventional handwriting recognition methods. They are thus suitable for either document image retrieval or document image categorization.

Recognition-based word spotting approaches have to deal with four main issues: text line detection, text line segmentation into words, word recognition, and rejection of word hypotheses that do not match the query. A first strategy is based on the sequential application of these four steps, as in [START_REF] Rodríguez-Serrano | Handwritten word-spotting using hidden markov models and universal vocabularies[END_REF]. In this work, text lines are segmented into words using heuristics and a distance criterion, then words are recognized, and finally word hypotheses are spotted using normalized scores. The major drawback of this strategy is that the word segmentation errors are often irreversible, and will therefore affect significantly the recognition performance. Moreover, the approach presented in [START_REF] Rodríguez-Serrano | Handwritten word-spotting using hidden markov models and universal vocabularies[END_REF] is holistic, so the system is not able to spot words that are not present in the training set. Also, the decision rule to accept or reject a word hypothesis, which is the core of the approach, is difficult to design as it is data-dependent.

A second strategy performs every steps in one single global dynamic programing based optimization process that carry out segmentation/recognition/rejection decisions at the line level, using a line model able to describe both the keywords and the non-keywords [START_REF] Fischer | Lexiconfree handwritten word spotting using character hmms[END_REF][START_REF] Thomas | An information extraction model for unconstrained handwritten documents[END_REF][START_REF] Fischer | Hmmbased word spotting in handwritten documents using subword models[END_REF]. In [START_REF] Fischer | Lexiconfree handwritten word spotting using character hmms[END_REF], a line-based HMM word spotting system is applied to handwritten documents 1 . The approach is based on a line model made of a left and right "filler" model surrounding the keyword model. Line decoding based on Viterbi dynamic programing is performed twice: with and without the keyword, leading to two recognition scores. The likelihood ratio of the two scores allows to decide whether the line contains the keyword or not. The main drawback of this approach is its limited local discriminative abilities as the Gaussian mixture model used to compute emission probabilities is trained in a non-discriminative way.

However, it has been shown that embedding a discriminative stage in HMM model can significantly improve the recognition performance. Discrimination between models can be introduced using either discriminative training [START_REF] Woodland | Large scale discriminative training of hidden markov models for speech recognition[END_REF][START_REF] Do | Maximum margin training of gaussian hmms for handwriting recognition[END_REF][START_REF] Keshet | Discriminative keyword spotting[END_REF], or by replacing the GMM classifier by a discriminative classifier such as a SVM [START_REF] Ganapathiraju | Hybrid svm/hmm architectures for speech recognition[END_REF][START_REF] Huang | A hybrid hmm-svm method for online handwriting symbol recognition[END_REF] or a neural network [START_REF] Boquera | Improving offline handwritten text recognition with hybrid hmm/ann models[END_REF][START_REF] Graves | A novel connectionist system for unconstrained handwriting recognition[END_REF][START_REF] Marukatat | Sentence recognition through hybrid neuro-markovian modeling[END_REF]. In this study, we propose to combine the modeling ability of HMMs with the discriminative ability of a deep neural network architectures in order to design an efficient "query by string" spotting system that will operate at the line level. This is an extension of our preliminary work [Hidden ref] 1 An extension of this approach has been proposed in [START_REF] Frinken | A novel word spotting method based on recurrent neural networks[END_REF], where the HMM is replaced by a combination of a LSTM neural network with a CTC algorithm.

, where the fully generative GMM-HMM architecture is now replaced by the hybrid deep HMM architecture that significantly improves the performance. We also provide a comparison between pure HMM, standard hybrid MLP HMM and deep HMM approaches, using different feature sets and larger lexicon sizes. We finally show that the deep HMM architecture provides the best results, while avoiding the tedious design of an efficient handcrafted feature set dedicated to handwritten word recognition.

When compared with previous word spotting systems, our approach exhibits several advantages. Firstly, the system is lexicon-free, i.e., it is not restricted to spot some specific keywords for which the system is tuned during a preliminary training stage, but it can rather spot any arbitrary keywords. Secondly, the system is segmentation-free, i.e., text lines are not required to be segmented into words. Thirdly, it embeds a discriminative classifier which outperforms the standard Gaussian mixture model. Fourthly, the system is able to learn some discriminative features from the raw data. To the best of our knowledge, it is the first time such hybrid HMM / DNN architecture is applied on a handwriting recognition task. Such a deep HMM architecture benefits from both modeling and discriminative skills of these two models. We show on the French public incoming mail RIMES database [START_REF] Grosicki | Icdar 2009 handwriting recognition competition[END_REF] that this deep HMM architecture significantly improves the keyword spotting results when compared with the pure HMM model. In addition, the system is able to spot multiple keywords at a time through a competitive modeling of keywords by introducing a lexicon of keyword in the model. This is a way to cope with the limitation of several keyword-spotting approaches, especially the one presented in [START_REF] Fischer | Lexiconfree handwritten word spotting using character hmms[END_REF], which is only able to spot one keyword at a time. To spot multiple keywords, [START_REF] Fischer | Lexiconfree handwritten word spotting using character hmms[END_REF] proposes to iterate the spotting process for each keyword of the keyword lexicon, but the authors do not discuss how to solve conflicts when two detected keywords overlap.

This paper is organized as follows. We introduce our generic statistical text line model in Section 2. In Section 3, we describe the deep HMM architecture. An overview of the whole system is described in Section 4. In Section 5, we report the evaluation of our system on a multiple keywords spotting task. Conclusion and future works are given in Section 6.

A generic text line discriminative model for multiple keywords spotting

A keyword spotting system should be able to detect the position of each occurrence of a keyword in a document image, as depicted on Figure 1. Keyword spotting systems rely on the analysis of text lines, in order to decide whether it contains the searched keyword or not. As already mentioned in the introduction, we believe that the segmentation of the text lines into words cannot be reliable enough without recognition, and that a global segmentation/recognition decision should be taken at the line level. A successful strategy in the literature consists in building a text line model in order to allow recognition driven segmentation. For example, in [START_REF] Fischer | Lexiconfree handwritten word spotting using character hmms[END_REF][START_REF] Fischer | Hmmbased word spotting in handwritten documents using subword models[END_REF], a text line model composed of the keyword model surrounded by a filler model is proposed. Line decoding is performed twice, with the text line model and with the filler model, leading to two likelihood scores. Their ratio is used as a decision score compared to a threshold, in order to decide whether a keyword is present or not. In [START_REF] El-Yacoubi | A statistical approach for phrase location and recognition within a text line: An application to street name recognition[END_REF], the authors also use a text line model made of the keyword, a "left-extraneous model" and a "right-extraneous model" to spot street names in lines of postal addresses. Once again, the extraneous models act as filler models describing any sequence of characters. The decision is also taken by comparing the global line score to a threshold after normalization.

As we want to spot multiple keywords in one shot, the model should integrate the keyword lexicon, as well as a filler model able to model the irrelevant informa-tion (namely Out of Keyword Lexicon (OKL) word sequences and noise). In order to depict a line of text as a sequence of keywords and irrelevant information, a space model must also be integrated in the model. It is an ergodic model that accounts for any sequence of alphanumeric characters, including uppercase and lowercase letters, digits and punctuation. Keywords in KL may occur with a lower frequency than words in OKL. This is why the two models compete in parallel in the line model, with parameter G standing for the proportion of relevant information that can appear in the text. Finally, the line model concatenates any KL or OKL words with a white space model before going to the next possible word in the line through the loopy backward transition. The model loops until the end of line is reached. Let us emphasize that as the model simulates a competition between the OKL and each word of the KL, it prevents from the conflicts that may occur between two keywords, or between a keyword and the OKL. It also avoids the use of numerous thresholds for accepting/rejecting the keywords hypotheses. The setting of parameter G is discussed in Section 5. [START_REF] Choisy | Dynamic handwritten keyword spotting based on the nshp-hmm[END_REF] As one can see, the handwritten modeling is text line oriented, and fits particularly well with the gen-erative modeling of Hidden Markov Models (HMM). Indeed, HMM are a well known and established probabilistic tool in sequence modeling [START_REF] Rabiner | A tutorial on hidden markov models and selected applications in speech recognition[END_REF]. Thanks to an efficient embedded learning through Baum-Welch algorithm, HMM have been widely used for the recognition of handwritten words [START_REF] Kessentini | Off-line handwritten word recognition using multi-stream hidden markov models[END_REF] or sentences [START_REF] Vinciarelli | Offline recognition of unconstrained handwritten texts using hmms and statistical langage models[END_REF]. However, despite its appealing generative modeling ability, HMM suffers from a poor discrimination ability at the local frame level. Indeed, the local likelihood P (o|q i ) that stands for the emission probability of observation o when the model is in state q i is modelled using Gaussian Mixture Models (GMM) that are learned in a generative way. As a consequence, GMM of several states may overlap and are not trained to discriminate between these states. Furthermore, estimating a GMM in high dimension requires a huge amount of training data and one generally limits the dimension of the feature space to prevent from this difficulty. To overcome these limitations, many works have successfully proposed to replace the GMM by a discriminative classifier, most of the time a neural network [20-22, 27, 28], but SVM have also been proposed [START_REF] Ganapathiraju | Hybrid svm/hmm architectures for speech recognition[END_REF]. In this paper, we propose to replace the generative stage of HMM by a discriminative stage based on a deep neural network (DNN). This kind of architecture has recently been applied for speech recognition in [START_REF] Mohamed | Acoustic modeling using deep belief networks[END_REF][START_REF] Dahl | Context-dependent pre-trained deep neural networks for large vocabulary speech recognition[END_REF] and has shown good performance.

As depicted on Figure 2, two kinds of information must be learnt: the character models (black circles) and the transition probabilities between them (black and grey arrows). Transition probabilities between characters of the KL model are all equal to one as they stand for a lexicon driven recognition. Transition probabilities of the OKL model must be set to build a shallow language model that stands for a lexicon free recognition. Therefore, they are learnt from a general large lexicon. Character models are hybrid left-right DNN-HMM models, where the DNN provides a posteriori probabilities P (q i |o) that the frame belongs to the HMM state q i . They require a discriminative training stage on locally labeled data (at the frame level) for the DNN part, while state transition probabilities can be learnt as it is the case for standard HMM using the Baum-Welch algorithm. The main difficulty in training a DNN architecture is to provide a labeled training data at the frame level. This is discussed further in the next section.

The deep HMM architecture

Let us consider an observation sequence O = {o 1 . . . o T } of length T , extracted from a line of text L, whatever the nature of the information encoded (a continuous feature vector sequence or simply a sequence of sub images with white and black pixels as will be described further). Let us denote L opt the optimal sequence of labels (KL words and OKL words), that can be associated to O. It can be computed thanks to the equation 1:

L opt = arg max L {P (L|O)} (1)
= arg max

L P (O|L)P (L) P (O) (2)
= arg max

L {P (O|L)P (L)} (3)
where P (O|L) is the probability of a sequence of observations O given the sequence of words L. Even if P (O) is difficult to compute, it is constant and thus does not affect the search of the best state sequence and can be removed from the computation. In equation 3, P (L) is the prior probability of a word sequence. We assume equal priors for every word sequence, and therefore P (L) can be ignored.

Let us assume that L contains N character sequences K n belonging to KL, M character sequences W m belonging to OKL, and P spaces S. Then L opt can be found by maximizing over all segmentation/recognition path hypotheses:

L opt = arg max G N × N n=1 P (O n |K n ) × (1 -G) M × M m=1 P (O m |W m ) × P p=1 P (O p |S) (4)
where

P (O|K) = T t=1 P (o t |q i )P (q i |q i-1 ) (5)
where {q i } are the states involved in the character sequence K, and T is the number of frames. Finally, finding the best character sequence L opt is performed using the Time Synchronous Beam Search algorithm presented in [START_REF] Ortmanns | A word graph algorithm for large vocabulary continuous speech recognition[END_REF], by maximizing over the whole line the quantity P (O|L)P (L). It depends on:

the model and the a priori information, i.e. the G parameter and the lexicon constraints modelled by the transition probabilities between states P (q i |q i-1 );

the local discrimination ability of the system P (o t |q i ).

In order to find the best character sequence, one needs to estimate the local likelihood quantities P (o t |q i ).

A statistical classifier such as a neural network for example generally provides a posteriori probabilities P (q i |o y ) and the Bayes law is applied to rescale its outputs into local likelihood: P (o t |q i ) = P (qi|ot)×P (ot) P (qi)

. During recognition, the term P (o t ) is a constant for all classes and does not change the classification, it can be omited for every q i . Finally, the scaled likelihood P (qi|ot) P (qi) can be used as an emission probability for the HMM.

Discriminative deep architecture

Let us recall that discriminative deep architectures [START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF] have proven to achieve very high performance on various applications such as digit recognition [START_REF] Ciresan | Convolutional neural network committees for handwritten character classification[END_REF][START_REF] Niu | A novel hybrid cnnsvm classifier for recognizing handwritten digits[END_REF], audio classification [START_REF] Lee | Unsupervised feature learning for audio classification using convolutional deep belief networks[END_REF], or action recognition [START_REF] Le | Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis[END_REF]. They are based on the (old) idea that training a neural network with many hidden layers is a good way to (i) handle high dimensional input, and (ii) estimate complex decision boundaries between classes. The novelty lies in the way the hidden layers are trained. Indeed, when using the standard back-propagation algorithm on many layers, the energy of the errors is too weak to train the whole network [START_REF] Bengio | Greedy layer-wise training of deep networks[END_REF]. This issue is also known as "the vanishing gradient". Therefore, for training a deep neural network with Λ-hidden layer, a two-step training has been proposed [START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF][START_REF] Bengio | Greedy layer-wise training of deep networks[END_REF][START_REF] Ranzato | Sparse feature learning for deep belief networks[END_REF]:

-The first Λ-1 hidden layers are sequentially trained in an unsupervised way. Each layer is fed by the output of the previous layer and is trained as an auto-associator2 . These layers are called model layers, while their training is called pre-training. Taking raw data as input, a high level representation of the data can be obtained at the output of the top layer. This pre-training process can be considered to be an automatic feature extraction process [START_REF] Bengio | Greedy layer-wise training of deep networks[END_REF]. -Then the last layer (called decision layer ) is added to the network. It is trained by unlocking all the parameters of the model layers and performing supervised gradient back-propagation on the whole network. This operation allows to learn the discriminant decision function, and is known as fine-tuning.

Definition of the network

More formally, let us define a deep network containing Λ layers, where each layer computes its outputs H λ , λ ∈ {1, . . . , Λ}. The first layer is fed with the input, while the last layer computes the output of the network H Λ . Let {N 1 , . . . , N λ , . . . , N Λ } denote the number of neurons of each layer. Each layer computes the vector output H λ = {h λ i }, ∀λ ∈ {1, . . . , Λ}, i ∈ {1, . . . , N λ } , where h λ i denotes the output of the i th neuron of layer λ, which is computed using the following equation:

h λ i = f λ   N λ-1 j=0 w λ i,j . h λ-1 j   ∀i ∈ {1, . . . , N λ } (6)
where w λ i,j ∈ R are the weights, and f λ is a non-linear activation function over the weighted sum: typically a sigmoid function for the Λ -1 first layers, and a softmax function for the last layer, so that the outputs sum to one and can approximate a posteriori probabilities. Note that w λ i,0 is the bias associated to the i th neuron of layer λ through a fictitious input h λ-1 0 = 1. The deep network computes iteratively from layer 1 to layer λ the equation 6 that can be rewritten at the layer level using matrices as:

H λ = f λ ( W λ . H λ-1 ) ∀λ ∈ {1, . . . , Λ} , (7)
where W λ = {w λ i,j } ∈ R N λ ×N λ-1 is the matrix of parameters to be learnt.

Note that the model layers and the decision layers compute the same quantities, they only differ by the way they are learnt. Let us now described the two-stage training procedure of the DNN.

Training the model layers

The auto-encoder framework allows to learn non linear feature extractors in an unsupervised way. It learns an encoder function e, that maps an input x to a hidden representation e(x), and a decoder function d, that maps back e to the input space as x = d(e(x)). The estimation x is the reconstruction of x through the layer. The encoder and decoder parameters are learnt to minimize a reconstruction error on the training set, similarly to the training of an MLP using back-propagation of the error.

Once trained, the encoder function e is stacked as a new model layer in the deep architecture, while the decoder function d is eliminated from the network. The output of the new stacked layer is fed at the input of a new auto-encoder of the next new layer. Many layers can be stacked, like in Fig. [START_REF] Rodríguez-Serrano | Handwritten word-spotting using hidden markov models and universal vocabularies[END_REF], depending on the complexity of the input space.

Training the DNN-HMM combination

The DNN and HMM models have to be learnt so that the DNN provides the emission probabilities to the HMM. Training a neural network-HMM combination can be performed either using joint-training [START_REF] Bengio | Lerec: a nn/hmm hybrid for on-line handwriting recognition[END_REF][START_REF] Knerr | A neural network-hidden markov model hybrid for cursive word recognition[END_REF][START_REF] Marukatat | Sentence recognition through hybrid neuro-markovian modeling[END_REF], or independently. Joint training iteratively refines the objectives of the DNN during the training of the HMM, as EM algorithm does for GMM. The independent training scheme first trains a standard HMM-GMM model, and then trains the neural network using the frame label derived from a Viterbi forced alignment on the character sequence ground truth using the trained HMM-GMM character models. In any case, the a posteriori probabilities are divided by the prior state probabilities in order to be rescaled into likelihoods. Note that some variants have also been proposed such as the tandem training [START_REF] Schenk | Novel Hybrid NN/HMM Modelling Techniques for On-line Handwriting Recognition[END_REF][START_REF] Dreuw | Hierarchical hybrid MLP/HMM or rather MLP features for a discriminatively trained gaussian HMM: a comparison for offline handwriting recognition[END_REF], where the outputs of a neural network are used as features on which a standard GMM-HMM is learnt. Training a DNN requires many more iterations than training a HMM, therefore in this work the two models have been trained independently. Once the DNN and HMM have been trained, a few joint-training iterations have been performed, but no significant improvement was observed.

Word-spotting system overview

We now describe the whole keyword spotting system. Its input is a document image, in which the set of keywords is spotted, while the output is the location hypotheses of each keyword in the document. It has been implemented using various HMM architectures: a standard GMM-HMM system, an hybrid MLP-HMM, and the DNN-HMM. In the following sections we will provide the details of the processing steps: line segmentation, pre-processing, feature extraction and training of the models.

Line segmentation

The line segmentation process is an important and difficult task, mainly due to variable skew angle along the document or even along the text line, and adjacent text lines. We have implemented a line segmentation method based on a connected component analysis (see [START_REF] Paquet | A categorization system for handwritten documents[END_REF] for details).

Pre-processing

Pre-processing is applied to text line images in order to eliminate noise and to ease the feature extraction procedure. In an ideal handwriting model, the words are supposed to be written horizontally and with ascenders and descenders aligned along the vertical direction. In real data, such conditions are rarely respected. We use skew and slant correction so as to normalize the text line image [START_REF] Kimura | A lexicon directed algorithm for recognition of unconstrained handwritten words[END_REF] (See Figures 5 and6). A contour smoothing is then applied to eliminate small blobs on the contour. This is because GMM models cannot cope with high dimensional binary data.

The feature set is inspired by [START_REF] Al-Hajj | Combination of hmm-based classifiers for the recognition of arabic handwritten words[END_REF], and has shown its efficiency in the 2009 ICDAR word recognition competition [START_REF] Kessentini | Off-line handwritten word recognition using multi-stream hidden markov models[END_REF]. It is based on density and concavity features. It is made of 26 features:

-9 are baseline independent (for instance: black pixels density in the window and in every column of this window, position of the writing pixels) -17 are baseline dependent (for instance: black pixel density upon and under baselines, local pixels configurations regarding baselines, ...)

Models training

In this study, an independent training of the HMM and the DNN has been carried out, as discussed in section 3.1.3. The HMM training has been performed in a traditional way. We have considered N c = 71 characters models: 26 lower case letters, 26 capital letters, 10 digits, a space model, accented letters (é, è, ê, à) and punctuation marks (.,', -, /) in order to model the lines of text. All these character models have been trained with a standard GMM/HMM embedded training at the line level using each text line image generated from the ground truth location of the words in the document.

The hyper-parameters of the HMM have been experimentally determined in order to maximize the word recognition rate on a validation dataset. We found that the best combination of parameters was N s = 4 states per character and 5 gaussians per state, and that the best frame window width was N p = 8 pixels with an overlapping factor of N r = 5 pixels.

We now provide the implementation details for training the networks, and especially the DNN architecture. Our implementations were carried out on a standard desktop computer using the Theano library [START_REF] Bergstra | Theano: a CPU and GPU math expression compiler[END_REF].

Data preparation:

The networks have to be trained so as to provide a posteriori probabilities of each character model state. A frame-labeled training database is therefore needed. It is build using a forced Viterbi alignment of the character sequence groundtruth on the image using a first GMM/HMM trained on the database. This frame labeling of the learning document dataset has led to more than one million frames.

The training of the network requires a huge amount of memory as it needs to store the network weights and the training samples. Therefore, the number of training samples has been limited to approximately 340.000 frames in order to make training affordable on standard desktop computer.

A common training issue with neural networks is that rare classes in the training dataset ('z', 'w', 'k' for Fig. 7 The DNN-432 architecture. The raw pixels from 54 × 8 pixels frames are taken directly as inputs. 3 model layers (resp. 400, 350 and 300 neurons) provide a high level representation of the data, while the last layer outputs the posteriors p(q i |o) for each of the 284 classes.

example for the french language) tend to be excluded from the network output decisions. To avoid this problem, the classes have been balanced to 1.200 samples per class. For classes with less than 1.200 samples, random duplication of samples was performed.

Features: The MLP and DNN have been trained using the same feature set as for GMM (26 features), and using a contextual feature set made of the features from three consecutive frames (3 × 26 = 78 features). The DNN has also been trained directly on pixels values from the whole frame, leading to a 54-pixels height × 8-pixel width = 432 feature set3 , in order to evaluate its ability to learn high level features from raw data. This configuration is illustrated on Figure 7. These networks are called thereafter DNN-26, DNN-78 and DNN-432, MLP-26 and MLP-78. The number of outputs of each network is the number of states of the HMM.

Sizing the networks: Optimizing the DNN architecture is not an easy task due to the lack of theoretical rules to set the hyper-parameters: number of hidden layers, and number of neurons for each hidden layer. The parameters are optimized by maximizing the word recognition rate of the whole system (network + HMM) over the RIMES 2009 word validation dataset (7000 words).

Regarding the choice of the optimal size of the network, several attempts have been made according to some experimental results found in the literature. The number of hidden layers of each DNN has been set to 3 as proposed in [START_REF] Bengio | Learning deep architectures for ai[END_REF]. We have also tested deeper architectures with 4 and 5 hidden layers, but the training was much longer, without any significant improvement of the error on the validation dataset. Therefore, we have kept for the remaining experiments a 3-hidden layer network.

The tuning of the number of neurons in the hidden layers is also difficult. In [START_REF] Larochelle | Exploring strategies for training deep neural networks[END_REF], three different strategies have been tested for recognizing handwritten digits from the MNIST database: the three architectures have increasing / constant / decreasing layer sizes from the bottom to the top layers, and the best results were obtained with layers of same size. We did not performed an extensive search, but for our problem the best results were achieved for a decreasing number of neurons (or increasing when the number of outputs is higher than the number of inputs, as it is the case for DNN-26 and DNN-78).

Finally, Table 1 shows the set of optimal hyperparameters for a three hidden layers network. Gradient parameters: Deep architectures have also a set of parameters controlling the training phase of the network, namely the weight decay parameter η and the stopping criterion. We classically used a weight decay that decreases with the number of iterations [START_REF] Larochelle | Exploring strategies for training deep neural networks[END_REF] according to the following equation :

Network N 0 (inputs) N 1 N 2 N 3 N 4 (
η n = η 0 n × η0
10 + 1 where η n is the weight decay at iteration n, and η 0 is the initial weight decay, set to 0.001.

We have used a common stopping criterion that consists in evaluating the mean square error between the outputs of the network and its target (resp. itself for the model layers). Training is stopped when the error is lower than an ε value.

Concerning the stopping criterion devoted to the model layers, we have observed in our experiments that ε should not be too low in order to limit the overfitting of the model layers. A value of 0.01 has been retained 4 . 4 An alternative to this strategy is to use a small ε, with a maximum number of iterations, as it is the case in [START_REF] Bengio | Learning deep architectures for ai[END_REF] For the fine tuning (final backpropagation applied to the whole network), a very small improving ratio is more suitable, to prevent from stopping the training too early. An ε = 0.0001 has been chosen, coupled with a validation procedure to retain the best network.

Experiments and results

In order to compare the performance of our approach, several spotting systems have been evaluated: standard GMM-HMM, hybrid MLP-HMM, and DNN-HMM. These systems have been trained on the public RIMES database. We evaluate our approach on a multiple keywords spotting task.

RIMES database

The RIMES database includes 1150 french incoming mails (see Figure 8) from different writers [START_REF] Grosicki | Icdar 2009 handwriting recognition competition[END_REF], as well as 36000 annotated isolated words extracted from these mails. As the coordinates of the words inside the mail images were not available, we have recovered them using a simple pattern matching method. For our experiments, 950 mails have been used for training, 100 documents for validation and 100 for test.

Experimental protocol

In order to evaluate the keyword spotting system in a database of D documents, we compute recall and precision measures. Different operating points are obtained by balancing the competition between relevant and irrelevant information in the model through the variation of the hyper-parameter G (see Figure 2). It enables to describe a whole recall/precision curve: a value of G close to 1 gives an advantage to the relevant information at the expense of the rejection model and thus will favor recall against precision. On the contrary, values of G close to 0 will favor precision. From an applicative point of view, the user of the system may choose the suitable value of G given its application constraints.

R mean = 1 D d N ok (d) N (d) (8)
P mean = 1 D d N ok (d) N ok (d) + N f d (d) (9)
Computing statistically significant recalls and precisions requires to have enough keywords to spot in a document database. However, we have observed that the most frequent words do not exceed a few occurrences in the whole database. Therefore, spotting a lexicon of L keywords over the database would lead to a too small N (d) value to correctly estimate recall and precision. For that reason, we have turned towards the use of a document-based protocol.

The experimental protocol is the following: a Lkeyword lexicon is generated by picking up randomly 10 words among the words of the document to spot in. Small words with less than 5 characters are discarded. These 10 keywords are completed with L -10 words which are not present in the current document under study.

Multiple keywords spotting results

We present the spotting performance for three different keyword lexicon sizes: 10, 100 and 500. Figures 9,10 and 11 show the recall-precision results for a lexicon containing respectively 10, 100 and 500 keywords. A first comparison between the different systems show that the deep HMM system clearly outperforms the other classifiers, whatever the value of G. These experiments show that the behavior of the system is the same whatever the lexicon size: the neural based discrimination always provides better results than GMM. The deep HMM system provides the best results if used directly with raw pixels. A more compact presentation of our results is given in Table 2 for each system through the classical break even point evaluation measure corresponding to the operating point where Recall = Precision. The first column (lexicon size = 1) shows that as expected, spotting a single word leads to better retrieval results. Note that a simple MLP used with some context (previous and next frames) also performs well. As a conclusion, one can say that the deep architecture is not only able to handle high dimensional inputs, but is also able to learn smart discriminating features that provide better results than a dedicated feature set. An expected result is the behavior of the system when the lexicon size increases: the system performance decrease as the lexicon size increases since more false alarms are generated. A more remarkable result is that the system performance don't fall down when using a 500-word lexicon, allowing an efficient content-based categorization of document images, as shown on [START_REF] Paquet | A categorization system for handwritten documents[END_REF].

Fig. 12 Example of a multiple keywords spotting result. The lexicon is listed on the lower part of the figure, with two words that do not appear in the document ("lampadaire" and "novembre"). No confusion is made but one utterance of each word, Madame and facture, is missed. In this example, recall is 6/8 = 75% and precision is 8/8 = 100%

Conclusion

A multiple keywords spotting system in handwritten unconstrained documents has been introduced. It relies on a global modelling based on HMMs allowing a dual representation of the relevant and the irrelevant information. As we model an entire line of text, a global segmentation/recognition/rejection decision is taken at the line level, avoiding the word segmentation problems. In order to strongly discriminate the local information, an efficient deep neural network has been introduced. The decoding stage of the proposed framework is vocabulary-independent, meaning that the keyword lexicon is independent of the training process and can be set given the application needs.

Results on the RIMES database illustrate the power of the combination scheme. The best results are obtained with the deep architecture trained on raw data, which demonstrates the ability of this architecture to automatically infer high level representation of handwriting.

Future works might investigate alternative learning strategies, such as HMM discriminative learning or Conditional Random Fields (CRF), to propose a fully discriminant method, from the input data to the sequence label. Knowing the language of the documents, possible improvement might also result in the use of language models in order to enhance the performance. Regarding the target application, the proposed approach could also be extended to the search for regular expressions including joker characters ' ?' and ' ', for example by searching for the sequence "canc " in order to spot any of the words "cancel", "cancellation" or "cancelled". We believe it could be of interest for applications such as automatic document categorization or indexation.

Fig. 1

 1 Fig.1Expected result for the spotting of the word "commande" in a french handwritten mail.

Fig. 2

 2 Fig.2Global line model containing the keywords lexicon KL (upper part), and the filler model used for modeling everything else (Out of Keyword Lexicon OKL, on the lower part). The line model is also made of a space model ("SP"), and structural initial and final models ("Ei" and "Ef").

Figure 2

 2 Figure 2 represents our model for modeling the two kinds of information that can be encountered within a text line. The Keyword Lexicon (KL) is represented in the upper part of the scheme. It is a parallel model of keywords represented as character sequences that are equally likely to occur in the text. The irrelevant information is represented in the lower part of the Figure. It is used to model any word that does not belong to the Keyword Lexicon, namely the Out of Keyword Lexicon (OKL).It is an ergodic model that accounts for any sequence of alphanumeric characters, including uppercase and lowercase letters, digits and punctuation. Keywords in KL may occur with a lower frequency than words in OKL. This is why the two models compete in parallel in the line model, with parameter G standing for the proportion of relevant information that can appear in the text. Finally, the line model concatenates any KL or OKL words with a white space model before going to the next possible word in the line through the loopy backward transition. The model loops until the end of line is reached. Let us emphasize that as the model simulates a competition between the OKL and each word of the KL, it prevents from the conflicts that may occur between two keywords, or between a keyword and the OKL. It also avoids the use of numerous thresholds for accepting/rejecting the keywords hypotheses. The setting of parameter G is discussed in Section 5.[START_REF] Choisy | Dynamic handwritten keyword spotting based on the nshp-hmm[END_REF] As one can see, the handwritten modeling is text line oriented, and fits particularly well with the gen-

Fig. 3

 3 Fig. 3 left: A standard MLP with one hidden layer and one output layer. right: A Deep Neural Network, with Λ -1 model layers (trained in an unsupervised way) and a final decision layer (trained in a supervised way).

Fig. 5 Fig. 6

 56 Fig. 5 Deskew correction

Fig. 4

 4 Fig.4(i) An auto-associator (e, d) is learnt on the input x. The encoder function e has been stacked to produce a hidden representation H 1 of the data, while d is dropped. (ii) A new auto-associator is then learnt on the hidden units H 1 to produce Ĥ1 . (iii) This process is repeated to get a compressed representation of the data. (iv) At last, a decision layer is added and a standard supervised gradient back-propagation is applied on the whole network.

  Given a document d, let N (d) be the number of keywords to spot, N ok (d) be the number of correctly spotted keywords in this document, and N f d (d) the number of false detections. Recall and precision means (respectively R mean and P mean ) for a whole database containing D documents are computed as follows (equations 8

Fig. 8

 8 Fig.8Incoming mails from RIMES database[START_REF] Grosicki | Icdar 2009 handwriting recognition competition[END_REF].

Fig. 9

 9 Fig.9Recall-precision results for all combinations with a 10keyword lexicon size.

Fig. 10

 10 Fig.10Recall-precision results for all combinations with a 100keyword lexicon size.

Fig. 11

 11 Fig.11Recall-precision results for all combinations with a 500keyword lexicon size.

Table 1

 1 Number of neurons N i for each hidden layer h i for the three DNN and the two MLP: DNN-26 and MLP-26 are trained on the 26 features set, DNN-78 and MLP78 are trained on 3*26 features set over frames at t -1, t and t + 1, and DNN-432 is trained on the raw pixels. The number of outputs of each network is the number of states of the HMM: 4 states * 71 characters = 284.

	outputs)

Table 2

 2 Break even point for different keyword lexicon sizes, for all the systems.

Note that a Restricted Boltzman Machine (RBM) can also be used, in this case they are called Deep Belief Networks[START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF].

A vertical scaling procedure is applied to normalize the height of the text line image to h = 54.


Exploring multiple feature combination strategies with a recurrent neural network architecture for off-line handwriting recognition

Keywords: Neural networks, Recurrent networks, Feature combination, System Combination, Handwriting recognition

The BLSTM-CTC is a novel recurrent neural network architecture that has outperformed previous state of the art algorithms in tasks such as speech recognition or handwriting recognition. It has the ability to process long term dependencies in temporal signals in order to label unsegmented data. This paper describes different ways of combining features using a BLSTM-CTC architecture. Not only do we explore the low level combination (feature space combination) but we also explore high level combination (decoding combination) and mid-level (internal system representation combination). The results are compared on the RIMES word database. Our results show that the low level combination works best, thanks to the powerful data modeling of the LSTM neurons.

INTRODUCTION

Handwriting recognition deals with the extraction (localization and recognition) of words from images containing a graphical representation of these words. The recognition of offline handwriting is difficult due to the variability between writers for a same word, presence of overlapping letters, and complex long term context dependencies. However, handwriting systems have been successfully applied to different tasks, such as bank cheque recognition [START_REF] Knerr | The A2iA intercheque system: courtesy amount and legal amount recognition for French checks[END_REF] and postal address recognition. [START_REF] El-Yacoubi | Conjoined location and recognition of street names within a postal address delivery line[END_REF] Indeed bank cheques and addresses follow a strict format and have a very limited vocabulary : the prior knowledge on these task is extensively used to boost the recognition performances. However, the recognition of unconstrained handwriting that can occur in various documents, such as letters, books, notes, is a very complex task. Exploring recognition in context free systems is now a main interest of researchers.

A simple handwriting system is generally divided in four steps : preprocessing, feature extraction, recognition and post-processing. In this paper we focus on the recognition stage, for which the Hidden Markov Models 3 (HMM) have been massively used. [START_REF] Kundu | Recognition of handwritten word: first and second order hidden Markov model based approach[END_REF][START_REF] El-Yacoubi | An HMM-based approach for off-line unconstrained handwritten word modeling and recognition[END_REF][START_REF] Bunke | Offline recognition of unconstrained handwritten texts using HMMs and statistical language models[END_REF] HMMs are states machines that combine two stochastic processes: a stochastic event that cannot be observed directly (the hidden states) is observed via a second stochastic process (the observations).

These models have several advantages. First they provide a way of breaking out of Sayre's paradox. [START_REF] Vinciarelli | A survey on off-line cursive word recognition[END_REF] Indeed recognizing a letter in a word image requires its prior detection. But the reverse is also true, thus leading to an ill posed problem known as Sayre's paradox in the literature. Methods using HMMs break up word images in atomic parts, either as graphemes or frames extracted using a sliding window. This will cause a difference between the input sequence length and the output sequence length. However, HMMs are able to cope with this difference in length, they can label unsegmented sequences. They do so by recognizing every window as a character or part of a character and modeling character length. A second advantage of HMMs is their robustness to noise since they are statistical models. Third the algorithms used for HMMs decoding integrate language modeling, lexicon check or N-gram models, which makes them very powerful tools for handwriting recognition.

However, they also have some shortcomings. First of all, they are generative systems, which means that when compared to other classifiers, they have a lesser ability to discriminate between data since they provide a likelihood measure to decide . Secondly, within the HMMs framework the current hidden state only depends on the current observation and the previous state (markovian condition) which prohibits the modeling of long term dependencies.

Recently a new system based on recurrent neural networks 8 has overcome these shortcomings. The Bidirectional Long Short Term Memory neural network 9, 10 (BLSTM) consists in combining two recurrent neural networks with special neural network units. The output of these networks are then processed by a special Softmax layer, the Connectionist Temporal Classification 11 (CTC), that enables the labelling of unsegmented data. The composed system, referred as the BLSTM-CTC, has shown very impressive results on challenging databases. [START_REF] Graves | Supervised sequence labelling with recurrent neural networks[END_REF][START_REF] Graves | A novel connectionist system for unconstrained handwriting recognition[END_REF][START_REF] Grosicki | ICDAR 2009 Handwriting Recognition Competition[END_REF][START_REF] Menasri | The A2iA French handwriting recognition system at the Rimes-ICDAR2011 competition[END_REF] In this paper, we present two baseline systems using the same architecture, built around the BLSTM-CTC network. These systems only differ by the features that are extracted using a sliding window method. These baseline systems enable a direct comparison between the two sets of features, which enables us to prefer one set of features over the other. However, it is well known that combining systems or features may improve the overall results. [START_REF] Menasri | The A2iA French handwriting recognition system at the Rimes-ICDAR2011 competition[END_REF][START_REF] Gehler | On feature combination for multiclass object classification[END_REF] In this paper we explore different ways of combining an ensemble of F BLSTM-CTC. We explore different levels of information combination, from a low level combination (feature space combination), to mid-level combinations (internal system representation combinations), and high level combinations (decoding combinations). The experiments are carried out on the handwriting word recognition task WR2 of the ICDAR 2009 competition. [START_REF] Grosicki | ICDAR 2009 Handwriting Recognition Competition[END_REF] We first present the baseline system used for handwriting recognition. We then detail the different level of combinations available throughout the BLSTM architecture, finally we present the results and analyse them.

THE BASELINE PROCESSING SYSTEM

In this section we present the baseline system we use to recognize handwritten word images. We first give an overview of our system, we then describe each step with further detail.

Overview

Our system is dedicated to recognizing images of isolated handwritten words. In order to do so we implemented a very common processing workflow that is represented in Fig. 1.

First the image is preprocessed in order to remove noise and normalize character appearance between different writing styles. Given that our main interest is the combination of systems we did not focus on the preprocessing, hence we used well known algorithms that have been widely used in the literature. After cleaning the image we use a sliding window to extract the features : a window of length P is used to extract a subframe from the image, this subframe is then analyzed and transformed by mathematical processes into a M dimensional vector representation. For instance a 2D image of length N is transformed into a sequence of N feature vectors of dimension M . This sequence is then processed by a BLSTM-CTC network, in order to transform this M dimensional signal into a sequence of labels (characters). This output is finally processed by a HMM in order to apply a directed decoding. It has to be stressed that it is possible to add a lexicon to the BLSTM-CTC [START_REF] Graves | A novel connectionist system for unconstrained handwriting recognition[END_REF] decoding stage. However, we did not opt for this solution since the HMMs enable us to be more modular, e.g. offering the possibility to integrate language models with various decoding strategies available on standard decoding platforms such as HTK, [START_REF] Young | The HTK book[END_REF] Julius 18 or Kaldi, [START_REF] Povey | The kaldi speech recognition toolkit[END_REF] without deteriorating the performances.

We now go into further details for each part.

Preprocessing

The preprocessing of images consists in a very simple three step image correction. First a binarization by thresholding is applied. Secondly a deslanting process 20 is applied. The image is rotated between [-Θ; Θ] and for each angle θ the histogram of vertical continuous traits H θ is measured. The deslanting angle is determined by the H θ with the maximum number of continuous traits. Thirdly a height normalization technique is applied to center the baseline and normalize the heights of the ascenders and descenders.

Feature extraction

Among numerous feature representations, we selected two efficiency proven features : the features presented in [START_REF] Graves | A novel connectionist system for unconstrained handwriting recognition[END_REF] and the Histogram of Oriented Gradients [START_REF] Dalal | Histograms of Oriented Gradients for Human Detection[END_REF][START_REF] Ait-Mohand | Combining structure and parameter adaptation of HMMs for printed text recognition[END_REF] (HOG). The two feature sets are extracted using a sliding window method on the images.

The features presented in [START_REF] Graves | A novel connectionist system for unconstrained handwriting recognition[END_REF] are referred as Simple Pixel Features (SPF). It is a very simple set of features, based on a spatial representation of a single colon of pixels: the mean grey value of the pixels, the position of the center of gravity, the second order vertical moment of black pixels, the positions of the uppermost and lowermost black pixels, the rate of change of these positions (with respect to the neighbouring windows), the number of black-white transitions between the uppermost and lowermost pixels, the proportion of black pixels between the uppermost and lowermost pixels. SPF are a low level feature set, very basic information is extracted from the pixels. It mainly compresses the information for faster processing. We decline this descriptor over three colons of pixels, hence composing a vector of 3 × 9 = 27 dimensions.

HOG features are extracted from windows of 8 × 64 pixels. HOG describes the window by dividing it into sub-windows of n × m pixels. Within each sub-window a histogram is computed, it describes the distribution of the local intensity gradients (edge direction). The histograms from all sub-windows are then concatenated to form the final representation. We used 8 non-overlapping sub-windows of 8 × 8 using 8 directions, hence composing a feature vector of 64 dimensions. HOG features are medium level features, they are more complex features than SPF, they describe the local context of the window.

BLSTM-CTC

After extracting the features from a database (RIMES database) they are then independently used to train a BLSTM-CTC network. The network transforms a sequence of unsegmented data into a one dimensional output vector, e.g. it transforms a sequence of SPF features of length N into a word of length L. We first present the BLSTM network and then the CTC.

BLSTM network

The BLSTM is a complex memory management network, it is in charge of processing a N dimensional input signal to produce an output signal that takes into account long term dependencies. In order to do this the BLSTM is composed of two recurrent neural networks with Long Short Term Memory neural units.

One network processes the data chronologically while the other processes the data in reverse chronological order. Therefore at time t a decision can be taken by combining the outputs of the two networks, using past and future context. For handwriting recognition having a certain knowledge of previous and future possible characters is important since in most cases characters follow a logical order induced by the underlying lexicon of the training database. Instead of modeling this order at a high level using N-grams they are integrated at a low level of decision inside the BLSTM networks. Moreover, handwritten characters have various length which can be modeled efficiently by the recurrent network.

These networks integrate special neural network units : Long Short Time Memory 11 (LSTM). LSTM neurons consist in a memory cell an input and three control gates. The gates control the memory of the cell, namely : how an input will affect the memory (input gate), if a new input should reset the memory cell (forget gate) and if the memory of the network should be presented to the following neural network layer (output gate). The gates enable a very precise and long term control of the memory cell. Compared to traditional recurrent neural network layers, LSTM layers can model much longer and more complex dependencies. A LSTM layer is a fully recurrent layer, the input and the three gates receive at each instant t the input at time t from the previous layer and the previous output t -1.

The combination of the bidirectional networks with LSTM units enables the BLSTM to provide a complex output taking into account past and future long term dependencies.

CTC layer

The CTC is a specialized neural network layer dedicated to transforming BLSTM outputs into class posterior probabilities. It is designed to be trained using unsegmented data, such as handwriting or speech. It is a Softmax layer [START_REF] Bishop | Neural networks for pattern recognition[END_REF] where each output represents a character, it transforms the BLSTM signal into a sequence of characters. This layer has as many outputs as characters in the alphabet plus one additional output, a "blank" or "no decision" output. Therefore it has the ability to avoid taking a decision in uncertain zones instead of continuously being forced to decide on a character signal in a low context area (e.g. uncertain).

The power of the CTC layer is in its training method. Indeed for handwriting recognition, images are labelled at a word level, therefore making it impossible to learn characters individually. The CTC layer provides a learning mechanism for such data. Given a word, e.g. "hand", first this word is transformed to integrate the blank label, in order to learn to model this label. Using an operator B the blank label is inserted between each character and at the beginning and end of a word w of length L. For instance the word "hand" is replaced by " h a n d " where " " represents the blank label.

Unsegmented data has a disparity between the input sequence length N and the replaced word length 2L + 1, with N ≥ 2L + 1. In order to train the CTC network an output has to be provided for the complete input sequence. Inspired by the HMM algorithms, the CTC uses an objective function integrating a forward and backward variable to determine the best path through a lattice of possibilities. This lattice describes all possible transitions in a word, hence representing all paths π 1:T possible over time. The authorized transitions are from a character to a blank, from a character to another character, staying on the same character or on the blank. The forward and backward variable are calculated using the authorized transitions. The forward variable is defined as the summed probability of all paths ending in position s of the modified word at time t in the sequence:

α t (s) = B(π1:t) t t ′ =1 y t ′ π t ′ , (1)
with y t ′ π t ′ the output of the CTC layer at time t ′ . The backward variable is defined as the summed probability of all paths beginning in position s of the modified word at time t in the sequence:

β t (s) = B(πt:T ) T t ′ =t+1 y t ′ π t ′ . (2)
These variables enable the calculation of all possible outputs at all times, hence covering the disparity of length between the input and output signal. Given a sequence x, a label output l, the label sequence probability at time t is defined as:

p(l|x) = |B(l)| s=1 α t (s)β t (s) (3)
An objective function can then be calculated to backpropagate the error through the CTC and the BLSTM network using the Back Propagation Through Time 24 algorithm.

After being trained the BLSTM-CTC is able to output for an unknown sequence x a label output l ′ .

HMM

In order to perform lexicon directed recognition we use a modeling HMM. This stage is usually performed by the CTC using a Token Passing Algorithm. [START_REF] Graves | A novel connectionist system for unconstrained handwriting recognition[END_REF] However, we substitute this correction step by a modeling HMM, enabling us to have more flexibility in regards of the decoding strategy applied without deteriorating the results.

The BLSTM-CTC character posteriors substitute the Gaussian Mixture Models 25 (GMM), prior to this step we simplify the outputs of the CTC. As said previously the blank output covers most of the response of a CTC output signal, whereas character labels only represent spikes in the signal. We therefore remove all blank outputs above a certain threshold, this enables us to keep some uncertain blank labels that may be used for further correction by the HMM lexicon directed stage. This new CTC output supersedes the GMMs used to represent the underlying observable stochastic process. Subsequently the HMM use a Viterbi lexicon directed decoding algorithm (implementing the token passing decoding algorithm) and outputs the most likely word among a proposed dictionary.

SYSTEM COMBINATION

In the previous section we described the BLSTM-CTC neural network. Our main interest in this paper is to combine F feature representations in a BLSTM-CTC system to improve the recognition rate of the overall system. The BLSTM-CTC exhibits three different levels at which we can combine the features. 1-low level combination can be introduced through the combination of the input features. 2-mid level combination can be introduced by combining the BLSTM outputs into one single decision stage. This combination strategy assumes implicitly that each BLSTM provides specific features on which an optimized CTC based decision can take place. 3-finally a high level combination at the CTC decoding stage using the direct output of this level. Fig. 2 represents all the combinations we explored. This section describes the different combinations we propose using F different feature sets.

Low combination

First we consider the feature combination. The feature vectors of the different representations are concatenated into one unique vector, this method is also known as early integration. Combining at this level has the advantage of being the most straightforward method.

Mid level combination

The mid level combination concerns the BLSTM level combination. This method is inspired by previous work on deep neural networks, especially auto-encoders [START_REF] Bengio | Learning Deep Architectures for AI[END_REF] and denoising auto-encoders. [START_REF] Vincent | Extracting and composing robust features with denoising autoencoders[END_REF] An ensemble of F BLSTM-CTC is first trained on the F different feature representations. Then the CTCs are removed, hence removing the individual layers that transform each feature signal into a label sequence. The remaining BLSTMs output are then combined by training a new architecture containing a CTC. The BLSTMs weights stay unchanged during the training of the new architecture, we consider they are high level feature extractors that are independently trained to extract the information from each feature set.

We consider the addition of two different architectures for this level of combination. First a single CTC layer is added, this CTC level is a simple feature combination layer. Second a complete BLSTM-CTC architecture is added, hence building a new feature extractor using the previously acquired context knowledge as an input.

High level combination

Lastly we consider the combination of the CTC outputs, a late fusion stage. An ensemble of F BLSTM-CTC is first trained on the F different feature representations. The outputs of each BLSTM-CTC are then combined to form a new output signal. We propose five different combinations for the outputs : averaging outputs, averaging outputs and training a new BLSTM-CTC, combination of outputs and training a new BLSTM-CTC, selecting the maximum output at all times, selecting the maximum output and training a new BLSTM-CTC. The first combination is a simple average of outputs at every instant. The second combination adds a BLSTM-CTC in order to measure the ability of the BLSTM-CTC to learn and correct long term dependencies errors at a character level. The third combination is a simple concatenation of the F CTC outputs at every instant. Therefore creating a new output representation, if we consider an alphabet A of size |A|, the new representation is of dimension (|A| + 1) × F . This new output representation is largely superior to the HMM input dimensions capacity. Hence we use a BLSTM-CTC to learn and correct errors as well as reduce the signal dimensionality. The fourth combination selects at every instant the maximum output between the F output labels, the results The different combination levels previously presented all have theoretical advantages and drawbacks. We now present the experimental results and we explain why certain combinations outperform the others.

EXPERIENCES

In this section we describe the database and recognition the task we used to compare the various combination performed on the BLSTM-CTC.

Database and task description

In order to compare our results we use the RIMES database. This database consists in handwritten letters addressed to various administrations and registration services. From these letters, single words were isolated and annotated. The database is divided in three different subsets : training, validation and test. Each set contains respectively 44197 images, 7542 images and 7464 images. The RIMES database was built to be a low noise database, i.e. the background and foreground pixels are easily identifiable. However, the intraword variation is very important since 1300 different writers participated for writing the letters. The word length distribution among the three sets is very similar. It has to be stressed that the words with three and less characters contribute to more than 40% of the database. This will affect the HMM lexicon correction since it is harder to correct short words.

We compare our results on the ICDAR 2009 WR2 task. [START_REF] Grosicki | ICDAR 2009 Handwriting Recognition Competition[END_REF] This task consists in finding the correct transcription of a word using a lexicon of 1600 words. The measure used is the Word Error Rate (WER) on the Top 1 outputs and Top 10 outputs.

In order to learn the system we use the whole training database to train each part of the system. This may cause some overfitting, however splitting the training base in order to train each system on separate bases causes some global loss on the learning performances.

Results

We now present the results of the various combination schemes on the RIMES database for BLSTM-CTC feature combination. Table 1 presents the results of the different combinations by displaying the top 1 and top 10 Word Error Rate (WER). We also provide the raw WER output of the last CTC layer prior to the HMM layer, i.e. the system without the lexicon decoding. The best combination is the low level feature combination. The reason is that the BLSTM-CTC is able to model very efficiently the data. In a very similar fashion to deep neural networks, it models very well the spatial and time domain dependencies of features. In order to combine several features using a BLSTM-CTC it is therefore preferable to directly combine the features. Table 2 demonstrates the differences between the input and output of the two baseline systems and this combination. These examples show that this combination learns the best and worst of both features, improving the overall results but some errors are kept or added. An interesting result concerns the mid levels combinations. The raw WER points out to the fact that the best mid level combination is by simply adding a new CTC. However, after applying the HMM the combination using a BLSTM-CTC to combine outputs bests the combination by CTC. Therefore, the combination by CTC corrects some words but adds noise on other, hence it hinders the HMM lexicon directed decoding. Relearning some long term dependencies (using a BLSTM-CTC) improves by 0.2 the recognition after the HMM lexicon directed decoding.

The outcome of the high level combinations is more contrasted. On the one hand the first three and the last high level combination deteriorate the results, they inject noise in the outputs hindering the directed decoding.

The averaging in particular is notably bad, the HMM lexicon directed decoding is unable to correct this combination of features. This can be explained by the CTC output signal : most of the time the blank label output is on a plateau of value 1, while all other outputs (labels) are close to null. As soon as the BLSTM-CTC has collected enough information to be certain that a character is present, then the CTC output of this character spikes to 1 and all other outputs become 0. This behavior can be explained by the modification of the training target. Inserting blanks between each character of the real target produces this spiked signal. However, the two systems may not output the labels at the same timestep, therefore a good output label may be lost after the averaging stage.

Relearning long term dependencies from the averaging slightly improves the results, but not enough to be of interest. The BLSTM-CTC is able to learn some information post averaging, but not enough to improve the overall performance.

On the other hand maximum selection and normalization improves the results. In this case the signal produced by this stage includes much weaker spikes on uncertain zones, compared to the spiked output of the baseline systems. As can be seen on Fig. 3, the output signals of this combination seems much more balanced among classes on a given timestep. Thus, producing more uncertainty between characters at any time, enabling the HMM lexicon decoding to explore more paths.

As a final note it is interesting to see that system 3.3 is not able to outperform a maximization and normalization step, we can therefore deduce that the BLSTM-CTC cannot build an internal representation of such a function.

CONCLUSION

In this paper we presented different combinations of feature representations using a BLSTM-CTC. The best result is achieved using a low level feature combination(early integration), indeed the internal spatial and time modeling ability of the BLSTM network is very efficient. A second system outperforming the baseline systems, only when using a lexicon directed decoding, is achieved by a maximum selection and a normalization step. However, a mid level combination is also quite interesting, as we mentioned previously we did not relearn the internal weights of the BLSTM after adding the new output layer, future work will explore this possibility as well as adding more features. Indeed adding too many features may lead to a saturation of a single BLSTM-CTC, hence training multiple BLSTM layers with low dimensional inputs may prove a better solution than working with high dimensional input. Future work may also investigate the combination of combinations. As we can see from Fig. 3 and Table 2 different combinations produce different outputs, especially after a lexicon directed decoding, therefore a combination of these two steps may improve further the results.

Figure 1 .

 1 Figure 1. The baseline system

Figure 2 .

 2 Figure 2. The different BSTM-CTC combinations

Table 1 .

 1 Results of the different BLSTM-CTC combination experiments on the RIMES database using Word Error Rate

	System	Raw WER Top 1 Top 10
	Single features :
	HOG	39.91	14.31	4.95
	SPF	45.44	17.95	7.01
	1-Low Level combinations	33.93	12.19	3.6
	2-Mid level combinations :
	2.1-Output combination and CTC	37.75	14.27	6.16
	2.2-Output combination and BLSTM-CTC	38.62	14.06	5.48
	3-High Level combinations :
	3.1-Averaging	94.2	48.76	17.72
	3.2-Averaging and BLSTM-CTC	40.4	23.86	6.9
	3.3-Output combination and BLSTM-CTC	43.94	26.05	9.28
	3.4-Maximum and normalization	60.0	13.96	4.1
	3.5-Maximum, normalization and BLSTM-CTC	38.2	15.2	6.2

Table 2 .

 2 Examples of different outputs between the baseline systems, the low level combination system and the maximum system

	Image	Ground truth HOG output SPF output Low	level	Maximum and
					combination	normalization
					output	output
		suite	autre	vente	suite	attente
		prie	prie	plume	prie	prie
		je	je	Je	Je	Je
		de	je	de	de	de
		un	rue	un	une	un