Solène Tarride

Aurélie Lemaitre

Bertrand Coüasnon

Sophie Tardivel

A comparative study of information extraction strategies using an attention-based neural network

Keywords: Document image analysis, Historical documents, Information extraction, Handwriting recognition, Named entity recognition

This article focuses on information extraction in historical handwritten marriage records. Traditional approaches rely on a sequential pipeline of two consecutive tasks: handwriting recognition is applied before named entity recognition. More recently, joint approaches that handle both tasks at the same time have been investigated, yielding state-of-the-art results. However, as these approaches have been used in different experimental conditions, they have not been fairly compared yet. In this work, we conduct a comparative study of sequential and joint approaches based on the same attention-based architecture, in order to quantify the gain that can be attributed to the joint learning strategy. We also investigate three new joint learning configurations based on multi-task or multi-scale learning. Our study shows that relying on a joint learning strategy can lead to an 8% increase of the complete recognition score. We also highlight the interest of multi-task learning and demonstrate the benefit of attention-based networks for information extraction. Our work achieves state-of-the-art performance in the ICDAR 2017 Information Extraction competition on the Esposalles database at line-level, without any language modelling or post-processing.

Introduction

In recent years, many European libraries, museums and archives have undertaken to digitize their collections of historical documents [START_REF] Nauta | Europeana DSI 2-Access to Digital Resources of European Heritage -Report on[END_REF], as a way to ensure the preservation of our cultural heritage. Indeed, digital documents do not need to be physically handled, thus reducing the potential damage to fragile collections, and can be quickly and easily accessed from any location. However, searching for a specific document among millions of digitized entries remains a challenge. As a result, there is a need for data indexation, which would allow users to browse a collection and retrieve specific documents using keyword queries. A practical application is the search of specific information contained in population records. These documents hold a strong value for genealogists around the world, as they provide precious information about our ancestors. The ability to search for a specific name in a collection of records would substantially ease genealogical research. Additionally, a macro-analysis of such documents could provide an interesting perspective for historians and demographers.

This raises the question of how to extract information from large collections of historical records. Several platforms have been proposed for collaborative manual annotation, such as Transcribathon3 and CrowdHeritage 4 . Yet, even with collaborative tools, manual extraction of information from large collections still requires a lot of effort. Recently, the advances in computer vision have opened the way for automatic document understanding using computational models. These systems can be trained to recognize a sequence of handwritten characters, and to assign a semantic label to each predicted word. However, these approaches are not entirely reliable when applied to challenging documents, such as historical documents. Indeed, historical records can have various layouts and handwriting styles, and often feature paper and ink degradations, abbreviations, inter-line annotations, or crossed-out words.

The interest for the task of information extraction in historical documents has been reinforced by the 2017 Information Extraction in Historical Handwritten Records (IEHHR) competition [START_REF] Fornés | ICDAR2017 Competition on Information Extraction in Historical Handwritten Records[END_REF], based on Esposalles database [START_REF] Romero | The ESPOSALLES database: An ancient marriage license corpus for off-line handwriting recognition[END_REF]. The database is a collection of historical handwritten marriage records from the Archives of the Cathedral of Barcelona from the 17th century. The aim of the competition is to extract relevant information about the wife and husband and their parents, such as their name, occupation and place of origin, as illustrated in Fig. 1.

Fig. 1. Example from the database proposed for the IEHHR competition [START_REF] Fornés | ICDAR2017 Competition on Information Extraction in Historical Handwritten Records[END_REF]. Each image is associated with a transcription, and two semantic labels are associated with each word: category and person.

Most of the approaches submitted to this competition are based on a sequential approach, in which the handwriting recognition task is performed upstream of the named entity recognition task [START_REF] Fornés | ICDAR2017 Competition on Information Extraction in Historical Handwritten Records[END_REF]. However, several researchers have highlighted the possibility of combining these two tasks, by training a model to output characters and contextual tags [START_REF] Carbonell | Joint Recognition of Handwritten Text and Named Entities with a Neural End-to-end Model[END_REF][START_REF] Rouhou | Transformer-based approach for joint handwriting and named entity recognition in historical document[END_REF]. These joint approaches yield competitive results, however it is impossible to assert whether the improvement comes from the architecture used by the authors or from the joint learning strategy.

In this work, we conduct a comparative study of these two training strategies using the same attention-based model, in order to quantify the improvement linked to the joint learning strategy. We also introduce three additional joint learning configurations based on multi-task and multi-scale learning.

This article is organized as follows. Section 2 introduces the works related to automatic information extraction and handwriting recognition in historical documents. Section 3 introduces the attention-based sequence-to-sequence architecture proposed in this work. Section 4 presents our comparative study and introduces three joint learning strategies based on multi-task and multi-scale learning. The results of our experiments are presented and discussed in Section 5. Finally, we summarize this work and propose future directions.

Related Works

Automatic document understanding is a challenging research area that includes document layout analysis, handwriting recognition, and information extraction. As we work on pre-segmented line images, we focus this study on handwritten text recognition (HTR) and information extraction (IE).

Handwriting recognition

State-of-the art HTR models are currently based on deep neural networks. One of the most popular architectures is composed of a convolutional neural network (CNN) to extract features from the image, and a recurrent neural network (RNN) to capture sequential information. This architecture is often referred to as CRNN-CTC, as it is trained using the Connectionist Temporal Classification (CTC) loss function [START_REF] Graves | Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks[END_REF]. This architecture has become very popular over the recent years [START_REF] Bluche | Gated Convolutional Recurrent Neural Networks for Multilingual Handwriting Recognition[END_REF][START_REF] Graves | A Novel Connectionist System for Unconstrained Handwriting Recognition[END_REF][START_REF] Puigcerver | Are Multidimensional Recurrent Layers Really Necessary for Handwritten Text Recognition[END_REF]. More recently, attention-based sequence-to-sequence (seq2seq) networks have been investigated for HTR. Contrary to the CRNN-CTC architecture, attention-based models learn to align image pixels with the target sequence. As a result, the network learns to focus on a small relevant part of the feature vector to predict each token [START_REF] Michael | Evaluating Sequence to Sequence Models for Handwritten Text Recognition[END_REF][START_REF] Poulos | Character-Based Handwritten Text Transcription with Attention Networks[END_REF]. Another strength of this architecture is that the recurrent decoder learns an implicit language model at character-level. The seq2seq architecture has also demonstrated its ability to handle multiple tasks at once [START_REF] Luong | Multi-task Sequence to Sequence Learning[END_REF], which is interesting in the context of information extraction. Finally, the Transformer architecture is also gaining a lot of attention from the HTR community [START_REF] Kang | Pay Attention to What You Read: Non-recurrent Handwritten Text-Line Recognition[END_REF], although this architecture requires many training images.

Information extraction

The task of information extraction (IE) consists in extracting semantic information from documents. For structured documents, such as tables, forms or invoice, semantic information can be derived from the word localization. In this case, it is common to build end-to-end models that localize each word, transcribe them, and derive context from localization features [START_REF] Palm | Attend, Copy, Parse End-to-end Information Extraction from Documents[END_REF][START_REF] Yu | PICK: Processing Key Information Extraction from Documents using Improved Graph Learning-Convolutional Networks[END_REF]. But for semi-structured documents, such as marriage records, context can only be derived from phrasing. For example, a surname generally comes after a name. In this scenario, the first step is to localize textual zones, such as paragraphs, text-lines or words. Then, handwriting recognition and named entity recognition (NER) is performed.

Three main strategies have been considered so far. The traditional strategy is a sequential approach where handwriting recognition is performed before named entity recognition [START_REF] Fornés | ICDAR2017 Competition on Information Extraction in Historical Handwritten Records[END_REF]. Another sequential approach consists in classifying each word into semantic categories, then applying handwriting recognition techniques [START_REF] Toledo | Information extraction from historical handwritten document images with a context-aware neural model[END_REF]. The last strategy is a joint approach, where both tasks are tackled at the same time, using an end-to-end model [START_REF] Carbonell | A Neural Model for Text Localization, Transcription and Named Entity Recognition in Full Pages[END_REF][START_REF] Carbonell | Joint Recognition of Handwritten Text and Named Entities with a Neural End-to-end Model[END_REF].

Transcription before word semantic classification The most common approach relies on a HTR model to predict a transcription, and on natural language processing (NLP) techniques to classify each word into named entities using textual features. The drawback of this sequential approach is that there is no contextual information during the transcription stage. Rather, the context is used in the post-processing stage, using category-based language modelling. Several methods have been proposed in the ICDAR2017 competition on Information Extraction [START_REF] Fornés | ICDAR2017 Competition on Information Extraction in Historical Handwritten Records[END_REF]. The Hitsz-ICRC team developed an approach at word-level. First, a bi-gram based CNN is trained on word images to recognize characters, without any language model. Then, words are classified using the CRF sequence tagging method. The CITlab ARGUS team developed an approach at line-level. Images are passed into a CRNN-CTC architecture for handwritten text recognition, then regular expressions are used to decode and classify each word.

Word semantic classification before transcription This approach consists in labelling each word before predicting the transcription. The interest is that knowing the semantic category beforehand helps the system to predict the right transcription. It is particularly suited when dealing with word image, as each image can be easily classified before being transcribed. Toledo et al. [START_REF] Toledo | Information extraction from historical handwritten document images with a context-aware neural model[END_REF] extract semantic context from word images, and obtain the transcription of each word using semantic context of precedent words. The authors observe that their system benefits from knowing the semantic category. For example, if it makes sense to read a male name, the word "John" will be more likely than the word "born", even if the handwritten word looks more like "born". The main drawback of this approach is that it requires word bounding box annotation.

Joint transcription and word semantic classification This approach consists in producing a transcription and a semantic category at the same time. A method has been proposed in [START_REF] Carbonell | A Neural Model for Text Localization, Transcription and Named Entity Recognition in Full Pages[END_REF], where an end-to-end model is used to localize word bounding boxes and jointly classifying and transcribing them. A major drawback is that it requires word-level segmentation for training. Another approach based on line-level images [START_REF] Carbonell | Joint Recognition of Handwritten Text and Named Entities with a Neural End-to-end Model[END_REF] relies on a CRNN-CTC network to predict a transcription with semantic categories, using tags located before important words. If this approach is very promising, we believe that it could benefit from using an attention-based network, which would allow the network to learn the appropriate features to predict the tags. More recently, this joint approach has been successively used with a Transformer network at line-level and record-level [START_REF] Rouhou | Transformer-based approach for joint handwriting and named entity recognition in historical document[END_REF]. We believe that this combined approach is very promising, and benefits from an attention-based network to contextual features that are relevant to predict the semantic tags.

Our statement

In this work, we address different aspects of information extraction at line-level.

First, we want to assess the interest of the joint approach for information extraction. Our intuition is that combining handwriting recognition and named entity recognition should be helpful, as both tasks are related and could benefit from shared contextual features. Indeed, semantic context can be derived from the transcription, but knowing the semantic context beforehand should also make the transcription more reliable. Recently, competitive results have been achieved using this joint approach [START_REF] Carbonell | Joint Recognition of Handwritten Text and Named Entities with a Neural End-to-end Model[END_REF][START_REF] Rouhou | Transformer-based approach for joint handwriting and named entity recognition in historical document[END_REF], which indicates its relevance for information extraction. However, it is challenging to assert if the improvement comes from the neural network architecture or from the combination of HTR and NER. To validate this intuition, we compare sequential and joint approaches using the same neural network. Our study aims to measure the improvement that can be attributed to the learning strategy.

Secondly, we present a simple seq2seq architecture with attention, and demonstrate the interest of attention-based models for the task of information extraction. Indeed, the attention mechanism learns to focus on relevant zones of the image at each step, which should facilitate the recognition of named entities based on visual features. Moreover, this architecture can learn an implicit language model, which is convenient for semi-structured documents. To this end, we compare its performance with other neural network architectures, mainly CRNN-CTC [START_REF] Carbonell | Joint Recognition of Handwritten Text and Named Entities with a Neural End-to-end Model[END_REF] and Transformer [START_REF] Rouhou | Transformer-based approach for joint handwriting and named entity recognition in historical document[END_REF].

Finally, we investigate three additional joint learning strategies based on our seq2seq architecture using multi-task and multi-scale learning. Indeed, we believe that these strategies can be helpful for joint handwriting recognition and named entity recognition, as they have been proved efficient for neural translation and image captioning [START_REF] Luong | Multi-task Sequence to Sequence Learning[END_REF].

Our contributions are summarized as follows: -We show the benefit of joint strategies for information extraction and handwriting recognition. -We highlight the interest of attention-based models for joint handwriting and named entity recognition. -We propose three additional joint strategies for HTR and NER using multitask and multi-scale learning strategies.

-We obtain state-of-the-art results on the IEHHR competition benchmark at line-level [START_REF] Fornés | ICDAR2017 Competition on Information Extraction in Historical Handwritten Records[END_REF], without any post-processing.

The attention-based seq2seq architecture

The seq2seq architecture is an encoder-decoder network which was initially proposed for automatic translation [START_REF] Sutskever | Sequence to Sequence Learning with Neural Networks[END_REF]. It recently gained popularity for speech recognition, image captioning and neural translation [START_REF] Bahdanau | Neural Machine Translation by Jointly Learning to Align and Translate[END_REF][START_REF] Chorowski | Attention-Based Models for Speech Recognition[END_REF][START_REF] Xu | Show, Attend and Tell: Neural Image Caption Generation with Visual Attention[END_REF]. The architecture has since been adapted for HTR [START_REF] Michael | Evaluating Sequence to Sequence Models for Handwritten Text Recognition[END_REF][START_REF] Poulos | Character-Based Handwritten Text Transcription with Attention Networks[END_REF], as illustrated in Fig. 2. In this work, we use an architecture adapted from [START_REF] Michael | Evaluating Sequence to Sequence Models for Handwritten Text Recognition[END_REF]. Input Line images are pre-processed and augmented before entering the encoder. First, they are resized, padded and normalized. Then, random augmentations are applied on the fly, including DPI adjusting, random perspective, random transform, elastic distortion, dilation, erosion, sign flipping, brightness and contrast adjustment.

Encoder The role of the encoder is to extract features from the image. Our encoder is a convolutional recurrent neural network (CRNN) based on a ResNet-101 backbone, which is pre-trained on ImageNet [START_REF] Deng | Imagenet: A largescale hierarchical image database[END_REF]. Only the last two convolutional blocks are trained to reduce the number of training parameters. We add to this architecture a bi-dimensional Long Short Time Memory (BLSTM) network to capture the sequential information. The final feature vector is then fed to the attention mechanism and the decoder.

Attention The attention mechanism has been introduced by Bahdanau et al. [START_REF] Bahdanau | Neural Machine Translation by Jointly Learning to Align and Translate[END_REF] as a way to focus on parts of the feature vector, without any manual segmentation. For every step, the attention mechanism allows the network to rely on contextual features that are useful for the task at hand. Several attention mechanisms have been proposed over the years [START_REF] Bahdanau | Neural Machine Translation by Jointly Learning to Align and Translate[END_REF][START_REF] Chorowski | Attention-Based Models for Speech Recognition[END_REF][START_REF] Luong | Effective Approaches to Attention-based Neural Machine Translation[END_REF]. In this work, we use Chorowski attention (hybrid) [START_REF] Chorowski | Attention-Based Models for Speech Recognition[END_REF], as it yields better results for HTR [START_REF] Michael | Evaluating Sequence to Sequence Models for Handwritten Text Recognition[END_REF].

Decoder The role of the decoder is to generate a textual sequence from the attention-aware feature vector. The decoder is a simple LSTM cell trained with an embedding layer. For each time step, the output is passed through a softmax layer to obtain probabilities for each character of the alphabet.

Training settings The seq2seq is trained using the hybrid loss proposed in [START_REF] Michael | Evaluating Sequence to Sequence Models for Handwritten Text Recognition[END_REF], and using the Adam optimization algorithm. We use teacher forcing in the decoder during training. Early stopping is used to stop the training if no improvement is observed on the validation loss for 20 epochs. During inference, we use the best validation weights and beam search decoding with a beam size of 5.

Strategies for information extraction

In this section, we present the two main approaches compared in this work. We also introduce three additional learning strategies for joint handwriting and named entity recognition. No language modelling or post-processing is used in this work.

Comparing the sequential and joint approaches

We present the two main strategies for information extraction: the sequential approach and the joint approach. These two approaches are illustrated in Fig. 3 and will be compared using the same seq2seq architecture in Section 5.

(a) Sequential approach: the seq2seq network is used to predict characters and FLAIR is used to classify predicted words into named entities.

(b) Joint approach: the seq2seq network is used to predict characters and tags Sequential approach This strategy is the most popular approach for information extraction [START_REF] Fornés | ICDAR2017 Competition on Information Extraction in Historical Handwritten Records[END_REF]. Handwriting recognition and named entity recognition are addressed as separate subsequent tasks, as illustrated in Fig. 3a. Once handwritten text recognition is performed, each predicted word is classified into multiple semantic categories, based on textual features. Our implementation of this approach relies on our seq2seq network for handwriting recognition, and the FLAIR system [START_REF] Akbik | FLAIR: An easy-to-use framework for state-of-the-art NLP[END_REF] for named entity recognition, trained using catalan word embedding and FLAIR embedding.

Joint approach This approach, illustrated in Fig. 3b, was initially proposed in [START_REF] Carbonell | Joint Recognition of Handwritten Text and Named Entities with a Neural End-to-end Model[END_REF] using a CRNN-CTC neural network. In this strategy, the network is trained to predict tags as well as characters, with tags being located before each word of interest. In this work, we propose to reproduce this experiment using our seq2seq architecture trained to predict characters and tags. We believe that using an attention-based architecture is appropriate, as the network learns to focus on relevant parts of the feature vector to predict the tags. The tags are designed to encode the semantic category and the person relative to each word. We use the combined tags proposed in [START_REF] Carbonell | Joint Recognition of Handwritten Text and Named Entities with a Neural End-to-end Model[END_REF], e.g. <wife_name> is used to represent the wife's name.

Exploring additional joint learning configurations

Finally, we propose and evaluate three original joint configurations using multitask and multi-scale training. Our intuition is that learning multiple tasks, or from multiple scales, could help the encoder to extract richer contextual features.

Joint multi-task strategy without tags In this first learning configuration, three decoders are connected to the same encoder, as illustrated in Fig. 4a. Each decoder is trained for a specific task:

the htr decoder predicts the sequence of characters the category decoder predicts the sequence of categories (6 classes: name, surname, location, occupation, state, other) the person decoder predicts the sequence of persons (8 classes: wife, father, wife's father, wife's mother, husband's father, husband's mother, other person, none) Each decoder relies on a specific attention mechanism that focuses on relevant features for each subtask. The network is trained using a single loss function that is computed as the mean of the three individual loss functions. During inference, the sequences are merged to assign a category and a person to each predicted word. However, this approach has two possible issues. The first is a potential convergence issue, as the htr decoder typically learns slower than the other decoders. The second issue comes from the alignment of the predicted sequences: an error at the beginning of a sequence can potentially offset the entire prediction. Joint multi-task strategy with tags This second learning configuration is designed to overcome the issues of the last strategy. It is illustrated in 4b. Three decoders are connected to the same encoder, and each decoder is trained using a specific semantic encoding:

tag-category is a single tag for the category, e.g <name> tag-person is a single tag for the person, e.g <wife> tag-mixed is a combined tag for both category and person, e.g <wife_name> Each decoder relies on a specific attention mechanism that focuses on relevant features for each subtask. The network is trained using a single loss function that is computed as the mean of the three individual loss functions. During inference, only the tag-mixed branch is evaluated to ensure a fair comparison with the single-task joint strategy. In this scenario, the subtasks tackled by each decoder are balanced, which ensure proper convergence.

Joint multi-scale strategy with tags The multi-scale learning configuration is based on the joint strategy and is illustrated in Fig. 4c. In this scenario, the image is passed through the encoder at different scales to get contextual information at different levels. The three feature vectors are concatenated and passed through the attention network and the decoder.

Experiments

In this section, we evaluate our contributions for information extraction. First, we present results achieved by our attention-based network for handwriting recognition, showing that it is competitive. Then, we conduct a comparative study of sequential and joint approaches for information extraction based on our attention-based network, and evaluate three additional joint learning configurations. Finally, we compare this work with other participants in the IEHHR competition.

Handwriting recognition using seq2seq

First, we demonstrate the interest of our seq2seq architecture for HTR, as it is used for the sequential strategy. The evaluation is carried out on two public databases at line-level: IAM [START_REF] Marti | The IAM-database: An English sentence database for offline handwriting recognition[END_REF] (modern, English) and RIMES [START_REF] Augustin | RIMES evaluation campaign for handwritten mail processing[END_REF] (modern, French). We use the Character Error Rate (CER) as the evaluation metric. We compare our architecture with other state-of-the-art methods, without any postprocessing or language model, in Tab. 1. Our architecture yields state-of-the-art results on the IAM database. When comparing our seq2seq (encoder-decoder) with the corresponding CRNN-CTC (encoder-CTC) architecture, we observe that the seq2seq architecture is more efficient. Indeed, when using the attentionbased model, the CER drops from 6.1% to 5.2% on the IAM database, and from 5.2% to 4.4% on the RIMES database. This is likely due to attention mechanism and the implicit language model learned by the decoder. Consequently, we use the seq2seq architecture in the following experiments.

Information extraction using seq2seq

We analyse the four approaches described in Section 4 for information extraction on the Esposalles database. Then, we compare our best approaches with stateof-the-art methods.

Evaluation protocol Two tasks can be evaluated using the Esposalles database [START_REF] Romero | The ESPOSALLES database: An ancient marriage license corpus for off-line handwriting recognition[END_REF]. The first task is full text recognition, which is evaluated using the Character Error Rate (CER) and the Word Error Rate (WER). The second task is information extraction, which can be assessed using the metrics introduced in the ICDAR 2017 competition on Information Extraction in Historical Handwritten Records (IEHHR) [START_REF] Fornés | ICDAR2017 Competition on Information Extraction in Historical Handwritten Records[END_REF]. The objective of information extraction is to go beyond full text recognition, by assigning two semantic labels to each word. For the basic track, the aim is to recognize the category among name, surname, occupation, location, civil state and other. The score associated to the basic task is 100 -CER if the category is correctly identified, 0 otherwise. For the complete track, the role of the person must also be identified among: wife, husband, wife's father, wife's mother, husband's father, husband's mother, other person, none.

The score associated to the complete task is 100 -CER if both labels are correctly identified, 0 otherwise. It should be noted that only relevant words (e.g. not classified as other /none) are taken into account in this evaluation.

Comparing the sequential and joint approaches The results presented in Tab. 2 compare the two main strategies presented in this work. The results highlight the interest of the joint strategy for information extraction task, as it yields a 3% increase on the basic score and an 8% increase on the complete score of the IEHHR competition. We observe that the sequential approach tends to propagate and amplify transcription errors. Indeed, we have observed that words with errors are more likely to be misclassified: the complete score drops from 98.5% when FLAIR is applied to ground truth words to 86.7% when applied to predicted words. Another weakness of the sequential approach can be observed on the complete score. The score drops by 5% when the evaluation of persons is considered, suggesting that this task is particularly problematic for the FLAIR model at line level. Overall, the strength of the joint approach comes from the implicit knowledge regarding the context of each word during the prediction. We observe that this knowledge helps the handwriting recognition task, as the CER drops from 2.82% when predicting only characters to 1.81% when predicting characters and tags.

Evaluation of additional joint learning configurations

We observe that the multitask without tags approach obtains a high error rate, which confirms our intuition on the asymmetric convergence of the three branches: the two ner decoders converge much quicker than the HTR decoder. This causes either an overfitting of the two ner decoders or an underfitting of the htr decoder. Moreover, difficulties related to the alignment of the three predicted sequences may explain the low scores on the IEHHR competition. An interesting contribution of our study is the joint multi-task with tags strategy. This learning strategy yields better results than the joint strategy, even though we only evaluate the tag-mixed branch. This highlights that the network benefits from learning different semantic representations at once. This result also confirms the observation of Luong et al. [START_REF] Luong | Multi-task Sequence to Sequence Learning[END_REF] who observed that multi-task learning improves the performance of seq-to-seq models for neural translation. However, we must relativize the small improvement of this strategy with its high computational cost. Finally, the joint multi-scale with tags strategy does not meet our expectations. Our intuition was that extracting features at different scales would help to get more contextual information. However, we observe that this makes it more difficult for the attention network to select relevant features. As a result, the final performance is quite poor.

Benchmarking information extraction (IEHHR)

We now compare our work with other participants in the IEHHR competition in Tab. 3. Our joint multi-task with tags strategy achieves state-of-the-art results at line-level, without any post-processing and language model. This result shows the interest of multitask learning for information extraction, despite a high computational cost. Another interesting point comes from the comparison of our joint strategy with the articles that rely on the same strategy using a different architecture, mainly CRNN-CTC [START_REF] Carbonell | Joint Recognition of Handwritten Text and Named Entities with a Neural End-to-end Model[END_REF] and Transformer [START_REF] Rouhou | Transformer-based approach for joint handwriting and named entity recognition in historical document[END_REF]. When compared to the CRNN-CTC [START_REF] Carbonell | Joint Recognition of Handwritten Text and Named Entities with a Neural End-to-end Model[END_REF], our seq2seq model is able to boost the complete score from 89.40% to 94.0% using the same methodology. When compared to the Transformer [START_REF] Rouhou | Transformer-based approach for joint handwriting and named entity recognition in historical document[END_REF], our seq2seq model obtain competitive results, with a lower basic score but a higher complete score. This observation highlights the interest of attention-based architectures, such as seq2seq or Transformer, for information extraction. Another valuable observation comes from the analysis of the attention maps. We observe that the attention is very narrow and well focused on the corresponding pixels when predicting characters. However, when the network predicts semantic tags, the attention spreads over the previous words, showing that it learns to attend over relevant visual features to predict semantic categories.

It should be noted that a work submitted on the IEHHR competition website5 outperforms our approaches, although the methodology has not been published yet. Moreover, the joint approach based on a Transformer model [START_REF] Rouhou | Transformer-based approach for joint handwriting and named entity recognition in historical document[END_REF] achieves state-of-the-art results at record-level, as the network benefits from a larger bidimensional context.

Conclusion

This study compares joint and sequential approaches for information extraction.

Our results demonstrate the interest of using joint approaches, as we show that training a network for handwriting recognition and named entity recognition increases performance on both tasks. Compared to the traditional sequential approach, our joint strategy yields to an 8% increase in complete recognition score and a significant decrease of the Character Error Rate (from 2.82% to 1.81%). In addition, this work highlights the interest of seq2seq architectures. Indeed, we obtain a substantial performance increase when the joint strategy is applied using a seq2seq network, as compared to the CRNN-CTC approach [START_REF] Carbonell | Joint Recognition of Handwritten Text and Named Entities with a Neural End-to-end Model[END_REF]. This is because seq2seq networks rely on an attention mechanism to extract relevant visual features, as well as a recurrent decoder to learn the implicit language model. Its performance is comparable to the Transformer proposed in [START_REF] Rouhou | Transformer-based approach for joint handwriting and named entity recognition in historical document[END_REF], although our approach yield a better complete score. Finally, we explore different joint learning configurations and observe that multi-task learning from multiple semantic encodings helps the network to extract relevant features for each task. Indeed, our joint multi-task with tags approach yields a complete score of 94.4% on the IEHHR competition at line-level. As a consequence, we believe that multi-task seq2seq architectures should be investigated in more depth. We obtain state-of-the-art results on the IEEHR competition [START_REF] Fornés | ICDAR2017 Competition on Information Extraction in Historical Handwritten Records[END_REF], without any postprocessing or external language modeling. We believe that future work should also focus on information extraction at paragraph-level to take advantage of the recurrent phrasing at record-level. Recent work shows that relying on incremental learning strategies could ease information extraction at record-level [START_REF] Rouhou | Transformer-based approach for joint handwriting and named entity recognition in historical document[END_REF].

Fig. 2 .

 2 Fig. 2. Seq2seq architecture used in this work, where v is the feature vector, L is a linear function, (ci) and (hi) are the context vector and the decoder's hidden state at time step i, and n is the number of decoding steps.

Fig. 3 .

 3 Fig. 3. Illustration of the two main strategies compared in this work. Legend: wife's father occupation, wife's father location, wife's mother name. Figure best viewed in color.

( a )

 a Joint multi-task without tags: A single encoder is connected to three decoders. Each decoder is specialised for the prediction of characters, categories and persons. (b) Joint multi-task with tags: A single encoder is connected to three decoders. Each decoder is specialised for the prediction of category-based tags, person-based tags, and mixed tags. (c) Joint multi-scale strategy: the image is fed to the encoder at multiple scales.

Fig. 4 .

 4 Fig. 4. Illustration of three joint strategies proposed in this work. Legend: WF for wife's father

Table 1 .

 1 Benchmark comparison of methods for handwriting recognition at line-level on the test set and without language modeling or post-processing.

	System	Method	IAM RIMES
	CRNN-CTC Wigington et al. [28] 6.4	2.1
	CRNN-CTC Puigcerver [23]	5.8	2.3
	CRNN-CTC Dutta et al. [11]	5.2	5.1
	CRNN-CTC Ours	6.1	5.2
	Seq2seq	Poulos et al. [22]	16.6 12.1
	Seq2seq	Chowdhury et al. [9] 8.1	3.5
	Seq2seq	Bluche [4]	7.9	2.9
	Seq2seq	Michael et al. [19]	5.2	-
	Seq2seq	Ours	5.2	4.4
	Transformers Kang et al. [15]	7.6	-

Table 2 .

 2 Comparison of various learning strategies for information extraction on the Esposalles database at line-level. The first table presents results for handwriting recognition and information extraction using scores from the IEHHR competition. The second table details these scores for each semantic category.

		CER WER Basic score Complete score
	Sequential	2.82 8.33	91.2	86.7
	Joint	1.81 6.10	94.7	94.0
	Joint multi-task without tags 7.75 17.38	61.8	48.1
	Joint multi-task with tags	1.74 5.38	95.2	94.4
	Joint multi-scale with tags	5.61 15.13	83.0	80.3
		Name Surname Location Occupation State
	Sequential	94.6	85.9	91.7	92.3	90.4
	Joint	96.1	91.0	93.7	95.3	97.8
	Joint multi-task without tags 63.2	41.0	48.2	69.8	86.7
	Joint multi-task with tags	97.0 92.6	94.5	95.3	96.7
	Joint multi-scale with tags	87.4	61.1	84.2	87.5	96.9

Table 3 .

 3 Benchmark comparison of the IEHHR competition on the Esposalles database at line level.

	Method	Model	Strategy Basic score Complete score
	Baseline HMM [12]	HMM+LM Sequential	80.2	63.1
	CITlab-ARGUS-1 [12]	CRNN-CTC Sequential	89.5	89.2
	CITlab-ARGUS-2 [12]	CRNN-CTC Sequential	91.9	91.6
	CITlab-ARGUS-3 [12]	CRNN-CTC Sequential	91.6	91.2
	CVC (tags) [7]	CRNN-CTC Joint	90.6	89.4
	InstaDeep (tags) [25]	Transformer	Joint	95.2	93.3
	Joint (ours)	Seq2seq	Joint	94.7	94.0
	Joint multi-task with tags (ours) Seq2seq	Joint	95.2	94.4

https://www.transcribathon.com/en/

https://crowdheritage.eu/en

https://rrc.cvc.uab.es/?ch=10

Acknowledgements

Solène Tarride is partly funded by the CIFRE ANRT grant No. 2018/0896.


MD, PhD Gilles Allali
email: gilles.allali@einstein.yu.edu

MD, PhD Cédric Annweiler

PhD Helena M Blumen

PhD Michele L Callisaya

MD Anne-Marie De Cock

MD Reto W Kressig

PhD Velandai Srikanth

PhD Jean-Paul Steinmetz

MBBS Joe Verghese

MD, PhD Olivier Beauchet

Gait phenotype from MCI to moderate dementia: results from the GOOD initiative

Keywords: Dementia, mild cognitive impairment, aging, Alzheimer's disease, gait disorders, quantitative gait parameters, motor control, cohort studies

Background-The differences in gait abnormalities from the earliest to the latter stages of dementia and in the different subtypes of dementia have not been fully examined. This study aims to compare spatio-temporal gait parameters in cognitively healthy individuals, patients with amnestic (aMCI) and non-amnestic (naMCI) MCI, and patients with mild and moderate stages of Alzheimer's disease (AD) and non-Alzheimer's disease (non-AD).

Methods-Based on a cross-sectional design, 1719 participants (77.4±7.3 years, 53.9% female) were recruited from cohorts from seven countries participating in the "Gait, cOgnitiOn & Decline" initiative. Mean values and coefficients of variation of spatio-temporal gait parameters were measured during normal pace walking with the GAITRite system at all sites.

INTRODUCTION

Gait abnormalities in dementia are described in the more advanced stages, and are caused by vascular or neurodegenerative subcortical lesions [START_REF] Scherder | Gait in ageing and associated dementias; its relationship with cognition[END_REF]. With the recent advent of instrumental devices that allow easy quantification of spatio-temporal gait parameters, subtle gait abnormalities have been demonstrated at the earliest stages of dementia, and even in the prodromal stage of dementia like mild cognitive impairment (MCI) [START_REF] Montero-Odasso | Gait and cognition: a complementary approach to understanding brain function and the risk of falling[END_REF][START_REF] Bridenbaugh | Quantitative gait disturbances in older adults with cognitive impairments[END_REF][START_REF] Beauchet | Motor phenotype of decline in cognitive performance among community-dwellers without dementia: population-based study and meta-analysis[END_REF][START_REF] Montero-Odasso | The motor signature of mild cognitive impairment: results from the gait and brain study[END_REF][START_REF] Muir | Gait assessment in mild cognitive impairment and Alzheimer's disease: the effect of dual-task challenges across the cognitive spectrum[END_REF][START_REF] Verghese | Quantitative gait dysfunction and risk of cognitive decline and dementia[END_REF]. While gait abnormalities have been associated with neurological conditions [START_REF] Allali | Frontotemporal dementia: pathology of gait?[END_REF][START_REF] Beauchet | Gait analysis in demented subjects: Interests and perspectives[END_REF], the specific profile of quantitative gait parameters has not been defined in the different subtypes and stages of cognitive decline. The use of various gait measurements, the inclusion of patients from different dementia stages, and small sample sizes contribute to this knowledge gap.

Unlike dementia subtypes, specific cognitive domains, like executive function or memory, have been linked to spatio-temporal gait parameters [START_REF] Beauchet | Motor phenotype of decline in cognitive performance among community-dwellers without dementia: population-based study and meta-analysis[END_REF][START_REF] Verghese | Quantitative gait dysfunction and risk of cognitive decline and dementia[END_REF][START_REF] Allali | Frontotemporal dementia: pathology of gait?[END_REF][START_REF] Beauchet | Gait control: a specific subdomain of executive function?[END_REF]. Based on factor analysis, executive functions have been linked with a pace factor (reflecting gait velocity and length measures), whereas memory functions with a rhythm factor (reflecting cadence and gait timing) [START_REF] Verghese | Quantitative gait dysfunction and risk of cognitive decline and dementia[END_REF]. Executive functions, memory and gait parameters have also been linked to different brain systems [START_REF] Holtzer | Neuroimaging of mobility in aging: a targeted review[END_REF]. Prefrontal regions play a key role in executive functions and gait control using different neuroimaging techniques, including structural [START_REF] Callisaya | Brain structural change and gait decline: a longitudinal population-based study[END_REF][START_REF] Srikanth | Cerebral white matter lesions, gait, and the risk of incident falls: a prospective population-based study[END_REF] and functional [START_REF] Allali | The neural basis of age-related changes in motor imagery of gait: an fMRI study[END_REF][START_REF] Blumen | Behavioral and neural correlates of imagined walking and walking-while-talking in the elderly[END_REF] MRI, as well as functional near-infrared spectroscopy [START_REF] Holtzer | Online fronto-cortical control of simple and attention-demanding locomotion in humans[END_REF]. Interestingly, memory-related brain structures (i.e. hippocampus) have been linked to both increases [START_REF] Beauchet | Hippocampal volume, early cognitive decline and gait variability: which association?[END_REF] and decreases [START_REF] Zimmerman | MRI-and MRS-derived hippocampal correlates of quantitative locomotor function in older adults[END_REF] in gait control. The various underlying neuropathologies of the studied population seem to explain these contrasting results that highlight the importance to better define the gait phenotypes of the different subtypes and stages of early cognitive decline. Thus, to define the spatio-temporal gait parameters in these different subtypes and stages of cognitive decline, we have the opportunity to use the unique setting of the Biomathics consortium combining, in the called "Gait, cOgnitiOn & Decline" (GOOD) initiative [START_REF] Beauchet | From industrial research to academic discoveries, toward a new concept of partnership: the Biomathics model[END_REF], several international teams of physicians and researchers studying gait with the GAITRite® system in more than 2700 older individuals with and without dementia. The present study aims to measure and to compare spatio-temporal gait parameters in cognitively healthy individuals (CHI), in patients with the amnestic (aMCI) and the non-amnestic (naMCI) forms of MCI, in patients with mild stage of Alzheimer's disease (AD) and non-Alzheimer's disease (non-AD), and in patients with moderate stage of AD and non-AD. Given previous research demonstrating (a) greater gait instabilities in patients with impaired executive functions [START_REF] Allali | Frontotemporal dementia: pathology of gait?[END_REF][START_REF] Beauchet | Gait control: a specific subdomain of executive function?[END_REF] (i.e. behavioral variant of frontotemporal dementia) in comparison to patients with the classical amnestic form of AD, and (b) greater gait disturbances in patients with more advanced stages of dementia compared to patients with milder forms [START_REF] Scherder | Gait in ageing and associated dementias; its relationship with cognition[END_REF][START_REF] Beauchet | Motor phenotype of decline in cognitive performance among community-dwellers without dementia: population-based study and meta-analysis[END_REF][START_REF] Beauchet | Gait analysis in demented subjects: Interests and perspectives[END_REF], we hypothesized that patients with more advanced forms of non-AD dementia would present the most disturbed gait parameters in comparison to patients with AD or patients with the milder forms of cognitive disturbances.

METHODS

Study design and population

This cross-sectional study used data from the GOOD initiative which combines data from 7 countries (i.e., Australia, Belgium, France, India, Luxembourg, Switzerland and United States) and recruited non-demented and demented older individuals from the "Tasmanian Study of Cognition and Gait" (TASCOG -community-dwellers) (Menzies Research Institute, Hobart, southern Tasmania, Australia), from Mechelen memory clinic (outpatients with cognitive complaints), from the "Gait and Alzheimer Interactions Tracking" (GAIT) study and Angers memory clinic (Angers memory clinic -community-dwellers and outpatients with cognitive complaints), from the "Kerala-Einstein Study" (KES) (Kozhikode city, Kerala, India -community-dwellers), from the Center for Memory and Mobility (CeM 2 , Luxembourg-city, Luxembourg -outpatients and inpatients with cognitive complaints), from the "Central Control of Mobility in Aging" (CCMA -communitydwellers) (New York, lower Westchester county, US), and from Basel mobility center (University Center for Medicine of Aging Basel, Felix Platter Hospital, Basel -outpatients with cognitive complaints). Participants were included from 2005 (TASCOG study) to 2014 (GAIT study). Inclusion criteria for the present study were aged 60 years and over, participants able to walk without personal assistance, information on clinical characteristics and cognitive status (i.e., CHI, patients with aMCI and naMCI, or mild and moderate AD or non-AD dementia) and gait assessment with the GAITRite® system. From the 2717 participants initially recruited, we excluded 618 because spatio-temporal parameters, clinical characteristics or cognitive status were missing, and we also excluded 380 due to the absence of information on the subtypes or stages of dementia or MCI. After exclusions, a total of 1719 (63.3%) participants (77.4±7.3 years, 53.9% female) were included in the present study (Figure 1). The ethics committee of Angers university hospital approved the GOOD initiative. Furthermore, each center involved in the GOOD initiative obtained individual approval from their local ethics committee. Clinical trials registration number is NCT02350270.

Gait measurements

Spatio-temporal gait parameters were measured at steady state walking using the GAITRite®-system in each center. All centers followed the European guidelines for spatiotemporal gait analysis in older adults [START_REF] Kressig | Guidelines for clinical applications of spatio-temporal gait analysis in older adults[END_REF]: the participants walked at their usual self-selected walking speed in a quiet, well-lit environment wearing their own footwear. The GAITRite®-System is an electronic walkway-integrated and pressure-sensitive electronic surface providing spatio-temporal gait parameters on a length ranging from 4.6 (TASCOG study) to 7.9 (GAIT study) meters active recording area. Spatio-temporal gait parameters from the GAITRite®-system present excellent test-retest reliability [START_REF] Brach | The reliability and validity of measures of gait variability in community-dwelling older adults[END_REF]. Participants walked one trial in all cohorts except in TASCOG study, where they walked six trials. In TASCOG study, the mean values of spatio-temporal gait parameters of six trials were used for the analysis. Based on the factor analysis performed in a previous study [START_REF] Verghese | Quantitative gait dysfunction and risk of cognitive decline and dementia[END_REF], we focused on the following spatio-temporal parameters: stride length, stride time, swing time, stance time, single support time, double support time, stride width, and stride velocity. Mean values and coefficients of variation (CoV = (standard deviation / mean) x 100) of spatio-temporal gait parameters were the main outcomes.

Dementia and cognitive assessments

Participants from all centers followed a comprehensive neuropsychological assessment that permit to classify them, as CHI, aMCI, naMCI, mild AD, mild non-AD, moderate AD and moderate non-AD (see the supplementary table showing the cognitive tests performed in each cohort involved in GOOD initiative). CHI presented normal cognitive function with all cognitive scores at 1.5 SDs or above the age-appropriate means. Amnestic MCI and naMCI were diagnosed if the participants reported spontaneous cognitive complaints and presented an objective impairment respectively in the memory or the non-memory domains (i.e., defined as a score 1.5 SDs or more below the age-appropriate mean), without impairment into the activities of daily living [START_REF] Winblad | Mild cognitive impairment--beyond controversies, towards a consensus: report of the International Working Group on Mild Cognitive Impairment[END_REF]. We adopted this classification because MCI status presents with a variety of symptoms [START_REF] Cummings | International Work Group criteria for the diagnosis of Alzheimer disease[END_REF]. Thus, when memory loss was the predominant deficit, patients were classified as aMCI, and when memory loss was not the predominant symptom and/or was combined with other cognitive dysfunctions, patients were classified as naMCI. This classification was done because aMCI is frequently seen as a prodromal stage of Alzheimer disease (AD) [START_REF] Cummings | International Work Group criteria for the diagnosis of Alzheimer disease[END_REF]. AD and non-AD type dementia were diagnosed according to the Diagnostic and Statistical Manual of Mental Disorders, fourth edition at a consensus diagnostic case conferences in all centers, except in TASCOG study: self-report, review of medical history, cognitive testing, and/or clinical interview, followed by interview of proxy if available were used to diagnose dementia. Mild and moderate stages of AD and non-AD were defined by a mini mental status examination (MMSE) score ≥20 and between 19 and 10, respectively, by a Clinical Dementia Rating (CDR) score 1 and 2, respectively, or by review of medical history.

Clinical covariates

Age, gender, body mass index in kg/m2, number of drugs taken per day, use of psychoactive drugs including benzodiazepines or antidepressants or neuroleptics, depressive symptoms assessed using the 4-item, the 15-item, or the 30-item Geriatric depression scale (score≥1 or ≥5 or ≥10 indicated the presence of depressive symptoms, respectively) [START_REF] Pomeroy | The effectiveness of very short scales for depression screening in elderly medical patients[END_REF], and history of falls in the past year [START_REF] Beauchet | How to manage recurrent falls in clinical practice: guidelines of the French Society of Geriatrics and Gerontology[END_REF] were collected during the clinical assessment. A fall was defined as unintentionally coming to rest on the ground, floor, or other lower level and not as the result of a major intrinsic event or an overwhelming hazard [START_REF] Beauchet | How to manage recurrent falls in clinical practice: guidelines of the French Society of Geriatrics and Gerontology[END_REF].

Statistics

Baseline characteristics and spatio-temporal gait parameters were summarized using means and standard deviations or frequencies and percentages, as appropriate. Normality of data distribution was checked using the Shapiro-Francia test. Between-group comparisons were performed using unpaired t-test or Chi-square test, as appropriate. Multiple linear regressions exploring the association between each gait parameters (dependent variable) and cognitive status (independent variable) adjusted on participant's characteristics (i.e., age, gender, number of drugs taken per day, body mass index, use of psychoactive drugs, depression symptoms and history of falls) were performed. P-trends between the seven groups (i.e., MCI amnesic, MCI non-Amnesic, mild dementia AD, mild dementia non-AD, moderate dementia AD, moderate dementia non-AD) are graphed in Figure 2 for each spatio-temporal gait parameter. Subgroups (i.e., aMCI, naMCI, mild and moderate AD and mild and moderate non-AD) and total group "effect size" of spatio-temporal gait parameters are presented in Figure 3 (Review Manager version 5.1, The Nordic Cochrane Centre, Copenhagen, Denmark), in order to identify the largest values of spatio-temporal gait parameters and the homogeneity of spatio-temporal gait values associated with each cognitive status. Finally, in order to quantify the amplitude of spatio-temporal gait values in each one of the six pathological groups, comparisons between mean values and the coefficient of variation of spatio-temporal gait parameters were performed using a paired ttest. P-values <0.05 were considered as statistically significant. All statistics were performed using SPSS (version 15.0; SPSS, Inc., Chicago, IL).

RESULTS

Clinical characteristics and spatio-temporal gait parameters are presented in Table 1, by comparing the diagnostic subtypes per type and stages of cognitive decline. Age, number of drugs taken per day, psychoactive drugs and depression symptoms were similar per stages between the 2 profiles (i.e. aMCI vs naMCI and AD vs non-AD). Non-amnestic MCI showed significantly worse gait performances than aMCI for mean value of stride length and CoV of stride length and stride width (after adjustment on participant's characteristics). For the mild stage of dementia, non-AD walked with significant worse gait performance than AD for CoV of stride length, mean value of stride width, and CoV of stride velocity (after adjustment on participant's characteristics). For the moderate stage of dementia, non-AD presented significantly more disturbed gait parameters in comparison to AD for walking speed, mean value of stride length, mean value of stance time, and mean value of stride velocity (after adjustment on participant's characteristics) (Table 1). The decline of gait parameters from normal cognition to the most advanced stages of dementia is illustrated by multivariate regressions examining separately the association between each gait parameter and cognitive status using CHI as the reference group (Table 2). The results showed that naMCI, as well as non-AD dementia, and moderate stages of dementia presented the worse gait performances in comparison to healthy older adults This degradation of performance of gait parameters from aMCI to naMCI and from AD to non-AD across stages of cognition, and from CHI to moderate dementia, showed that the Ptrends between groups were significant for all mean values and coefficients of variation of gait parameters, except for the mean values of the swing time (p=0.254) and the single support (p=0.278) and the CoV of swing time (Figure 2).

The effect sizes of the association of gait parameters and cognitive status are illustrated in Figure 3. The overall effect size for the MCI groups was 0.35 (p<0.001) with the largest effect sizes for the CoV of stride velocity (0.5) in the aMCI group; and for the CoV of stance time and the CoV of stride velocity (0.55) in the naMCI group (Figure 3a). The overall effect size for the mild dementia groups was 0.87 (p<0.001) with the largest effect sizes for the mean value of stride length (1.44) in the mild AD group; and for the CoV of stride length (1.66) in the mild non-AD group (Figure 3b). The overall effect size for the moderate dementia groups was 1.11 (p<0.001) with the largest effect sizes for the mean value of stride length (1.79) in the moderate AD group; and for the mean value of stride length (2.09) in the moderate non-AD group (Figure 3c).

The amplitude (mean value) and the homogeneity (coefficient of variation) of individual mean values and individual coefficients of variation of spatio-temporal gait parameter are illustrated in Table 2. The amplitude of both mean values and coefficient of variation increases with the progression of cognitive decline with less amplitude for the aMCI in comparison to the naMCI and with less amplitude for AD in comparison to non-AD. The amplitude of the CoV of spatiotemporal gait parameters was more affected than the mean value for the aMCI, the naMCI and the moderate AD groups. But, they were affected similarly for the other groups, with a tendency in favor of greater amplitude for the CoVs than the mean values. We observed similarities (i.e. non-significant differences) between mean values and CoV of gait parameters for aMCI and mild AD groups, whereas we found dissimilarities (i.e. significant differences) for the other groups.

DISCUSSION

The findings of the present study showed that spatio-temporal gait parameters are affected in parallel to the type and stage of cognitive impairment from MCI to moderate dementia. Indeed, patients with naMCI presented more disturbed gait parameters than aMCI. Whatever the MCI subgroup, they performed better than patients with dementia. In addition, patients with non-AD dementia had worse gait performance than AD dementia. Furthermore, we showed that this pattern of degradation of gait parameters is homogeneous in the earliest stages of cognitive decline, but becomes heterogeneous in the most advanced stages, especially in the non-AD subtypes.

Gait parameters presented a progressive degradation from normal aging to moderate stage of dementia -with a walking speed declining from 104.7±22.2 cm/s in CHI to 61.7±20.3 cm/s in the moderate stage of non-AD patients, and with a progressive increase of the magnitude of the effect sizes of the gait parameters from aMCI to moderate stage of non-AD dementia. This progressive decline of gait parameters paralleling the evolution of dementia confirms the findings of previous studies [START_REF] Scherder | Gait in ageing and associated dementias; its relationship with cognition[END_REF][START_REF] Beauchet | Gait analysis in demented subjects: Interests and perspectives[END_REF]. Different factors have been suggested to explain this parallel decline between cognition and gait: the progressive decline of the frontal lobe functions [START_REF] Coelho | Gait and risk of falls associated with frontal cognitive functions at different stages of Alzheimer's disease[END_REF][START_REF] Sheridan | The role of higher-level cognitive function in gait: executive dysfunction contributes to fall risk in Alzheimer's disease[END_REF], the involvement of the basal ganglia [START_REF] Burns | The pathology of the substantia nigra in Alzheimer disease with extrapyramidal signs[END_REF] or the accumulation of vascular lesions [START_REF] Srikanth | Cerebral white matter lesions, gait, and the risk of incident falls: a prospective population-based study[END_REF][START_REF] Inzitari | Cerebrovascular disease and gait and balance impairment in mild to moderate Alzheimer's disease[END_REF]. This progressive gait degradation with the course of dementia has been associated with poor clinical outcomes such as falls [START_REF] Camicioli | Motor impairment predicts falls in specialized Alzheimer care units[END_REF], institutionalization [START_REF] Scarmeas | Motor signs predict poor outcomes in Alzheimer disease[END_REF] and mortality [START_REF] Scarmeas | Motor signs predict poor outcomes in Alzheimer disease[END_REF]. The differences between the diagnostic subgroups (aMCI versus naMCI or AD versus non-AD) are more pronounced in the latest stage of dementia for the majority of the gait parameters, such as walking speed, stride length stance time, or stride width. These gait parameters reflects the various components of gait, including rhythmic control and dynamic postural control [START_REF] Beauchet | Gait variability among healthy adults: low and high stride-to-stride variability are both a reflection of gait stability[END_REF]. Among the spatio-temporal parameters, stride length (mean value) presented the highest effect size in both subtypes of dementia in the mild and the moderate stages. This parameter has been specifically associated with disruption of white matter integrity in patients with cerebral small vessel disease [START_REF] De Laat | Loss of white matter integrity is associated with gait disorders in cerebral small vessel disease[END_REF], subcortical hyperintensities in AD [START_REF] Nadkarni | Gait and subcortical hyperintensities in mild Alzheimer's disease and aging[END_REF] and with hippocampal volume in non-demented older adults [START_REF] Zimmerman | MRI-and MRS-derived hippocampal correlates of quantitative locomotor function in older adults[END_REF], suggesting that stride length seems to track both vascular and neurodegenerative components.

Non-amnestic MCI, like non-AD dementia, presented with greater gait decline in comparison to aMCI and AD respectively. These quantitative findings are supported by the observations that clinical gait abnormalities are more prevalent in non-AD dementia than in AD [START_REF] Allan | Prevalence and severity of gait disorders in Alzheimer's and non-Alzheimer's dementias[END_REF], and that they predict non-AD dementia [START_REF] Verghese | Abnormality of gait as a predictor of non-Alzheimer's dementia[END_REF]. It is possible that BMI could contribute in part to these disturbed quantitative gait parameters in naMCI and non-AD dementia, as higher BMI was noticed in naMCI in comparison to MCI and a tendency to higher BMI was also noticed between AD and non-AD dementia. The different cognitive profiles between aMCI and naMCI, as well as AD and non-AD could also explain this observation. Memory decline is the hallmark of aMCI and AD, whereas non-memory functions, and mainly executive function, are usually affected in naMCI and non-AD dementia [START_REF] Weakley | Analysis of verbal fluency ability in amnestic and non-amnestic mild cognitive impairment[END_REF]. Increased stride time variability -a marker of disturbed gait rhythmicity [START_REF] Beauchet | Gait variability among healthy adults: low and high stride-to-stride variability are both a reflection of gait stability[END_REF] -has been associated with executive functioning in healthy older adults [START_REF] Beauchet | Gait control: a specific subdomain of executive function?[END_REF] and in patients with dementia [START_REF] Sheridan | Influence of executive function on locomotor function: divided attention increases gait variability in Alzheimer's disease[END_REF]. In addition, by contrasting executive function and memory performances in non-demented older adults, only executive function has been linked to stride time variability [START_REF] Hausdorff | Walking is more like catching than tapping: gait in the elderly as a complex cognitive task[END_REF]. From a functional perspective, this strong link between executive functioning and gait in aging has been previously illustrated by a high correlation between the activation of the primary motor cortex during a mental imagery task of gait and the performance to the Stroop task in healthy older adults [START_REF] Allali | The neural basis of age-related changes in motor imagery of gait: an fMRI study[END_REF] and also by a greater activation of the prefrontal cortex shown by healthy older adults during a mental imagery task of gait in comparison to younger adults [START_REF] Allali | The neural basis of age-related changes in motor imagery of gait: an fMRI study[END_REF].

The heterogeneity of gait changes increases in parallel with the progression of cognitive deterioration with a decline in performances being more marked in non-AD dementia than in AD, and more marked in naMCI than in aMCI. In aMCI, we observed a similar variability of changes between mean values and coefficients of variation of gait parameters, whereas in naMCI and more advanced stages of dementia, the variability of changes in mean values of gait parameters was greater than the variability of changes in coefficients of variation of gait changes, suggesting a chaotic modification of gait parameters. With the progression of cognitive decline due to the progression of neuropathological or vascular lesions, we can interpret this chaotic modification of gait parameters by the diffuse spreading of these lesions affecting in a non-harmonic way the brain regions controlling the various steps involved in the goal-directed action of walking: integration of the sensory afferents, motor programming and generation of accurate foot movement [START_REF] Scherder | Understanding higher level gait disturbances in mild dementia in order to improve rehabilitation: 'last in-first out[END_REF]. Interestingly, in the earliest stages of cognitive decline, the AD-related profiles (aMCI and mild AD) affect gait parameters in a more homogeneous way than non-AD. In such earlier stages, hippocampal functioning is typically affected in AD [START_REF] Sarazin | Amnestic syndrome of the medial temporal type identifies prodromal AD: a longitudinal study[END_REF]. Hippocampus constitutes a key region for gait control [START_REF] Callisaya | Brain structural change and gait decline: a longitudinal population-based study[END_REF][START_REF] Beauchet | Hippocampal volume, early cognitive decline and gait variability: which association?[END_REF][START_REF] Zimmerman | MRI-and MRS-derived hippocampal correlates of quantitative locomotor function in older adults[END_REF] that is mainly due to its essential role in spatial navigation [START_REF] Ekstrom | Cellular networks underlying human spatial navigation[END_REF]. The disturbance of this higher-level functional control affecting spatial navigation acts as a central supervisor that explains its homogeneous effect on the different substructures controlling gait -reflected by a similar effect on mean values and coefficients of variation of spatio-temporal gait parameters.

Comparing quantitative gait parameters collected using similar methods and equipment in a large sample of MCI and demented patients from multiple nations, and divided on cognitive subtypes and stages represent the main strengths of this study. However, the absence of autopsy-confirmed diagnoses constitutes a main limitation. Indeed, although the diagnoses of dementia were made according to standardized criteria, clinical misclassification of subtypes of dementia is still possible. Although the associations between cognitive status and gait parameters were controlled for many covariates, information on some clinical characteristics, such as presence of major outcomes following falls or presence of any prosthesis that could influence gait parameters, is lacking. Furthermore, the absence of information concerning major comorbid conditions affecting gait, such as osteoarthritis or peripheral neuropathy, represents another limitation of this study. Although our findings are controlled for medication, we do not know if demented patients are taken any specific antidementia drugs. Finally, identifying stages of dementia based on the MMSE or the CDR represents also a limitation, because both scores did not take into account the level of education that could induce misclassification between mild and moderate stages of dementia.

In conclusion, this study found that spatio-temporal gait parameters deteriorated with the progression of dementia from MCI to moderate dementia, and that they are more affected in the non-AD dementias than in AD. Furthermore, in the most advanced stages and in the non-AD subtypes, this deterioration is more heterogeneous than in the earlier stages and in AD subtype. These findings suggest that quantitative gait parameters could be used as a surrogate marker improving the diagnosis process of dementia. Future analyses need to further examine the respective gait characteristics among the non-AD subtypes of dementia and to study the respective contributions of clinical covariates to gait deterioration in the course of dementia.

Supplementary Material

Refer to Web version on PubMed Central for supplementary material. P-trend between groups of participants categorized on cognitive status (i.e., cognitively healthy individuals, patients with amnestic-mild cognitive impairment, patients with nonamnestic mild cognitive impairment, patients with Alzheimer disease mild dementia and patients with non-Alzheimer disease mild dementia, patients with Alzheimer disease moderate dementia and patients with non-Alzheimer disease moderate dementia) for each spatio-temporal parameter (n=1719).

Figure 1 .

 1 Figure 1.Flow chart showing the selection of participants included in the analysis.

Figure 2 .

 2 Figure 2.

Figure 3 .

 3 Figure 3. Effect size of the association of spatio-temporal parameters and cognitive disorders in patient groups with (a) mild cognitive impairment, (b) mild dementia and (c) moderate dementia (n=1719)

Table 1

 1 Comparisons of the participants' characteristics separated into four groups based on cognitive status (n=1719) Comparison between groups based on unpaired t-test or Chi-square test, as appropriate; † based on multiple linear regression exploring the association between each spatio-temporal gait parameters* (dependent variable) and cognitive status (independent variable; MCI) using amnestic mild cognitive impairment as the reference group, adjusted on participant's characteristics (age, gender, number of drugs taken per day, body mass index, use of psychoactive drugs, depression symptoms and history of falls); ‡ based on multiple linear regression exploring the association between each spatio-temporal gait parameters* (dependent variable) and cognitive status (independent variable; mild dementia) using mild Alzheimer disease as the reference group, adjusted on participant's characteristics (age, gender, number of drugs taken per day, body mass index, use of psychoactive drugs, depression symptoms and history of falls); ¶ based on multiple linear regression exploring the association between each spatio-temporal gait parameters* (dependent variable) and cognitive status (independent variable; moderate dementia) using moderate Alzheimer disease as the reference group, adjusted on participant's characteristics (age, gender, number of drugs taken per day, body mass index, use of psychoactive drugs, depression symptoms and history of falls);

	Dementia		P-Moderate P-	value * value * AD (P-(P-(n=177) value ‡ ) value ¶ ) Non-AD (n=91)	0.280 83.9±5.6 83.3±5.2 0.411	0.029 121 (68.4) 52 (57.1) 0.069	0.330 4.6±2.6 4.4±2.3 0.147	0.054 25.8±4.6 26.2±3.8 0.069	0.454 79 (44.6) 46 (50.5) 0.358	0.187 93 (52.5) 50 (54.9) 0.709	0.119 86 (48.6) 61 (67.0) 0.004	0.245 (0.107) 68.1±20.6 61.7±20.3 0.015 (0.005)		0.298 (0.063) 85.3±21.6 79.6±20.3 0.038 (0.008)	0.025 (0.015) 6.8±4.2 7.1±4.0 0.587 (0.992)		0.515 (0.793) 1273.9±187.6 1321.1±189.2 0.053 (0.058)	0.641 (0.313) 5.3±3.4 5.7±4.1 0.486 (0.952)		0.260 (0.465) 401.3±60.7 406.5±55.0 0.489 (0.271)	0.677 (0.290) 10.2±10.1 10.6±9.9 0.742 (0.782)		0.227 (0.841) 872.6±156.8 914.6±159.1 0.040 (0.043)	0.327 (0.222) 5.9±2.5 6.8±4.4 0.040 (0.458)	0.255 (0.464) 401.9±59.8 407.2±54.6 0.483 (0.357)	0.644 (0.299) 10.0±9.9 10.5±9.6 0.692 (0.923)
	MCI	P-	CHI value * Amnestic Non-amnestic Mild	(n=735) (P-(n=108) (n=286)	value † ) AD (n=196) Non-AD (n=126)	73.9±6.3 76.7±7.9 75.5±6.6 0.117 82.5±5.1 81.9±5.1	374 (50.9) 40 (37.0) 134 (46.9) 0.080 134 (68.4) 71 (56.3)	1.2±1.4 0.7±1.0 0.8±1.0 0.322 4.2±2.3 4.6±2.5	27.5±5.3 26.4±4.8 27.7±5.1 0.020 26.2±4.4 27.2±4.9	93 (12.7) 24 (22.2) 70 (24.5) 0.640 85 (43.4) 60 (47.6)	134 (18.2) 30 (27.8) 98 (34.3) 0.220 105 (53.6) 58 (46.0)	178 (24.2) 34 (32.4) 84 (29.4) 0.558 93 (47.4) 71 (56.3)	104.7±22.2 96.7±26.2 92.4±25.9 0.138 (0.089) 74.1±18.9 71.6±20.4		121.9±20.2 116.4±23.8 111.3±24.7 0.067(0.016) 93.0±19.5 90.6±21.7	2.8±1.7 3.5±2.3 3.8±3.2 0.265 (0.031) 5.4±3.6 6.3±3.7		1175.6±135.2 1224.5±153.1 1225.1±159.1 0.976 (0.818) 1270.1±163.7 1282.6±173.3	2.6±1.4 3.2±1.9 3.6±2.8 0.196 (0.076) 4.5±3.5 4.7±4.3		411.7±45.4 422.8±41.8 416.3±51.7 0.242 (0.382) 414.1±49.4 407.3±57.3	4.4±2.7 5.4±3.9 5.8±6.0 0.540 (0.182) 8.3±8.7 7.9±4.6		764.0±103.8 801.8±128.9 808.8±131.1 0.633 (0.970) 856.1±135.1 875.3±145.0	3.3±1.7 4.1±2.4 4.6±3.5 0.229 (0.135) 5.3±3.0 5.8±5.5	411.7±45.4 423.0±41.2 416.4±51.4 0.229 (0.367) 414.1±49.2 407.3±57.3	4.4±2.7 5.3±3.8 5.8±5.9 0.503 (0.165) 8.3±8.7 7.9±4.6
						Age (years), mean±SD	Female gender, n (%)	Number of drugs taken per day, mean±SD	Body mass index (kg/m2), mean±SD	Use of psychoactive drugs # , n (%)	Depression symptoms § , n (%)	History of falls ** , n (%)	Walking speed (cm/s), mean±SD	Stride length	Mean value (cm)	Cov (%)	Stride time	Mean value (ms)	Cov (%)	Swing time	Mean value (ms)	Cov (%)	Stance time	Mean value (ms)	Cov (%)	Single support time	Mean value (cm)	Cov (%)	Double support time

*

Table 2

 2 Multiple linear regressions showing the association between each spatio-temporal gait parameters* Eur J Neurol. Author manuscript; available in PMC 2017 March 01. CHI cognitively healthy individuals; MCI: mild cognitive impairment; AD: Alzheimer's disease; CoV: coefficient of variation; CI: confidence interval; * separated model for each gait parameter with each model adjusted on age, gender, number of drugs taken per day, body mass index, use of psychoactive drugs, depression symptoms and history of falls; β: coefficient of regression indicating a decrease or increase in value of gait parameter associated with cognitive decline, the reference group being cognitively healthy individuals; P-value significant (i.e., P < 0.05) indicated in bold.

	(dependent variable) and cognitive status (independent

Eur J Neurol. Author manuscript; available in PMC 2017 March 01.

Table 3

 3 Mean and coefficient of variation of magnitude * of decline in spatio-tempral gait parameters using cognitively healthy individuals as the reference value among participants categorized on cognitive disorders (i.e., amnestic mild cognitive impairment, non-amnestic mild cognitive impairment, Alzheimer disease mild dementia, non-Alzheimer disease mild dementia, Alzheimer disease moderate dementia, non-Alzheimer disease moderate dementia)

	P-
	Mean value

Value † Coefficient of variation (%) P-value † Mean value Coefficient of variation Mean value Coefficient of variation

	0.201	0.018	0.074	0.003	0.027	0.008
	94.45±188.70	22.97±20.68	15.43±19.33	10.75±5.78	9.81±8.29	9.37±5.49
	39.08±13.11	32.9±24.8	47.40±66.69	33.14±43.63	16.78±14.73	33.22±45.89
	<0.001	0.008	0.090	0.077	0.022	0.092
	0.34±0.15	0.42±0.15	0.88±0.34	1.09±0.39	1.20±0.44	1.50±0.48
	0.28±0.07	0.32±0.17	0.72±0.56	0.87±0.57	0.92±0.61	1.15±0.77
	Amnesic mild cognitive impairment	Non-amnesic mild cognitive impairment	Alzheimer disease mild dementia	Non-Alzheimer disease mild dementia	Alzheimer disease moderate dementia	Non-Alzheimer disease moderate dementia

* calculated with effect size mean value and standard deviation and pooling all spatio-temporal parameters; † based on paired t-test Eur J Neurol. Author manuscript; available in PMC 2017 March 01.

Eur J Neurol. Author manuscript; available in PMC 2017 March 01.

Fall in the past year of assessment; P-value significant (i.e., P < 0.05) indicated in bold.Eur J Neurol. Author manuscript; available in PMC 2017 March 01.

Acknowledgments

Funding: G. Allali is supported by a grant from the Geneva University Hospitals and the Resnick Gerontology Center, Albert Einstein College of Medicine, Yeshiva University. The Kerala-Einstein Study was funded by the National Institutes of Health, USA (R01 AG039330). The CCMA study was funded by the National Institutes of Health, USA (R01AG036921, RO1AGO44007-01A1). TASCOG was funded by the National Health and Medical Research Council (NHMRC grant number 403000 and 491109) and the Royal Hobart Hospital Research Foundation. M.L. Callisaya is funded by an NHMRC Early Career Fellowship (1034483); V. Srikanth is funded by an NHRMC CDF/HF Future Leader fellowship.

Conflict of Interest: None

HHS Public Access