Licia Amichi †⋆

Aline C Viana

Mark Crovella

Antonio F Loureiro

Mobility profiling: Identifying scouters in the crowd

The prediction of individuals' dynamics has attracted significant community attention and has implication for many fields: e.g. epidemic spreading, urban planning, recommendation systems. Current prediction models, however, are unable to capture uncertainties in the mobility behavior of individuals, and consequently, suffer from the inability to predict visits to new places. This is due to the fact that current models are oblivious to the exploration aspect of human behavior. This paper contributes better understanding of this aspect and presents a new strategy for identifying exploration profiles of a population. Our strategy captures spatiotemporal properties of visits -i.e. a known or new location (spatial) as well as a recurrent and intermittent visit (temporal) -and classifies individuals as scouters (i.e., extreme explorers), routineers (i.e., extreme returners), or regulars (i.e., with a medium behavior). To the best of our knowledge, this is the first work profiling spatiotemporal exploration of individuals in a simple and easy-to-implement way, with the potential to benefit services relying on mobility prediction.

INTRODUCTION

Many prediction models have been proposed to forecast individuals trajectories. However, they all show limited bounded predictive performance [START_REF] Cuttone | Understanding predictability and exploration in human mobility[END_REF]. Regardless of the applied methods (e.g., Markov chains, Naive Bayes, neural networks), the type of prediction (i.e., next-cell or next place) or the used data sets (e.g., GPS, CDR, surveys), accuracy of prediction never reaches the coveted 100%. The reasons for such limitations in the accuracy are manyfold: the lack of ground truth data, human beings' complex nature and behavior, as well the exploration phenomenon (i.e., visits to never seen before places) [START_REF] Cuttone | Understanding predictability and exploration in human mobility[END_REF][START_REF] Schmidt | Human Factors in Complex Systems The Modelling of Human Behaviour[END_REF][START_REF] Scherrer | Travelers or locals? Identifying meaningful sub-populations from human movement data in the absence of ground truth[END_REF]. In this paper, we focus on the exploration problem, which has rarely been tackled in the literature but indeed, represents a real issue [START_REF] Cuttone | Understanding predictability and exploration in human mobility[END_REF]. By construction, most prediction models attempt to forecast future locations from the set of known places, which hinders predicting new unseen places and by consequence, reduces the predictive performance. In [START_REF] Pappalardo | Returners and explorers dichotomy in human mobility[END_REF], the authors reported the existence of two mobility profiles: (i) returners and (ii) explorers, and suggested that the probability of exploring new areas is correlated with the number of frequently visited places. However, this classification can be unsuitable; for instance, a person who regularly visits two different locations and usually explores many new areas is considered to be a returner, while a person who spends most of her time between eight different locations and rarely visits new ones can be viewed as an explorer. The authors in [START_REF] Scherrer | Travelers or locals? Identifying meaningful sub-populations from human movement data in the absence of ground truth[END_REF] corroborate the results drawn in [START_REF] Pappalardo | Returners and explorers dichotomy in human mobility[END_REF] and shown the existence of two distinct groups of individuals: (i) travelers, who move around extensively, and (ii) locals, who move in a more constrained area and revisit many of their locations. Nevertheless, they do not bring any understanding of the exploration behavior of individuals. Although their approach does not classify all individuals and results in five groups of individuals, only two groups were interpreted and considered to be significant. In [START_REF] Cuttone | Understanding predictability and exploration in human mobility[END_REF], an exploration prediction model was proposed based on random guessing of explorations. Still, this model suggests that all individuals have the same probability to explore, which contradicts what was shown in [START_REF] Pappalardo | Returners and explorers dichotomy in human mobility[END_REF][START_REF] Scherrer | Travelers or locals? Identifying meaningful sub-populations from human movement data in the absence of ground truth[END_REF].

Thus, when considering the exploration problem, previous studies either did not provide any consideration of the exploration factors of individuals, or divided the population based on properties that are not always consistent, or assumed that all individuals have the same propensity to explore. Our main goal in this work is to understand the exploration phenomenon and answer the following question: What type of visits characterize the mobility of individuals? Using newly designed metrics capturing spatiotemporal properties of human mobility -i.e., known/new and recurrent/intermittent visits -our strategy identifies three groups of individuals according to their degree of exploration: scouters, routineers, and regulars. In the future, we plan to deeply investigate the mobility behavior of individuals in each profile and to assign to each individual an exploration factor describing her susceptibility to explore.

PROPOSED METHOD

To understand human mobility dynamics and identify the circumstances inciting individuals' propensity to break their routine and explore new spots, we divide human moves into two complementary movements: explorations and returns. Indeed, at each instant, an individual has two choices: she either walks back to a place she visited in the past, or explores a new site. Hereafter, we define (i) an exploration as a visit to a never seen before location, i.e., a location that is not present in the history of a given individual and (ii) a return as a visit to a previously seen locality.

Formalization

Let M be the Finite-State Automaton (FSA) describing an individual movements, as shown in Fig. 1, with two possible states: exploring What do the metrics α and µ capture? The intermittency µ captures the transition patterns of individuals between the states U and R. The more distant an individual is from the origin, the steadier she is. When #U or #R increases the sum #U + #R increases, indicating that fewer shifts occur between U and R. Therefore, the intermittency metric reveals whether the individual is versatile or prefers to be steady. For instance, the individual 2 (i.e., with µ 2 ) in zone 3 (i.e., Z3) in Fig. 2, is more intermittent than the individual 1 (i.e., with µ 1 ) in zone 1 (i.e., Z1). The degree of return reports the exploration habits of an individual compared to her returns, whether she relatively performs more explorations or returns compared to the average statistics raised from the population. In Fig. 2, individual 1 is more prone to explore than individual 2 (α 1 < α 2 ).

Preliminary evaluation

For each individual, we first measure her intermittency and degree of return. Next, we use the Gaussian mixture probabilistic model to investigate whether we can split the population into distinct cohesive and significant groups.

Dataset: Our first dataset source is an anonymized trace collected by the MACACO project [START_REF] Jaffr'es-Runser | Crowdsensing Mobile Content and Context Data: Lessons Learned in the Wild[END_REF] during approximately 34 months. It contains timestamped GPS-like coordinates of 99 individuals. The second dataset contains the timestamped geolocalized trajectories of 100 volunteers collected by the Privamov project [START_REF] Ben Mokhtar | PRIVA'MOV: Analysing Human Mobility Through Multi-Sensor Datasets[END_REF] during 14 months. We consider only participants that appear with more than 500 measurements with at least 10 days of contiguous data and a frequency of sampling equal to 5min, resulting in 25 individuals for MACACO and 29 individuals for Privamov. In this work, we tessellate the concerned geographical regions in the datasets with grids of side 600m, which results in an assignment to each GPS coordinate, a cell with a unique identifier.

Results: Fig. 3 and4 show that our metrics identify three distinct profiles in terms of human mobility dynamics. The first profile is scouters or extreme explorers, whose degree of return is relatively low and who are intermittent and constantly shifting from a state to another. These individuals are more prone to explore new areas.

The second is routineers or extreme-returners, who have a surprisingly large degree of return and remain steady in the different states. These individuals rarely perform explorations and prefer to stick among the common and known places. Finally, regulars are individuals who have a medium behavior alternating between explorations and revisits. Our metrics results in a natural clustering of individuals, although having a different number of frequently visited locations, individuals who usually break their routines to explore are viewed as scouters, unlike in [START_REF] Pappalardo | Returners and explorers dichotomy in human mobility[END_REF] where some can be clustered as explorers and others as returners. Contrary to [START_REF] Scherrer | Travelers or locals? Identifying meaningful sub-populations from human movement data in the absence of ground truth[END_REF] our approach captures three major mobility features that fully describe the exploration phenomenon: uniqueness of visits ( i.e. explorations), intermittency between returns and explorations ( its importance was shown in [START_REF] De C. Teixeira | Human Factors in Complex Systems The Modelling of Human Behaviour. In Deciphering Predictability Limits in Human Mobility[END_REF] as stationarity), and the ratio of explorations compared to returners and splits the populations accordingly.

Discussions and Future Work

In this study, we split the population according to their propensity to explore: How often does an individual explore? How many new places does she visit consecutively? This profiling resulted in three distinct classes: (i) scouters, who are more adventurous and like to discover many new places sequentially; (ii) routineers, who are more steady and rarely leave their comfort zone to explore new ones and (iii) regulars, who have a medium behavior alternating between explorations and revisits. In the future work, we aim to assess the effectiveness of our clustering method by investigating each group independently and measuring new spatiotemporal features -e.g., the duration of visits, the number of stops, the ratio of places visited only once or distances walked -and identifying the features that are specific to each mobility profile. Further, we aspire to understand the exploration phenomenon and to associate to each individual a factor that given her mobility profile and history, can tell whether she is more susceptible to return to a previously know place or perform a visit to a new region, and this can be a prime mover in improving the accuracy of prediction.

  CoNEXT '19 Companion, December 9-12,2019, Orlando, FL, USA L. Amichi et al.

Figure 1 :

 1 Figure 1: State Diagram of Human movements


Member, IEEE Li Liu

Gang Feng

Denis Beautemps

Senior Member, IEEE Xiao-Ping Zhang

A New Re-synchronization Method based Multi-modal Fusion for Automatic Continuous Cued Speech Recognition

Keywords: Cued Speech, Multi-modal fusion, Resynchronization procedure, Automatic CS recognition, CNN, MSHMM

Cued Speech (CS) is an augmented lip reading complemented by hand coding, and it is very helpful to the deaf people. Automatic CS recognition can help communications between the deaf people and others. Due to the asynchronous nature of lips and hand movements, fusion of them in automatic CS recognition is a challenging problem. In this work, we propose a novel re-synchronization procedure for multi-modal fusion, which aligns the hand features with lips feature. It is realized by delaying hand position and hand shape with their optimal hand preceding time which is derived by investigating the temporal organizations of hand position and hand shape movements in CS. This re-synchronization procedure is incorporated into a practical continuous CS recognition system that combines convolutional neural network (CNN) with multi-stream hidden markov model (MSHMM). A significant improvement of about 4.6% has been achieved retaining 76.6% CS phoneme recognition correctness compared with the state-of-the-art architecture (72.04%), which did not take into account the asynchrony issue of multi-modal fusion in CS. To our knowledge, this is the first work to tackle the asynchronous multi-modal fusion in the automatic continuous CS recognition.

I. INTRODUCTION

C OMMUNICATION is one of the most important parts of human life, and more and more attention has been paid to improve communications among the disabled people in nowadays' society. It was reported by the World Health Organization (WHO) [START_REF]Deafness and hearing loss[END_REF] that more than 5% of the world's population (466 million people) has disabling hearing loss (432 million adults and 34 million children) in the world. As one of the most common communication ways for deaf people, lip reading [START_REF] Barbara | Hearing by eye: The psychology of lip-reading[END_REF], [START_REF] Gaye | Cued speech and the reception of spoken language[END_REF] helps the deaf or hearing impaired people access spoken speech, and has undoubtedly improved communication of these people a lot. However, a significant drawback is that lip reading can only provide insufficient information in most cases. In fact, it cannot distinguish some contrasts, for example [p] vs. [b], which is caused by the similarity of labial shapes. As a result, this problem makes it L. Liu and X.-P. Zhang are with the Electrical and Computer Engineering, Ryerson University, Toronto, ON, Canada. E-mail:liliu.math@gmail.com, xzhang@ee.ryerson.ca G. Feng and D. Beautemps are with Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, 38000 Grenoble, France. E-mail:gang.feng@gipsalab.grenoble-inp.fr, Denis.Beautemps@gipsa-lab.grenoble-inp.fr L. Liu and X.-P. Zhang are the corresponding authors. Part of this work has been presented in conference Eusipco, 2019.

difficult for deaf or hearing impaired people to access speech only by lip reading.

To overcome insufficient information of lip reading and improve reading ability of deaf children, in 1967, Cornett [START_REF] Richard | Cued speech[END_REF] invented the first Cued Speech (CS) system for the American English, which complements lip reading and makes all the phonemes of a spoken language clearly visible. This system is based on four hand positions and eight hand shapes in which two major criteria were set: the minimum effort for encoding, and the maximum visual contrast. In the French CS named Langue franc ¸aise Parlée Complétée (LPC) [START_REF] Lasasso | Cued Speech and Cued Language Development for Deaf and Hard of Hearing Children[END_REF] (see Fig. 1), five hand positions (i.e., mouth, chin, throat, side, cheek) are used to encode vowel groups, and eight hand shapes are used to encode consonant groups. For British English CS, four hand positions are used to code monophthong vowel groups and eight hand shapes are designed to code consonant groups. By using CS systems, sounds that may look identical on lips (e.g., /y/, /u/ and /o/), can be distinguished using hand information, and thus it is possible for the deaf people to understand a spoken language using visual information alone.

CS has drawn increasing attention from all over the world and has been adapted to more than sixty spoken languages for the moment. Another widely used communication method in deaf community is Sign language (SL) [START_REF] William | Sign language structure: An outline of the visual communication systems of the american deaf[END_REF], [START_REF] Scott | American sign language: The phonological base[END_REF], [START_REF] Valli | Linguistics of American sign language: an introduction[END_REF] which was developed in the early 18th century. SL is a language with its own grammar and syntax, while CS is a visual representation of spoken languages. The deaf people are able to master and learn the native language more easily by using CS [START_REF] Lasasso | Cued Speech and Cued Language Development for Deaf and Hard of Hearing Children[END_REF], [START_REF] Elizabeth | An examination of cued speech as a tool for language, literacy, and bilingualism for children who are deaf or hard of hearing[END_REF].

This work investigates the framework of asynchronous multi-modal fusion [START_REF] Peruffo Minotto | Multimodal multi-channel on-line speaker diarization using sensor fusion through svm[END_REF], [START_REF] Wu | Multimodal web aesthetics assessment based on structural svm and multitask fusion learning[END_REF], [START_REF] Tang | Integration of diverse data sources for spatial pm2. 5 data interpolation[END_REF] applied to automatic continuous CS recognition. Since the multi-modal streams in CS (i.e., lips, hand shape and hand position) that need to be fused are naturally asynchronous [START_REF] Attina | A pilot study of temporal organization in cued speech production of french syllables: rules for a cued speech synthesizer[END_REF], [START_REF] Attina | Temporal measures of hand and speech coordination during french cued speech production[END_REF], the feature fusion of CS recognition posses a major challenge. The supervised CS recognition system will be interfered if the lips and hand information are misaligned.

In the existing literature, CS feature modalities are assumed to be synchronous in the recognition task. In [START_REF] Heracleous | Cued speech automatic recognition in normal-hearing and deaf subjects[END_REF], [START_REF] Heracleous | Continuous phoneme recognition in cued speech for french[END_REF], direct feature fusion (i.e., direct concatenation of the features) was applied to the isolated 1 CS recognition without taking into account the asynchrony issue. In the state-of-theart [START_REF] Liu | Visual recognition of continuous cued speech using a tandem cnn-hmm approach[END_REF], a tandem architecture that combines convolutional neural network (CNN) [START_REF] Lecun | Deep learning[END_REF], [START_REF] Goodfellow | Deep learning[END_REF] with multi-stream hidden 1 Isolated CS recognition means that the temporal segmentation of each phoneme to be recognized are given at the test stage. markov model (MSHMM) [START_REF] Potamianos | Recent advances in the automatic recognition of audiovisual speech[END_REF] was used for the continuous CS recognition, and is referred as S 3 in this work. In [START_REF] Liu | Visual recognition of continuous cued speech using a tandem cnn-hmm approach[END_REF], MSHMM merges different features by giving weights to three feature modalities of CS, but it does not take into account the asynchrony issue between them. Therefore, there is still room for improvement regarding the CS recognition performance by exploring a pre-processing approach to tackle the fusion of asynchronous multi-modalities.

In this work, we propose a new architecture based on a novel re-synchronization method for asynchronous multimodal feature fusion in CS. The novelty stems in investigating the temporal organization of hand movement, which allows us to obtain the optimal hand preceding time (HPT) for both vowels and consonants. More precisely, our method composes of two main stages.

1) First of all, we build a hand preceding model (HPM) by analyzing the HPT on the database containing four CS speakers, and show that this model provides efficient segmentations of hand movements for all four speakers. Our proposed model significantly improves hand position recognition accuracy, and present the first main contribution of this work. 2) Secondly, we propose an efficient re-synchronization procedure based on the HPM to align the hand feature stream, so that the hand and lips movements are statistically synchronous, providing a good fusion condition. By incorporating this re-synchronization procedure into the tandem CNN-MSHMM architecture (see S re in Fig. 2), the automatic continuous CS recognition obtains a significant improvement compared with the S 3 in the stateof-the-art [START_REF] Liu | Visual recognition of continuous cued speech using a tandem cnn-hmm approach[END_REF]. This is the second main contribution of the present work. To our knowledge, the proposed resynchronization procedure is the first method to tackle the asynchrony issue of the CS multi-modal fusion applied to the automatic continuous CS recognition.

II. RELATED WORKS

The literature on automatic CS recognition can be classified into three main categories: feature extraction, multi-modal temporal modeling and CS recognition modeling. We will discuss them separately in this section.

A. Feature extraction in CS

In the literature on the automatic CS recognition, video images were recorded with artifices applied to the CS speaker (blue sticks on the lips, blue marks on the hand and forehead) to mark the pertinent information and make their further feature extraction easier. For example, in [START_REF] Heracleous | Cued speech automatic recognition in normal-hearing and deaf subjects[END_REF], [START_REF] Heracleous | Continuous phoneme recognition in cued speech for french[END_REF], lips feature was extracted by tracking the color marks on speaker's lips. Then a threshold was applied to the gray level images to segment the blue lips. The coordinates of the color marks on the finger were used as features for the hand shape and hand position modeling. In [START_REF] Burger | Cued speech hand gestures recognition tool[END_REF], the speaker wore a one-colored glove in order to help the hand segmentation. Stillitano et al. [START_REF] Stillittano | Lip contour segmentation and tracking compliant with lip-reading application constraints[END_REF] used active contours combined with parametric models to extract the lips contour in CS.

In our recent works [START_REF] Liu | Extraction automatique de contour de lèvre à partir du modèle clnf[END_REF], [START_REF] Liu | Automatic tracking of inner lips based on clnf[END_REF], [START_REF] Liu | Inner lips parameter estimation based on adaptive ellipse model[END_REF], several methods to get rid of these artifices on the speaker's lips and hand were explored. A modified constrained local neural fields (CLNF) model was proposed to extract the inner lips height and width, and an adaptive ellipse model was proposed for inner lips parameters extraction in CS. In this work, we adopt the deep CNN for the feature extraction of lips and hand shape, and use the artificial neural network (ANN) [START_REF] Lecun | Deep learning[END_REF], [START_REF] Svozil | Introduction to multi-layer feed-forward neural networks[END_REF] to process the hand position feature.

B. Multi-modal temporal modeling in CS

The temporal organization of hand movements in CS was studied in [START_REF] Attina | A pilot study of temporal organization in cued speech production of french syllables: rules for a cued speech synthesizer[END_REF], [START_REF] Aboutabit | Hand and lip desynchronization analysis in french cued speech: Automatic temporal segmentation of hand flow[END_REF]. For CV syllables 2 , it was found that the hand position reaches its target before the vowel being Fig. 2: Overview of the proposed automatic CS recognition architecture S re in this work with the re-synchronization procedure. T (L) , T (P) , T (S) are the temporal segmentations/boundaries for phonemes in case of lips, hand position and hand shape, respectively. Compared with the state-of-the-art architecture S 3 in [START_REF] Liu | Visual recognition of continuous cued speech using a tandem cnn-hmm approach[END_REF], the temporal relationship modeling (dotted box) and the re-synchronization procedure (i.e., the tier before the MSHMM-GMM decoder) are added.

visible at lips on average 239ms [START_REF] Attina | A pilot study of temporal organization in cued speech production of french syllables: rules for a cued speech synthesizer[END_REF] (based on the non-sense syllables logatome, like 'mamuma'), or 144.19ms [START_REF] Aboutabit | Hand and lip desynchronization analysis in french cued speech: Automatic temporal segmentation of hand flow[END_REF] (based on the syllables extracted from French continuous sentences). In this work, we focus on not only the HPT of vowels, but also that of consonants.

In our previous work [START_REF] Liu | Automatic temporal segmentation of hand movement for hand position recognition in french cued speech[END_REF], the relationship between the HPT of vowels and their target time instant (i.e., position in time) was analyzed. It was found that HPT follows a Gaussian distribution that remains almost the same for all the instants of vowels, except a small time interval (about 1s) before the end of each sentence. In this work, instead of following the piece-wise linear relationship, which gives different HPT for each vowel, we make a simple but efficient assumption that the mean value of the Gaussian distribution is suitable for all vowels. Moreover, based on the HPT for vowels in [START_REF] Liu | Automatic temporal segmentation of hand movement for hand position recognition in french cued speech[END_REF], we explore the optimal HPT of consonants, and then propose a re-synchronization procedure to align the hand features with lips feature for CS multi-modal feature fusion.

C. CS recognition modeling

The early work on the CS recognition is for the isolated vowel recognition in [START_REF] Aboutabit | Reconnaissance de la Langue Franc ¸aise Parlée Complété (LPC): décodage phonétique des gestes main-lèvres[END_REF], [START_REF] Beautemps | Telma: Telephony for the hearingimpaired people. from models to user tests[END_REF]. Then, the isolated CS phoneme recognition was realized in [START_REF] Heracleous | Cued speech automatic recognition in normal-hearing and deaf subjects[END_REF], and the continuous CS speech recognition based on an corpus of isolated words was conducted in [START_REF] Heracleous | Continuous phoneme recognition in cued speech for french[END_REF] using the context-independent HMM. For the automatic continuous CS recognition based on a corpus of continuous sentences, a tandem CNN-HMM architecture extracting the CS feature from raw image was proposed in [START_REF] Liu | Visual recognition of continuous cued speech using a tandem cnn-hmm approach[END_REF]. One-stream context-dependent HMM and MSHMM are both used for the multi-modal feature fusion, and MSHMM obtains better performance. However, none of them takes into account the asynchrony issue of the multi-modalities in CS.

In this work, we propose a new automatic CS recognition architecture S re by adding a re-synchronization procedure to process the features extracted by CNNs and ANN before feeding them to the context-dependent MSHMM-GMM decoder. The experimental results show that this re-synchronization procedure significantly improves the CS recognition performance compared with both the state-of-the-art of isolated [START_REF] Heracleous | Cued speech automatic recognition in normal-hearing and deaf subjects[END_REF], [START_REF] Heracleous | Continuous phoneme recognition in cued speech for french[END_REF] and continuous CS recognition [START_REF] Liu | Visual recognition of continuous cued speech using a tandem cnn-hmm approach[END_REF].

III. TEMPORAL ORGANIZATION OF HAND MOVEMENT

In this section, we investigate the temporal organization of hand movement in CS, which is important for establishing the HPMs for vowels and consonants in Section IV. We first illustrate the hand preceding phenomenon using an example of CS where the speaker utters fait des ([f E d e]) in a French sentence Il fait des achats. It can be seen in Fig. 3 that, when the CS speaker's hand points to the chin position for the vowel [E], she is only lip-reading the consonant [f], and the vowel [E] has not begun yet. This phenomenon are also observed for the syllables [d e] in this example.

Let the rectangles on the 3rd row (hand position tier) denote the time intervals in which the hand reaches its target position, and the rectangles on the 4th row (hand shape tier) denote the time intervals in which the hand prepares its shape to indicate consonants. It can be seen in Fig. 3 that the corresponding rectangles on the 3rd and 4th rows are aligned on the right end, and the bottom one is longer. The underlying reason is that in our database (will be introduced in Section V-A), we observe the following two facts. [START_REF]Deafness and hearing loss[END_REF] In the hand movement, the hand shape reaches its final shape at the same time as the hand position reaches its target for a vowel. [START_REF] Barbara | Hearing by eye: The psychology of lip-reading[END_REF] The hand stays at the target position only for a small time interval. During this period, the hand shape is almost formed, but the hand continues moving and rotating. Therefore, these intervals are relatively long (about 200ms). In fact, when the hand begins to leave the target position, the fingers move quickly to prepare the next hand shape.

As a consequence, the hand position is more sensitive to the asynchrony issue than the hand shape. This may be due to the intrinsic fact that the hand often stays in its target position for a relatively short time, while the full hand shape keeps a longer time in the coding process of CS. Now, we give some definitions and notations for HPMs. Without loss of generality, we only consider the CV syllables [START_REF] Richard | Cued speech[END_REF]. We are interested in how long time the hand precedes the lips movement. As shown in Fig. 4, for a vowel in a syllable, the middle instant of this vowel in the audio signal is denoted by t v , and the target instant in the hand position movement is denoted by t tar v . D v is the time interval in which the hand reaches its target position, which is set to be 60ms experimentally. Then the HPT (in ms) for vowels is

∆ v = t v -t tar v . (1)
For a consonant in a syllable, the middle instant of this consonant in the audio signal is denoted by t c . Since the hand preceding phenomenon, t c precedes t v . We assume that the complete hand shape is realized at the same time as the target position t tar v . However, this complete hand shape does not correspond to a single instant but a certain time interval naturally. This time interval is before t tar v because after this moment, the hand shape begins to change immediately. Let D c be the time interval corresponding to a given hand shape, and D c0 be the time interval in which a hand shape is almost formed but continues moving and rotating. The middle instant of D c is denoted by t tar c , which is the target instant in the hand shape. Then the HPT (in ms) for consonants is

∆ c = t c -t tar c . (2)

IV. NOVEL RE-SYNCHRONIZATION PROCEDURE

In this section, we first formulate the problem of multimodal fusion in CS recognition. Then, based on the temporal organization of hand movement in Section III, and by studying the HPT for vowels and consonants, we use them to explore the HPMs and build a new re-synchronization procedure.

A. Problem formulation

In the automatic continuous CS phoneme recognition, features of lips O (L) , hand position O (P) and hand shape O (S) are merged and fed into the phonetic decoder. Let the phoneme Υ extracted from a continuous French sentence at time t be determined as

Υ = arg max Υ P (O (LPS) |Θ Υ ), (3)
where

O (LPS) = [O (L) T , O (P) T , O (S) T
] T is the merged feature and Θ Υ is the model parameter for Υ.

As we introduced in Section III, hand position and hand shape features are both asynchronous with lips feature in CS. Therefore, at time t, the direct concatenated feature will be interfered and not suitable to train one particular phoneme class Υ. For example, in Fig. 3, the vowel [E] in lips feature is merged with [e] in hand position feature to train the reference vowel [E] if a direct concatenation fusion is applied to these features.

This work aims to propose a way to align O (P) and O (S) with O (L) , i.e., to build transformations τ 1 and τ 2 such that

O (P) resy = τ 1 (O (P) ), (4)
O (S) resy = τ 2 (O (S) ), (5)
are both synchronous with O (L) . Then the merged feature

O (LPS) resy = [O (L) T , O (P) T resy , O (S)
T resy ] T for phoneme Υ can be used to train the model of this phoneme without any interference.

B. Hand preceding model for vowels

Now we establish the relationship between HPT and t v based on our database (will be introduced in Section V-A). For this purpose, it is necessary to measure t tar v . To make them precise enough, we determine them manually. A manual temporal segmentation of vowels and consonants for each sentence is accomplished by using the movie editor Magix [START_REF] Daniel V Abreu | Podcasting: contemporary patient education[END_REF], [START_REF] Naylor | Magix Movie Edit Pro 2014 Revealed[END_REF]. Here, we set t tar v to be the middle instant of the temporal target interval, which contains several images around the hand target position.

All the t v can be obtained by the audio-based segmentation. Then ∆ v can be calculated by [START_REF]Deafness and hearing loss[END_REF]. Taking the LM speaker as an example, ∆ v of 1066 vowels extracted from 138 sentences are plotted in Fig. 5(a) with respect to t v . In this figure, all the points are aligned by their end (i.e., the instant 0) instead of the beginning, since in this way we can find a common rule between ∆ v and t v for all the vowels. More precisely, from the beginning of a sentence to a certain instant (about one second before the end), the statistical repartition of ∆ v is almost the same in this period. Then, ∆ v decreases until the end of the sentence and finally converges. By aligning all the sentences by their end, this phenomenon becomes very evident. Indeed, the distribution of short sentences and long sentences are superposed entirely at the end of sentences. Besides, by comparing the ∆ v distribution of other three speakers, we find that they follow the same repartition as shown in Fig. 5(b)-(d).

Based on these observations, we build the HPM as

∆(t) = ∆ v 0 < t < t 0 , at + b t 0 < t < L, ( 6
)
where t is the time instant of vowel, and ∆ v is the mean value of the ∆ v before the turning time instant t 0 . As shown in Fig. 5, we can see that the HPT of all vowels before t 0 follows a Gaussian distribution. Because we observe that the mean value ∆ v can reflect the main character of these points, we take it as the model. After t 0 , we make a linear regression based on these points since the HPT for vowels decreases linearly. a and b in ( 6) are the slope and intercept of the linear line after t 0 , respectively. In our case, t 0 stays at about 0.88s before the end of a sentence. The HPM ( 6) fits all sentences of four CS speakers. The proposed HPM for the vowel will be evaluated in Section VI-A, where a hand position recognition experiment is carried out. In this experiment, we propose a temporal segmentation method for the hand position movement based on the HPM. More precisely, from the audio-based segmentation, each temporal segmentation of vowels is shifted by ∆(t) using the HPM in [START_REF] William | Sign language structure: An outline of the visual communication systems of the american deaf[END_REF] according to the vowel instant in the sentence. Besides, the proposed temporal segmentation is also used to improve the efficiency of the ANN training for the hand position feature, as well as the CNN training for the hand shape feature (with respect to the HPM for the consonants) in the automatic CS recognition.

C. Hand preceding model for consonants

To investigate the HPM for the consonant, we first perform a statistical study on

δ cv = t v -t c , (7)
where t v and t c are the middle instant of the vowel and consonant in the audio signal.

Based on the LM corpus, we randomly choose 10 sentences containing about 100 syllables. δ cv of all these syllables are calculated and vary in a broad range. Therefore, we only consider its mean value (about 110ms). Since the optimal ∆ v is about 140ms, from Fig. 4, we can deduce that t tar v precedes t c by about 30ms (i.e., the difference between 140ms and 110ms). We assume that D c is about 60ms (i.e., three images) based on a large number of observations. Consequently, t tar c precedes t c by about 60ms, which is an theoretical estimation of the optimal ∆ c for all consonants.

Then we experimentally determine the optimal ∆ c . To do this, we perform a hand shape recognition experiment using the CNN-based hand features (will be introduced in Section V-C) and multi-Gaussian classifier. In the recognition, we apply several different temporal segmentations, which are derived by shifting the audio-based segmentation with different ∆ c (from 0 to 160ms with a step of 10ms). We denote by ∆ * c the ∆ c that gives the best hand shape recognition accuracy. A hand shape recognition experiment based on the LM corpus of all 476 sentences is conducted. 80% of the data is used for training and the rest 20% is for test. The recognition results are shown in Fig. 6, which gives the recognition accuracy as a function of the monotonically increasing ∆ c . We observe a convex curve (red curve) with a local maximum value of about 60ms. This is coherent with the theoretical analysis of ∆ c . Indeed, the peak region of this curve is relatively smooth, but the clear maximum value confirms that ∆ * c exists. The temporal segmentation of hand shape is obtained using the same way as for the hand position. This temporal segmentation of hand shape will be used in the CNN training for hand shape feature extraction in Section V-C.

D. Re-synchronization procedure

Instead of following the piecewise linear relationship in [START_REF] William | Sign language structure: An outline of the visual communication systems of the american deaf[END_REF], which gives different ∆ v for the vowels after t 0 , we propose a simple but efficient assumption that ∆ * v is suitable for all vowels, as introduced in Section IV-B. Similarly, we assume that ∆ * c is suitable for all consonants. Now we propose the re-synchronization procedure to align the hand shape and position features with lips feature based on the above HPMs for vowels and consonants. This procedure contains two steps:

1) Positively shift O (P) by ∆ * v , i.e., O (P) resy

(t) = τ 1 (O (P) )(t) = O (P) (t -∆ * v ), (8)
where

∆ * v = 140ms. 2) Positively shift O (S) by ∆ * c , i.e., O (S) resy (t) = τ 2 (O (S) )(t) = O (S) (t -∆ * c ), (9)
where ∆ * c = 60ms. We take the vowel case in Fig. 7 as an example to illustrate this procedure. Fig. 7(a) shows the audio signal of the French sentence Ma chemise est roussie with the phonetic annotations. The lips feature is assumed to be synchronous with the audio signal [START_REF] Tinwell | The effect of onset asynchrony in audio-visual speech and the uncanny valley in virtual characters[END_REF]. In Fig. 7(b), the hand position is defined to be the X coordinate of the hand back point. It is clear that the hand position stream is not synchronous with the audio signal. Therefore, a direct fusion of these two streams will cause some interference. In Fig. 7(c 140ms as in [START_REF] Valli | Linguistics of American sign language: an introduction[END_REF]. It turns out that the hand position stream is re-synchronized with the audio signal on average. For consonants, the alignment of the hand shape feature is similar, but with ∆ * c = 60ms. Even though the value of ∆ * v and ∆ * c may vary for different speakers, the proposed re-synchronization procedure is new, simple and efficient to improve the multi-modal fusion for continuous CS recognition.

V. EXPERIMENTAL SETUP

In this section, we first introduce the database and the experimental metric for the HPMs as well as the CS continuous recognition experiments. Then technical details for the automatic continuous CS recognition are presented.

A. Database

The database contains the recording data of three French CS speakers LM, SC, MD and one British English CS speaker CA. Three French CS corpora were recorded in a sound-proof room of GIPSA-lab, France. Color video images of the speaker's upper body are recorded at 50 fps, with a spatial resolution of 720×576 pixels RGB images. The LM speaker pronounces and codes a set of 238 French sentences in CS derived from a corpus described in [START_REF] Aboutabit | Reconnaissance de la Langue Franc ¸aise Parlée Complété (LPC): décodage phonétique des gestes main-lèvres[END_REF], [START_REF] Gibert | Analysis and synthesis of the three-dimensional movements of the head, face, and hand of a speaker using cued speech[END_REF]. Each sentence is repeated twice resulting in a set of 476 sentences (about 11770 phonemes, and 120K images totally). The SC corpus is made of 267 sentences from the same database as LM. The MD corpus contains videos of 50 French words made of numbers and daily words. The corpus is uttered 10 times. It should be mentioned that the LM corpus is recorded without using any artificial mark, while the SC and MD corpora are earlier recorded with artificial marks [START_REF] Aboutabit | Hand and lip desynchronization analysis in french cued speech: Automatic temporal segmentation of hand flow[END_REF], [START_REF] Aboutabit | Reconnaissance de la Langue Franc ¸aise Parlée Complété (LPC): décodage phonétique des gestes main-lèvres[END_REF]. However, in this work, the SC and MD corpora are only used to establish and evaluate the HPM and they do not attend the automatic CS recognition. Therefore, in this work, their artificial marks are not used at all although they exist.

The CA corpus is the first British English CS corpus 3 , which is recorded for this work in Cued Speech UK4 association, without using any artificial mark. The professional CS speaker CA (with no hearing impairment) is asked to simultaneously utter and encode a set of 97 British English sentences (e.g., I feel it is a time to move to a new chapter in my career). Color video images of the speaker's upper body are recorded at 25 fps, with a spatial resolution of 720x1280. There are totally 907 monophthongs and 138 diphthongs in this corpus.

In Section IV, we take a subset of the whole database to build the HPM, which contains 138 sentences including 88 short sentences 5 and 50 long sentences from LM corpus (totally 1066 vowels), and 44 short sentences (196 vowels) from SC corpus. The hand position is manually determined in the rule that it is assumed to be the 2D position of the middle finger if the middle finger appears; otherwise, it is assumed to be the index finger. Moreover, the hand back point is also tracked manually. The advantage of this point is that it is always visible in the CS coding process, and thus all the hand back points can be collected without any interruption.

For the CS phoneme recognition, all 476 sentences of LM corpus are used with 34 French phoneme classes (14 vowels and 20 consonants). We have made this database publicly available 6 . The French CS is described with 8 lips visemes (as defined in [START_REF] Aboutabit | Reconnaissance de la Langue Franc ¸aise Parlée Complété (LPC): décodage phonétique des gestes main-lèvres[END_REF]), 8 different hand shapes, and 5 different hand positions (as defined in Fig. 1). The phonetic transcription is extracted automatically using Lliaphon software [START_REF] Béchet | Lia phon: un systeme complet de phonétisation de textes[END_REF] and post-checked manually to adapt it to the pronunciation of the CS speaker. The audio-based temporal segmentation of each vowel is extracted from the conventional audio speech recognition (ASR) system in HTK 3.4 [START_REF] Steve | The HTK hidden Markov model toolkit: Design and philosophy[END_REF]. Using forced alignment, the audio signal synchronous with the video is automatically labeled.

B. Experimental metric

For the experiments including the hand shape recognition (for investigating the ∆ * c in Section IV-C), the hand position recognition (for evaluating the HPM in Section VI-A) and the CS vowel, consonant, phoneme recognition (for evaluating the proposed re-synchronization procedure in Section VI-B and Section VI-C), we randomly select 80% of the sentences as the training set while the rest 20% is the test set. Because each sentence is recorded twice, we allocate each sentence and its repetition either simultaneously to the training set or to the test set. All the results are repeated ten times with different training sets and test sets, so that the standard deviation (std) of these results can be controlled. It will be seen in Section VI that the stds of all the experimental results are less than 0.5%.

In CS phoneme recognition, the correctness of the MSHMM-GMM decoder

T corr = (N -D -S)/N ( 10
)
is chosen as the metric, where N is the number of phonemes in the test set, D is the number of deletion errors, and S is the number of substitution errors.

C. Automatic continuous Cued Speech recognition

The continuous CS recognition (see Fig. 2) is carried out to evaluate the proposed re-synchronization procedure, and it contains three steps: feature extraction, multi-modal feature fusion and phonetic decoding. Note that the re-synchronization procedure is used to deal with the multi-modal feature fusion. Now we only introduce the methods for feature extraction and phonetic decoding.

1) Cued Speech feature extraction: a) Lips and hand shape feature extraction: We first extract the regions of interest (ROIs) of lips and hand shape. The lips ROI is extracted using Kanade-Lucas-Tomasi (KLT) feature tracker [START_REF] Shi | Good features to track[END_REF] (with a dezooming process), and the hand shape ROI is extracted using Adaptive Background Mixture Model (ABMM) [START_REF] Stauffer | Adaptive background mixture models for real-time tracking[END_REF], which is able to track the variableshape object. Then, lips and hand shape ROIs are transferred to 2D gray images and resized to a fixed size using the cubic interpolation.

CNN is very powerful in extracting high-level features of images [START_REF] Lecun | Deep learning[END_REF]. In order to get rid of using the artifices (e.g., color marks on speaker's lips and hand), in this work, CNN is used to extract the lips and hand shape features from their raw ROI (i.e., nature images without marks). The CNN architecture in this work is almost the same as that in the state-of-the-art [START_REF] Liu | Visual recognition of continuous cued speech using a tandem cnn-hmm approach[END_REF] (see Fig. 8). The only difference is that, instead of using the hand movement temporal segmentation derived by the simple procedure (i.e., the left temporal boundary of each phoneme is forced to extend to the left boundary of its previous phoneme), in this work, we use the HPM-based temporal segmentations of hand position and hand shape movement for CNN training.

b) Hand position feature extraction: In [START_REF] Heracleous | Cued speech automatic recognition in normal-hearing and deaf subjects[END_REF], [START_REF] Heracleous | Continuous phoneme recognition in cued speech for french[END_REF], hand position was tracked by detecting the color marks on the hand, and no automatic method was used. In our previous work [START_REF] Liu | Visual recognition of continuous cued speech using a tandem cnn-hmm approach[END_REF], ABMMs were used to automatically extract the hand position in CS on raw hand ROIs. It is then processed by a simple feed-forward ANN, which contains two standard fully connected layers with ReLU activation function. Two hidden layers with four neurons in each layer are used for the hand position feature processing. After the softmax layer, it will output the posterior probabilities of the target classes with dimension 6, which is the final hand position feature. A mini-batch gradient descent algorithm based on the Root Mean Square Propagation (RMSprop) adaptive learning rate method is used for the parameter optimization in ANN.

2) MSHMM-GMM phonetic decoding: As in the S 3 architecture [START_REF] Liu | Visual recognition of continuous cued speech using a tandem cnn-hmm approach[END_REF], in the proposed S re architecture, each phoneme is modeled by a context-dependent triphone MSHMM (i.e., takes into account the contextual information about the preceding and following phonemes) [START_REF] Young | Tree-based state tying for high accuracy acoustic modelling[END_REF]. Three emitting states are used with GMM to model the features of lips, hand position and hand shape together with their first derivatives. The main difference between S 3 and S re is that MSHMM-GMMs are used to model the re-synchronized multi-modal features in S re , while in S 3 , MSHMM-GMMs are used to model the asynchronous multi-modal features. In S re , the emission probability at state j is bj(O

(LPS) t ) = S 0 s=1 Ms m=1 cjsmN(O (LPS) st ; µjsm; Σjsm) λs , (11)
where

O (LPS) t = [O (L) T , O (P) T resy , O (S)
T resy ] T is the merged feature at time t, and S 0 is the number of feature streams, which is set to be 3 in this work. N(O (LPS) st ; µ jsm ; Σ jsm ) is the probability density function of the observation O (LPS) st for feature stream s and time t. µ jsm is the mean value and Σ jsm is the covariance matrix of the Gaussian probability density function with state j, mixture Gaussian component m and stream s. For stream s, M s gaussians are used to model the observation, with weight c jsm . In the experiment, M s iteratively increases from 1 to 4 in the training, and is finally set to be 2 which achieves the best performance. λ s is the weight for feature stream s, which Fig. 8: Implementation details of the CNNs used to extract the hand shape features from hand ROI in CS.

is optimized using the cross-validation. These weights should satisfy

3 s=1 λ s = 1, 0 ≤ λ s ≤ 1. (12)
Finally, the optimal weights are set to be 0.4, 0.4 and 0.2 for lips, hand shape and hand position, respectively. It should be mentioned that, in this work, we do not take into account the pronunciation dictionary or language model in order to directly compare the CS recognition results with the state-of-the-art [START_REF] Liu | Visual recognition of continuous cued speech using a tandem cnn-hmm approach[END_REF], which does not incorporate them.

VI. RESULTS AND DISCUSSIONS

In this section, we first evaluate the performance of HPM for vowels. Then a two-step evaluation of the proposed CS recognition architecture is carried out to evaluate the performance of the proposed re-synchronization procedure.

A. Evaluation of the HPM-based temporal segmentation of hand position

To evaluate the HPM-based temporal segmentation of hand position, we compare it with both the ground truth and audiobased segmentations. (1) For the ground truth, the target instants of all vowels are determined manually, which constitutes a golden reference for the hand position recognition in CS. [START_REF] Barbara | Hearing by eye: The psychology of lip-reading[END_REF] In the literature [START_REF] Heracleous | Cued speech automatic recognition in normal-hearing and deaf subjects[END_REF], [START_REF] Heracleous | Continuous phoneme recognition in cued speech for french[END_REF], the temporal segmentation of hand position is always based on the audio signal. It is important to compare this segmentation with the HPM-based segmentation to see the potential benefits of HPM.

We first evaluate the proposed HPM by visualizing the hand position distributions in a 2D image using different temporal segmentations. Then, we apply the simple Gaussian classifier to hand position recognition.

Hand position spatial distributions of vowels for four CS speakers using the target finger points and different temporal segmentations are shown in Fig. 9. It is reasonable to assume that the better temporal segmentation corresponds to more distinguishable and separable hand position distributions. It can be seen that the Gaussian ellipse becomes more and more distinguishable from left to right for all these four speakers. Note that the second speaker SC codes CS using her left hand while other three speakers use their right hand. Taking the LM case corresponding to (a), (b) and (c) as an example, the points in (a) have large parts of overlaps for five positions, while the points in (b) are significantly separable. In particular, the points at the throat and chin positions are divided. Moreover, the distribution of the Gaussian ellipse in (b) is very close to that in (c) with only a few intersections between the ellipses. These distributions efficiently illustrate a satisfactory performance of the HPM.

For further evaluation, we apply these three temporal segmentations to the hand position recognition experiments for four speakers. The experiment results are reported in Table I. Five multi-gaussian models are trained for the five positions based on a sub-database containing 138 sentences from LM corpus, a sub-database containing 44 sentences from SC corpus and all vowels from 50 words of MD corpus. For the British English CS corpus CA, four multi-gaussian models are trained for the four positions based on the CA corpus.

Using the target finger positions and the ground truth temporal segmentation, recognition accuracies of all four speakers are higher than other two cases which use the audio-based and HPM-based segmentations. More precisely, for speakers LM, SC, MD and CA, the highest accuracies of 96.9%, 99.9%, 98.3% and 92.02%, are achieved (see Fig. 9(c), (f), (i) and (l)), respectively. This constitutes golden references to the hand position recognition in CS for these four speakers. However, when the audio-based segmentation is used, the recognition accuracies become lower 64.23%, 75.1%, 65.8% and 54.61% (see Fig. 9(a), (d), (g) and (j)), respectively. When using the proposed HPM-based segmentation, compared with the audiobased segmentation, the accuracies increase significantly to 82.4%, 83.9%, 73.1% and 65.03% (see Fig. 9(b),(e),(h) and (k)), respectively. All these results show that the audiobased segmentation is indeed not suitable for the hand position recognition, and our proposed HPM improves the performance of hand position recognition significantly.

Note that for MD corpus, the improvement (around 7.5%) using the HPM-based temporal segmentation is not as evident as LM and SC corpora. This is because the MD corpus is composed of single words with a comparably short time duration, while LM and SC corpora are made of continuous sentences. Therefore the hand preceding phenomenon in MD corpus is not as strong as other two corpora.

Above all, we see that the HPM-based temporal segmentation can significantly improve the performance of hand position recognition. B. Evaluation of the re-synchronization procedure applied to the vowel and consonant recognition in CS Now, we evaluate the proposed re-synchronization procedure by carrying out the first step, i.e., vowel and consonant recognition experiments based on the fused features of two streams. We use MSHMM-GMM to see the benefits of this re-synchronization procedure.

1) Vowel recognition based on the fusion of lips and hand position: We mainly discuss the vowel recognition based on the fusion of lips and hand position. The vowel recognitions using only lips and only hand position information are also presented.

Lips and hand position features are merged and fed to a twostream MSHMM with triphone context-dependent modeling. 14 vowels (i.e., [i, e, E, a, y, ø, u, o, O, Ẽ, oe, ã, Õ, @]) plus silence are the target labels. Results are shown in Table II. We see that the vowel recognition using the re-synchronization procedure (8) obtains a higher correctness (74.65%) than the one without using it (70.12%). An improvement of about 4.5% is achieved. For the single stream case, using only lips and only hand position give very similar and low correctness. This is reasonable since these two streams are supposed to be combined to recognize the vowels, and every single stream only carries the viseme-level information. TABLE II: Vowel recognition results using lips and hand position. T corr is the correctness and '-' means that this case does not exist. The shorthand 'one-stream' means the CS vowel recognition using only one stream feature. 'non-resyn' means the CS recognition without using the re-synchronization procedure, while 'resyn' means using this procedure. 2) Consonant recognition based on the fusion of lips and hand shape: Lips and hand shape features are merged and fed to a two-stream MSHMM-GMM with triphone contextdependent modeling. 18 consonants (i.e., [p, t, k, b, d, f, s, S, v, z, Z, m, n, l, g, r, j, w]) plus silence are the target labels. In Table III, we can see that using the re-synchronization procedure achieves a higher correctness (82.28%) than the one without using it (80.33%). For the single stream case, using only lips and only hand shape obtain a much lower correctness compared with the fused feature. TABLE III: Consonant recognition using lips and hand shape. T corr is the correctness and '-' means that this case does not exist. The shorthand 'one-stream' means the CS consonant recognition using only one stream feature. 'non-resyn' means the CS recognition without using the re-synchronization procedure, while 'resyn' means using this procedure. To further evaluate the proposed re-synchronization procedure, a new multi-modal CS phoneme recognition architecture S re is investigated (see Fig. 2). We compare the result of S re with the state-of-the-art S 3 in [START_REF] Liu | Visual recognition of continuous cued speech using a tandem cnn-hmm approach[END_REF], which does not take into account the asynchrony issue in CS. The results are shown in Fig. 10. Now we make some analysis. We first focus on the case when the hand positions given by ABMMs are used, as this is the case in [START_REF] Liu | Visual recognition of continuous cued speech using a tandem cnn-hmm approach[END_REF]. Using S 3 architecture, the phoneme recognition correctness is 71.0%, without using any re-synchronization procedure. When the proposed resynchronization procedure is incorporated, the recognition correctness increases to 72.67%, which shows a minor improvement of 1.67% (see columns 3 and 4 in Fig. 10). We realize that in S re , the triphone context-dependent modeling of MSHMM is helpful to correcting the recognition errors, which could be the co-articulation or the asynchrony of multimodalities [START_REF] Schwartz | Multimodal speech: Two or three senses are better than one[END_REF]. Therefore, using the context-dependent modeling may hide the effect of the re-synchronization procedure. In order to avoid this phenomenon, we examine the phoneme recognition correctness without using this context-dependent modeling. Then only 60.4% is obtained without any resynchronization procedure, while it increases to 64.38% when using the proposed re-synchronization procedure (see columns 1 and 2 in Fig. 10). It can be seen that the improvement of about 4% is more evident than the case of using the contextdependent modeling (1.67%).

The above improvement is weak. This may be because (1) only a weight of 0.2 is applied to the hand position stream, and (2) the hand position extracted by ABMMs has some errors (see Section V-C). For the first reason, as mentioned in Section III, the hand position feature is much more sensitive to the asychrony problem in CS. The small weight reduces the importance of the re-synchronization procedure. For the second reason, the errors directly reduce the efficiency of the re-synchronization procedure and cause some interference of the CS recognition system, since the hand position target can be recognized accurately only if the correct hand position feature is used at a good temporal boundary for a particular vowel. Note that in this work, we only consider the interference caused by the hand position errors because the CNNbased features for lips and hand shape are shown to be correct in [START_REF] Liu | Visual recognition of continuous cued speech using a tandem cnn-hmm approach[END_REF].

In order to see the effects when the hand position is correct, we use the ground truth hand position instead of that given by the ABMMs. The CS phoneme recognition results in this case are shown in Fig. 10, from column 5 to column 8. Now we make some discussions. In this case, without the context-dependent modeling or re-synchronization procedure, the correctness is 62.33%, which is close to the result of 60.4% based on the hand positions by ABMMs. This can be explained by the above first reason. However, when the re-synchronization procedure is used, a correctness of 70.1% is achieved, which shows a significant improvement of about 7.7% (see columns 5 and 6 in Fig. 10). This shows benefits of the proposed re-synchronization procedure.

When using the context-dependent modeling (see columns 7 and 8 in Fig. 10), the correctness is 72.04% without the re-synchronization procedure. However, when using both of them, a significant correctness of 76.63% is obtained (with an improvement of 4.6%) compared with the state-of-the-art [START_REF] Liu | Visual recognition of continuous cued speech using a tandem cnn-hmm approach[END_REF] in the automatic continuous CS phoneme recognition case. Moreover, this result also outperforms the state-of-the-art 74.4% [START_REF] Heracleous | Cued speech automatic recognition in normal-hearing and deaf subjects[END_REF] which is for the automatic isolated CS phoneme recognition.

We also calculate the CS phoneme recognition correctness when using only the lips information. The result is low (around 30%). It can be seen that by using the CS, the phoneme recognition performance increases by a large margin of around 46.63% compared with the case of using lip reading only. This confirms that CS can significantly help the Deaf/Hard of hearing on speech perception and production.

VII. CONCLUSION

In this work, a novel re-synchronization procedure is proposed for the CS multi-modal feature fusion applied to a practical automatic continuous French CS recognition system (i.e., CNN-MSHMMs). First, by exploring the relationship between the HPT and the time instants of phonemes in French continuous sentences, we obtain the optimal HPTs for all vowels (140ms) and for all consonants (60ms), and develop new HPMs. Then, the re-synchronization procedure is proposed by delaying the hand position and shape streams with these two different optimal HPT, respectively. The lips and hand features can be re-synchronized on average. By incorporating the proposed re-synchronization procedure to the CNN-MSHMMs based CS recognition system, we propose a new architecture S re that takes into account the asynchrony issue in the CS recognition. The evaluation of the automatic CS phoneme recognition using S re achieves a significant improvement (about 4.6%) compared with the state-of-the-art architecture S 3 . In the future work, we aim at investigating the proposed fusion methods in CS of other languages (e.g., the American English) and generalizing it to be speakerindependent. Fig. 10: Results of the continuous CS phoneme recognition with and without using the proposed re-synchronization procedure and the context-dependent modeling. The shorthand 'non-resyn' means the CS recognition without using the re-synchronization procedure, while 'resyn' means using the re-synchronization procedure. The error bars are the stds of the results (less than 0.5) and all the differences are statistically significant.

Fig. 1 :

 1 Fig. 1: Hand coding in French Cued Speech system. Five hand positions (mouth, chin, throat, side, cheek) are used to code vowel groups. Eight hand shapes are used to code consonant groups. The * in the side position is used to code a single consonant, and the ** in the full hand shape is used to code a single vowel.

Fig. 3 :

 3 Fig. 3: Illustration of the asynchrony phenomenon in the CS lips-hand movement. Two syllables [f E] and [d e] are extracted from the French sentence Il fait des achats. The red vertical lines show the target instants of the vowels and consonants in the audio signal, hand position and shape streams, respectively.

Fig. 4 :

 4 Fig. 4: Illustration of different parameters concerning the hand movement temporal organization in CS. ∆ c and ∆ v are the HPT for consonants and vowels. D v and D c are the time intervals for target hand position and hand shape.

Fig. 5 :

 5 Fig. 5: HPT distribution and HPM. The abscissa is the vowel instant in sentences. All these points are aligned at the end of sentences, where the instant is 0. Y-axis: the ∆ v (in seconds). In (a), the red circles show the distribution of 50 long sentences, and the blue stars show the distribution of 88 short sentences from LM corpus. The black curve shows the HPM. In (b), the blue stars show the distribution of 88 short sentences from LM corpus and the magenta stars show the distribution of 44 short sentences from SC corpus. In (c), the blue stars show the distribution all 83 vowels of the MD corpus made of 50 single words. In (d), the blue stars show the distribution all 1045 vowels of the CA corpus made of 97 British English sentences.

Fig. 6 :

 6 Fig. 6: Hand shape recognition accuracy using different segmentations in function of ∆ c (red curve). The green circle highlights the optimal recognition accuracy when ∆ c ≈ 60ms.

Fig. 7 :

 7 Fig. 7: Illustration of the re-synchronization procedure. (a) The audio speech with its temporal segmentations and phonetic annotations. (b) The original hand position trajectory (i.e., X coordinate of the hand back point). (c) The re-synchronized hand position derived by shifting the original hand position in (b) with ∆ v . Two green lines correspond to the audio based temporal segmentation of vowel [i].

Fig. 9 :

 9 Fig. 9: Hand position (target finger position) distributions with different temporal segmentations for four CS speakers LM, SC, MD and CA, respectively. (a), (d), (g) and (j): the audio-based segmentation. (b), (e), (h) and (k): the HPM-based segmentation.(c), (f), (i) and (l): the ground truth temporal segmentation. For French CS speakers LM, SC and MD, five groups of points correspond to different hand positions. Red points: cheek; green points: mouth; black points: throat; cyan points: chin; blue points: side. Note that the speaker SC uses her left hand to code CS, while other three use their right hand. In British English CS, only four hand positions are used to code all vowels. Thus, for the speaker CA, the meanings of four colors are listed as following. Red points: mouth; black points: throat; green points: chin; blue points: side.

TABLE I :

 I Hand position recognition results using multi-Gaussian classifier based on different temporal segmentations of hand position movement for four CS speakers. Ground truth target finger is used as the hand position. Audio-based segmentation is the temporal segmentation used in the previous work[START_REF] Heracleous | Cued speech automatic recognition in normal-hearing and deaf subjects[END_REF],[START_REF] Heracleous | Continuous phoneme recognition in cued speech for french[END_REF] and the HPM-based segmentation is the temporal segmentation predicted by the proposed HPM for hand movement.

	Recognition accuracy (%) audio-based segmentation HPM-based segmentation	ground truth segmentation
	LM	64.23	82.35	96.93
	SC	75.11	83.93	99.91
	MD	65.77	73.10	98.26
	CA	54.61	65.03	92.02

syllables consist of a consonant followed by a vowel.

This database will be made publicly available on Zenodo.

http://www.cuedspeech.co.uk/

In this work, a short sentence means a sentence with less than 4 vowels. Otherwise, it is a long sentence.

https://doi.org/10.5281/zenodo.1206001.

ACKNOWLEDGEMENT

The authors would like to thank the CS speakers for their time spent on the French CS data recording, and Thomas Hueber for his help in CNN-HMM. The authors would like to thank the referees for their valuable comments and suggestions. This work is supported by a PhD thesis grant of Université Grenoble Alpes in France, and in part by the Natural Sciences and Engineering Research Council of Canada under Grant RGPIN239031.


M Bertrand

La A Mojica

H Muhr

E Plasari

F Auger

Modelling of

Keywords: Precipitation, Fluidized bed reactor, Modelling

HAL is

Introduction

Modelling of precipitation reactions requires the knowledge of flow pattern inside the reactor, the thermodynamic properties as well as the kinetic laws of the different mechanisms (nucleation, crystal growth, agglomeration…). Thanks to this information, population balance can be solved in support of design and operating of industrial process. Precipitations performed in fluidized bed reactor are commonly used, as fluidized bed leads to solid particles of excellent properties, in terms of morphology, granulometry or density. In this study we propose a new simple model to describe precipitations carried out in continuous fluidized bed reactor equipped with a mechanical stirring: the Mixed Suspension Separated Product Removal (MSSPR). Inspired by the classical Mixed Suspension Mixed Product Removal (MSMPR), this model was applied to the precipitation of uranium peroxide. This latter is involved in the processing of uranium ores to recover uranium from purified leach solution as yellow cake [1].

Thermodynamic study

The uranium peroxide precipitation is achieved at pH ~ 3 by mixing hydrogen peroxide and uranyl sulfate solutions according to the following equilibrium:

UO 2 SO 4 + H 2 O 2 𝐻2𝑂 ↔ UO 2 (O 2 ) .4 H 2 O + H 2 SO 4
(1) As the ore solutions are concentrated and multicomponent with a high ionic strength, the deviation from the ideality has to be taken into account to calculate the driving force. The supersaturation ratio, S, is linked to the activities of the uranyl and peroxide ion. The uranyl ion activity is obtained as the product of the uranyl activity coefficient and the uranyl free concentration, which are calculated from the Specific Interaction Theory and the total concentration balance, respectively [2].

Kinetic study

The kinetic laws for the primary nucleation were identified using a specific apparatus patented by the French Atomic Energy Commission and AREVA. As the primary nucleation occurs very rapidly, this apparatus allows the reagent mixing state at molecular scale to be controlled. It has been shown to achieve very efficient micromixing time, inferior to the millisecond, with a high degree of precision and reproducibility. The homogeneous and heterogeneous nucleation rates are in agreement with the classical theory proposed by Volmer and Weber. The crystal growth rate was studied in a batch stirred tank reactor. It was found to be controlled by the surface integration into the crystal lattice, according to a screw dislocation mechanism [2]. The crystal growth rate is first order with respect to the supersaturation and independent from the impeller speed. Dealing with the agglomeration kernel, it was determined from continuous experimental runs and the mathematical treatment of the crystal size distributions. It appears to be particle size independant, directly proportional to the crystal growth rate and inversely proportional to the shear rate [3]. Experimental runs performed at pilot scale in a continuous fluidized bed reactor showed that the breakage is negligible.

The MSSPR model

While the feed flow rate entering the column in our conditions was not sufficient to create fluidization, the use of a high recirculation rate meant a fluidized-bed was obtained in which a good reagent mixing took place. The fluidized bed precipitator operating in a different way from the perfect mixed reactor, the classical equations of population balance have to be adapted. That is why, we propose a new model which takes into account different outlets for the solid and liquid phases as shown in Figure 1 where Q is the total feed flow rate and q is the outlet flow rate of the suspension. The MSSPR model is based on a separation coefficient between the liquid phase and the suspension: α=Q/q. For a MSSPR precipitator in presence of nucleation, crystal growth and agglomeration, the population balance in terms of moment can be expressed as follows:

𝑚 𝑘 𝛼𝜏 = 𝑅 𝑁 • 𝐿 * 𝑘 + 𝑘 • 𝐺𝑚 𝑘-1 + 𝛽𝐼 𝑘 (2)
with

𝐼 𝑘 = ∫ ∫ [ 1 2 (𝐿 3 + 𝜆 3 ) 𝑘/3 -𝐿 𝑘 ] • (𝐿) • (𝜆) • 𝑑𝐿 • 𝑑𝜆 ∞ 0 ∞ 0 (3)
where  is the residence time (s -1 ), L and  are the particle sizes, (L) is the number particle size density, m k is the k-order moment, G is the crystal growth rate, R N is the nucleation rate,  is the agglomeration kernel. (5) Figure 2 shows an example of the comparison between the experimental results and the equation ( 4) and points out a good agreement.

Conclusion

We present in this study a new model adapted to any precipitation reaction performed in a continuous fluidized bed reactor with a recirculation and a mechanical stirring. The successful application of this model reveals its potential for sizing fluidized-bed reactors at the industrial scale.

Acknowledgments:

The authors gratefully acknowledge the help of AREVA in sponsoring this study.

References:

[1] L.A. Mojica-Rodriguez, M. Bertrand, J.P. Gaillard, F. Auger, H. Muhr, E. Plasari, Experimental study on the uranium peroxide precipitation in a fluidized-bed reactor, Nuclear Engineering and Design, Vol. 293, pp 150-158 (2015).

[2] S. Planteur, M. Bertrand, E. Plasari, B. Courtaud, JP. Gaillard, Thermodynamic and crystal growth kinetic study of the uranium peroxide, CrystEngComm -The Royal Society of Chemistry, Vol. 15 (12), pp 2305-2313(2013).

[3] M. Bertrand, L. Mojica, E. Plasari, F. Auger, H. Muhr, Study on Agglomeration Kinetics of Uranium Peroxide, Procedia Chemistry, pp. 263-270 (2016).

Figure 1 .

 1 Figure 1. Schematic diagram of the MSSPR model.Figure 2. Comparison between experiments (circles) and the MSSPR model (line)

Figure 2 .

 2 Figure 1. Schematic diagram of the MSSPR model.Figure 2. Comparison between experiments (circles) and the MSSPR model (line)

  The ability of the MSSPR model to describe the precipitation in continuous stirred fluidised bed with recirculation was assessed from several experiments at pilot scale. Equation (2) can be written in logarithmic scale as follows: log(Y k ) = log() + log(• 𝐿 * 𝑘 -𝑘 • 𝐺 𝑚 𝑘-1