Lei Chen
email: lei.chen@univ-ubs.fr

Sylvie Gibet
email: sylvie.gibet@univ-ubs.fr

Camille Marteau
email: cam.marteau@hotmail.fr

CONDUCT: An Expressive Conducting Gesture Dataset for Sound Control

Keywords: Corpus, sound-control gestures, expressive gesture, conducting

come

Introduction

The use of 3D mid-air gestures for controlling sound interactively has gained attention over this past decade (R. [START_REF] Aigner | Understanding mid-air hand gestures: A study of human preferences in usage of gesture types for hci[END_REF] One of the main difficulties to design such interfaces is the lack of guidelines to help designers in creating a set of gestures that can be significant for the given application and adopted by a large number of users. The purpose of this study is to be able to control and modulate a live sound with expressive gestures. In live performances indeed the use of human movements to control both the visual and sound outputs have brought an intuitive and natural dimension to the interaction, leading to unprecedented performances, rich in body sensations and artistic creation. Among gestures controlling sound we are seeking a subset of meaningful and expressive gestures performed in the air, without any instrument. Furthermore, we expect the message conveyed by each gesture to correspond to an underlying desired function, and the quality encoded in the movement to correspond to an understandable expressive intent. The gestures should also be sufficiently different from each other, so that they may be automatically discriminable and recognizable. And finally, as in high-level languages, the structure of the produced gestures should allow to efficiently carry expressive variations by changing only a few gesture features such as kinematics, dynamics, geometry, or hand shape features. Our search for a gestural language to drive sound applications naturally led us to consider conducting gestures performed by orchestral conductors. These gestures are of particular interest, since they gather most of the mentioned properties. Moreover, they are highly coded gestures which are potentially understandable by most of musicians around the world [START_REF] Meier | The Score, the Orchestra, and the Conductor[END_REF]. Even if each conductor has his own style and defines his own set of gestures, we may find a subset of those gestures that share common meaning and form.

Most descriptions of the conducting gestures are concerned by gestures executed by the dominant hand, i.e. beating gestures that indicate the structural and temporal organization of the musical piece (tempo, rhythm) and lead to precise, efficient, and unambiguous orders to the orchestral musicians. Other gestures performed by the non dominant hand are dedicated to show other aspects of the music execution and interpretation, among which variations in dynamics and intensity, musical phrasing or articulation, accentuation, entries and endings, sound quality and color, etc.These musical cues are sometimes noted as specific symbols or semantic terms in the score, but there is no agreement between musicians, musicologists or composers to precisely define the meaning of these additional notations on the scores. When the musical piece is performed, these indications can be translated into gestures that express the desired musical expression. For the musician, it comes down to interpreting, through his instrumental gesture, the nuance indicated on the score. For the conductor, the interpretation is understood more generally at the level of the ensemble of the musicians and the intention of the musical excerpt, and his gestures indicate in a slightly anticipatory manner the nuance that the musicians must achieve. We propose here to define a new data set of expressive gestures that are inspired by these conducting gestures performed by the non dominant hand. Our gesture selection has been partly guided by the joint work of professional conductors and linguists studying sign languages [START_REF] Braem | A pilot study of the ex-pressive gestures used by classical orchestra conductors[END_REF]. However, since we target non musicians in our interactive application, we chose intuitive, expressive, and easy to perform gestures, in order to efficiently control a musical performance.

In this paper, we present the design and construction of this new multimodal data set, called CONDUCT, composed of associated gestures and musical excerpts. To this end, we defined a set of interacting gestures that are likely to express and control the performance of music excerpts. We have identified four functional categories of gestures that corre- spond to classically used musical effects. These categories relate to indications in Articulation, Intensity, Attack, or Cutoff. Within each category, we have characterized several expressive variations. The paper is organized as follows: section 2 describes the main principles that have led to the design of the data set, both for the sound excerpts and for the design of associated gestures. Section 3 details the experimental setup, section 4 presents a first analysis of the data and section 5 concludes the paper and raises the main perspectives.

Overview of the Study

Figure 1 illustrates the way the data are collected (motion capture and sound recording data) and the future use of the database (recognition of actions and their variations). Concerning the experimental protocol, we first selected a set of appropriate musical excerpts that illustrate the main categories of sound effects (Articulation, Intensity, Accentuation, Cutoff) and their variations within each category. Our grammar of gestures follows the same protocol, by defining one specific gesture per category and variation. Many researchers have designed experiments to record and analyze music-related gestures. Previous work has shown that gesture-sound correspondence is mutual, i.e. musical gestures help to understand the music, and, conversely, gestures can be used in turn to control the music. [START_REF] Godøy | Playing "air instruments": mimicry of sound-producing gestures by novices and experts[END_REF] have studied air-instrument playing gestures where subjects were asked to observe real musician performances and imitate them as if they were playing the piano. In another study, they have asked the subjects to listen to the sound and draw induced sketches on a tablet [START_REF] Godøy | Exploring music-related gestures by sound-tracing: A preliminary study[END_REF]. Both studies contribute to the understanding of gesture-sound mapping. F. Bevilacqua et al. also proposed a procedure to learn the gesture-sound mapping where the users' gestures are recorded while they are listening to specific sounds [START_REF] Bevilacqua | Wireless sensor interface and gesturefollower for music pedagogy[END_REF] (Bevilac-qua et al., 2011). On the basis of these theoretical studies, J. Franc ¸oise et al. have design several gesture-tosound applications [START_REF] Franc ¸oise | A hierarchical approach for the design of gesture-to-sound mappings[END_REF][START_REF] Franc ¸oise | A multimodal probabilistic model for gesture-based control of sound synthesis[END_REF]. This work supports the approach we have adopted, by building the database from the music guiding the conductor rather than vice versa. As illustrated in the right part of Figure 1 (Analysis), the aim of this gesture database is to serve as a basis for further gesture recognition for live sound control. Each recognized action will directly affect the sound effect (whether real or synthesized), and each detected gesture variation will correspond to a sound variation. In our approach the gestures are codified following sign language compositional structure, by identifying for each gesture the couple of components (hand movement, hand shape). With such a structure, it is possible to express the gesture variations as one modulation of either the hand movement (varying for example the form of the trajectory or the kinematic of the movement), or the hand shape (varying the shape of the hand). Previous work in sign recognition from video has demonstrated some success on isolated and continuous signing. Adding some linguistic knowledge about the composition of lexical signs considerably improves the performances of the recognition system [START_REF] Dilsizian | A new framework for sign language recognition based on 3d handshape identification and linguistic modeling[END_REF]. Although these results are not achieved on motion capture data, they are promising for our training recognition system based on similar linguistic components.

Corpus Design

The purpose of this new data set was to be able to modulate a live sound with expressive gestures. From this hypothesis, the first questions that arose were: Which variables of the sound should we change? How much? In which categories should we consider such variations? Which variations should we take into account within the category? What qualifies as an expressive gesture? How is it related to the sound variation? Instead of directly defining a set of non dominant hand gestures, as was done in [START_REF] Braem | A pilot study of the ex-pressive gestures used by classical orchestra conductors[END_REF], we first listed different sound effects and grouped them in main functional categories. Among each category, we identified different variations that are commonly found in musical scores. We have then defined possible conducting-like gestures corresponding to these categories and variations, strongly inspired by the structural and iconic description of deaf sign languages.

Sound Categories and Variations

A musical score provides all the information needed to build an interpretation of the musical piece. Through this score, musicians (instrumentalists, singers) are able to read a musical piece and transform it in a gestural performance with musical sound. The challenge of the conductor is to have a global idea of the composer's musical intention, to imagine sounds and colors, and to read and understand all the scores of all the instruments. Among the information not contained in the temporal organization (tempo, rhythm) of the musical excerpt, we have identified four main categories: Articulation, Intensity, Attack, and Cutoff.

• The Articulation category is related to the phrasing of the musical discourse which is strongly dependent on the style of the piece. This expresses the way in which specific parts or notes in a piece are played within the musical phrasing, how they are linked and co-articulated, taking into account the timing and the quality of the musical sequencing. Among the techniques of articulation, we retained three of them: Legato (linked notes), Staccato (short and detached notes), and Tenuto (completely held and sustained). In our examples, we are aware that these terms and their meaning might differ according to the instrument and the musical context.

• The Intensity category, also called Dynamics in musicology, characterizes the music loudness. In our study we are interested in variations of intensity. These variations can be progressive (smooth) or abrupt with an increase or decrease in intensity. Four Intensity variations have been retained: Long Crescendo, Long/Medium Decrescendo, Short Crescendo, Short Decrescendo.

• The Attack category gathers different types of accents which are indicated in the score by different symbols but also by terms such as sforzato (sfz). In our study, we have identified two main discriminating attacks: Hard hit, Soft Hit.

• The Cutoff category expresses the way a musical phrase ends. We have retained two main variations within this last category: Hard Cutoff, Soft Cutoff.

These categories and variations have been retained for two kinds of musical excerpts:

• Orchestral classical music; some of the excerpts are taken from conducting scores [START_REF] Meier | The Score, the Orchestra, and the Conductor[END_REF];

• Two musical phrases with different variations played on a piano (one variation at a time, keeping the same tempo); we have chosen excerpts extracted from work of J. S. Bach: Prelude no. 1 in C Major, and Cantate Bwv 147.

We also aim to add to this sound database two synthesized musical phrases similar to the above piano playing excerpts from J.S. Bach, along with corresponding generated variations for the four categories of actions.

Grammar of Gestures and their Modulation

For the conductor, the challenge is to interpret along the musical piece the sound effects and to give proper orders in terms of physical gestures to the musicians. Our aim was to find a set of intuitive and expressive gestures, in adequacy with the sound categories and variations. Beyond command and coverbal gestures (those that accompany the speech), several gestural languages have been developed for the exclusive expression by means of gestures. This is the case with gestures used by conductors or choirmasters, for whom it is impossible to express verbally. In addition, the need to teach the musical direction and to transmit it over time requires the definition of a form of codification of gestures and the definition of a written grammar. Another community that invented languages to express themselves and communicate with gestures is the one that practices the sign languages of the Deaf. Interestingly is the fact that there are strong similarities between these communities expressing themselves through gestures: they say by showing, they exploit the iconicity of gestures, and they mimic actions that make sense. Inspired by expressive conducting gestures we chose a subset of gestures that are shared by most of musicians around the world [START_REF] Meier | The Score, the Orchestra, and the Conductor[END_REF]. In this study, we selected gestures composed of short actions, so that they can be used as isolated gestures, or repetitively as flows of actions. These gestures should implicitly contain the richness and the expressiveness of the conducting gestures, and they should be sufficiently simple to be codified and used in real-time through an automatic recognition process.

A novel grammar of gestures has therefore been defined in Table 1, whose structure is closely linked to sign language gestures. These gestures share indeed some common properties with conducting gestures. They are both visualgestural languages: the produced information is multimodal and conveyed by hand-arm movements, facial expression and eye's gaze direction, and the perceivable information is interpreted by visual receptors, which means that the gestures have to be well articulated and visually comprehensible. Both of them have also a metaphoric function, i.e. they may fall into the category of describing objects (iconic gestures that describe the shape and the size of an object), or manipulating objects (like hitting or touching an object). As they may accurately express various meanings, they are characterized by a linguistic structure that can be described by a set of components also called phonological units in sign languages, including the location of the gesture, the movement, the hand shape, the wrist orientation and the facial expression. Therefore, as in high-level lan- guages, the combination of these components allow to efficiently express various gestures and sequences of gestures. Modulating these gestures to express various meanings and nuances in the desired performances can be achieved by changing only one or few components (like the hand shape, or the kinematics and dynamics of the movement). Our gesture database is composed of hand-arm gestures, and we only retain as structural components the hand shape and the hand movement. In sign languages the number and nature of hand shapes change according to the context of the corpus (around 60 basic hand shapes are identified in French Sign Language). This number is much more limited for conducting gestures. In our database we selected five basic hand shapes, as illustrated in Figure 2.

Experimental Protocol

To capture the conducting gestures, we used a Qualisys motion capture system which is a marker-based system composed of 12 high speed and high resolution cameras and one video camera. As these gestures mostly involve the hands and the upper body, we have fixed 8 markers on each hand, and 18 markers on the upper body, which counts for 34 markers in total. Figure 3 shows the MoCap marker setting. The MoCap frequency rate was set to 200 Hz to deal with the rapid movements of the hands. Two subjects of different musical levels (one non musician, and one musician highly graduated in instrument playing) participated in the experiment, both of them being right-handed. We call S 1 and S 2 these subjects.

Previously to the experiment, one expert musician (called E), with extensive experience in orchestra, chamber music and instrumental violin practice, chose the musical excerpts, designed the corresponding gestures, and evaluated them qualitatively. The MoCap recording experimentation can be described as follows: the musical excerpts were played as many times as necessary and each subject S 1 and S 2 , in two different sessions (each session lasted about four hours), was instructed to execute the corresponding gestures according to a specific musical effect. For each gesture, several variations were defined, induced by the musical excerpts. The users had to perform each gesture after a short training session, by visualizing some referent videos performed by the expert musician E. Each variation was expressed in several different musical excerpts. For example, for the orchestra music, we had about three excerpts per variation. Moreover, within the same musical excerpt, we could have different nuances of the same variation played at different times of the excerpt (for example several attacks, or several cutoffs). Each musical excerpt was played several times (up to 10 times).

For each subject, two recording sessions were considered, for two classes of classical music pieces:

• 30 classical orchestra music excerpts; these excerpts were validated by the expert musician E;

• two piano excerpts from J.S. Bach played by a piano expert with the instructed variations.

All these excerpts covered the four sound control effects corresponding to the four categories (Articulation, Intensity, Attack, Cutoff), and for each category they covered the previously identified variations. Note that for the orchestra music excerpts, there were other variations related to the musical piece (style, articulation, rhythm, etc.) and the interpretation (tempo, loudness, etc.), whereas for the Bach piano pieces, the variations were more constrained. In the latter case, indeed, the same phrases were replayed several times with the instructed variations, but only one variation at a time, with the same tempo.

To sum up, we captured 50 gesture sequences from each participant, which corresponds to 50 musical excerpts. For each gesture sequence, each participant repeated the gesture at least 5 times. After pre-processing and manual cut-ting, we obtained a conducting gesture data set of 1265 gesture units.

Analysis

Human movement can be represented by a regularly sampled sequence (also called time series) of T body postures.

Each body posture corresponds in turn to a set of m joint positions x(t) = {x 1 , x 2 , ..., x m }(t), where x k (t) (with 1 ≤ k ≤ m) is the 3D Cartesian position associated to the k th joint and 1 ≤ t ≤ T . Thus, a movement amounts to a matrix X = {x(1), x(2), ..., x(T )} of dimensionality 3×m×T . The variation in movement category is illustrated

Feature Extraction and Vectorization

In order to characterize the expressive content of a given movement X, we wished to consider a significant variety of descriptors, both for the recognition of gestures and for the detection of expressiveness, while ensuring that the selected quantities could be computed for several distinct movement categories and variations.

Based on a review of the motion descriptors traditionally used to characterize gestures [START_REF] Karg | Body movements for affective expression: A survey of automatic recognition and generation[END_REF], [START_REF] Kapadia | Efficient motion retrieval in large motion databases[END_REF], [START_REF] Larboulette | A review of computable expressive descriptors of human motion[END_REF], we selected, (i) for the hand movement, kinematic features such as position, velocity, acceleration, and curvature, and, (ii) for the hand configuration, geometrical features measuring distances between the wrist and the extremity of the fingers (thumb, index finger, and middle fingers), as well as the volume covered by the hand. The selected hand movement features have proven to be generic and sufficient to cover most variations of affect in movements [START_REF] Carreno-Medrano | End-effectors trajectories: An efficient low-dimensional characterization of affective-expressive body motions[END_REF]. For example, for a right hand Cutoff gesture, illustrated by the trace of its captured data in Figure 5(a), we show the four kinematic features, i.e. the norm of the position, velocity, acceleration, and jerk (see Figure 5(b)). With regard to hand configuration features, they are specifically defined for the hand shapes of Figure 2.

In addition to these low-level descriptors, the Laban Effort descriptors [START_REF] Maletic | Body-space-expression: The development of Rudolf Laban's movement and dance concepts[END_REF] are frequently used to characterize bodily expressiveness, especially in music-related gestures [START_REF] Glowinski | Toward a minimal representation of affective gestures[END_REF]. Concerning conducting gestures, an experiment was also conducted to show how Laban Effort-Shape descriptors can help young conductors build expressive vocabulary and better recognize the level of expressiveness of different artistic disciplines [START_REF] Jean | The effect of Laban Effort-Shape instruction on young conductors' perception of expressiveness across arts disciplines[END_REF]. On the basis of these results, Laban Effort descriptors become a good candidate for characterizing the expressive variations of the conducting gestures. Focusing on the quality of motion in terms of dynamics, energy and expressiveness, these descriptors are defined by four subcategories (Weight, Time, Space, and Flow). The Weight Effort refers to physical properties of the motion; the Time Effort represents the sense of urgency; the Space Effort defines the directness of the movement which is related to the attention to the surroundings; the Flow Effort defines the continuity of the movement. In summary, the chosen descriptors, whether they are kinematic, geometric, or inspired by Laban's Effort components, are computed from the equations defined in [START_REF] Larboulette | A review of computable expressive descriptors of human motion[END_REF]. They are listed in Table 2.

Category Features Kinematics

Normalized velocity, acceleration, curvature, and jerk norms.

Geometry and space

Displacements between hand joint (wrist) and each finger extremity. Laban Effort Flow, Space, Time, and Weight.

Table 2: List of features used to characterize gesture and motion expressiveness.

Recognition

For recognition purposes, we had to cut the gestures in elementary units, each cut gesture corresponding to a specific command and variation that will be used for controlling sound in live performances. The splitting has been achieved through a semi-automatic process, using the velocity information for all the hand traces. Moreover, to manage the variation of duration in our recognition process, we proceeded in two different ways, whether we consider off-line or on-line recognition.

Validation through Off-line Recognition

In the scope of validating our database, both for the gesture categories and the gesture variations within each category, we used an off-line classification approach applied on our gesture units. Two methods were considered to deal with the temporal variation of the gesture units. In the first method, we divided each gesture unit into an equivalent number of segments (1, 5, 10 or 20 divisions), and computed a vector of normalized descriptors for each segment composing the gesture, using average and standard deviations of the kinematic and geometric features. In this way, each gesture is represented by a vector of the same size, which facilitates the recognition process. We plan to use in a second method elastic distances associated to Support Vector Machines methods.

On-line Recognition

Secondly, for real-time recognition, we used a moving window and compute the average and standard deviation in this window.

Conclusion and Perspectives

In this article we have designed and recorded a novel database of expressive gestures induced by sound effects. The originality of this database lies in the fact that we have separated the functional components of the gestures from their expressive variation. This may lead to interesting applications where it is possible to control sound both from the recognition of the action (gesture category) and from the recognition of the gesture expressiveness. In order to quantitatively validate the database, a recognition system is currently under development, using machine learning algorithms. This ongoing work currently focuses on two aspects: finding the best feature set for on-line classification, and adapting the algorithms for real-time recognition.

In the near future, we also intend to evaluate perceptually the database. The gestures will be evaluated by qualitative information related to Laban Effort components (Energy, Timing, Flow), and by multi-choice questionnaires using semantic terms, while the music excerpts will be evaluated through linguistic questionnaires.

Figure 1 :

 1 Figure 1: Work flow to collect the data and analyze them for the application needs.

Figure 2 :

 2 Figure 2: List of the five selected handshapes.

Figure 3 :

 3 Figure 3: Mocap marker settings and marker-model reconstruction.

Figure 4 :

 4 Figure 4: Right hand traces of one subject performing repetitively different gestures among the four categories: Articulation, Intensity, Attack, Cutoff.

  (a) 3D hand trace of the right hand. (b) Feature sets.

Figure 5 :

 5 Figure 5: Kinematic features of a Cutoff sequence.

  Figure 6 shows how the division method (part (a) of the Figure) or a moving window (part (b) of the Figure) works on a Cutoff gesture. These classification methods for off-line and on-line recognition are currently being implemented and tested.

  (a) Equal division of two Cutoff gestures. (b) Moving window on a Cutoff gesture.

Figure 6 :

 6 Figure 6: Kinematic features of a Cutoff sequence.

Table 1 :

 1 List of gestures with their movement and handshape description.

	Gesture	with	Hand move-	Handshape
	demo		ment
	Articulation		Phrasing ges-	Flat/Bent for
			ture:	Legato	Legato	or
			(sagital plane);	Tenuto, O for
			Staccato (hori-	Staccato
			zontal); Tenuto
			(vertical)
	Intensity		Upward		Alternating a
			(crescendo)	Flat and a Bent
			or downward	handshape
			(decrescendo)
	Attack		Metaphor	of	O for a hard
			"hitting a hard	attack and Fist
			object"		for a soft attack
	Cutoff		Metaphor	of	Flat handshape
			"taking		an	closes to a
			object out of	Pursed	(soft
			the view"	cutoff) or O
					(hard	cutoff)
					handshape.


Sylvie Gibet
email: sylvie.gibet@univ-ubs.fr

Building French Sign Language Motion Capture Corpora for Signing Avatars

Keywords: Corpus, MoCap, Sign Language, Signing avatar

This paper describes four corpora that have been designed and built in our research team. These corpora have been recorded using motion capture (MoCap) and video equipment, and annotated according to multi-tiers linguistic templates. Each corpus has been designed for a specific linguistic purpose and is dedicated to data-driven synthesis, by (i) replacing signs or groups of signs within an utterance, (ii) replacing phonetic or phonological components and in this way modifying the grammatical or semantic aspects of the phrase, or (iii) altering prosody in the produced sign language utterances.

Introduction

The design of traditional corpora for linguistic analysis aims to provide living representations of sign languages across deaf communities and linguistic researchers. Most of the time, the sign language data is video-recorded and then encoded in a standardized and homogenous structure for open-ended analysis (statistical or phonological studies). With such structures, sign language corpora are described and annotated into linguistic components, including phonology, morphology, and syntactic components [START_REF] Johnston | Researching the linguistic use of space in Auslan: Guidelines for annotators using the Auslan corpus[END_REF][START_REF] Crasborn | Annotation of the video data in the Corpus NGT[END_REF][START_REF] Efthimiou | GSLC: Creation and annotation of a Greek Sign Language corpus for HCI[END_REF][START_REF] Wolfe | Synthetic and acquired corpora: Meeting at the annotation[END_REF][START_REF] Hanke | Where does a sign start and end? Segmentation of continuous signing[END_REF].

Conversely, motion capture (MoCap) corpora provide researchers the data necessary to carry on finer-grained studies on movement, thus allowing precise, and quantitative analysis of sign language gestures as well as sign language (SL) generation. One the one hand, motion data serves to validate and enforce existing theories on the phonologies of sign languages. By aligning temporally motion trajectories and labelled linguistic information, it thus becomes possible to study the influence of the movement articulation on the linguistic aspects of the SL, including hand configuration, hand movement, coarticulation or synchronization within intra and inter phonological channels. On the other hand, generation pertains to sign production using animated virtual characters, usually called signing avatars.

Although MoCap technology presents exciting future directions for SL studies, tightly interlinking language components and signals, it still requires high technical skills for recording, post-processing data, and there are many unresolved challenges, with the need to simultaneously record body, hand motion, facial expressions, and gaze direction. Therefore, there are still few MoCap corpora that have been developed in the field of sign language studies. Some of them are dedicated to the analysis of articulation and prosody aspects of sign languages, whereas recent interest in avatar technology has led to develop corpora associated to data-driven synthesis. In particular, [START_REF] Lu | Collecting and evaluating the cuny asl corpus for research on american sign language animation[END_REF] collected an ASL corpus and discussed how linguistic challenges in ASL generation could be addressed through this corpus. To improve avatar movement, kinematic and linguistic cues were retrieved from motion capture data and incorporated into a data-driven technique, thus leading to a more realistic animation [START_REF] Mcdonald | A new tool to facilitate prosodic analysis of motion capture data and a data-driven technique for the improvement of avatar motion[END_REF]. More recently, a MoCap dataset on French sign language (LSF) has been collected [START_REF] Limsi | Mocap1. ORTOLANG[END_REF]. 25 pictures are described in a spontaneous way, and first analysis are conducted. However these corpora do not capture simultaneously the multiple channels involved in SL gestures, i.e. body movement, hand configuration, facial expression, and gaze direction, which remains an important challenge and is necessary to address these highly coded sign languages using multi-tiers linguistic elements, both for recognition [START_REF] Dilsizian | A new framework for sign language recognition based on 3d handshape identification and linguistic modeling[END_REF] or synthesis [START_REF] Gibet | The signcom system for data-driven animation of interactive virtual signers : Methodology and evaluation[END_REF].

In this article, we describe four motion capture corpora in French sign language that were designed and built in our research team during the last decade. The technical aspects of the MoCap acquisition are described. Then the linguistic rules that guide the corpora design for the synthesis of new sentences in LSF are discussed and illustrated with examples. Most of these linguistic issues are applied to datadriven generation. However, in all our studies, it is important to emphasize that we adopted a synthesis-by-analysis approach. That is to say, the improvement of our synthesis models led us progressively to refine our methods of segmentation and labeling, to better understand the mechanisms responsible for the formation of the signs, as well as the processes of coarticulation [START_REF] Naert | Coarticulation analysis for sign language synthesis[END_REF].

Motion Capture Datasets

Four corpora in French Sign Language (LSF) and their corresponding MoCap databases have been designed by a team of researchers that includes linguists and computer scientists, Hearing and Deaf. Before describing these corpora we describe hereafter the motion capture databases that were collected over the last ten years in the context of national research projects, with different objectives concerning the linguistic aims and the level of avatar synthesis.

Mocap Devices and Experimental protocols

Different MoCap setups and experimental protocoles were defined in the context of three projects. For all of them, we used a Vicon MX infrared camera technology to capture the 3D displacements of a set of markers. The main differences between the setups of the projects are the number of cameras, of markers, and the frequency of acquisition. Table 1. gives an overview of the projects with the experimental setups. For capturing precisely movements of the hands and facial expressions, it is necessary to use more cameras and to place them closer to the subject so that they can detect all the markers. This is why we increased the number of cameras as we gained experience and mastered the motion capture systems. In addition, the frequency of acquisition has to be large enough to capture the subtle variations of the movements, for example when changing a facial expression, or moving rapidly one hand. We therefore tried to determine the appropriate frequency of acquisition, trying to keep a good compromise between the spatial accuracy and the speed of the cameras. Finally, in all our experimental setups, we considered pairing MoCap with video recordings, assuming that parallel recordings would aid in the ulterior data annotation processes.

HuGEx project

In HuGEx project [START_REF] Gibet | Virtual agent for deaf signing gestures[END_REF]), the Vicon system was composed of 12 infrared cameras cadenced at 120Hz. For the body movements 24 reflective markers were placed on standardized anatomical landmarks. We also recorded facial expressions using 39 small semi-spherical markers (3mm) at locations compliant with Mpeg4 specification. As we had no experience with hand capture, hand movements were recorded using two Cyber gloves (Ascension technologies), each one composed of 22 sensors (see Fig. 1). The fusion of the different signals (body, left and right hand) was realized after reconstruction and synchronization (resampling at 60 Hz). The different information sources (body and hands) were then converted into BVH format. During the recording session, about forty minutes of LSF gestures were captured on one expert deaf signer. This one, who was a trainer in LSF, signed on texts that he himself transcribed into a sequence of glosses. Two databases were built: (i) the TRAIN database (about 10 min), aimed at building sentences with predefined replaceable parts; (ii) the METEO database (about 40 min), aimed at studying the variation in prosody of the LSF phrases. For both datasets, the mean duration of a sequence was about 60 seconds.

SignCom project

The SignCom project [START_REF] Gibet | The signcom system for data-driven animation of interactive virtual signers : Methodology and evaluation[END_REF]) also used a Vicon MX system with 12 high definition cameras to capture the movements of our LSF signers at a frequency rate of 100Hz. We had 43 markers for the body, and 41 markers of small diameter for the face. Instead of using data gloves which lack precision and exhibit significant drift when used for a long time, we captured the hand movements by fixing 6 markers per hand (see Figure 2). As for the previous projects, we used an additional video camera to have video recordings in addition to MoCap data. This is necessary for annotation. Body movements, hand and finger movements, and facial expressions were recorded simultaneously by the Vicon system. Two professional deaf linguists signing in LSF designed the corpus, and learned it by heart. During the recording session, the information was presented to the deaf signers through images projections, so that the signers were able to recall the scenarii without reading any text-based translation. 68 motion sequences were recorded on one signer. This constitutes the SignCom database containing about one hour of Mo-Cap data. From this data (in C3D format), a skeleton was reconstructed, and the data (body and hand movements, as well as facial expressions) was converted into the formats BVH and FBX.

Sign3D project

The Sign3D project [START_REF] Gibet | Interactive editing in French sign language dedicated to virtual signers: requirements and challenges[END_REF] used a Vicon T160 system with 16 high definition cameras at a frequency rate of 100 Hz, combined with a head-mounted oculometer (MocapLab MLab 50-W), designed to track the gaze direction. Facial expressions, body and finger motions were again simultaneously recorded. For recording precisely hand movements and hand configurations, we used a much larger number of markers (19 per hand against 6 in the SignCom project). The LSF gestures of one expert deaf signer were recorded for about 10 minutes to form the dabase Sign3D. The motion capture settings associated to the skeleton reconstruction is illustrated in Figure 3. The motion skeleton data including body and hand movements) was converted into the FBX format. The facial expressions data was converted into blendshape coefficients.

Corpora Design

We describe hereafter the various corpora that we have designed in the context of the former research projects. Through these projects, we developed a complete concatenative data-driven synthesis pipeline that enables the assembling of motion elements, from signs and parts of sentences, to motion chunks retrieved from different channels and body parts (hand movements, hand configurations, body movements, facial expressions, and gaze direction), representing phonetic or phonological components. Our corpora follow the objectives of synthesis by replacing signs or groups of signs in sentences, by composing phonetic or phonological components, and finally by analyzing and generating prosody in sentences carried out in different stylistic contexts.

Motivation

For the purpose of corpus design, three main questions have been addressed in the three former projects. The first one concerns the corpus content itself and the compromise that exists between breadth and depth in its design. The second question concerns the nature of the sign variability which is of paramount importance if we want to create new sentences in different discourse contexts. The third question concerns the acted or spontaneous nature of the produced SL utterances. Concerning the first question, we wanted to have control over the signs or gloses that appeared into the corpora, and therefore we would prefer a limited vocabulary of given signs, and multiple instances for each sign played in different discourse contexts. We also chose to incorporate standard signs into our lexicon, as they were easier to handle for synthesis. Given the difficulty of capturing large corpora (a tedious and time-consuming process, both in terms of capture, post-processing, and annotations), we also opted for a limited set of utterances or sequences of signs. Therefore, in parallel with the design of our sentences, we had to think deeply about the mechanisms of editing signs and constructing new sentences. The question of variability can be approached in different ways: (i) by constructing sentences containing the same signs appearing in different contexts; (ii) by repeating the sentences several times and with different subjects; and (iii) by enriching the initial corpus with new constructed sentences.

To answer the third question, in all our corpora, the scenarii were scripted by deaf persons, and the produced sign language utterances were acted. Table 2 gives an overview of the corpora, indicating the level of annotation, the topic, and the linguistic application.

Replacing Signs or Groups of Signs in Sentences

The first experimental ideas for synthesizing new statements from original sign language data were to insert replaceable parts into a sentence, such as signs, or groups of signs. This was first achieved in the HuGEx project where the corpus was composed of a set of phrases expressing incident reports relatively to the railway traffic, with a set of additional signs representing French towns. Two excerpts are shown below. The brackets delimit the variable elements.

The weather / personal accident).

In the Sign3D project, we collected a corpus of French sign language utterances, describing various events (exhibitions, inaugurations, cultural events) taking place in various buildings and monuments (swimming pool, theater, town hall, museum, etc.), indicating the opening and closing hours, entrance fees, their location relative to each other, and the potential occurrence of an incident (weather, work, etc.). In this latter corpus, the aim was also to build new sentences by replacing signs (hours, buildings, etc.), or groups of signs (events, incident causes). To preserve the linguistic coherence of the LSF statements, while optimizing the number of variants of the different sentences, the corpus was designed by declining a limited set of syntactic patterns (the brackets delimit the variable elements).

The [LOCATION] is [ABSOLUTE OR REL-ATIVE LOCATION]; it opens at [TIME] and closes at [TIME]. Access is [PAYING / FREE], [ENTRY CONDI-TION]. The [EVENT] in [LOCATION] is moved to [LOCATION] due to [CAUSE]. In case of [CAUSE], the [EVENT] in [LOCA-TION] will be moved to [LOCATION].

where the variables ([LOCATION], [TIME], etc.) may be replaced by values belonging to a given set of signs. Thirteen sample sentences were then signed by a deaf LSF expert. This corpus can easily be extended by enriching it with the synthesized variant sentences. This represents about 10 minutes of continuous LSF.

Altering Phonological Components of Signs

The objective of the SignCom project was also to design new utterances in LSF, and to animate a virtual signer, using both raw motion and annotated data. The idea was similar to the previous projects, but instead of manipulating signs, the aim was to re-assemble phonetic or phonological elements of signs, while keeping the global coherence and realism of the produced sequences. The corpus contains three thematic scenarii: the Cocktail monologue, and the Galette and Salad dialogues. The scenarii were scripted by two expert deaf people who designed the scenes using comic stories that were displayed on the back wall of the room, thus avoiding lowering the head for reading the scenarii. Both deaf people trained for several days before the recording sessions, hence they executed the motion as acted sequences. A total of 68 sequences was captured and annotated, following a multi-tier template with different levels of labeling (gloss, phonological and phonetic elements), for each hand separately, and for the two-hands. As signed languages are by nature spatial languages, forming sign sequences requires a signer to understand a set of spatial-temporal grammatical rules and inflection processes. These processes have oriented the range of LSF signs recorded for the project. This brought us to include a number of various linguistic inflection mechanisms into the corpus that allow for creating novel sentences from our original corpus. After defining a delimited vocabulary, we chose to introduce spatial references (for example depicting and indicating verbs which are modulated in the context of dialog situations), changes in hand configurations, and changes in hand movements.

Spatial references: directional verbs and pointing movements

We included in the dataset directional verbs and depicting verbs as well as personal and possessive pronouns. This gave us the possibility to build new sentences by conjugating the verbs. For example, the sign INVITE can be modified grammatically to become "I invite you", "You invite me", etc. Our vocabulary thus contains several instances of the directional verbs shown in Table 3. A certain number of pointing gestures in different parts of the signing space were also included. These targets are labeled with their 3D location.

Changes in hand configurations

Many hand configurations, possibly associated to verbs, allow for designing different objects, or indicate size or

Salad

Cocktail

Directional verbs

22 × SALAD 8 × COCKTAIL GIVE 20 × PRO-1 8 × DRINK TAKE 19 × WHAT 7 × GLASS PROPOSE 8 × PLATE 7 × FRUIT INVITE 6 × TOMATO 3 × ORANGE COMMUNICATE 12 × POUR 3 × JUICE PUT 11 × WANT 7 × FILL EXPLAIN 9 × CHEVRE 7 × THERE-IS QUESTION 9 × VARIOUS 3 × NEXT 3 × AVOCADO 4 × ALCOHOL 5 × ADD 2 × WITHOUT
Table 3: Two first columns: some tokens with their occurrence in the Salad and Cocktail scenarii (SignCom corpus); Third column: directional verbs mainly found in the dataset.

shape specifiers. Given our inclusion of signs that take multiple hanshapes, like GIVE, we introduced in the corpus different hand configurations from other signs that can be substituted to the original handshapes. In the case of GIVE, most often signed in our dataset as if the signer was handing a glass to someone, a hanshape substitution could yield addition meanings, such as giving a piece of paper or giving an object with a cylindrical shape. In particular, the expression GIVE A GLASS is performed in our corpus with glasses of different sizes and forms (for example a large glass, a thin long glass, or a champagne flute).

Changes in movement kinematics

Analyzing hand movements has shown regular shapes (bell-shapes) which differ whether they belong to strokes (within-sign) or transitions (inter-signs). Moreover, for strokes, toward-target movements differ from backward movements [START_REF] Duarte | Reading between the signs: How are transitions built in signed languages?[END_REF]. These observations have led us to introduce many kinematic variations of movements in the corpus, so that it becomes possible to analyze and annotate these patterns, and to retrieve the appropriate movement from the database that preserve the temporal coherency of the reconstructed phrase.

The corpus also contains reversal verbs, as for example the sign GIVE which can be reversed in the sign TAKE, or the sign LIKE which can be reversed in DO-NOT-LIKE in LSF.

Composition process to build new sentences

An overview of the most frequent tokens in the Cocktail and Salad scenarii is provided in Table 3. With this variety and frequency of our related lexemes, we are able to produce a number of novel utterances based on the thematic subjects. (1) ends; the direction of movement is reversed, and the shape of the hand is changed to handle a big glass instead of a thin glass. In the second example, different channels are combined, by keeping the torso/lower-body/left-arm of one sequence (3), and substituting the head, facial expression and right arm movements of another sequence (4). The sign DO-NOT-LIKE is reversed from the sign LIKE. In such a composition process, the spatial constraints should be preserved, in particular the sign ORANGE should be executed near the corresponding body part (head), whatever the torso or the head orientation is. This clearly reveals that the combination process should be driven at a more abstract level, expressed by rules or constraints incorporated into the animation engine.

Altering the Prosody of Sentences

The objective of the HuGEx project was to animate with a data-driven approach a virtual signer endowed with expressive sign language gestures. Our attention focused on the prosody of the LSF gestures, and on its influence on the semantic comprehension. The corpus METEO was composed of a set of sentences describing weather forecasts, performed with different variations of expressiveness: neutral, angry, emphasis, and tired. The mean duration of a sequence was 60 seconds. We took as referent performance the first sequence performed according to neutral style. An example retrieved from the corpus is given below.

Today, July 6th, here is the weather forecast. In the morning, clouds will cross Brittany. In the afternoon, it will rain. Tomorrow, the sun will shine. It will be hot and dry.

An analysis of the LSF prosody was achieved on the expressive sentences, through a temporal alignment process using an adaptive dynamic time warping algorithm [START_REF] Héloir | Temporal alignment of communicative gesture sequences[END_REF]. Using machine learning techniques, it would be possible to learn the sequences performed with different styles and then transfer the style of one sequence into another one.

Conclusion

In this article, we described four corpora with different linguistic purposes that have been designed and built in our research team over the last ten years. These corpora were recorded using MoCap data, post-processed and manually annotated, so that they could be used for different goals: linguistic analysis, automatic annotation, or generation. With the increasing interest of linguistic or computer science researchers using sign language motion capture data, there is a need to provide motion capture databases that can be shared by the different communities. Following the approach adopted by other research teams in movement sciences that have made available raw motion, videos, and tools, with exchangeable data formats, we want to share our experience on the design of corpora and the construction of MoCap databases. We also propose to make available soon our LSF MoCap corpora.

Concerning the annotated data, we have used schemes inspired from the linguistic community, and we are currently enriching these schemes by developing automatic annotation methods. These annotated schemes (manual or automatic) with the documentation explaining the structure and the coding system of the annotation should also be shared by the different research communities.

Other more focused corpora are also currently designed and collected in our research team. They are dedicated to the automatic annotation of two specific channels: facial expressions and hand configurations, and will be used for animating a signing avatar [START_REF] Naert | Per channel automatic annotation of sign language motion capture data[END_REF].

Acknowledgements

The motion capture datasets presented in this paper have been realized, thanks to the HuGEx project funded by the CNRS, the SignCom project funded by the French National Research Agency (ANR), and the Sign3D project, funded by the French Ministry of Industry.

Figure 1 :

 1 Figure 1: Photo of the MoCap settings in HuGEx project.

Figure 2 :

 2 Figure 2: Photo of the MoCap settings in SignCom project.

Figure 3 :

 3 Figure 3: Photo of the MoCap settings in Sign3D project.

  Examples of construction of new sentences from the above transformations is shown in the following examples: I GIVE-YOU a THIN-GLASS (1) I TAKE a LARGE-GLASS (2) I LIKE FRUIT JUICE (3) I DO-NOT-LIKE ORANGE JUICE (4) In this first example, only the right arm is involved. The movement (2) begins at the position where the movement

Table 1 :

 1 MoCap databases built at IRISA.

	Projects Markers	Cameras frequency Databases Year Size
		Capture device number	Hz		min
	HuGEx	24 (body)	12	120	TRAIN	2005 10'
		2 Cybergloves			METEO	40'
		39 (face)
	SignCom 43 (body)	12	100	SignCom	2009 60'
		2x6 (hands)
		41 (face)
	Sign3D	40 (body)	16	100	Sign3D	2013 10'
		2x19 (hands)
		40 (face)
		gaze direction

Table 2 :

 2 Corpora in LSF.

	train from [CITY] to [CITY] is delayed by
	[DURATION], due to [CAUSE].
	The train [NUMBER] is being prepared; the
	starting lane will be displayed in [DURATION].

It was then possible to build programmable sentences by choosing the departure and arrival cities ([CITY]) among a given set of pre-recorded cities, the number of the train ([NUMBER]), or the nature of the incident ([CAUSE]) belonging to the following set: a technical incident / bad


Lucie Naert
email: lucie.naert@univ-ubs.frlsf.irisa.fr

Clément Reverdy
email: clement.reverdy@univ-ubs.frlsf.irisa.fr

Caroline Larboulette
email: caroline.larboulette@univ-ubs.frlsf.irisa.fr

Sylvie Gibet
email: sylvie.gibet@univ-ubs.frlsf.irisa.fr

Per Channel Automatic Annotation of Sign Language Motion Capture Data

Keywords: automatic annotation, automatic segmentation, motion capture, French Sign Language, linguistic channels

Manual annotation is an expensive and time consuming task partly due to the high number of linguistic channels that usually compose sign language data. In this paper, we propose to automatize the annotation of sign language motion capture data by processing each channel separately. Motion features (such as distances between joints or facial descriptors) that take advantage of the 3D nature of motion capture data and the specificity of the channel are computed in order to (i) segment and (ii) label the sign language data. Two methods of automatic annotation of French Sign Language utterances using similar processes are developed. The first one describes the automatic annotation of thirty-two hand configurations while the second method describes the annotation of facial expressions using a closed vocabulary of seven expressions. Results for the two methods are then presented and discussed.

Introduction

Whether we want to linguistically study sign languages, use digital data to identify salient linguistic components, recognize or synthesize continuous signing, an annotation of the data is needed. The annotation of sign language is a two-step process. The first step, called segmentation, consists in dividing the stream of data in segments of interest. Those segments are then identified in a second step called labeling. This annotation might be done manually but is a fastidious, time consuming and expensive task. Not only does it require the skills of language experts, but it is also subject to inaccuracies and mistakes as the experts may not have exactly the same segmentation criteria. In the particular case of sign language, the annotation process needs to be done by experts in sign language and gesture annotation. In addition, sign languages are expressed simultaneously on multiple channels (manual configuration, wrist orientation, body posture, facial expression, etc.), thus complicating the task of the annotators. When comparing the duration of the annotation process to the duration of the data to annotate, (Dreuw and Ney, 2008) introduces a real-time factor of 100 (i.e. all the manual and non manual features of a 1 minute video of sign language will be annotated in 100 minutes). We propose to automatically annotate each channel separately following the scheme of Figure 1.

Automatic annotation of sign languages could reduce the annotation costs but is still a challenging and yet to be solved task. One way to tackle this problem is the machine learning approach (e.g. use of Bayesian/statistical models or artificial neural networks) which aims at automatically learning the parameters of a model from a sample of manually annotated data. This model is then used to automatically annotate new data. The corpus intended to train the machine learning model must thus be designed carefully before recording sign language data using either video or motion capture technologies. Despite being easy to use and relatively cheap, video does not preserve the third spatial dimension of motion. Furthermore, the Figure 1: Our intended annotation scheme: currently, we automatically annotate the hand configuration and affect channels (boxes with bold borders). resolution of classic video recordings is rarely high enough to obtain a precise segmentation. Motion capture (MoCap) technologies offer a better precision, both spatially and temporally, and spatial information that is not available with 2D cameras. MoCap data can be used for sign language analysis -using motion descriptors computed from the 3D data, such as distances between joints, velocity or curvature of selected joints -as well as for synthesis by using the captured motions to animate a signing avatar. In this paper, we will address the problem of the automatic annotation of sign language MoCap data, focusing more specifically on two main channels: facial expression, and hand configuration.

Previous work on automatic annotation on video and Mo-Cap data are described in section 2. The specification, capture and manual annotation of the input sign language data used in this study is introduced in section 3. In section 4, the annotation methodology is presented and illustrated with the examples of the automatic annotation of the hand configurations and the facial expressions in French Sign Language. Section 5 describes the results of the automatic annotation applied to those two channels. Finally, section 6 discusses the perspectives and challenges of our method.

Related Work

In this section, previous studies on automatic annotation of sign language are presented. For gesture segmentation in general, a very complete framework is developed in [START_REF] Lin | Movement primitive segmentation for human motion modeling: A framework for analysis[END_REF]. It provides a general overview and a comparison of several works in human motion segmentation using different data sources (camera, MoCap, sensors, etc.) but does not address the problem of "per channel segmentation" or the particular application of sign language processing. Continuous signing segmentation and recognition can be opposed to isolated sign language recognition. Coarticulation effects present in continuous signing and absent from isolated signs make the segmentation of the former harder. Most of the existing work on the automatic segmentation of continuous signing relies on video footage to segment at a gloss-level. [START_REF] Kim | Continuous korean sign language recognition using gesture segmentation and hidden markov model[END_REF] take advantage of Hidden Markov Model (HMM) to segment a continuous stream in Korean Sign Language into signs segments. Similarly, [START_REF] Yang | Detecting coarticulation in sign language using conditional random fields[END_REF] perform sign segmentation of continuous American Sign Language using Conditional Random Fields (CRF). In their article they demonstrate the superiority of the CRF approach (85% accuracy) compared to HMM (60%). A different approach was developed by [START_REF] Lefebvre-Albaret | Toward a computer-aided sign segmentation[END_REF]. It presents a computer-aided segmentation of sign language sequences based on the detection of motion cues such as symmetry, repetition and hand trajectories templates. The algorithm is helped by the punctual intervention of an operator who has to specify one frame belonging to each sign. However, those segmentation approaches do not take into account the multichannel aspect of sign languages and lead to segmentation schemes highly dependent on the context of the utterance, i.e. the segments implicitly contain the coarticulation effects of the sequential signs. The resulting segmentation is thus hardly reusable in a different context, for example to synthesize new utterances. A lower-level segmentation, based in particular on phonological elements would facilitate sign composition in various contexts in order to produce new utterances. However, although several studies address the issue of the annotation of sign language video data at a gloss level, little attention has been given to the automatic annotation of the different linguistic channels of sign languages. The work of [START_REF] Stokoe | Sign language structure: An outline of the visual communication systems of the american deaf[END_REF] gives a phonological structure to signs by specifying three linguistic parameters to describe all signs: hand motion, hand configuration and hand placement. Each feature can take a discrete value in a limited vocabulary. Two complementary features were later added, hand orientation (Battison, 1978) and non manual features such as shoulder tilt or facial expressions. Many phonological structures use those five features to define signs which can be used as a basis for video annotation : the segments are of a finer level than the gloss segmentation and retain a linguistic value. In an early work, [START_REF] Vogler | A framework for recognizing the simultaneous aspects of american sign language[END_REF]) break signs into "phonemes" (close to the previous five features) and use HMM on the combination of the phonemes to recognize signs. The channels are processed separately but the ultimate purpose is gloss recognition and not channel annotation. In addition, this work is based on the Movement-Hold model which has been later replaced by a more precise phonetic model [START_REF] Johnson | A segmental framework for representing signs phonetically[END_REF]. Furthermore, due to the difficulty of capturing the finger movements, the hand configuration channel was not processed and the authors of the article also chose not to deal with non manual features. More recently, (Dilsizian et al., 2014) propose to add some linguistic knowledge about the composition of lexical signs to considerably improve the performances of the recognition system.

Work on sign language MoCap data is scarcer than on video. For gloss-level segmentation, [START_REF] Naert | Coarticulation analysis for sign language synthesis[END_REF] use some kinematic properties of the two wrists that can be extracted from MoCap data. At a finer level, [START_REF] Héloir | Captured motion data processing for real time synthesis of sign language[END_REF] focused on the segmentation of hand configurations using Principal Composant Analysis (PCA) but the work is restricted to fingerspelling segmentation.

The recognition of facial expressions has received increasing attention in recent years, mainly from the computer vision community. Regarding the existing datasets, the availabiliy of 2D recording devices made possible the creation of large data collections, such as the Cohn-Kanade Dataset (posed facial expressions) and its extension [START_REF] Lucey | The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression[END_REF] (posed + non-posed facial expressions) or the MUG database (Aifanti et al., 2010) (posed + non posed) with many (up to hundreds) actors. High resolution 3D facial databases with expressions have also been created using binocular/structured light cameras [START_REF] Zhang | Bp4d-spontaneous: a high-resolution spontaneous 3d dynamic facial expression database[END_REF], [START_REF] Yin | A high-resolution 3d dynamic facial expression database[END_REF]. The frame rate of such optical device is often limited to 60 or less frames per second (usually 20/30 fps) which may be insufficient for those who are interested in dynamic expressive variations. MoCap techniques can capture movements up to 200 fps and more, which makes them much more powerful for analyzing fine expressive variations. Nevertheless, the publicly available facial MoCap databases are still scarce. The multimodal database described in (Busso et al., 2008), which includes facial MoCap with speech and elicited emotional expressions is one of the few existing ones. This paper presents the automatic annotation of continuous signing in French Sign Language using MoCap data on two linguistic channels : the hand configurations and the facial expressions.

Input Data

This section describes the definition and recording of sign language data as well as the specification of the two corpora that have been used for the studies.

General Considerations

To develop models for automatic annotation, the first solution that might come to mind is to record all the existing signs in all the possible contexts in order to cover exhaustively all the possible cases of sign production. While this can be attempted (with varying degrees of success) for oral languages by, for example, retrieving huge databases from the Internet (e.g. Wikipedia pages, Twitter posts, etc.), this is impossible for sign languages. Indeed, (i) sign language databases are scarce, especially MoCap databases, (ii) sign languages use the 3D space and the temporal dimension which leads to the production of an infinite number of combinations of the different physical channels, and (iii) sign language cannot be limited to their standard, reference signs: many sign language mechanisms such as classifierpredicates are as important as standard signs and depend strongly on the context of the sentence. Instead of collecting a large set of random data, it might be more relevant to design a corpus specifically suited to the studied problem. One way to reduce the complexity of the capture is to consider each chosen channel separately. Indeed, each channel can display a limited number of different behaviors. For example, we can enumerate a limited number of different hand configurations in sign languages (often less than fifty in French Sign Language). As a consequence, a corpus designed to study and automatically annotate the hand configurations would focus only on a small number of signs to cover all the possible hand configurations. To sum up, for the application of automatic annotation, a corpus containing many repetitions of a limited number of different occurrences of the studied element will be preferred. The variability induced by a different context in the element production (for example, by capturing the same hand configuration in different signs) or by a different signer, must be recorded in order to improve the generalization capacity of the resulting model.

Corpora

Two different corpora were used for the presented work: Sign3D (Gibet et al., 2015) to annotate hand configurations, and FEeL, a novel corpus that is still under development, for the facial expressions.

Characteristics

The Sign3D corpus contains eight sequences of motion. Each of these sequences is composed of one to five French Sign Language utterances. The utterances are messages about the opening hours and entrance fees of various town places (swimming pool, museum, etc.), as well as the description of various events (exhibitions, theatre play etc.). The capture was performed on one signer using a Vicon MoCap system and an eye-tracking device to follow gaze direction. Facial expressions, body and finger motions were simultaneously recorded during approximately 9 minutes at 100 fps (around 54000 frames in total). The FEeL corpus has been captured using two signers (learner level). Three kinds of sequences -corresponding to different sets of instructions given to the signers -were recorded. We worked exclusively on the affect channel of the face and chose to analyze this channel within the Ekman framework with six categorical classes of basic emotions (i.e. anger -A, disgust -D, fear -F, joy -J, sadness -Sa, surprise -Su and neutral -N), which was easier to annotate and more understandable by humans than continuous models (e.g. the Pleasure -Arousal -Dominance framework). Three kinds of sequences were recorded: i) Isolated Expressions -IE: the signers were asked to perform a given expression five times, each expression had to be maintained several seconds before returning to neutral (e.g. for joy we have: N -J -N -J -N -J -N -J -N -J -N ). Six sequences were recorded per signer, one for each class of affect. ii) Sequences of Expressions -SE: the signers were asked to alternate a given expression with each of the five other expressions (e.g. for joy:

N -J -Su -J -A -J -F - J -Sa -J -D -J -N ).
Five sequences were recorded per signer. iii) Expressive Utterances -EU: Sign language sentences with emotional content were prepared. The signers were asked to repeat three times each sentence with a given affect (e.g. it was asked to the signer to sign the following sentence with disgust : "There is a spider on my pizza! Yuk!"). 18 sequences were recorded per signer, one for each sentence. The corpus has been recorded via a Qualysis MoCap system. A total of 40 facial markers were tracked at 200 fps.

Manual Annotation

Manual annotations are used as reference and training data for our automatic annotation. It is thus necessary to have a thorough annotation. The Sign3D and the FEeL corpora have been annotated using the ELAN software (Max Planck Institute for Psycholinguistics, 2017). The Sign3D corpus has been annotated on several channels including, but not restricted to, gloss, hand placement, hand orientation, mouthing, facial expressions and hand configurations. To reduce the error rate and to have a more consistent annotation, two annotators knowledgeable in French Sign Language validated each others' work. Concerning the annotations of the FEeL corpus, we focused our efforts on the affect channel and, so far, a single annotator has been involved in the process. This annotator has been instructed to "subjectively annotate what he saw" with respect to the two following rules: (i) we distinguish two kinds of segments: the transition segments where the class vary from a starting expression to an ending expression, and the stable segments where the class doesn't vary along time; (ii) the name of a transition segment is the concatenation of the name of the starting class and of the name of the ending class (e.g. NA means that the transition come from the neutral class to the anger class). A stable segment is named according to the maintained expression displayed (e.g. Sa stands for sadness).

Automatic Annotation

This section describes the principal steps to automatically annotate a sign language channel for a given corpus. Motion descriptors are first computed in order to segment and then label the sign language data. The examples of the annotation of the hand configuration and of the affect channels are detailed.

Descriptors

The raw data collected from motion capture is the vector of the 3D positions of the body markers along time and might not be the best representation to study either the hand configurations or the facial expressions. Indeed, it is often required to transform the initial data in order to get a descriptor that depends only on the phenomenon that we intend to analyze. For instance, if the system is supposed to recognize hand configurations, it should not be sensible to morphological differences between the signers.

Hand Configuration Descriptors

While the positions and orientations of the joints vary according to the chosen reference frame, the Euclidean distances between two articulations are invariant. The hand configurations are therefore described by the vector of the Euclidean distances between each joints of each hand. In our model, each hand has 26 joints (five per finger and one for the wrist) resulting in a total of 325 possible combinations. However, some distances are more relevant than others. For example, distances between two consecutive joints (e.g. the second and third joints of the middle finger) are physiologically similar to bones. Those distances only undergo small variations (due to noise in the data) and are not relevant to discriminate hand configurations. Different subsets of the total number of distances have been tested in order to find the optimal features. A subset of the 29 most discriminating features was preferred (see. fig. 2). It consists of the distances between:

(1) the wrist and the extremities of the fingers (5 distances) to evaluate the bending of the fingers on the palm,

(2) the extremity of one finger with its neighbors (5 distances) to measure the gap between the fingers,

(3) the extremity of each finger and its corresponding knuckle (5 distances) to evaluate the bending of the fingers with respect to the knuckles, and

(4) the extremity of the thumb and the joints of the other fingers (14 distances) to specify the behaviour of the thumb.

The Sign3D corpus that has been used to study the hand configurations contains the data of a unique signer but, in order to have more generic results and to give each distance the same weight, it is necessary to normalize our features.

The normalization was performed by dividing each of the 29 types of distances by its maximal value in the corpus.

All the distances have therefore a value between 0 and 1.

Those distances are then used to segment and label the hand configurations.

Facial Descriptors

A common approach to animate facial expressions of a virtual character is the blendshapes method. An expression Expr can be expressed as the sum of the mesh B 0 representing the neutral expression and a weighted linear combination of n basic deformations b i expressed differentially from the neutral expression (see also fig. 3):

Expr = B 0 + n i=1 w i • b i (1)
This method has the advantage of providing a light representation (in our case only 51 basic deformations) which leads to faster computations and facilitates storing in our database. In order to automatically obtain the appropriate set of parameters {w 1..n } at each frame we have to face two problems: i) the targeted avatar and the signer don't have exactly the same morphology (the retargeting problem), ii) for one given expression E there might be multiple existing We dealt with the retargeting problem as in (Bickel et al., 2007) or (Deng et al., 2006). Given one frame where the signer shows a neutral expression, a RBF regression is trained in order to make the link between the position of any point of the signer's face and the position it would have on the avatar's face:

M = F RBF (M ) = K k=1 u k f k (M ) (2)
where M is the estimated position of the signer's corresponding marker M retargeted on the avatar's mesh and {u 1..K } are the optimized weights associated to the radial basis functions {f 1..K }.

The non unicity problem is formulated as a minimization problem in which the parameters {w 1..n } are optimized so that the distances between the retargeted marker positions M1..40 and the corresponding vertices of the mesh V 1..40 are reduced. To ensure that the optimal weights found with this method do not generate visual artifacts, some constraints (e.g. non-negativity constraint) and/or some regularization energy that penalizes weights outside the [0, 1] range can be incorporated. In our case, we introduced the Thin-shell model (Botsch and Sorkine, 2008) as a regularization energy that doesn't directly ensure that the weights stay between 0 and 1 but penalizes the bending and stretching deformations of the initial mesh:

Ŵ = arg min {w 1..K } (dist eucl ( M1..40 , V 1..40 ) 2 + E T S ) (3)
with Ŵ the optimal vector of weights {w 1..K } for the given expression and E T S the Thin-shell energy. The vector of blendshape weights Ŵ is the chosen descriptor for the analysis of the affect channel.

Automatic Segmentation

The localization of segments of interest in a stream of sign language data is called the segmentation. It is done by detecting manually or automatically the temporal points corresponding to the beginning and the end of a behaviour (hand configuration or facial expression in our case). The coarseness of the behaviour to detect depends on the chosen annotation scheme. For example, sign language data can be segmented at a gloss level by detecting the beginning and end of a sign, or at a finer level such as facial expression, by detecting the beginning and end of an affect.

Segmentation of the Hand Configuration Channels

While signing, the signer alternates between stable periods where hand configurations stay the same (no or little motion of the fingers) and transitions between two consecutive hand configurations. The segmentation step therefore consists in separating the continuous signing sequences in two types of segments : hand configuration or transition. Only the hand configuration segments are labeled in the labeling step. To perform the segmentation, we assume that the variation of the distances is discriminating, i.e. the variation will be small during stable configurations and high during transitions. For each frame f , and for each selected distance SD, the variation of the distances d(i, j) between two joints (i and j) is computed between the frame f -1 and f . Those variations are summed (see Equation 4for the right hand RH).

VarDist f,RH = i∈RH j∈RHd(i,j)∈SD |d(i, j) f -d(i, j) f -1 | (4)
The segmentation relies on the use of a threshold. If VarDist is above this threshold, a transition segment will be detected. If VarDist is below the threshold, the segment will be recognized as a hand configuration segment. To select the value of the threshold and to evaluate our segmentation, we used the Simple Matching Coefficient (SMC) metric. It measures the similarity between two sets of values (here, the manual and the automatic segmentations). The SMC is the ratio of the number of overlapping frames between the two segmentations on the total number of frames. Figure 4 shows the variation of the SMC of the whole corpus with respect to the chosen threshold for the 29 normalized distances. The maximum (SMC = 81%) is reached for a threshold of 0.2. As manual annotation is performed by a human being on video footage, automatic annotation may be more accurate than manual annotation. Therefore, we consider this threshold satisfactory and it will be the one used in the following steps.

Segmentation of the Affect Channel

The affect channel is segmented in a similar way, before the labeling step . We aim at detecting the frames that are located at the border between two segments. Since the border frames are related to the transitions from one expression to another we consider the energy curves of the velocity and acceleration of the n blendshape coefficients:

E V elBS = n i=1 | dw i dt | (5)
E AccBS = n i=1 | d 2 w i dt 2 | (6)
In order to detect the local peaks, we consider the local optima of the curve E 2 V el BS + E 2 Acc BS , and only keep those for which the local variation is important. This detection procedure is achieved by computing the derivative values on a window surrounding the detected optima, and applying a threshold. The orange and turquoise curves in Figure 5 show an example of segmentation using this method.

Automatic Labeling

The identification of the previously defined segments of interest is called the labeling. This task is highly dependent Manually segmented and labeled sequence Automatically segmented and labeled sequence Figure 5: An example of automatic annotation: "What? I won 1000 e!" repeated 3 times (white: surprise, blue: joy, cyan: fear; the orange curve stands for the sum of accelerations, the turquoise one for the sum of velocities; the vertical brown lines represent the limits of each segment; each vertical black line stands for 0.5 second). on the chosen annotation scheme. Typically, it will consists in selecting the right label from a closed vocabulary to identify a segment.

Labeling the Hand Configuration Channels

A supervised machine learning approach is used to identify the handshape on each frame of the hand configuration segments. 32 classes were defined corresponding to 32 different handshapes (see fig. 6). Like for the segmentation step, the algorithm takes as input the chosen distances between the joints to characterize the hand configurations. Our machine learning classifiers are trained on 23533 manually annotated frames presenting those 32 configurations. The test set is composed of 3927 frames which amount to 14% of the total number of available examples. Our labeling approach sorts each frame in one of the 32 categories. We tested three different machine learning algorithms : logistic regression, support vector machine (SVM) with a linear kernel and k-nearest neighbors (kNN) on different subsets of our descriptors. Figure 7 shows the accuracy (number of correct predictions divided by the total number of predictions) on the test set depending on the machine learning algorithm and the subset of distances selected. We can see that the "29 distances" subset presented in Section 4.1. gives the best results with the 3NN approach (91.2%) while the SVM on the 325 distances have the overall best accuracy (92.3%) (but the duration of the classifier training is longer). Some configurations are more sensible to confusion than others. For example, the 'K' and 'V' configurations are often mistaken (in the two configurations, the middle and index fingers are raised; in the 'K', there is a contact between the thumb and the base of the middle finger while there is not in the 'V'). The 'B' and 'Pi' configurations are also confused as only the thumb position is discriminant between the two configurations.

Labeling the Affect Channel

The facial channel labeling is a supervised learning task aiming at identifying the correct class among the 7 defined in section 3.2.. Different methods were tested: kNN (1NN and 3NN), SVM (linear and RBF kernels) and Random Forests (RF). The sequences recorded on each signer were processed separately. For each signer, the sequences IE and SE which represent roughly 50% of the data were used as the training set while the EU sequences were used as the test set. Whereas during the training phase, each frame of the training set with its corresponding manually annotated label was considered as a training sample, the test examples were constituted of the average along time of the frames composing each segment. These segments have been previously obtained according to the method presented in sec-tion 4.2.. Each of these segments was classified as a whole represented by its average vector of blendshape weights:

W = F f =1 Ŵf F (7)
with F the number of frames of the considered segment. Figure 5 shows an example of classification using this method; results are detailed in next section.

Results

The results of the automatic annotation of the hand configurations and of the facial expression channels are presented in this section.

Automatic Annotation of the Hand Configurations

To automatically annotate hand configurations during continuous signing, (i) the stream is segmented to distinguish hand configuration from transitions segments (section 4.2.), then, (ii) the hand configuration of each frame of the hand configuration segments is labeled (section 4.3.), and finally, (iii) each hand configuration segment is labeled according to the predominant class in the segment (see fig. 8).

Figure 8: Overview of the automatic annotation of hand configurations.

Figure 9 shows three utterances of French Sign Language manually and automatically annotated. The segmentation threshold has been fixed to 0.2 and the machine learning algorithm used here is 3NN (3-nearest neighbors). We worked with the 29 distances described in section 4.1.. While the results can differ from one utterance to another, the recognized labels and segments are mainly consistent with the manual annotation. The results given by the metrics (i.e. accuracy, recall and precision) are computed with respect to the manual annotation and are therefore limited by the 80% overlap of the automatic segmentation with the manual segmentation. There are very few errors in terms of recognition of hand configurations (accuracy of 90%). A perceptual evaluation could give a better assessment of our results.

Automatic Annotation of the Affect channel

The sequences are first segmented according to the methods described in section 4.2.. Each segment is then labeled independently of the others, according to the following procedure. For each segment, we compute the average vector descriptor over time: W . This vector is used as input of the classification model (kNN or SVM or Random Forest) previously trained on the basis of the learning set. The classifier then returns the label associated with this segment. In order to evaluate the error due to the segmentation, the automatic annotation is performed on both the automatically detected segments and the manually defined ones. For both segmentations, Figure 10 gives the accuracy of the classifier for each tested algorithm. It shows that the best results are obtained with the Random Forest algorithm.

Figure 10: Accuracy on the test set for the affect channel depending on the machine learning algorithms and the segmentation.

Conclusion

We designed an approach to automatically annotate sign language MoCap data by processing each channel separately. We detailed the specific examples of hand configuration and facial expression annotation.

There are still many challenges to overcome. Using machine learning models, the automatic annotation could be significantly improved by increasing the size of the dataset, so that the training phase would be more efficient. In addition, following the approaches developed in language processing, we could also use models that learn the dynamics of the sequences, such as Hidden Markov Models, Con-ditional Random Fields, or Recurrent Neural Networks. However, these methods require large databases. Another challenge concerns the evaluation of the annotation results. Indeed, for the manual annotation, we rely on a ground truth which may be subject to errors or imprecision. This problem occurs for most recognition or annotation tasks. One solution could be to define a ground truth from a set of previously trained annotators, following strict instructions. In the near future, we plan to validate our annotations by defining quantitative metrics, both for hand configurations and facial expressions. As a complement to assess the quality of the annotation, we also plan to perceptually evaluate the results.

Figure 2 :

 2 Figure 2: The subset of the 29 distances.

  b i that minimize the distances between the markers and the corresponding vertices of the mesh (the non unicity problem).

Figure 3 :

 3 Figure 3: Synthesizing expressions from a linear combination of blendshapes.

Figure 4 :

 4 Figure 4: SMC with respect to the threshold values. The maximum (SMC = 81%) is reached for a threshold of 0.2.

Figure 6 :

 6 Figure 6: The 32 hand configurations played on an avatar.

Figure 7 :

 7 Figure 7: Accuracy on the test set for the hand configurations channel depending on the machine learning algorithm and selected distances.

Figure 9 :

 9 Figure 9: Result of the automatic annotation of hand configurations on three different utterances.

Acknowledgements

Part of the observations of this paper are based on the motion capture data and annotations of the Sign3D project (Gibet et al., 2015).

Aifanti, N., Papachristou, C., and Delopoulos, A. (2010). The mug facial expression database. In 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10, pages 1-4, April. Battison, R. (1978). Lexical borrowing in American sign language. ERIC. Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M., Pfister, H., and Gross, M. (2007). Multi-scale capture of facial geometry and motion. In ACM SIGGRAPH 2007 Papers, New York, NY, USA. Botsch, M. and Sorkine, O. (2008). On linear variational surface deformation methods. IEEE Trans. on Visualization and Computer Graphics, 14(1):213-230, January. Busso, C., Bulut, M., Lee, C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J., Lee, S., and Narayanan, S. (2008). Iemocap: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42(4):335-359, 12. Deng, Z., Chiang, P.-Y., Fox, P., and Neumann, U. (2006). Animating blendshape faces by cross-mapping motion capture data. In Proce. of the 2006 Symposium on Interactive 3D Graphics and Games, pages 43-48, New York, NY, USA. Dilsizian, M., Yanovich, P., Wang, S., Neidle, C., and Metaxas, D. (2014). A new framework for sign language recognition based on 3d handshape identification and linguistic modeling. In Nicoletta Calzolari (Conference Chair), et al., editors, Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), Reykjavik, Iceland, may. European Language Resources Association (ELRA). Dreuw, P. and Ney, H. (2008). Towards automatic sign language annotation for the elan tool. In 3rd Workshop on the Representation and Processing of Sign Languages. Gibet, S., Lefebvre-Albaret, F., Hamon, L., Brun, R., and Turki, A. (2015). Interactive editing in French sign language dedicated to virtual signers: requirements and