Frédéric Elisei
email: frederic.elisei@gipsa-lab.inpg.fr

Gérard Bailly
email: gerard.bailly@gipsa-lab.inpg.fr

Alix Casari

Stephan Raidt
email: stephan.raidt@gipsa-lab.inpg.fr

Towards eyegaze-aware analysis and synthesis of audiovisual speech

Keywords: eye gaze, eyelids, face-to-face interaction, talking heads, embodied conversational agents, audiovisual speech corpus analyzing

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

When interacting, people mostly gaze at face and gesturing. While speech is clearly audiovisual [START_REF] Stork | Speechreading by Humans and Machines[END_REF], facial expressions and gaze also inform us about the physical, emotional and mental state of the interlocutor. Together with speech and gesturing, face and gaze participate in signaling discourse structure, ruling turn taking and maintaining mutual interest. This paper will focus on the eye region, studying the eyelid shape deformations that we could observe, along with eye gaze changes, in an expressive audiovisual speech corpus used to create a talking head (Figure 1).

Our long term research goal is to build an accurate control model for our next generation of context-aware ECAs (embodied conversational agents). We want them to be able to collaborate on some tasks using voice and pertinent gaze patterns (keeping eye contact, maintaining mutual attention, pointing at objects, displaying feeling of thinking...) with humans in their environment.

Therefore, audiovisual synthesis should be able to provide the eye region of the face with realistic appearance as well as pertinent behavior and control models. In the same interactive loop (imposing real-time scene analysis), ECAs should be reactive to the gaze patterns of their interlocutors and implement a complex set of interaction rules, as noted in [START_REF] Thórisson | Natural turn-taking needs no manual: computational theory and model from perception to action[END_REF]. For analysis as well as for synthesis, this remains a challenge covered by many researches from various domains.

Related research results

Importance and role of eye gaze

Eye gaze production and perception in human-human interaction has already attracted a lot of psychophysical and psychological studies. In conversation, gaze is involved in the regulation of turn taking, accentuation and organization of discourse [START_REF] Kendon | Some functions of gaze-direction in social interaction[END_REF][START_REF] Argyle | Gaze and mutual gaze[END_REF]. Most data on eye movement of perceivers during audiovisual speech perception have been gathered using non-interactive audiovisual recordings [START_REF] Vatikiotis-Bateson | Eye movement of perceivers during audiovisual speech perception[END_REF]. Several experiments have however shown that gaze patterns in live interaction are significantly different from screening: social rules have in fact a strong impact on communication when interacting face-to-face [START_REF] Gullberg | Visual attention towards gestures in face-to-face interaction vs on screen[END_REF]. Gaze and eye-contact are also important cues for the development of social activity and speech acquisition [START_REF] Carpenter | Joint attention, cultural learning and language acquisition: Implications for children with autism[END_REF].

Perceived gaze direction is biased by the head orientation [START_REF] Langton | The mutual influence of gaze and head orientation in the analysis of social attention direction[END_REF]. To our knowledge, no study has measured the importance of the eyelid movements for gaze perception, although gaze direction clearly influences eyelid appearance.

Animating eye gaze or eyelids

Some statistical properties of the studied or measured eye gaze trajectories and blink rates have been evidenced and implemented to drive avatars and ECAs [START_REF] Cassell | ANIMATED CONVERSATION: Rule-based Generation of Facial Expression, Gesture & Spoken Intonation for Multiple Conversational Agents[END_REF][START_REF] Colburn | The Role of Eye Gaze in Avatar Mediated Conversationnal Interfaces[END_REF][START_REF] Lee | Eyes alive[END_REF][START_REF] Garau | The Impact of Avatar Realism and Eye Gaze Control on Perceived Quality of Communication in a Shared Immersive Virtual Environnment[END_REF][START_REF] Bilvi | Communicative and statistical eye gaze predictions[END_REF]. These gaze control models usually distinguish between several states of the user-ECA interaction (at least talking, listening, but sometimes much more complex ones gathered in the literature). While they often use rather realistic-looking 3D face models, the eyelid shapes are often not realistically correlated with the eye gaze direction. For example, Lee et al. [START_REF] Lee | Eyes alive[END_REF] report: « During the analysis of eye-tracking images, we noticed a high correlation between the eyes and the eyelid movement which could be incorporated ».

The eye pupils and eyelids synthesis has gained increased interest in computer vision and image-based rendering with the potential development of audiovisual communication software (video calls and virtual conference systems). Modifying the displayed gaze orientation, and possibly the head orientation, is almost mandatory to restore the eye contact and all the related behavior. Many proposals that worked at correcting the pupil position and appearance have reported that not correcting the eyelids results in inappropriate expressive attitude being perceived [START_REF] Gemmell | Gaze awareness for video-conferencing: A software approach[END_REF], for example looking surprised if eyelids remain wide open while the gaze is modified to look downwards. In a shared mixedreality space, Tateno et al. [START_REF] Tateno | Enhanced eyes for better gaze-awareness in collaborative mixed reality[END_REF] demonstrated the benefit on gaze awareness of enhancing eyelid shapes.

Many 3D animated models of the eyes region have been used and proposed by computer graphics industry and researchers. Correcting eyelids to avoid interpenetration with eyeballs during the animation is necessary [START_REF] Parke | Section 6.3, Implementation of a Direct Parametrized Model[END_REF], but such algorithms fail to reproduce reality. Some highly detailed 3D models exist [see reviews in 17, 18], but eye and eyelids joint-control is usually not addressed. Moriyama et al. [START_REF] Moriyama | Meticulously detailed eye model and its application to analysis of facial image[END_REF], whose model is generic enough to adapt across large subject variability as well as eye movements, uses independent sets of parameters to lower or raise either the eyelids or the eyeball. In [START_REF] Adamo-Villani | EMOES: Eye Motion and Ocular Expression Simulator[END_REF], a somewhat high-level dynamic controller is proposed for their advanced EMOES model, but eyelid and eyeball controls seem to remain independent. In [START_REF] Itti | Photorealistic Attention-Based Gaze Animation[END_REF], Itti et al. report that they tried to improve the visual animation of their 3D head by raising eyebrows (effectively acting on eyelids) when gaze is lifted up.

Tracking eye gaze

Capturing real eye gaze data has never been so easy, with many industrial solutions using infrared light and sensors now available. But the least invasive solutions do not solve the generic problem of recovering the point looked at in space by a subject, they only capture the point looked at on screen (or any other plane surface) which may be sufficient for mediated face-to-face interaction [START_REF] Raidt | Mutual gaze during face-to-face interaction[END_REF]. More complex but invasive equipments fail to capture ecological face-to-face gazes when subject's face or eyes are partly hidden or look distracting. Some hardware or software relies on tracking where the head or the eyes are in space, but they generally do not capture how the eyes and eyelids are looking (ocular expressions…)

Computer vision research develops new solutions to compute more accurate/general solutions, but using multiple or single cameras seems promising [START_REF] Tan | Appearancebased Eye Gaze Estimation[END_REF][START_REF] Wang | Eye Gaze Estimation from a Single Image of One Eye[END_REF][START_REF] Djeraba | State of the art of Eye tracking[END_REF].

Overview of our proposal

In the real world, eye gaze is important and eyelid deformations (as well as head movements) participate to the elaboration of the displayed and perceived gaze direction. In an ECA context, synthesized or captured gaze patterns should be rendered so that human partners may perceive the intended multimodal deixis and mutual attention. Adequate prediction of eyelid deformations together with gaze direction will reinforce the perception of spatial cognition.

For our audiovisual studies, we chose to capture and model motor activities of the upper face of one subject with a data-based approach. Section 4 presents the labeling of the training dataset, consisting of images that span the space of gaze directions and capture the eyelid deformations, either cooccurring or independent. Section 5 proposes a two-stage nonlinear model that can cope with pure rotations (as undergone by points on iris contour) or more complex transformations. Section 6 presents a preliminary analysis-bysynthesis algorithm, used to collect speech articulation and co-occurring gaze patterns from free speech video sequences.

The training dataset

The gaze related eyelid deformation model presented here is extending a 3D articulatory face and lips model of a German speaker. This talking head is presented and evaluated in another paper of these proceedings [START_REF] Fagel | Intelligibility of natural and 3D-cloned German speech[END_REF]. The initial model and the extended one were built using a common corpus. In this corpus, 400 colored beads were used as markers, glued on the subject's face and covering the area from one ear to another and from the forehead to the neck. Some of the markers were placed around the eyebrows, as can be seen on Figure 1.

To capture the face model as well as dynamic eye gaze stimuli, three synchronized analog video cameras (720x576 pixels, 50Hz, interlaced) were used to record the subject. In this setup, the subject's chest was hold in place and can be considered as static relatively to the three cameras. Using a calibration object, intrinsic parameters, epipolar geometry and relative positions of the video cameras were extracted. 3D coordinates in the world frame can then be estimated easily.

Using a selected subset of uttered visemes from the calibrated video cameras, we will first build a 3D articulatory face-only model, as described in [START_REF] Bailly | Degrees of freedom of facial movements in face-to-face conversational speech[END_REF]. In this methodology, the rigid movements of the head defined in reference to the bite plane are accurately subtracted from the measured 3D coordinates. Then, a modified PCA (principal component analysis) approach is applied, that recovers a set of 6 articulatory speech-related dimensions (jaw opening, lips rounding, upper lip control, lower lip control, jaw advance and throat lowering) and how they correlate with the displacements of the beads. In the reconstruction process, the rigid movements of the head, relative to the static chest, are also put back in the model with 6 extra movement parameters. They drive non rigid deformations in the neck area.

Extra data for the eye region

For the eye-area modeling phase, data with better spatial accuracy was collected using a single numerical photo camera (1200x700 pixels, not interlaced) to grab static postures of the full face (which are also used for texturing the face). In this setup, the subject was sitting on a desk chair that could be rotated: his position relative to the photo camera is not static anymore. The subject was successively asked to hold a constant gaze in nine fixed directions (far front, as much as possible on the right/left/up/down and the remaining corners). Closed eyelids were also recorded. For each hold gaze direction, the seat was slowly rotated to allow high-resolution images to be captured by the photo camera from three viewpoints (left, front and right views). Intrinsic parameters of the photo camera and its projection matrix were also recovered using our calibration object. Of course, 3D coordinates cannot be reconstructed from a single view. as will be explained later.

In our eye-modeling procedure, we first use the built face model to recover the spatial relationship of every coherent image triplets (images featuring the same recorded gaze). Fitting the model recovers the facial articulation (constant for image triplets) as well as the head/chest pose of every image. This is performed with Matlab, by minimizing the prediction error of beads positions while optimizing the articulatory and head posture parameters. In practice, this is quite equivalent to having three calibrated photo cameras in different positions for every recorded gaze. More important is the fact that this also recovers the head orientation and computes the head frame in which the modeling has to be conducted.

On every set of three coherent images and with the help of epipolar lines, an interactive editor is used to draw 3D Bezier curves. We outlined the iris contour, the upper and lower eyelids, as well as an eyelid anchor line under the eyebrows (see Figure 1).

To increase the statistical completeness, 24 image triplets were also selected in the dynamic low resolution corpus (obtained with the three video cameras) in order to span a more complete range of gaze directions. 3D Bezier curves in the eye region were also edited on these images.

The iris contour will be used to estimate the gaze direction. The other contours will shape the 3D model. At this point, the curves are resampled using regularly spaced curvilinear coordinates to get a canonical representation with a constant number of points. All points have been transformed and expressed in the head coordinate system.

Modeling the eyeballs/eyelids

Our modeling hypotheses

Our first set of modeling hypotheses concerns the shape of the iris contour and its relation to the gaze direction. Like many researchers, we assume that this external iris contour is planar and circular. We also assume that the gaze direction is equal to the normal traversing the iris contour circle in its center. From these, we will capture the eye gaze directions and the eyeball rotations by observing the external iris contour.

Our second modeling hypothesis is that the eyelid deformations that correlate with the eyeball rotations can be captured in a two stage non-linear model, as was done for hand articulation in a cued speech context [START_REF] Gibert | Analysis and synthesis of the 3D movements of the head, face and hand of a speaker using cued speech[END_REF]. In the first stage of the model, angular parameters are driving trigonometric functions. The second stage of the model will be linear, predicting 3D coordinates from the trigonometric values. Such a two-stage model can capture pure rotations without a loss, but more complex anisotropic displacements or deformations can also be modeled.

A third modeling hypothesis is that remaining eyelid deformations can be modeled as extra linear contributions that add to the previous deformations. We expect at least one for blinking. This is not a strong assumption as a complex nonlinear deformation can always be modeled by several linear contributions (it just adds more parameters than necessary and adds unnecessary complexity to control models).

Our last hypothesis will only affect the eyeballs rendering process: the subject's eyeball geometry can not be measured, so we used an average generic ratio to derive his eyeballs geometry from his iris radius.

Results of the modeling

From the labeled iris contours and for each eye, we compute the 3D coordinates of each iris center, the direction that is normal to the iris plane, as well as an iris radius. Fitting circles is robust enough that we recover an almost constant radius for the two irises across the high resolution images. This value has been used to ease the labeling in the 24 lower resolution images. The normal vector is expressed in the head frame as two angles a1 (azimuth) and a2 (elevation). In our two-stage modeling formalism, we are interested in the linear regression between 3D coordinates and a set of trigonometric values. Ideally, we would retain the sine and cosine from a1, from a2, from their difference and from their sum. These are necessary and sufficient to reconstruct the rotation matrix characterized by (a1,a2) by an additive formalism (because cos(a1).cos(a2) and other similar product functions are linear combinations of the retained functions). In practice, we do not have enough training data to robustly recover all the model coefficients that would drive the 3D coordinates from the full set of trigonometric values. For robustness, we only retain sin(a1+a2), sin(a1-a2), cos(a1) and cos(a2), as they best reconstruct the eyelid data variance. A linear regression of the centered training 3D coordinates using the trigonometric values as predictors recovers 66% and 61% respectively of the left and right eyelid coordinates variance.

We perform a standard principal component analysis (PCA) on the residual data, to catch extra degrees of freedom. As we expected, the first principal component acts along a blink dimension, narrowing or widening the upper and lower eyelids. This captures 40.9% and 30.8% of the residual data, for the right and left eyelids respectively. As the next principal components do not display clearly interpretable movements, we only retain the first principal component.

Using three parameters per eye (two angular ones and a linear one), 79.9% and 73% of the original data variance is captured (for the right and left eye, respectively).

Figure 2 presents the effects of our three parameters on eyelids geometry and gaze orientation. Both the upper and lower eyelid contours are deforming, especially for horizontal gaze movements. For vertical gaze movements, both are moving. The upper eyelid anchorage is not very active. Globally, the eyelid geometry preserves the vision field while protecting the rest of the eye. A geometric control model can easily generate the four low-level angular parameters from a target point in space to preserve the gaze vergence. Using homogeneous coordinates for the target point conveniently encodes cases with vergence or gazes at optical infinity.

Rendering of the model

Figure 3 presents textured 3D views of the merged model (6 speech-related articulatory parameters, 3 gaze and blink parameters per eye, and 6 head movement and orientation parameters). In the eye region, the crude mesh learned with beads only has been replaced by the animated high-definition eyelid region and 3D eyeballs. The eyelids are textured with a single static image. Rendering is performed using the OpenGL library. For each eyeball, we use a two-sphere model whose intersection matches the measured iris circle. The ratio of the radii of each pair of spheres reproduces an average ratio. To avoid 3D collisions between the eyeball and the eyelid geometries, the rendering is performed using OpenGL stencil buffer capability: the textured eyelids and face are rendered first and the stencil buffer is cleared. Then, the area comprised between the upper and lower eyelid is filled using invisible triangles to enable the stencil for these pixels. This defines two windows through which eyeball geometries will be rendered and become visible.

Our rendering is not photorealistic yet: eyelashes should be modeled either by their shape or by their appearance; eyelids need multiple textures to capture wrinkles and creases appearance changes, and pertinent glints should be computed for the eyeballs. Despite these drawbacks, subjective tests where the face-model eyes track a moving 3D target show that the face looks more natural and that its gaze is perceived much more accurately than with static eyelids. Of course, objective tests need to be conducted to assess these results.

An algorithm for corpus analysis

The model we developed is not to be used in a 3D rendering context only. The motivation of our meticulous modeling is to use it also to analyze recordings of the cloned subject. To illustrate the interest of such a model in this context, we have been experimenting on our currently available data with the very simple four-step algorithm presented here.

This algorithm can capture eye gaze accurately if eyelids are open. It can capture blinks, but fails to robustly capture eye gaze when eyes are almost closed. Experimenting with a bigger corpus in the future, we will try to solve the shortcomings of our actual algorithm.

Recovering facial articulation

First, we recover both the six head movement parameters and the six articulatory speech parameters using the simpler model (no eyelids). As the rendered 3D model perfectly matches the recorded images, analysis by synthesis could have been used [START_REF] Odisio | Tracking talking faces with shape and appearance models[END_REF]. We actually used a pattern matching algorithm to track the beads displacements first. Matlab is used then to minimize the bead positions reconstruction error while optimizing the 12 parameters. At that point, the face model matches the video stimuli by its face orientation and speech articulation, but not by its eye gaze.

Estimating the blink parameter

It might have been convenient to have markers on the eyelids of our subject (as used in motion capture studios to detect blinks) but our corpus features none there. With a video realistic enough 3D model, we might compare directly each RGB(x,y) pixel at position (x,y) in a camera image with its R S G S B S (x,y) counterpart in the synthesized image. We avoid using a prior segmentation of the images to evade the introduction of coarse segmentation errors. These errors seem very likely with the relatively low coverage of the eye region for a full-face corpus at video camera resolutions.

Instead, our second step uses an indirect analysis-bysynthesis loop to estimate the value of each blink parameter (1 per eye). We chose to reinforce the red component of the eyelid region to contrast with the eye ball region, filled by white and iris colors. We propose to represent a pixel at position (x,y) in a camera image by its normalized red component:

) , ( ) , ( ) , ( y x RGB y x R y x p ! (1)
For every triplet value (a 1 ,a 2 ,v) of gaze and blink values, the eyelid model and the camera model predict where eyeballs and eyelids are projected (see Figure 4 for the defined regions). For correct parameters values, pixels covered by the E3 region should have a stronger red bias than pixels in the E1 and E2 regions. E3 is defined so that its external contour does not depend from the gaze and blink parameters: the pixels covered by E1+E2+E3 depend only from facial articulation and head posture. Using a 1 =0 and a 2 =0 as initialization, we first optimize v only to minimize:

" # $ $ % % & ' ! 2 1 ) , ( 3 ) , ( 2 1 ) , ( ) , ( 1 )
, , (

E E y x E y x v y x p y x p v a a Err ! (2)
The inversion process is to be continued with the two following steps only if the resulting v value does not correspond to a completely closed eyelid.

Estimating the gaze direction

In the "window" opened by the eyelids, iris and pupil pixels contrast with the white part (sclera). Considering only the red planes (camera image and synthetic image), we found it robust enough to use a direct analysis-by-synthesis approach at this point. For this, we chose two suitable colors to draw the synthesized pixels corresponding to E1 and E2. Picking the average color of the two regions in a camera image and amplifying their contrast was successful enough with our subject.

Preserving the previous value of the blink parameter v, together with the defined E1 and E2 regions, we optimize for the two eye gaze angles a1 and a2 to globally match the image iris contour by minimizing the image reconstruction error (in the red channel):

$ % ' ! ) , 0 , 0 ( 2 ) , 0 , 0 ( 1 ) , ( 2 1
) , ( ' ) , ( ) , , (

v E v E y x a y x R y x R v a a Err ! (3)
Contrary to v variations that might have subtle influences, a1 and a2 usually have easily visible influence in some of the camera images, and non marginal effect on equation (3).

Final inversion

At this point, the three parameters values are only estimations, not having been optimized at the same time. Using them as initial values to minimize again equation ( 2) then (3) as in the linesearch algorithm leads to better solution.

Results

The proposed algorithm has been implemented using OpenGL. Using a PCI x16 3D accelerator, a new set of 6 gaze/blink parameters is recovered every 800ms, using our three views setting. Figure 5 shows some results of our tracking procedure, with a few example frames. The accompanying AVSP DVD (see also [29]) includes videos of analyzed and reconstructed audiovisual recordings. Figure 6 shows the trajectories of the parameters for a subpart of the sequence.

One can notice that eyelashes can lead to over-estimation of the blinking parameter. This is no surprise, as the pixels covered by the dark hairs are classified as closer to the iris color than to the eyelid skin color. This bad classification might also explain the difficulties in recovering the gaze direction when the eyelids are almost closed.

Conclusions

We presented a methodology to construct an articulatory model of the shape deformation for the eyelids of a virtual talking head that takes into account the eye gaze direction. A model has been built from the data of a German speaker. It is driven by three low-level parameters per eye: two for gaze direction and one for the proper opening/closing movements of the eyelids.

We will assess the effectiveness of the eyelid deformation model and its positive impact on the perceived gaze in the deictic bootstrapping paradigm already used in [START_REF] Raidt | Does a virtual talking face generate proper multimodal cues to draw user's attention towards interest points?[END_REF] with our previous ECA generation.

This model was used to analyze and reconstruct some audiovisual recordings of the cloned subject in a face-to-face interaction. While the inversion algorithm needs to be corrected to cope with closed eyes (post-processing the data in the blinking intervals) or almost closed ones (drawing eyelashes), the model itself predicts satisfactorily the observed deformations. We might also compare the accuracy of the algorithm with that of our infrared based eye-tracker, or check the robustness of the algorithm (to slight head pose changes, image noise or image resolution) with synthetic images.

The recovered eye movements will complement the gestural score driving our ECAs. The automatic generation of such scores is particularly interesting for studying the gestural encoding of linguistic and paralinguistic information in discourse production with less manual labeling [START_REF] Martin | Manual Annotation and Automatic Image Processing of Multimodal Emotional Behaviours: Validating the Annotation of TV Interviews[END_REF].

Creating such a 3D model per subject is quite tedious, especially as it needs cloning the full face also. While we believe this data-based approach to be worth the effort when the accurate measurement or analysis of audiovisual stimuli is needed, we plan to reuse the already built model in a 3D synthesis context: with proper scaling and moderate change of orientation, the two eye regions can be merged on another talking agent and improve the perception of gaze.

Acknowledgements

Parts of this work were supported within the PROCOPE program, the GIS PEGASUS, the ELESA research federation and the Presence project of the Cluster Rhône-Alpes InfoLog. We want to thank Christophe Savariaux and Alain Arnal for their technical help in the capture process and Ralf Baumbach who helped with annotating the data. We are also grateful to Sacha Fagel for accepting to be our subject.

Figure 1 :

 1 Figure 1: Evidences from our learning corpus, showing different eyelid shapes and eye gaze directions. The color beads glued on the subject face serve as markers to construct the face model. White lines are 3D curves drawn manually on the images to recover the eye related contours (iris, eyelids…)

Figure 2 :

 2 Figure 2: Effects of each control parameters on the gaze and the eyelid geometry (in the figure, the same value is used for left and right eye parameters)

Figure 3 :

 3 Figure 3: Textured but non-photorealistic renderings of the model, while looking respectively far in front, on the left and slightly down, ahead of its mouth.F0 is the original mesh, connecting beads. E3 is mainly a skin-colored area, including the eyelids. E2 is mainly white (sclera from the eyeball), when visible (eyelids opened). E1 is the visible part of the iris and pupil.

Figure 4 :

 4 Figure 4: Details of the extended mesh (left eye)

Figure 5 :

 5 Figure 5: Tracking results of the proposed algorithm. The three used camera views are shown horizontally. Curves recovered from the model are displayed in bright yellow: eyelids anchorage, eyelids and complete iris circle. The two first lines show succesfull trackings. The last line demonstrates an inversion error, with almost closed eyelids (see text).

Figure 6 :

 6 Figure 6: Analyzing an extract of conversational speech. From top to bottom, using the same time-scale: acoustic signal with orthographic transcription, azimuth angle for the right eye, blink parameter for the right eye and head nodding parameter. The articulatory scores were all produced by automatic inversion processes from the images (3 views) and the models. Two blinks are enlightened in the parametric tracks. They cue end of utterances in order to signal turn keeping. Note that gaze patterns and head motion are quite nicely synchronized with phrases.


Z Alaverdyan

C Lartizien

Feature extraction with regularized siamese networks for outlier detection: application to lesion screening in medical imaging

Feature extraction with regularized siamese networks for outlier detection: application to lesion screening in medical imaging

Z. Alaverdyan, C. Lartizien

Introduction

For each voxel v, gather its feature vector of dimension 32 by using a patch centered at the voxel as input to the proposed feature extraction method (see the blue block) • Voxel-level oc-SVM For each voxel v, extract the feature vectors from all the healthy subjects and train a oc-SVM [START_REF] Schölkopf | Neural computation[END_REF] model with rbf kernel (x i is the feature vector of subject i) as in [START_REF] Azami | [END_REF]:

min w,ρ,ξ i 1 2 ||w|| 2 + 1 νn n i=1 ξ i -ρ subject to w • φ(x i ) ≥ ρ -ξ i , ξ i ≥ 0, i ∈ [1, n]
Feature extraction with regularised siamese networks (1 -α)cos(g(x 1 ), g(x 2 ))

• Feature vector Given an input patch x, the middle-layer representation in the subnetworks g(x) yields its feature vector

Results

Table 1: Test results for 9 patients with confirmed lesions reporting true positive/false positive clusters. Our rSNN features are compared to the handcrafted features in [START_REF] Azami | [END_REF] and those obtained with stacked denoising autoencoder(sDA).

Feature type

Conclusion and future work

• Automatic feature extraction allows detecting voxel-level outliers in brain MR images and yields at least equivalent epilepsy lesion detection rate as epilepsy-specific handcrafted features. • Slightly more false positive detections compared to the handcrafted features.

• Future work includes integrating other imaging modalities such as MR FLAIR images.

  Medical application contextDesign a Computer Aided Diagnostic (CAD) system for lesion screening in brain MR images ⇓ Assist clinicians in fast lesion detection in routine exams Specifics • No annotated pathological data =⇒ voxel-level outlier detection problem • Relatively small data set (around 100 training subjects) Objective Design an automatic feature extraction method to be used in outlier detection problem at voxel level CAD pipeline description

Figure 1 :

 1 Figure 1: CAD output for patients A, C and E. Maximum intensity projections of clusters onto slices of interest are shown. Lesion location is highlighted in red circles.

  Patches (x i , x j ) are similar if they are centered at the same voxel v

	• Strategy
	Randomly extract similar patches from different training subjects
	Feed into a regularized siamese neural network [1]
	• Architecture
	Our regularized siamese neural network (rSNN) is composed of two identical stacked
	denoising autoencoder (sDA) subnetworks with l hidden layers
	• Loss function
	In rSNN the proposed loss function for a single input pair (x 1 , x 2 ) is:
		regularizer	'close' representations for similar patches
	L(x 1 , x 2 ; Θ) = α	2 t=1 ||x t -xt || 2 2	-
		reconstruction error

• Definition: similar patches


Stephan Raidt
email: stephan.raidt@gipsa-lab.inpg.fr

Gérard Bailly
email: gerard.bailly@gipsa-lab.inpg.fr

Frédéric Elisei
email: frederic.elisei@gipsa-lab.inpg.fr

Analyzing and Modeling Gaze during Face-to-Face Interaction

Keywords: embodied conversational agents, face-to-face interaction, eye gaze, talking head, gaze model

We present here the analysis of multimodal data gathered during realistic face-to-face interaction of a target speaker with a number of interlocutors. During several dyadic dialogs videos and gaze have been monitored with an original experimental setup using coupled cameras and screens equipped with eye tracking capabilities. For a detailed analysis of gaze patterns we distinguish different regions of interest on the face. With the aim to understand the functions of gaze in social interaction and to develop a coherent gaze control model for our talking heads we investigate the influence of cognitive state and social role on the observed gaze behaviour.

Introduction

When interacting with an ECA (Embodied Conversational Agent) we do have strong expectations concerning its appearance that we interpret and evaluate according to natural human behavior. The more human-like the appearance of the ECA the higher are the interlocutors' expectations towards human like behavior. This is true for the entire interaction and not only while the ECA actively communicates. We perceive the ECA not only when it speaks but also the way it displays attentiveness, listens to us or its way of managing turn taking. The comprehension of the dialog and the credibility of the delivered information may be degraded by incorrect control strategies, imprecise interaction loops or an impoverished multimodal implementation. In this context we consider gaze as an essential component of face-to-face interaction and proper gaze generation pivotal to maintenance of mutual attention and prosperous interaction.

Eyes are very special stimuli in a visual scene and humans are especially sensitive to the orientation of eyes [START_REF] Argyle | Gaze and mutual gaze[END_REF][START_REF] Langton | Reflexive visual orienting in response to the social attention of others[END_REF]. The gaze direction carries a huge diversity of information. It reveals the center of interest of a subject and may guide the attention of an interlocutor [START_REF] Langton | Do the eyes have it ? Cues to the direction of social attention[END_REF][START_REF] Yarbus | Eye movements during perception of complex objects[END_REF]. Together with facial expression and context, it is interpreted to derive mental states of another person [START_REF] Baron-Cohen | Do children with autism use the speaker's direction of gaze strategy to crack the code of language?[END_REF]. During interaction it is important to the organization of discourse such as beginning and ending of speech, turn taking, or accentuation of utterances [START_REF] Argyle | Gaze and mutual gaze[END_REF][START_REF] Kendon | Does gesture communicate? A Review[END_REF].

In previous studies [START_REF] Raidt | Does a virtual talking face generate proper multimodal cues to draw user's attention towards interest points?[END_REF][START_REF] Raidt | Face-to-face interaction with a conversationnal agent: eye-gaze and deixis[END_REF], we tested the capability of our animated talking head [START_REF] Badin | Three-dimensional linear articulatory modeling of tongue, lips and face based on MRI and video images[END_REF] to direct the attention of a human observer by the means of head and gaze orientation. The work described here presents quantitative measurements of gaze patterns recorded during dyadic face-to-face conversations in relation to the course of the dialog. We show that in live interaction these patterns differ from those obtained when scrutinizing pre-recorded videos. The results are exploited for a first basic version of a gaze generation model of our talking head.

Motivation

The aim of our research is to understand the functions of gaze in social interaction and to develop a coherent gaze control model for our talking heads taking into account the results of detailed multimodal scene analysis. Such a coupling between multimodal scene analysis and multimodal scene synthesis is rather usual for anthropoid robots at least for the planning of movement. But there are very few proposals of that kind in the field of speech communication that comprise speech and gesture generation as well as facial animation. Humancomputer interaction is mainly focused on the dialog component and the exchange of symbolic information between information retrieval (speech recognition, etc.) and rendering (concept-to-speech synthesis, etc.). Signals gathered from a rich scene analysis are nevertheless crucial to provide a convincing sense of presence by linking virtual actions to the real world [START_REF] Bickmore | Social dialogue with embodied conversational agents[END_REF][START_REF] Thórisson | Natural turn-taking needs no manual: computational theory and model from perception to action, in Multimodality in language and speech systems[END_REF]. Since the pioneer work of Argyle and Cook [START_REF] Argyle | Gaze and mutual gaze[END_REF] and Kendon [START_REF] Kendon | Some functions of gaze-direction in social interaction[END_REF], few research has been conducted on gaze during face-toface interaction. Vertegaal et al. [START_REF] Vertegaal | Eye gaze patterns in conversations: There is more to conversational agents than meets the eyes[END_REF] analyzed gaze behaviour of a subject involved in multi-party conversation. Gullberg [START_REF] Gullberg | Visual attention towards gestures in face-to-face interaction vs on screen[END_REF] found differences in gaze patterns when involved in live interactions versus off-line video presentation. She concluded that these differences are partly due to social norms. Both studies only survey the gaze of one of the interacting subjects and thus are not able to take into account the gaze of the interlocutors. Furthermore the measurements are not as fine grained as an analysis distinguishing different parts of the face would afford. Bateson et al [START_REF] Vatikiotis-Bateson | Eye movement of perceivers during audiovisual speech perception[END_REF] however showed that the gaze of listeners alternates between eyes and mouth when scrutinizing off-line videos. The percentage of gaze towards eyes vs. mouth is influenced by the perception task. In return visual attention is also known to influence speech comprehension (see for example [START_REF] Paré | Gaze behavior in audiovisual speech perception: the influence of ocular fixations on the McGurk effect[END_REF], for sensitivity of McGurk effect to visual attention).

Different approaches have been proposed to model and generate gaze patterns. Itti et al. [START_REF] Itti | Realistic avatar eye and head animation using a neurobiological model of visual attention[END_REF] developed a gaze generation model coupled with a visual attention system that detects salient and pertinent points of interest in a natural scene and triggers exogenous saccades and fixations handled by a biological model of eye motion. Note that there is no special treatment or even detection of faces in this system. Bilvi and Pelachaud [START_REF] Bilvi | Communicative and statistical eye gaze predictions[END_REF] implemented a model for gaze generation in dialogue. It takes text tagged with labels of communicative functions as input combined with a statistical model to generate eye movements alternating between direct and averted gaze. Note that no saccadic model is included and that alternation is paced by phone boundaries. Lee et al [START_REF] Lee | Eyes alive[END_REF] uses a purely statistical approach. The statistics are based on analysis of a video recording of one subject during face-toface interaction. Bilvi and Pelachaud and Lee et al both take into account the cognitive activity of the ECA (e.g. speaking vs. listening, etc).

Eye gaze in face-to-face interaction 4.1 Experimental Setup

In order to investigate the functions of eye gaze during close dyadic face-to-face interaction, we developed an experimental platform where two subjects can interact via a crossed camera-screen setup (see Figure 1). The experimental set-up should give interlocutors the impression to be facing each other across a table.

A pinhole camera (1.5 cm 3 ) placed at the center of a computer screen films the subject facing the screen and displays the video image on a second screen. The second screen is symmetrically equipped. During the recording period the video signals are crossed. Prior to the beginning of each recording session, the screens function as inversed mirrors and subjects see their own video image to be able to adjust their position. In order to optimize gaze contact, subjects adjust their rest position in such a way that the middle of their eyebrows coincides with the position of the camera on the screen. A camera located on top of (or below) the screen would generate the impression of seeing the other subject from above (resp. below). This would make direct eye contact impossible [START_REF] Chen | Leveraging the asymmetric sensitivity of eye contact for videoconference[END_REF].

The audio signals are exchanged via microphones and earphones. Video and audio signals as well as gaze directions are recorded during the interaction. For this purpose we use computer screens by Tobii Technology ® with embedded eye trackers. Before the recording a calibration phase writes a synchronization time stamp to the data streams. Gaze patterns can thus be aligned with audiovisual data for off-line analysis. This particular setting (mediated interaction, 2D displays, non intrusive eye tracking) limits the working space but is fully compatible with our target application i.e. a virtual ECA displayed on a screen and able to interact face-to-face with a human interlocutor. We will make our ECA imitate the interaction strategies emerging from the analysis of human dyadic conversations. Comparative evaluation can then be performed.

According to our knowledge this is the first experimental setup that monitors both subjects during such a mediated faceto-face interaction.

Experiment

Scenario: The experiment involves two subjects into a sentence-repeating game. One subject (initiator) reads and utters a sentence that the other subject (respondent) should repeat subsequently in a single attempt. The initiator is advised to face the screen when uttering a sentence. Roles (initiator and respondent) are further exchanged. The subjects are told that their performance (number of correctly repeated sentences) will be evaluated at the end. Semantically Unpredictable Sentences (SUS) [START_REF] Benoît | The SUS test: A method for the assessment of text-to-speech synthesis intelligibility using Semantically Unpredictable Sentences[END_REF] are used to force the respondent to be highly attentive to the audiovisual signal.

With this rather restricted scenario of interaction we try to isolate the main elements of face-to-face interaction and to enhance the aspect of mutual attention. It imposes a clear chaining of turns and roles (reading, speaking, listening and repeating) that avoids complex negotiation of turn taking and eases state dependent gaze analysis. Subjects: We study inter-and intra-subject variability. In each dyad, we have a permanent target interlocutor, i.e. the female researcher that served as the target speaker for our talking head. She interacts with female subjects of the same social status and cultural background (French, European, and researcher) in order to control social relationship. Sessions: The advocated purpose of the experiment is the study of the importance of visual feedback for telecommunication. Each session consists in fact of an on-line interaction using the full experimental setup followed by a faked interaction where the subjects are confronted to a previously recorded stimulus. It is taken from an interaction of our target speaker with the main experimenter and is the same for all subjects. The subjects should not realize that parts of the stimulus are pre-recorded. The noticeable impoverished feedback is justified by an absence of video feedback.

Each subject faces thus three tasks of ten sentences each:

Data processing & labeling

Thanks to an original synchronization procedure we developed, the multimodal data (audio, video and gaze data of both subjects) are time-aligned. Multimodal scores are then computed, automatically labelled and downloaded into the ELAN® system [START_REF] Hellwig | EUDICO Linguistic Annotator (ELAN) Version 2.0.2 manual[END_REF] for subsequent additions and corrections.

The raw gaze data are processed to characterize fixations. Fixations are identified using a dispersion-based algorithm. An affine transform is applied to compensate for head movements determined by a robust feature point tracker. This permits to represent all fixations on a single reference image (see Figure 2). Elliptic regions are then defined by the experimenter on this reference image assigning the fixations to different regions of interest (ROI): left or right eye, mouth, face (other parts than the three preceding ones such as nose) or else (when a fixation hits other parts of the screen). Successive saccades can be labeled with the same ROI. Corrective saccades are very characteristic of attention (for example during reading [START_REF] Vergilino-Perez | Decision and metrics of refixations in reading isolated words[END_REF]). The accumulations of fixations assigned to the ROI may be displaced relative to the targets as they can be identified on the reference image and only overlap partly (see Figure 2). This discrepancy between measured fixation target and obviously intended ROI might be explained by imprecision of eye tracking. It might also be sufficient for the subjects to focalize close to the ROI and exploit peripheral view.

The speech data is aligned with the phonetic transcriptions of SUS sentences and sessions are further segmented into sequences assigned to six different cognitive states (CS): prephonation, speaking, listening, reading, waiting and thinking, also distinguishing role.

The role indicates which subject dominates the interaction. Differences are for example expected to occur during listening. The SUS that the initiator utters are unknown to the respondent whereas the initiator already knows the content of the SUS and might therefore be less attentive when the respondent repeats them.

Some of the states depend on role: waiting is the CS of the respondent while the initiator is reading or the CS of the initiator after having uttered a sentence while waiting until the respondent begins to repeat the sentence. There are also syntactic dependencies between CS: pre-phonation preceding speaking is triggered by pre-phonatory gestures such as lip opening, speaking state triggers listening state for the interlocutor, etc. Some CS appear only in one of the two roles. The CS reading only occurs while a subject is initiator (reading next sentence to utter) and the CS thinking only occurs while a subject is respondent (preparing in mind the sentence to repeat).

We also label blinks. Our automatic detection of blinks is triggered by short gaps of invalid gaze data between fixations lasting between 20 and 240 ms.

After the manual correction of the labels of CS and blinks with ELAN® they are exported into Matlab® for statistical analysis.

Results

Up to now we recorded interactive sessions of our target subject with 4 interlocutor subjects. The results clearly confirm the triangular pattern of fixations scanning the eyes and the mouth previously obtained by Vatikiotis-Bateson, Eigsti et al [START_REF] Vatikiotis-Bateson | Eye movement of perceivers during audiovisual speech perception[END_REF] for perception of prerecorded audiovisual speech (see Figure 2). They also confirm our choice to distinguish cognitive state and role.

Fixations and cognitive states

We define fixation profiles on a number of time intervals as the percentual distribution of fixations among the ROIs within this time span. For statistical analysis we only consider 4 ROIs: left eye, right eye, mouth and face. The ROI 'else' is disregarded since it almost never occurs during the analyzed sessions. We investigate the influence of the two factors role and cognitive state on the mean fixation profiles calculated for our target subject during the four interactions (see Figure 3). This means about 80 measurement samples of our target subject for each CS, of which 40 for each role. We compare the multivariate means of the fixation profiles of the CS (prephonation, speaking, listening, and waiting) that occur in both roles using MANOVA.

MANOVA returns an estimate of the dimension 'd' of the space containing the multivariate group means and p-values to indicate the significance of each dimension. We found that independent of role CS-specific profiles are significantly different from each other (d=3, p=0; 0; 0.03). Separating the data for role this is even more significant for the role initiator: d=3, p=0; 0; 0.0007 but less significant for the role respondent: d=2, p=0; 0; 0.12). All pair wise comparisons of CS are also significant.

To better be able to interpret the mean fixation profile we also computed the probability of occurrence of at least one fixation in a given ROI for each instance of a given cognitive state as displayed in Figure 4. A fixation to a specific ROI lasting the whole duration of every second instance of a specific CS would for example result in the same value of 50% in the fixation profile as a fixation lasting half of the duration of each instance of that CS. Considering a model for gaze control this means however an important difference that can be identified via the probability of occurrence.

To verify the impact of live feedback, we compared the fixation profiles measured during the online interaction (when the interlocutors of our target subject were acting as respondents) with the measurements during the faked interaction (using the pre-recorded stimulus). MANOVA showed for each interlocutor that independent of CS mean profiles of online and faked interaction are significantly different. When comparing live versus faked interaction separately by cognitive state, we found inter-subject differences. While one subject shows no difference in the direct comparison of CS at all, another subject has different gaze patterns for both listening and speaking (p=0.01; p=0.02) while the two others have only one significantly different CS, respectively speaking (p=0.02) and waiting (p=0.03) (see Figure 5).

Comments

First of all, these results confirm the eyes and mouth as dominant target zones [START_REF] Vatikiotis-Bateson | Eye movement of perceivers during audiovisual speech perception[END_REF]. We have shown that role has a significant impact on fixation profiles. When listening, respondents should for instance gaze towards the mouth to be able to decode articulation, while the initiators do not need to benefit from audiovisual speech perception since to them the content of the message is already known.

The segmentation of the discourse into cognitive states explains a large part of the variability of the gaze behavior of our reference subject. The high significance of these predictors justifies a posteriori our choice of cognitive states.

The comparison of the fixation profiles of online versus faked interaction indicates that faked interaction has an impact on gaze behavior even if gaze patterns of the interlocutor are natural (since videos are extracted from the recording of a real online interaction). This is largely subconscious: note that only one of the subjects actually realized that the second stimulus was prerecorded. We interpret this as an argument that a generic gaze model should not only use rich and pertinent internal states but also profit from scene analysis.

Figure 4 shows interesting variation between the probabilities of occurrence of fixations to the different ROIs. Given a role and a cognitive state, we can be almost certain that our reference subject will fixate certain regions of the face of her interlocutor but that she will never gaze at others. For instance she never fixates the mouth when speaking as initiator but always the left and right eye. Note also the strong tendency to look especially at the right eye of her interlocutor when preparing to speak as respondent and the important rise of probability to fixate the mouth when respondent.

The statistics of the fixation profile given above just consider the time spent on a given ROI on a frame basis. Fixations span of course across boundaries between cognitive states. In order to build an effective statistical model of saccade generation (cf. § 5.4), we also characterize the duration of fixations over region of interest and role. Statistical analysis with an ANOVA gives no reason to distinguish for role. The influence of ROI however is highly significant (df=9, F=18.84, p=0). Post hoc analysis shows that there is no difference of fixation duration between the ROI right eye and left eye. In comparison to them fixations to the face are significantly shorter and fixations to the mouth significantly longer.

Blinks and cognitive states

Blinking is a very important biological movement. Blinking helps the eyelids to spread a tear film over the eye. But blinks are also generated to protect the eye (blink reflex) and have been shown to co-occur with triggering of large gaze shifts and head movements (gaze-evoked blinks described by [START_REF] Evinger | Not looking while leaping: the linkage of blinking and saccadic gaze shifts[END_REF]). Most ECA generate blinks with a simple random event generator. Our data evidence however that blinking rate is highly dependent on cognitive state (see also [START_REF] Peters | Attention-driven eye gaze and blinking for virtual humans[END_REF]).

Assuming that blinks occur randomly or at a regular frequency their amount should be proportional to time. To test this hypothesis we used Chi-square goodness-of-fit test to compare the observed amount of blinks to the amount expected. Therefore we considered the added-up duration of the cognitive states and the mean frequency over the entire duration of the interaction. The cognitive states taken into account are pre-phonation, speaking, listening, and waiting. For all subjects this hypothesis was rejected at p < 0.01.

A detailed analysis of the influence of CS on blink rate showed that 'speaking' accelerates blink rate, whereas 'reading' and 'listening' slow it down or inhibit blinks. Especial in the role of respondent 'listening' seems to have a strong tendency to inhibit blinks. Strikingly often blinks occur at the change-over from reading to speaking (pre-phonation). This might be explained by the fact that the subjects wet the eyes in perspective of a long period of mutual attention. An alternative explanation is the linkage of blinking and major saccadic gaze shifts proposed by Evinger et al [START_REF] Evinger | Not looking while leaping: the linkage of blinking and saccadic gaze shifts[END_REF].

Modeling

We built a first gaze control model for our talking head (see Figure 8) by training and chaining role-and CS-specific Hidden Markov Models (HMM). Given a succession of CS with associated durations it computes parameters describing the fixations of the ECA towards the various ROI on the face of its interlocutor. HMM states equal to the different ROI and observations equal to the durations of fixations.

The transition probabilities of the HMM are computed from the transition matrix between the different ROI within a given CS and role as observed during the experiment. Figure 6 shows the transition matrices observed during the CS 'speaking' and 'listening' for the two roles instructor and respondent. The colons and lines represent respectively the current and subsequent possible fixation targets. The grey level is proportional to the transition probability between pairs: high probabilities are coded by dark gray. The amount 'n' of accumulated fixations to a target, denoted on bottom of each colon, indicates the number of items on which this estimation is based and thus gives the reliability of the probabilities displayed in the matrix colons. Fixations to the face for instance are very rare and thus the transition probabilities from face to other ROI not very reliable since they are calculated from only few observations. An initial state in each HMM has been added to cope with the particular distribution of the first fixation. The observation probabilities determine the duration of the fixation emitted by the HMM at each transition. The probability density functions of these durations are computed from fixations gathered from the interactions. Fixations to the mouth are for instance longer than fixations to the eyes.

Based on these parameters we use the same generation process as proposed by Lee [START_REF] Lee | Eyes alive[END_REF] to control the gaze of the clone of our target speaker (cf. Figure 8). Until now we have not yet evaluated the model experimentally but the distributions of fixations according to ROI and cognitive state obtained with this gaze control model are -as expected -very similar to the distributions observed during live face-to-face interactions (see Figure 7). [START_REF] Bailly | Degrees of freedom of facial movements in face-to-face conversational speech[END_REF]. The eyes and eyelids movements are controlled by 5 degrees-of-freedom that captures the correlations between gaze and eyelids deformations [START_REF] Bailly | Embodied conversational agents : computing and rendering realistic gaze patterns[END_REF].

Conclusions and Perspectives

Gaze and blinking are essential visible cues for sense of presence of ECA (Embodied Conversational Agent). They are important cues for signaling the ECA's awareness of its environment, the cognitive and emotional state of its interlocutors as well as its own states. We have shown that for the generation of realistic gaze direction the control model should at least be aware of its own cognitive states and its role in the interaction. We have settled a basis for a state-aware eye-gaze generator for controlling the eye movements of a virtual ECA based on these findings.

In order to develop an improved gaze generator we should isolate the significant events detected in the multimodal scene that impact the closed-loop control of gaze. We should notably investigate the influence of eye saccades produced by the interlocutor as potential extrinsic driving events of gaze. We expect for example to find patterns of gaze avoidance after periods of eye contact.

Furthermore we should implement other cognitive and emotional states as well as other functions of gaze (deictic or iconic gestures). This supposes a more elaborated and richer multimodal scene analysis as well as a deeper comprehension of the intentions of the interlocutors.

Figure 1 :

 1 Figure 1: Experimental setup: In contrast to video phones this setup enables real size rendering of video image and eye contact, as the camera is placed on the screen. With additional audio transmission it is therefore very close to a scenario where interlocutors face each other across a table.

  (1) repeating SUS given on-line by the target speaker; (2) uttering SUS and checking the correct repetition by the target speaker; (3) repeating SUS given off-line by the target speaker.

Figure 2 :

 2 Figure 2: Fixations of a whole session projected onto a reference image. With the ellipses the experimenter defines regions of interest that the fixations are assigned to. We distinguish between mouth, eyes, face and else. The size of the asterisks is proportional to the duration of fixations. Left: fixations of our target speaker on one subject. Right: the interlocutor's data.

Figure 3 :

 3 Figure 3: Fixation profiles of all interactions of our target speaker over role, ROI (face, right eye, left eye, mouth, else) and cognitive state CS (speaking, listening, waiting, reading, pre-phonation, thinking, else). The bars represent the means of the percentage of fixation time on ROI during an instance of a cognitive state. The diagram is completed by bars (ROI named "n") representing the means of percentage of time when no fixations are detected.

Figure 4 :

 4 Figure 4: Probabilities that fixations to ROI appear at least once during a cognitive state, calculated for all interactions of our target speaker. Bars of the same gray scale value constitute the fixation profiles of the different CS.

Figure 5 :

 5 Figure 5: Example of fixation profiles of the same subject acting as respondent during on-line interaction (left) vs. faked interaction (right).

Figure 6 :

 6 Figure 6: Distribution of probability over ROI of first fixation of a cognitive state and transition matrix inside the cognitive state for speaking and listening in both roles and over entire interaction for all recording sessions of our target speaker.

Figure 7 :

 7 Figure 7: Comparing original gaze patterns (top) with patterns generated using the statistical model driven by the sequence of cognitive states (bottom). Left: fixations labeled with ROI; right: fixations labeled with cognitive state.

Figure 8 :

 8 Figure8: Our virtual talking face is driven by 12 facial degrees-of-freedom[START_REF] Bailly | Degrees of freedom of facial movements in face-to-face conversational speech[END_REF]. The eyes and eyelids movements are controlled by 5 degrees-of-freedom that captures the correlations between gaze and eyelids deformations[START_REF] Bailly | Embodied conversational agents : computing and rendering realistic gaze patterns[END_REF].

Acknowledgments

This work is supported by the project Presence of the cluster ISLE financed by the Rhône-Alpes region.