Gérard Bailly
email: gerard.bailly@gipsa-lab.grenoble-inp.fr

Stephan Raidt

Frédéric Elisei

Gaze, conversational agents and face-to-face communication

Keywords: Conversational agents, face-to-face communication, gaze, eye fixations, blinks, deixis, mutual attention, games

HAL is

INTRODUCTION

Building Embodied Conversational Agents (ECA) able to engage in convincing face-to-face conversation with a human partner is certainly one of the most challenging Turing tests one can imagine [START_REF] Cassell | Embodied Conversational Agents[END_REF]. The challenge is far more complex than the experimental conditions of previous Loebner Prize competitions 2 where dialog is conducted via textual information: the ECA should not only convince the human partner that the linguistic and paralinguistic contents of the answers that are generated in response to human inquiries have been produced by a human intelligence, but also generate the proper multimodal signals that should fool human perception. We are however very close to being able to conduct such experiments (see for example the joint Interspeech/Loebner Speech & Intelligence Competition 2009). Automatic learning techniques that model perception/action loops at various levels of human-human interaction are surely key technologies for building convincing conversational agents. George, the talkative bot that won the Loebner Prize 2005, learned its conversation skills from the interactions it had with visitors to the Jabberwacky website, and through chats with its creator, Mr Carpenter. Similarly the first Turing test involving a non interactive virtual speaker [START_REF] Geiger | Perceptual evaluation of video-realistic speech[END_REF] has demonstrated that image-based facial animation techniques are able to generate and render convincing face and head movements. Combining a pertinent dialog management capability with convincing videorealistic animation is still not sufficient to produce a real sense of presence [START_REF] Riva | Being there: concepts, effects and measurements of user presence in synthetic environments[END_REF]. The sense of "being there" requires basic components of situated face-to-face communication such as mixed initiative, back channeling, turn taking management, etc. The interaction requires a detailed scene analysis and a control loop that knows about the rules of social interaction: the analysis and comprehension of an embodied interaction is deeply grounded in our senses and actuators and we have strong expectations about how action-perception loops are encoded by multimodal signals.

We describe here part of our efforts for designing virtual ECAs that are sensitive to the environment (virtual and real) in which they interact with human partners. We focus on the control of eye gaze and blinks. We describe the multiple scientific and technological challenges we face, the solutions that have been proposed in the literature and the ones we have implemented and tested.

EYE GAZE AND HUMAN-COMPUTER INTERACTION

Face-to-face interaction, attention and deixis

Eye gaze is an essential component of face-to-face interaction. Eyes constitute a very special stimulus in a visual scene. Gaze and eye-contact are important cues for the development of social activity and speech acquisition [START_REF] Carpenter | Joint attention, cultural learning and language acquisition: Implications for children with autism. Communicative and language intervention series. Autism spectrum disorders: A transactional perspective[END_REF]. In conversation, gaze is involved in the regulation of turn taking, accentuation and organization of discourse [START_REF] Kendon | Some functions of gaze-direction in social interaction[END_REF][START_REF] References Argyle | Gaze and mutual gaze[END_REF]. We are also very sensitive to the gaze of others when directed towards objects of interest within or even outside our field of view, notably when interpreting facial expressions [START_REF] Pourtois | Dissociable roles of the human somatosensory and superior temporal cortices for processing social face signals[END_REF]). In the Posner cueing paradigm [START_REF] Posner | Orienting of attention[END_REF][START_REF] Posner | The attention system of the human brain[END_REF], observers' performance in detecting a target is typically better in trials in which the target is presented at the location indicated by a former visual cue than in trials in which the target appears at an uncued location [START_REF] Driver | Gaze perception triggers reflexive visuospatial orienting[END_REF]. [START_REF] Langton | Reflexive visual orienting in response to the social attention of others[END_REF][START_REF] Breazeal | Sociable machines: expressive social exchange between humans and robots[END_REF] have also shown that observers react more quickly when the cue is an oriented face than when it is an arrow or when the target itself changes.

Eye gaze is thus capable of attracting visual attention whereas visual features associated with the objects themselves such as highlighting or blinking attract less attention, unless they convey important information for the recognition of a scene. Perceptual salience is thus not the only determinant of interest. The cognitive demand of a task has a striking impact on the human audiovisual analysis of scenes and their interpretation. Yarbus (1967) showed notably that eye gaze patterns are influenced by the instructions given to the observer during the examination of pictures. Similarly Vatikiotis-Bateson et al. (1998) showed that perceivers' eye gaze patterns during audiovisual speech perception are influenced both by environmental conditions (audio signal-to-noise ratio) and by the recognition task (identification of phonetic segments vs. the sentence's modality). Buchan et al. [START_REF] Buchan | Spatial statistics of gaze Fixations during dynamic face processing[END_REF]) replicated this experiment comparing judgments on speech and emotion.

Interacting with humanoids and avatars

The faculty of interpreting others' eye gaze patterns is thus crucial for humans and machines interacting with humans. For the "theory of mind" (TOM) as described by Baron-Cohen [START_REF] Premack | Does the chimpanzee have a theory of mind?[END_REF], the perception of gaze direction is an important element of the set of abilities that allow an individual, based on the observation of the actions and behavior, to infer the hidden mental states of another. Several TOM characterizations have been proposed (Baron-Cohen, [START_REF] Baron-Cohen | Does the autistic child have a "theory of mind[END_REF][START_REF] Leslie | ToMM, ToBY, and Agency: Core architecture and domain specificity. Mapping the Mind: Domain specificity in cognition and culture[END_REF]. Baron-Cohen proposes an Eye Direction Detector (EDD) and an Intentionality Detector (ID) as basic components of a Shared Attention Mechanism (SAM) that is essential to bootstrapping the TOM. The actual implementation of these modules requires the coordination of a large number of perceptual, sensorimotor, attentional, and cognitive processes. Scassellati (2001) notably applied the "theory of mind" concept to humanoid robots developing an "embodied theory of mind" to link high-level cognitive skills to the low-level motor and perceptual abilities of such a robot. The low-level motor abilities included for example coordinated eye, head and arm movements for pointing. The low-level perceptual abilities consisted in essentially detection of salient textures and motion for monitoring pointing and visual attention. This work still inspires much research on humanoid robots where complex behaviors emerge from interaction with the environment and users despite the simple tasks achieved by the robot such as expressing empathy for Kismet [START_REF] Breazeal | Sociable machines: expressive social exchange between humans and robots[END_REF] or following turn-taking for Robita [START_REF] Matsusaka | Conversation robot participating in group conversation[END_REF][START_REF] Fujie | Back-channel feedback generation using linguistic and nonlinguistic information and its application to spoken dialogue system[END_REF]

ECA

Figure 1: ECA-Human interaction scheme. The environment hosts human partner(s). The perceptionaction loops are regulated by three main layers: reactive, process and content (Thórisson 2002). The parameters of these modules are set according to the computed ECA profile that may vary according to social needs as put forward in the Communication Accommodation Theory (CAT, [START_REF] Giles | Language and Social Psychology[END_REF]. Note that a mutual belief space is built by converging representations of estimated and actual profiles and cognitive states of the interlocutors. The profile can for example instruct the content module to approach or recede from the phonological space of the interlocutor.

INTERACTING WITH AN ECA

Most ECAs derive their "theory of mind" from high-level linguistic information gathered during the dialog. These virtual agents are generally not equipped with the means to derive meaning from the implicitly and explicitly communicational gestures of a human interlocutor and are also not generally equipped to generate such gestures for communication purposes. For example, in the study by Os et al [START_REF] Os | Conversational agent or direct manipulation in human-system interaction[END_REF] an ECA is used as a virtual advisor. It gives advices, instructions and some back channel information but is not equipped with sophisticated scene analysis and computes his gesturing only on the basis of dialogic information. The failure of the ECA to gain users' satisfaction was attributed to the poor performance of individual components of the system (speech recognition, audiovisual synthesis, etc). Another possible way of improving users' commitment is via grounding. Eye gaze of ECAs can be generated without grounding these gestures in the scene by simply reproducing statistical properties of saccadic eye movements [START_REF] Lee | Eyes alive[END_REF][START_REF] Peters | A model of attention and interest using gaze behavior[END_REF]. Visual attention models have however been developed aiming at reproducing eye movement patterns of subjects viewing natural scenes. Notably, [START_REF] Itti | Realistic avatar eye and head animation using a neurobiological model of visual attention[END_REF] propose a model that couples physical scene analysis and control of eye gaze of a virtual ECA while preserving cognitive permeability of the analysis strategy thanks to the use of a so-called pertinence map. Our visual attention model [START_REF] Picot | Scrutinizing natural scenes: controlling the gaze of an embodied conversational agent[END_REF]) incorporates the identification of potential objects of interest around each fixation point: in particular a face recognition module has been added to elicit multimodal behaviour adapted to face-to-face communication.

In situations where context-aware face-to-face interaction is possible, an ECA should be able to give direct and indirect signs that it actually knows about where the interaction is taking place, who is its interlocutor and what service it may provide to the user considering the given environment. By signalling its ability to interpret human behavior, the system encourages the interlocutor to exhibit the appropriate natural activity. Such a complex face-to-face interaction requires intensive collaboration between an elaborate scene analysis and the specification of the task to be performed in order to generate appropriate and convincing actions of the ECA (see Figure 1).

Our perspective is to develop an embodied TOM for an ECA that will link high-level cognitive skills to the low-level motor and perceptual abilities and to demonstrate that such a TOM will provide the information system with enhanced user satisfaction and efficient interaction. The motor abilities are principally extended towards speech communication i.e. adapting content and speech style to pragmatic needs (e.g. confidentiality), speaker (notably age and possible communication handicaps) and environmental conditions (e.g. noise). If the use of a virtual talking head instead of a humanoid robot limits physical actions, it extends the domain of interaction to the virtual world. The user and the ECA can thus employ both physical and virtual objects -such as icons surrounding the virtual talking headin their interaction.

The two experiments described here analyze gaze behaviour during live face-to-face dialogs between naive subjects and one reference interlocutor. Our research strategy is in fact to analyse and model the adaptive behaviour of a few reference subjects when interacting with various subjects and in various conditions: ECA control and embodiment are data-driven and aim at reproducing behaviour of their human alter egos. In experiment I, the reference interlocutor is an ECA: we evaluate here its ability to elicit mutual visual attention with its human interlocutors. This experiment focuses on deictic gestures attracting attention to common objects in the mutual shared visual field. Despite the rough control strategy of the gaze patterns, we show that subjects benefit significantly from these deictic gestures and -as for the case of priming by human gaze -can not avoid using them. Experiment II focuses on eye contact and mutual attention. The reference interlocutor here is the human speaker who donated his voice and appearance to the ECA so that control strategies can be fed back into the ECA, notably to better handle the interplay between deixis and mutual attention in future replications of experiment I. Both experiments build on previous experimental work performed mainly with non interactive videos. They should be considered as a further step towards the automatic measurement and comprehensive modelling of multimodal behaviour of human and virtual agents during live face-to-face interactions.

EXPERIMENT I. ECA THAT ATTRACTS ATTENTION

The first experiment [START_REF] Bailly | Multimodal face-to-face interaction with a talking face: mutual attention and deixis[END_REF][START_REF] Raidt | Does a virtual talking face generate proper multimodal cues to draw user's attention towards interest points?[END_REF]) involves a face-to-face interaction between an embodied conversational agent (ECA) and several other naïve human interlocutors. The ECA is embodied as a virtual talking head obtained by cloning one target human speaker named HL in the following. This female speaker is also the target speaker of Experiment II. We test here the impact of assistance given by the ECA on task-oriented interactions. A card game was designed during which the multimodal deictic gestures of the ECA aims at attracting attention of subjects towards regions of interest that are either appropriate or inappropriate for the given task. Despite the fact that subjects were explicitly instructed not to take ECA behaviors into account (as in [START_REF] Langton | Do the eyes have it ? Cues to the direction of social attention[END_REF], we show that most of the subjects are drastically influenced by ECA hints and that the benefit in case of appropriate gestures -compared with inappropriate or no ECA gestures -is quite large: 10% of the task duration in case of deictic head and gaze movements, up to 20% when these deictic gestures are accompanied with a spoken instruction.

Figure 2 : Face-to-face interaction platform with an ECA (left) that provides deictic hints (see Figure 3) to a subject whose gaze is monitored. The hints help -or disturb -the subject in playing a virtual card game consisting of playing as fast as possible a card (white card here at the bottom of the screen) on one of eight possible positions (coloured boxes) according to different conditions.

Experimental Setup

Hardware

Sensors. The core element for the experiments described here is a Tobii 1750 eye-tracker3 realized as a standard-looking flat screen that discretely embeds infrared lights and a camera. It allows us to detect, at up to 60Hz4 , the eye gaze of the user whose head can move and rotate freely in a fairly unrestricted 3D volume having the shape of a 40cm edge cube centered at 50cm away from the screen. To ensure the required accuracy of tracking each user must follow a short calibration procedure. During interaction with the system, the user sits in front of the eye-tracker where our 3D talking head faces him, as shown in Figure 2. Hardware and software allow the user to interact with the system using speech, eye gaze, mouse and keyboard. Additional data inputs (video camera, speech recognition, etc.) are available for other experimental setups.

Actuators. The visual representation of our ECA is implemented as the cloned 3D appearance and articulation gestures of a real human [START_REF] Revéret | MOTHER: a new generation of talking heads providing a flexible articulatory control for video-realistic speech animation[END_REF][START_REF] Bailly | Audiovisual speech synthesis[END_REF], (see Figure 2). The eye gaze can be controlled independently to look at 2D objects on the screen or spots in the real world outside the screen. Ocular vergence is controlled and provides a crucial cue for inferring spatial cognition. The virtual neck is also articulated and head movements accompany the eye-gaze movements. Standard graphic hardware with 3D acceleration allows real-time rendering of the talking head on the screen. The ECA also uses speech: audiovisual utterances can either be synthesized from text input or mimic pre-recorded human stimuli. We expect that the proper control of these capabilities will enable the ECA to maintain mutual attention -by appropriate eye saccades towards the user or his/her points of interest -and actively draw the user's attention.

Software: scripting multimedia scenarios

The interaction during the virtual card game is described by an event-based language. This language allows the simple description and modification of multimedia scenarios. Compiled scenarios allow accurate recording of the multimodal input/output events and their timing.

In our event-based language a finite state machine (FSM) describes each scenario as a series of states with pre-conditions and post-actions. Each input device emits events according to user action and an internal model of the interaction space. Triggerable areas on the screen, such as selectable icons or areas of the talking head are defined and surveyed by the eye tracker. Each time the user looks at such a zone, the system posts new events, such as "entering zone" and "leaving zone" and may emit additional "zone fixation duration" events. The FSM is called each time an event is generated or updated.

As the user progresses in the scenario, the FSM specifies the events that can trigger entry into each state. Pre-conditions consist of conjunctions or successions of expected multimodal events as for instance recognized keywords, mouse clicks or displacements, eye movements or gaze directed to active objects. Each event is time-stamped. Pre-conditions can include tests on intervals between time-stamps of events. This allows, for example, association of speech items in terms of words that are identified as a sub product of speech recognition with a certain focus of attention. Post-actions typically consist of the generation of multimodal events. Time-stamps of these events can be used to delay their actual instantiation in the future (watchdogs for example). Post-actions can also generate phantom events, to simulate multimodal input, to share information or to trigger pre-conditions for following states of the FSM.

The interaction scenario

Design of the card game

To follow up on the findings of Langton and Driver about the special ability of human faces and eyes to direct attention, we designed an interaction scenario where an ECA should direct the user's attention in a complex virtual scene. Our aim was to investigate the effect of deictic gestures on the user's performance during a search and retrieval task. We chose an on-screen pair matching game, where the user is asked to locate the correct target position of a played card.

The card game consists in pairing cards. Eight target cards placed on the sides of the screen are turned over (with visible face up) once the played card initially positioned at the lower middle of the screen is selected with a mouse click (see Figure 3). The target cards are numbered from 1 to 8 and the play card has a random number between 1 and 8. The play card has then to be laid over one of the target cards with the same digit. To avoid memory effects, the target cards are shuffled before each turn. The target position is thus chosen randomly but uniformly distributed amongst the eight possibilities provided that the number of cycles is a multiple of eight. This should compensate for possible influences of the different positions on the users' performance. The background color of the cards is fixed for each position.

Experimental conditions

In absence of assistance from the ECA, users have to explore most of the possible target cards to find the good match. A condition with no ECA is thus used as reference. When the ECA is displayed, we instruct the subjects that the deictic behavior of the ECA is unpredictable and that they should not pay attention to it as in [START_REF] Langton | Do the eyes have it ? Cues to the direction of social attention[END_REF].

We tested different experimental conditions corresponding to different levels of assistance and help by the ECA that is displayed in the center of the screen when present. Screenshots of the game interface are given in Figure 3. The ECA can utter spoken commands and indicate directions with a rapid eye saccade combined with a slower head turn.

General information explaining the task is given as text on the screen at the beginning of the experiment. The user is instructed to play the chosen card on the target position as fast as possible but no strategy is suggested.

Each experimental condition has 24 measurement cycles. It is preceded by three training cycles for the subjects to become accustomed with the task and experimental conditions. The characteristics of the upcoming condition are described as text informing the user about the expected gaze behavior of the clone.

Data acquisition

For the evaluation of the experiments, the time-to-complete (TC) and the gaze behavior have been monitored. TC was measured as the time span between the first mouse click on the played card and the click on the correct target position. As the card game was displayed on the monitor with embedded eyetracking, the visual focus of the user on the screen was recorded. We thus computed which objects on the screen were looked at and how much time users spent on them. Possible objects were the eight cards on the sides and the face of the ECA. Eye gaze towards the played card was not monitored.

At the end of the experiment, which lasted about 15 minutes, participants were asked to answer a questionnaire. They notably had to indicate which condition they considered as the easiest to perform and which condition seemed the fastest.

Data processing

Time-to-complete. Before evaluating TC, extreme outliers (distances from median > 5 times inter quartile range) were detected and deleted from the remaining valid data. Such outliers may be due to the fact that a two screen setup was chosen for the experiment. The mouse pointer may leave the screen on one side to appear on the other screen. This happened occasionally when users overshot the target card which made them loose time while moving the mouse pointer back into view. The distribution of TC is log-normal. We thus analyse the logarithms of TC within each experiment.

Number of cards inspected. The number of possible target positions looked at while searching for the correct target position was determined to analyse the search strategy of the subjects. In order to verify the reliability of the data collected by the eye tracker, the percentage of time reliably monitored by the eye tracker during the duration of the search of a respective cycle was calculated. A threshold of 95% was chosen to consider blinks and classify measured data as valid. If this requirement was not fulfilled, the data of this cycle was just discarded. If less than 60% of all cycles of a condition were not valid, the data of this condition were entirely discarded. We characterized the log-normal distribution of the number of cards inspected during a game. ANOVA and T-test analysis was then performed on valid data and significant differences between pairs of distributions are indicated in figures with stars.

Run A: does our clone have cues to direct social attention?

This first run aimed at evaluating the capacity of our ECA to attract users' attention using facial cues and quantifying the impact of good and bad hints on the users' performance. This work builds on the psychophysical experiments on visual priming of Langton et al. [START_REF] Langton | Reflexive visual orienting in response to the social attention of others[END_REF][START_REF] Langton | Do the eyes have it ? Cues to the direction of social attention[END_REF] using photographs of faces looking at different parts of the screen. In Langton's experiments, the impact on reaction time was more effective for the vertical dimension and remained small (10ms). We show here that these findings are confirmed by more realistic conditions and that much larger benefits can be expected.

Conditions

The first series of experiments consisted of four different conditions, screenshots of which are displayed in Figure 3. For condition 1, no ECA was displayed. For condition 2, the ECA was visible and provided bad hints: it indicated randomly one of the non-matching positions with a facial gesture as soon as the user selected the played card. In condition 3, it indicated the correct target position. In condition 4, cards remained upside down and the correct visual cues provided by the ECA were the only ones to find the correct target position without a try-and-error strategy.

In all conditions where the ECA is displayed it encouraged the user with randomly chosen utterances alternating between motivation and congratulation after each turn. The utterances were generated offline to avoid computation delays.

We had strong predictions about the data to be collected. Corresponding to the design of the experiment we expected a negative influence on subjects' performance when the clone provided misleading cues and a positive influence when it provided good hints. The condition where no clone was displayed served as a reference. From the fourth condition, we expected to measure the precision with which the gaze direction of the ECA could be perceived.

Ten users (six male and four female) participated in the first series of experiments. Participants ranged from 23 to 33 years of age and most were students. All regularly used a computer mouse and none reported vision problems. The dominant eye was the right eye for all but one subject. Each user had to play the game with the four experimental conditions as described above. Order of conditions was counterbalanced among subjects. An ANOVA analysis distinguishes between 3 groups of conditions with highly significant differences for number of cards inspected (p<0.05).

Results

Errors. Most errors occur during condition 4 where users could only rely on the gestures of the ECA. In total there are 34 cases in which subjects clicked on a wrong card before finding the correct target position (15% error), although one subject accomplished the task without any errors. This indicates that users have difficulties to precisely interpret the gaze direction of the ECA. Nevertheless, all of these errors occurred between neighbouring cards. Since the average number of inspected cards is close to 1 (see Figure 4), we consider the assistance given by the facial gestures as sufficient as long as the user has additional information to localize the target position as it is the case in the other conditions. Time-to-Complete. There is no significant effect for conditions on mean TC for this run (F(3,36)=1; p>0.39). Conditions 1 and 2 lead in fact to similar results: giving no cues has almost the same effects as giving bad cues (see Figure 4): subjects just ignored consciously the behaviour of the ECA. Number of cards inspected. Conditions have a strong effect on this parameter (F(3,36)=36.96; p<0.001).

Pair-wise T-tests reveal a clear advantage for condition 3 over conditions 1 and 2. On average users indeed inspected 0.5 fewer cards with a correct gaze than with a wrong or absent deictic gaze (see Figure 4). We interpret this as a clear decrease of cognitive load since less cognitive resources are used for matching cards.

Questionnaire. 4 of the 10 subjects think they were faster with the helpful assistance of the ECA and preferred this condition to play the card game.

Run B : impact of concomitant speech on deixis?

This experiment aims at evaluating the benefit of multimodal deixis in drawing user's attention using facial cues together with a spoken instruction.

Conditions

This second series of experiments consists likewise of four different conditions. As a major difference with Run A, the head and gaze movements of the clone are accompanied by the uttering of the demonstrative adverb "là!" (engl.: "there!"). The signal duration is 150ms. Condition 1 with no clone was replicated for reference. In conditions 2 (wrong cues) and 3 (good cues) speech onset is initiated 150ms after the onset of the deictic gestures: this delay corresponds to the average duration of the eye saccade towards the target position implemented in our ECA. All other rewarding utterances are now omitted. Condition 4 of experiment I is replaced by a condition with correct hints, where an additional delay of 300 ms was introduced between the gestural and the following acoustic deictic gestures in order to comply with data on speech and gesture coordination [START_REF] Castiello | Temporal dissociation of motor responses and subjective awareness[END_REF]. We expect this natural coordination to enhance the ability of the ECA to attract user attention.

Fourteen users (ten male and four female) participated in this experiment. They range from 21 to 48 years of age and most are students. All regularly use a computer mouse and none reported vision problems. The dominant eye is the right eye for 8 subjects and the left eye for the other 6 subjects. As in Run A, order of presentations is counterbalanced.

Results

The average number of cards was greatly reduced for conditions 1, 2 and 3 in comparison with Run A while the average TC for the reference condition 1 was increased by 230 ms: a turn lasts on average 1.72 s for Run A vs. 1.95 s for Run B. We hypothesize that the spoken instruction forced subjects to take a first decision rapidly while the perception of the verbal prompt increased the response time. Time-to-Complete. Conditions have now a strong effect on this parameter (F(3,52)=3.76; p<0.02). The difference between TC for conditions 1 and 2 vs. 3 and 4 (see Figure 5) is now significant (p<0.007).

The detailed analysis of TC reveals a clear advantage for 7 of 14 subjects for condition 3 (with correct cues) vs. condition 2 (with misleading cues), and for 8 of 14 subjects compared to condition 1 (without the ECA). These users now gain a substantial amount of almost 400 milliseconds (∼20% of the mean duration of a game) at each turn when the target position was correctly cued.

Number of cards inspected.

Conditions have a clear effect on this parameter (F(3,52)=17.8; p<0.001).

Note that all subjects inspect on average 2 cards. Pair-wise T-test analysis reveals however a clear advantage (p<0.002) for condition 3 and 4 (correct cues) over conditions 1 and 2 (none or bad cues). On average users indeed inspect 0.7 fewer cards with a correct gaze than with a wrong or absent deictic gaze (see Figure 5). No clear tendency can be reported when considering influence of delay on performance.

Questionnaire. 11 of the 14 subjects estimated that they were quickest when correct hints were given by the ECA. Most of the subjects declared that they glance a lot at the ECA giving correct hints and discard gestures in condition 2 but that these cues have poor influence on their speed. The movements of the ECA were judged realistic.

Discussion

When considering the number of cards inspected and the number of wrong selections in condition 4 of Run A, the current control and rendering of deictic gestures of the ECA are sufficient for subjects to localize targets, as long as there is information available at the target position to make the final decision. Without such additional information the gestures of the ECA are sometimes not precise enough to discriminate between close neighboring objects. Apart from the limitations of 3D rendering on a 2D screen, this may be due to the synchronization between gaze and head orientation that are not yet derived from empirical data. An additional limitation is the poor rendering of the facial deformations around the eyes of the ECA when eye gaze deviates from head direction, e.g. eyelids lower when gaze direction is down. We have developed an eyelid model [START_REF] Elisei | Towards eyegaze-aware analysis and synthesis of audiovisual speech. Auditory-visual Speech Processing[END_REF]) that may resolve this problem. Further runs of this experiment will be used to evaluate the perceptual pertinence of these objective enhancements of the ECA control and embodiment.

Several subjects complained of being disturbed by rewarding utterances in Run A. Therefore these utterances do not contribute to maintain attention and increase naturalness of the interaction. A more appropriate feedback should be short and clear according to the instruction given to subjects to react quickly. The use of speech to complement gaze and head deixis in Run B demonstrates that the pertinence of the ECA behavior has a significant impact on subjects' performance.

The constraints on speech-gaze synchronization seem quite lax. Kaur et al. [START_REF] Kaur | Where is "it"? Event synchronization in gaze-speech input systems[END_REF] have shown that "the fixation that best identifies the object to be moved occurs, on average, 630 ms before speech onset with a range of 150 to 1200 ms for individual subjects". Note however that these large time lags concern fixations triggering natural planning of deictic gestures… not the final intended communicative gestures. Multimodal perceptual binding is known to be flexible and binding window is known to narrow with age [START_REF] Lewkowicz | Perception of auditory-visual temporal synchrony in human infants[END_REF]. Apparent performance -such as measured by reaction times -may be nevertheless quite insensitive to multimodal asynchrony while regions that mediates this binding may be more or less activated depending on time shifts [START_REF] Miller | Perceptual fusion and stimulus coincidence in the cross-modal integration of speech[END_REF].

Additional experiments should be conducted to have a better insight on the impact of multimodal synchrony on cognitive load.

The results of Run B show that the pairing performance could still be improved using multimodal deixis: imperative deixis using hand-finger pointing (Rochet-Capellan, Laboissière et al. 2008) is expected to further augment this benefit. A more important finding is the reduced number of cards inspected. The majority of participants managed to complete the task looking at significantly less cards when the ECA was providing helpful assistance. This means that even if they do not improve their speed, the search process is more efficient and probably more relaxed. We conclude that ECA assistance diminishes the cognitive load of the subjects. The answers to the questionnaire confirm this finding as the positive ratings for naturalness of the ECA and the preference for the condition where it is providing correct hints are revealed more clearly for Run B compared with Run A.

The experimental scenario presented here could likely be further improved by displaying more objects on the screen and using smaller digits. This should enhance the benefit of ECA assistance since this would require a closer examination of the objects and increase the number of objects to check in order to find the correct one without the assistance of the ECA. However, we consider the results with the current implementation as sufficient confirmation of our assumptions and encouraging motivation to study further possibilities to enhance the capabilities of the ECA.

EXPERIMENT II. MEDIATED FACE-TO-FACE COMMUNICATION

In Experiment I, subjects triggered saccades towards the ECA mainly to register gaze direction. Spoken instructions were very short and did not provide complementary information to gaze. During more complex interactions involving -or not -common objects of interest, interlocutors spend most of the time looking at each other. Gaze patterns during conversation are complex and depend on multiple factors such as cognitive load, cognitive and emotional state, turn, topic, role in the conversation as well as social factors [START_REF] References Argyle | Gaze and mutual gaze[END_REF][START_REF] Goodwin | Restarts, pauses, and the achievement of a state of mutual gaze at turn-beginning[END_REF][START_REF] Haddington | Congruent gaze points, mutual gaze and evasive gaze : some ways of using gaze in stancetaking sequences in a conversation[END_REF]. Experiment II was designed to collect precise data on mutual gaze in order to build and parameterize a model that could supplement our ECA with effective visual attention.

The second experiment thus involves a mediated face-to-face conversation between our target human speaker and several naïve human interlocutors. The face-to-face conversation is performed through a set of two screens, pinhole cameras and microphone/headphone pairs (see Figure 6). Videos, audio signals and gaze estimates are synchronized and recorded on both sides. Thanks to a very precise measurement of the gaze direction with eye trackers and image-based estimation of the position of eyes and mouth of the respective scrutinized video, we are able to distinguish between four regions of interest on the face (see Figure 7) as well as to detect saccades, fixations and blinks.

In order to be able to build an empirical model of visual attention, the number of parameters has to be minimized and number of observations maximized. We thus focused on simple pre-scripted dialogs where most of the factors influencing gaze behavior listed above are either kept constant or implicitly controlled: interlocutors are involved in a sentence repeating game where each speaker either instructs and listens or listens and repeats. Roles (initiator vs. respondent) are swapped at the middle of the game.

We show that gaze patterns and blinking rate are influenced by the respective roles and cognitive states of the interlocutors during mediated audiovisual face-to-face conversations.

Experimental Setup

The experimental platform is displayed in Figure 6. It gives interlocutors the impression of facing each other across a table. A small pinhole camera placed at the center of a computer screen films the subject facing the screen. The video image is displayed on the screen facing the interlocutor, which is equipped symmetrically. Prior to each recording session, the screens function as inversed mirrors so that subjects see their own video image in order to adjust their rest position. We determined that eye contact is optimal when the middle of the eyebrows of the video image coincides with the position of the camera on the screen. A camera located above (vs. below) the screen would generate the impression of seeing the interlocutor from above (vs. below). This would make direct eye contact impossible [START_REF] Chen | Leveraging the asymmetric sensitivity of eye contact for videoconference[END_REF].

The audio signals are exchanged via microphones and earphones. Video and audio signals as well as gaze directions are recorded during the interaction. For this purpose we use computer screens by Tobii Technology ® with embedded eye trackers. At the beginning of the recording a calibration phase writes a synchronization time stamp to the data streams. This particular setting (mediated interaction, 2D displays, non intrusive eye tracking) limits the working space but is fully compatible with our target application of an interactive ECA displayed on a screen. . Prephonation starts at the first lip opening gesture preceding the utterance (usually 300ms before any speech is produced). This cognitive state has been added for analyzing the particular gaze patterns at speech onsets. "Reading" starts at the first fixation downwards. "Else" states are added between each turn if necessary.

Reading

Scenario

One subject (initiator) reads and utters a sentence that the other subject (respondent) should repeat immediately in a single attempt. The initiator is instructed to face the screen when uttering a sentence. Roles of initiator and respondent are subsequently swapped. Semantically Unpredictable Sentences (SUS) [START_REF] Benoît | The SUS test: A method for the assessment of text-to-speech synthesis intelligibility using Semantically Unpredictable Sentences[END_REF] are used in this sentence-repeating game to force the respondent to be highly attentive to the audiovisual signal. With this rather restricted scenario of interaction we try to isolate the main elements of face-to-face interaction and elicit mutual attention. The scenario imposes a clear chaining of cognitive states and roles (see Figure 8) that avoids complex negotiation of turn taking and eases state-dependent gaze analysis. We study inter-and intra-subject variability. In each dyad the reference target interlocutor HL interacts with subjects of the same social status, cultural background and sex (French female senior researcher). We recall here that HL is the human model of the talking head used in experiment I.

Each session consists of an on-line interaction using the full experimental setup (Run A) followed by a faked interaction (Run B). Subjects were instructed that the experiment was designed to quantify the impact of visual feedback on visiophony. For Run B, the reference speaker HL was supposed to participate as instruction giver with live audio but without visual feedback. HL was thus expected to have ungrounded gaze behavior. But the subjects are in fact confronted with an audiovisual stimulus videotaped during a previous Run A by HL with one female colleague. Most subjects do not notice that the Run B stimuli are pre-recorded.

Each subject thus experiences three tasks of ten sentences each: (1) repeating SUS given on-line by the target speaker;

(2) uttering SUS and checking the correct repetition by the target speaker;

(3) repeating SUS given off-line by the target speaker.

Data processing & labeling

Fixations were identified in the raw gaze data using a dispersion-based algorithm (Salvucci and Goldberg 2000). An affine transform was applied to compensate for head movements determined by a robust feature point tracker. All fixations were projected back on a reference head position (see Figure 7). Dispersion ellipses for all regions of interest (ROI) were positioned a posteriori by the experimenter on this reference image given all fixation points produced during the interaction. Fixations were then assigned to one to these ROIs: 'left eye', 'right eye', 'mouth', 'face' (other parts than the three preceding ones such as nose), 'other' (when a fixation hits other parts of the screen) or 'none'.

The speech data were aligned with the phonetic transcriptions of SUS sentences and sessions were further segmented into sequences assigned to six different cognitive states (CS): 'pre-phonation', 'speaking', 'listening', 'reading', 'waiting' and 'thinking' (see Figure 8).

We also distinguish role (initiator vs. respondent). Differences are, for example, expected to occur during listening. When listening to the respondent, the initiator already knows the linguistic content of the SUS he has just pronounced, and visual benefit provided by lip reading is not as crucial for speech comprehension as it is when the message is not known. Note also that some states depend on role: 'waiting' is the CS of the respondent while the initiator is reading or the CS of the initiator after having uttered a sentence while waiting until the respondent begins to repeat the sentence. There are also sequential dependencies between CS: 'pre-phonation' preceding speaking is triggered by pre-phonatory gestures such as lip opening, 'speaking' triggers 'listening' for the interlocutor, etc. Some CS appear only in one of the two roles. The CS 'reading' only occurs while a subject is initiator ('reading' next sentence to utter) and the CS 'thinking' only occurs while a subject is respondent (mentally preparing the sentence to repeat). We also labeled blinks. Most ECAs generate blinks with a simple random event generator. We will however show that blinking frequency is highly modulated by CS and role.

Results

We recorded interactive sessions of our target subject with 12 interlocutors but only 9 of these pairs have valid gaze data for all sessions. The data analyzed in the following thus concerns these 9 subjects. The results clearly confirm the triangular pattern of fixations (see Figure 7) scanning the eyes and the mouth previously obtained by Vatikiotis-Bateson, Eigsti et al (1998). They also confirm our choice to distinguish cognitive state and role.

Run A: mediated one-line face-to-face

This run starts immediately after a first calibration procedure of the eye gaze tracker. HL first instructs from a list of 10 SUS then roles are exchanged and a further exchange of 10 SUS is performed.

Fixations and cognitive states. We define fixation profiles as the relative distribution of fixations among the ROI within a given activity. For statistical analysis we only considered 5 ROIs: 'left eye', 'right eye', 'mouth', 'face' and 'none'. The ROI 'else' was disregarded since it almost never occurred during the analyzed sessions. We investigated the influence of the two factors 'role' and 'cognitive state' on the mean fixation profiles calculated for our target subject during the nine interactions (see Figure 9). This means at least 180 fixations of our target subject for each CS (90 for each role). Using MANOVA we compared the multivariate means of the fixation profiles of the CS (pre-phonation, speaking, listening, and waiting) that occur in both roles. We found that fixation profiles are significantly influenced by role (F(5,871)=297, p<0.001) and CS (F(30,4375)=50.5, p<0.001).

Durations of fixations.

We also characterized the duration of fixations over region of interest and role (see Figure 11). The ANOVA provided no reason to distinguish for role (F(1,2981)=2.22, p>0.15). The influence of ROI however is highly significant (F(4,2986)=66.5,p<0.001). Post hoc analysis shows that there is no difference between durations of fixations of the ROIs "right eye" and "left eye" (F(1,2248)=0.13,p>0.7). In comparison, fixations to the face are significantly shorter and fixations to the mouth significantly longer.

Blinks. Increased cognitive load is known to be correlated with fewer blinks (Wallbott and Scherer 1991). Our data reveal that blink rate does not strongly depend on role (F(1,94)=7,p=0.01) but is highly dependent on cognitive state (F(6,94)=54,p<0.001) (see also [START_REF] Peters | Attention-driven eye gaze and blinking for virtual humans[END_REF]. A detailed analysis of the influence of CS on blink rate showed that 'speaking' accelerates blink rate, whereas 'reading' and 'listening' slow it down or even inhibit blinks (see Figure 11). Particularly, in the role of respondent, the CS 'listening' strongly inhibits blinks. Strikingly, blinks often occur at the change-over from reading to speaking (pre-phonation). This might be explained by the linkage of blinking and major saccadic gaze shifts proposed by [START_REF] Evinger | Not looking while leaping: the linkage of blinking and saccadic gaze shifts[END_REF].

Run B: faked face-to-face

To verify the impact of live feedback, we compared the fixation profiles measured during the online interaction with those of the faked interaction (using the pre-recorded stimulus). After finishing Run A naïve subject were instructed that HL would not have any visual feedback. In fact a pre-recorded stimulus was played back hoping that the sentence repeating game would keep the same rhythm. In fact no subject reported any awareness of this manipulation. Figure 12 shows that gaze behaviour confirms this conservative sense of presence: a MANOVA contrasting runs and CS indeed shows that interaction condition is not significant (F(5,56)=1.55, p=0.18) whereas CS has a significant impact on subjects' behaviour (F(25,280)=1.65, p<0.03). The only difference between interaction conditions is the spurious presence of 'Else' states during faked interactions due to minor desynchronizations between cognitive activities of the interlocutors that do not impact the global picture.

This control experiment confirms that the cognitive state is the main determinant of the gaze behaviour and this behaviour is sufficiently consistent -at least for HL -among interactions to fool the control of perception-action loops of the interlocutors.

Modeling

A generative model of visual attention during face-to-face interaction is proposed. It extends Lee et al.'s model [START_REF] Lee | Eyes alive[END_REF]) by taking into account both the cognitive states of the speaker as well as the statistical distribution of gaze fixations on important regions of interest on the face of the interlocutor. The control model for the gaze of our talking head was built by training and chaining roleand CS-specific Hidden Markov Models (HMM). Given a succession of CSs with associated durations it computes parameters describing the fixations of the ECA towards the various ROI on the face of its interlocutor. HMM states correspond to the different ROIs while observations specify the durations of the fixations (see Figure 13).

The transition probabilities of the HMM are computed from the transition matrix between the different ROI within a given CS and role as observed during the mediated face-to-face experiment. An initial state in each HMM was added to address the particular distribution of the first fixation as emphasized by the particular distribution of prephonatory gaze patterns as well as the blinking behavior. Statedependent observation probabilities determine the duration of the fixation emitted by the HMM at each transition. The probability density functions of these durations are computed from data gathered during the interactions: fixations to the mouth are for instance longer than fixations to the eyes. Based on these parameters we use the same generation process as proposed by Lee [START_REF] Lee | Eyes alive[END_REF] to control on-line the gaze of the clone of our target speaker: generators of random numbers with appropriate statistical distributions (uniform for state transitions, Gaussian for observations) are used to trigger state transitions and set durations of fixations and blink parameters.

Parameters of the HMM can then be switched and set according to cognitive state and role (see Figure 13) so as to signal scene comprehension by the appropriate statistical distribution of the ECA's gaze over regions of interest of its human interlocutor.

Discussion

These results confirm the eyes and mouth as dominant target zones (Vatikiotis-Bateson, Eigsti et al. 1998). Our data also confirm that the nose also attracts a significant number of fixations. Buchan et al [START_REF] Buchan | The effect of varying talker identity and listening conditions on gaze behavior during audiovisual speech perception[END_REF] suggest that "subjects might be using the nose as a central vantage point that permits a monitoring of the eyes and the face for social information" (see also [START_REF] Blais | Culture shapes how we look at faces[END_REF], for culture-specific gaze patterns contrasting holistic and analytical strategies). This momentary "dezooming" of the elements of the face can in fact be interpreted as a scan reset that checks for other facial information's complementary -or indirect connection -to speech communication such as facial expressions. Another possibility is that the pinhole camera and small wire may also have attracted visual attention. Note however that the seats of the interlocutors were positioned at the start of the experiment so that the camera coincides with the top of their nose ridge when their own video is displayed on their screen. The only way to test this hypothesis is to use high quality teleprompters. Apart from its cost, such an experimental setting also impacts on interaction distance.

Coordination of turntaking by gaze patterns has been mainly documented by human transcriptions of dialogs [START_REF] Goodwin | Restarts, pauses, and the achievement of a state of mutual gaze at turn-beginning[END_REF][START_REF] Rutter | Turn-taking in mother-infant interaction: An examination of vocalizations and gaze[END_REF][START_REF] Novick | Coordinating turn-taking with gaze[END_REF]: eye contact is claimed to signal beginning as well as end of turns. Using automatic eyetracking devices, our data suggest that eye contact with the right eye of the conversational partner may signal that the speaker is ready to speak. We did not observe such a specific trigger for the end. Prosody and other non verbal cues [START_REF] Duncan | Some signals and rules for taking speaking turns in conversations[END_REF] probably support this function. This eye preference is also no longer observed in the course of speech production. Our analysis is however focused on gaze distribution: with much more experimental data, data mining techniques such as time series analysis [START_REF] Povinelli | A new temporal pattern identification method for characterization and prediction of complex time series events[END_REF] could be used to search for signatures of state transitions by specific sequences of multimodal events including not only gaze patterns but also head movements, facial gestures or speech events. Such sequences of multimodal events are expected to be much more robust to monitor turns and joint attention than single unimodal events.

We have also shown that role has a significant impact on fixation profiles as well as on blinking rate. When listening, respondents should for instance gaze towards the mouth to benefit from lip reading, while the initiators do not need to rely on audiovisual speech perception since they already know the content of the message. The segmentation of the interaction into cognitive states explains a large part of the variability of the gaze behavior of our reference subject. In order to act as expected by human interlocutors, the ECA should thus at least be aware of its own cognitive state and its role in the interaction.

CONCLUSIONS AND PERSPECTIVES

Believability of context-sensitive ECAs is conditioned by both physical realism and social intelligence. Both impact the regulation of perception/action loops. As developed by [START_REF] Morgan | Signal to Syntax: an Overview[END_REF] for syntax: linguistic structures, scene and language understanding as well as social rules are learnt and established from the multimodal signals exchanged. The quality of signal production and perception is of prime importance for building efficient interactive agents. Virtual agents in turn benefit from a detailed analysis of multimodal input and output patterns observed during human-human interactions and from the interplay with their cognitive interpretation. Monitoring multimodal signals during live interactions and building comprehensive models that link them with information structure, communicative and social functions and that ground them with the environment in which the interaction takes place is a key issue for efficient situated face-to-face interaction involving artificial agents.

Based on our findings, we have established a basis for a context-aware eye-gaze generator for an ECA. In order to develop an improved gaze generator we should isolate the significant events detected in the multimodal scene that impact the closed-loop control of gaze. We should notably investigate the influence of eye saccades produced by the interlocutor as potential exogenous events that drive gaze. Furthermore other cognitive and emotional states as well as other functions of gaze (deictic or iconic gestures) should be implemented.

We will extend this study to other modalities including audible and visible speech, facial expressions, hands and head movements. We expect multimodal behaviors to be coordinated by extensive multilayered action-perception loops with different scopes and latencies. The control of the loops, the way we react and adapt to the others, is of course not only motivated by our role in the dialog, as shown in this paper, but more generally by social factors that have been voluntarily minimized in this study.

CondFigure 3 :

 3 Figure 3: Experimental conditions: The experiment was divided into four conditions with different levels of help and guidance by the clone.

Figure 4 :

 4 Figure4: Mean TC (left) and number of cards inspected (right) for our four conditions and all subjects pooled. Each bar represents 10 x 24 = 240 card pairings. An ANOVA analysis distinguishes between 3 groups of conditions with highly significant differences for number of cards inspected (p<0.05).

Figure 5 .

 5 Figure 5. Statistical analysis of TC and Number of cards inspected for each condition. Each bar represents 15 x 24 = 360 card pairings. * denotes significant differences.

Figure 6 .

 6 Figure 6. Mediated face-to-face conversation (from Raidt 2008). People sit in two different rooms and dialog through couples of cameras, screens, microphones and loudspeakers. Gaze of both interlocutors are monitored by two eye-trackers embedded in the TFT screens. Note that pinhole cameras and seats are positioned at the beginning of the interaction so that the cameras coincide with the top of the nose of each partner's face.

Figure 7 :

 7 Figure 7 : Analyzing gaze patterns after compensating for head movements. The four regions of fixation on the face: left and right eye, mouth and face. The later label is assigned to fixations towards the midsagittal plane above the nose. Regions are determined a posteriori by the experimenters by positioning dispersion ellipsis on fixation points gathered for each experiment after compensating for head movements.

Figure 8 :

 8 Figure8: Time chart of the "predefined" cognitive states for one elementary exchange of the one-line interaction scenario. Prephonation starts at the first lip opening gesture preceding the utterance (usually 300ms before any speech is produced). This cognitive state has been added for analyzing the particular

Figure 9 :

 9 Figure 9: Fixation profiles of all interactions of our target speaker over role (initiator, respondent), ROI (face, right eye, left eye, mouth, else) and cognitive state CS (speaking, listening, waiting, reading, pre-phonation, thinking, else). The bars represent the means of the percentage of fixation time on ROI

Figure 10 .

 10 Figure 10. Distribution of fixation durations as a function of ROI. Longer fixations to the mouth could be explained by smooth pursuit of a ROI in motion.

Figure 11 .

 11 Figure11. Blinking rate (average number of blinks per second) as a function of cognitive state for each interlocutor over role of our target speaker (left: initiator, right: respondent). Significant contrasts are evidenced such as less blinking when speaking compared to listening. Only data for 4 CS common for both roles are considered for analysis. Note that CS "reading" and "thinking" generate almost no fixations towards the face and that blinks are almost impossible to detect in this case.

Figure 12 .

 12 Figure 12. Comparing mean fixation profiles of all subjects acting as respondent in Run B (faked interaction at the left) vs. Run A (one-line interaction at the right). Same conventions as Figure9. The presence of 'Else' states during faked interactions could be explained by the absence of reactive chaining of instructions.

Figure 13 .

 13 Figure 13. Modeling duration of fixations and blink frequency as a HMM. Hidden states are possible ROI of the gaze (LE vs RE: left and right eyes; MO: mouth; FA: face and EL elsewhere). When entering a state a fixation is emitted (dotted arrows) according to a state-dependent distribution (dark gray). The transition matrix between states (dark arrows) specifies the CS-dependent scan of the interlocutor's face. The initial state (INI) does not emit any fixation but sets the average delay of the first blink and the blinking rate according to HMM-specific distributions (light gray). It also specifies the way the ECA will start to scan the face by transition probabilities (light gray arrows).

  ).

	Estimated profile of the other(s)	Profile of self		Settings
	Estimated cognitive state	Cognitive state	Content	Process	Reaction
	of the other(s)	of self
		Multimodal perceptual score	Multimodal gestural score
			Environment

Now with Carmeq GmbH, Carnotstr. 4 D-10587 Berlin

The Loebner Prize for artificial intelligence awards each year the computer program that delivers the most human-like responses to questions provided by a panel of judges over a computer terminal.

Please consult http://www.tobii.se/ for technical details.

The eye-tracker delivers in fact asynchronous gaze data with time stamps at a maximum rate of 60Hz. Gaze data are only sent when estimated gaze displacements exceed a fixed threshold. We post-process date off-line for further analysis by eliminating outliners and up-sampling at a fixed frequency of 60Hz.

ACKNOWLEDGEMENTS

We thank our colleague and target speaker Hélène Loevenbruck for her incredible patience and complicity. We also thank all of our subjects -the ones whose data have been used here and the others whose data have been corrupted by deficiencies of recording devices. Edouard Gentaz has helped us in statistical processing. This paper benefited from the pertinent suggestions of the two anonymous reviewers. We thank Peter F. Dominey and Marion Dohen for the proofreading. This project has been financed by the project Presence of the cluster ISLE of the Rhone-Alpes region.

 Salvucci, D. D. and J. H. Goldberg (2000). Identifying fixations and saccades in eye-tracking protocols. Eye

Tracking Research and Applications Symposium, Palm Beach Gardens, FL, 71-78. Scassellati, B. (2001)


Marion Dohen

Jean-Luc Schwartz

Gérard Bailly

Speech and Face-to-face Communication -An Introduction

published or not. The documents may come

this section provide insight to how this can be achieved. The results from these studies could have valuable applications in crucial domains such as language tutors and speech reeducation.

An important issue in human-machine interactions and especially in the synthesis of virtual speaking avatars is their perceived visual and auditory quality. Weiss, Kühnel, Wechsung, Fagel and Möller present a study of the factors which impact the quality of talking heads in the smart home domain. They focus on the importance of voice and head characteristics as well as on interactivity and media context and compare the perceived quality of three existing talking heads. A first experiment involves non-interactive ratings of videos of talking heads on the basis of speech quality, visual quality and overall quality. A second experiment tests talking head quality in an interactive dialogue setting between user and talking head. The results show a major effect of the degree of interactivity. It appears that the interactive setting prevents the users from objectively evaluating speech and visual quality. The authors also put forward an effect of the degree of media context: distraction of the user's attention impedes their ability to objectively rate talking head quality.

Badin, Tarabalka, Elisei and Bailly question the perception of augmented speech displays. Thanks to medical imagery and articulography, it is now possible to access the very details of the speech production chain: from cortical activities to the underlying movements of the speech organs. Badin et al. address the question of how this information can be used to supplement or improve speech perception. They present a study in which they explored whether seeing the tongue could improve overall speech perception in noise. It has been extensively shown that seeing lip movements during speech facilitates speech perception in noise. The visual articulatory information provided by the lips is incomplete. Since the tongue provides crucial articulatory information, the authors developed an original tongue display system using electro-magneto-articulography (EMA) and magnetic resonance imaging (MRI) data and imagined an augmented speech perception paradigm comparing speech perception in noise in four conditions: audio only, audiovisual natural (plain face visible), audiovisual + tongue and audiovisual + jaw. They find that users can extract valuable information from tongue movements although the benefit is relatively small compared to full face display, significant only for large noise levels and after some training. Users seem to implicitly and rapidly develop 'tongue reading' abilities. These results suggest that tongue displays could be effectively useful in a number of augmented speech applications.

Providing additional information can also help in ASR especially in applications dedicated to hearing impaired subjects. In this line, Heracleous, Beautemps and Aboutabit present a study on automatic recognition of French cued speech (hand supplement providing complementary visual articulatory information for deaf people) using hidden Markov models (HMM). Such recognition involves lip shape as well as hand gesture recognition and a crucial aspect is the integration of the information provided by these different channels. The authors investigate phoneme and isolated word recognition and show that adding hand shape information improves overall recognition compared to the use of lip shape information only. They find similar improvements for productions of normal hearing and deaf subjects.

Multimodal speech perception

As mentioned above, speech is multimodal and communication involves much more than acoustic features. Speech perception in humans involves the integration of many visual cues and it is necessary to understand the underlying processes involved in order to fully comprehend how face-to-face communication functions.

Several studies have shown that the visual information could participate in speech perception processes at very early stages: speech could be seen before it is heard. Troille, Cathiard and Abry aim at exploring the timing of vowel and consonant auditory and visual streams in a CVCV context. In doing so, they find that speech can also "be heard before it is seen". They present new data aiming at accounting for these apparent contradictions. They set the perceptuo-motor link at the core of this complex phenomenology by claiming that anticipatory mechanisms in speech production set the tempo for audio-visual speech perception.

Spoken communication is also a matter of word recognition and visual information could also contribute to lexical activation processes. In order to examine this question, Fort, Spinelli, Savariaux and Kandel present a study on the word superiority effect in audiovisual speech perception. Building on the fact that seeing facial gestures enhances phonemic identification in noise, the authors investigate whether the visual information regarding consonant identification could activate lexical representations. Using a phoneme monitoring task, they find that providing visual information facilitates consonant detection in noise and accelerates the phoneme detection process. Moreover, they show a specific advantage of consonant identification in words compared to pseudo-words in the audio-visual condition, which leads them to the conclusion that visual information on phoneme identity can contribute to lexical activation processes during word recognition.

Among other considerations, the fact that the visual information, and especially the articulatory visual information, is crucial in speech perception has led to suggest that the motor cortex would be involved in speech perception: the perceiver would recognize articulatory gestures rather than or complementary to acoustic targets. Sato, Buccino, Gentilucci and Cattaneo investigate the modulation of the activity in the tongue related primary motor cortex for matching versus incongruent audiovisual speech perception using transcranial magnetic stimulation, a technique enabling to monitor changes in the excitability of the cortical motor representations of tongue or lip muscles with a relatively high temporal resolution (around 10 ms). They measured motor-evoked potentials in the perception of phonemes involving tongue and/or lip movements. The results show that both visual and auditory information modulate the activity in the related primary motor cortex and suggest that visual and auditory information are recoded separately in the primary motor cortex. The authors discuss these results in terms of perception/action links and theoretical modelling of audiovisual interaction.

Communication: prosody and manual gestures

Prosody (intonation, rhythm and phrasing) plays a crucial role in spoken communication. It has long been considered to be purely auditory/acoustic. A number of recent studies have however showed that its production and perception are actually multimodal. It remains unclear exactly which facial information cues prosody. Several studies suggest an important role of visible articulatory information, others measured correlations between eyebrow movements and fundamental frequency as well as links between rigid head motion and prosodic cues. In this line, Flecha-Garcia presents a study in which she explores the relationship between eyebrow raises and discourse structure, utterance function and pitch accents in English face-to-face dialogues. She finds that eyebrow raises occurred more frequently at the start of high-level discourse segments and in requests for or acknowledgment of information. Moreover, eyebrow raises appear to be aligned with pitch accents.

From a perceptual point of view, Cvejic, Kim and Davis present a study in which they explore the perception of prosodic contrasts from the visual information provided by the upper part of the face. Two experiments explore pairing of prosodically matching fully textured or outlined videos (no sound). Two other experiments test auditory-visual prosodic matching capabilities. The results show that participants are able to match prosody corresponding videos and sounds. The authors suggest that rigid head motion is a strong visual cue to prosody.

A growing body of research suggests that manual gestures play a very important role in spoken communication and are actually part of the communicative intention. Colletta, Pellencq and Guidetti examine age-related changes (6 year olds, 10 year olds and adults) in the production of manual gestures and speech in narratives. From transcriptions of video taped narratives, the authors find that language complexity and the use of manual gestures in communication are greatly linked to age. Deeper analyses show that co-speech gestures and their combination to speech in narratives develop with age and play a crucial role in discourse cohesion and verbal utterance framing.

Tuning communication in interaction

As already showed by several studies published in the present issue, studies of spoken communication are more and more embedded in their natural context i.e. interaction. Even though this does not exclude controlled experiments to study the mechanisms involved in speech production and perception processes, it shows the necessity of considering spoken communication in an interactive framework. The papers grouped in this section put forward a number of original ideas, data and paradigms dealing with interaction in face-to-face communication.

A particular aspect of interaction is the adaptive behaviours of interlocutors involved in a conversation. In particular, speakers of different regional variations of a language manage to understand each other while identifying dialects and idiolects. Aubanel and Nguyen present a study in which they investigate the automatic recognition of phonological variation in regional accents in conversational interaction in French. They tested an automatic speech recognition technique combined to a Bayes classifier to automatically identify regional accents in natural interactive speech. Their study also confirms some of the phonetic differences between Northern and Southern French.

Further on in studying adaptive behaviours of speakers in interaction, Kopp discusses the concept of social resonance and embodied coordination in face-to-face conversation with artificial interlocutors. He analyses how and whether it would beneficial to include such considerations of adaptive behaviours of interlocutors in embodied conversational agents (ECA). He thus proposes a modelling framework both from a production and a perception point of view in a sensorimotor perspective and an implementation of this in an ECA especially in co-verbal gestures.

Many different features play a role in conversational interactions. It is necessary to understand how these features are used and which role they may play in the interaction. This will make it possible to design more natural and interactively efficient human-machine interaction systems such as ECAs. Bailly, Raidt and Elisei tackle this issue in a study on gaze behaviour of users involved in humanhuman and human-ECA face-to-face interactions. They describe two series of experiments in which they analyzed the interplay between speech and mutual gaze patterns during mediated face-to-face interactions. They hereby analyze both the effect of deictic gazes produced by a virtual agent on the interactant's behaviour and the interaction between cognitive state and communicative function on the measured gaze patterns.

"Un clin d'oeil" to Christian Benoît …

This special issue is dedicated to the memory of Christian Benoît who died in 1998. We will not recall his career here (for a biography, see e.g. http://www.icp.inpg.fr/ICP/_communication.en.html). Let us just mention that his research work intensively dealt with two of the four themes that structure the present issue: speech synthesis and audiovisual speech processing. He largely contributed to structure these domains within the speech community by organizing the first international workshops on these topics and creating the ISCA SIG groups. Even though he was less involved in the two other themes tackled in this issue, he did also work on prosody and manual gestures (through cued speech). Christian Benoît was obviously a typical face-to-face communication guy, who would certainly have entered the present field with extreme passion and enthusiasm!

Aubanel & Nguyen Kopp Bailly et al.

Acknowledgments

The guest editors of this special issue would like to thank all the authors who submitted papers. We had a lot of pleasure reading these submissions. We also thank all the reviewers who greatly helped in the process of selecting and revising the papers submitted. Of course, we thank the editors of the Speech Communication journal who gave us the opportunity of putting together such an issue, and particularly Marc Swerts, for constant confidence and support all over the editing process.

The "Speech and Face-to-Face Communication Workshop", which launched the idea of the present issue, was organised with the support of the Christian Benoît Association (ACB: http://www.icp.inpg.fr/ICP/ACB.en.html). We acknowledge the financial support of ACB, ISCA, AFCP, GIPSA-Lab, and Mrs Benoît.

Order of the papers in this special issue


Christophe Cassou
email: cassou@cerfacs.fr

THE NORTH ATLANTIC OSCILLATION : MECHANISMS AND SPATIO-TEMPORAL VARIABILITY

Over the middle and high latitudes of the northern hemisphere, the most prominent mode of atmospheric variability is the North Atlantic Oscillation (NAO). The spatio-temporal characteristics of the NAO variability are reviewed in this lecture. The general state of knowledge on the complex dynamical processes governing the NAO is described. Emphasis is laid on teleconnections and ocean signals known to modulate the temporal excitation of the NAO.

Introduction

The NAO commonly refers to swings in the atmospheric pressure difference between the Arctic and the subtropical Atlantic. The seesaw pattern is clearly pronounced during the boreal winter season and is associated with changes in the mean wind speed and direction as well as in the number of storms, their intensity, their preferred paths and their associated weather over the neighbouring continents. Significant anomalies in ocean surface temperature and heat content, ocean currents and related heat transport, ocean deep convection and sea ice cover in subarctic seas etc., are also linked to a great extent to the NAO.

The NAO across time scales

The NAO is one of the oldest known weather patterns as some of the earliest descriptions of it were from seafaring Scandinavians several centuries ago [START_REF] Stephenson | The history of scientific research on the North Atlantic Oscillation[END_REF]. There is no single way to define the NAO. The simplest one often referred to as "NAO index" has been proposed by Walker and Bliss (1932) and recently updated by [START_REF] Rogers | The association between the North Atlantic Oscillation and the Southern Oscillation in the northern hemisphere[END_REF] and [START_REF] Hurrell | Decadal trends in the North Atlantic Oscillation: regional temperatures and precipitation[END_REF] among others; it is calculated as the difference in normalized mean sea level pressure (SLP) between Azores (or Portugal) and Iceland thus capturing the north-south redistribution of atmospheric mass between the two main centres of action in the North Atlantic sector. Most studies of the NAO focus on the boreal winter, when the atmosphere is most active dynamically and perturbations grow to their largest amplitude. In the so-called positive phase, higher-than-normal surface pressures south of 55 o N combine with deeper Iceland Low to enhance the climatological meridional pressure gradient leading to stronger-than-average surface westerlies across the middle latitudes and enhanced penetration of the mild and moist oceanic influence towards most of Europe. During NAO+, northward shifted storm track leads to cooler and drier conditions over the Mediterranean basin.

A disadvantage of station-based indices is that they are fixed in space. They do not account for the seasonal migration of the NAO centres of action and they are polluted by transient meteorological phenomena not related to the NAO, especially in summertime and autumn. An alternative approach is Empirical Orthogonal Function (EOF) analysis [START_REF] Hurrell | An overview of the North Atlantic Oscillation[END_REF]; the NAO is often identified as the leading eigenvector computed from the time variations of gridpoint values of SLP or some other climate variables (geopotentiel, zonal wind etc.). The associated principal component is used to evaluate the temporal evolution of the NAO. While EOFs clearly capture the spatial and large-scale coherence of the NAO pattern, a well known shortcoming stands however in the linear constraints of the method ; EOFs assumes preferred atmospheric circulation states come in pairs in which anomalies of opposite polarity have the same spatial structure. A solution among others to overcome this simplification is to use classification techniques or clustering, which seek for recurrent patterns of a specific amplitude and sign. Objective analyses based on these methods are often used to identify number, spatial structure and frequency of occurrence of recurrent anomalous circulations referred to as "weather regimes". Travelling pressure systems or storms contribute to a significant fraction of the daily to interannual variability of the extratropical climate. Those are linked to the unstable nature of the upper-level westerly jet stream and interact with circulation patterns of large scale, or "weather regimes", in which they are embedded. Those regimes can be interpreted as quasi-stationary atmospheric circulations during which the character of the synoptic storms is unusually persistent [START_REF] Reinhold | Dynamics of weather regimes: quasi-stationary waves and blocking[END_REF]. They are recurrent, spatially well defined and limited in number and ideally correspond to statistical-dynamical equilibria in the climate phase space. Over the North Atlantic-European (NAE) domain when applied to daily atmospheric circulation fields, they traditionally leads to four regimes [START_REF] Vautard | Multiple weather regimes over the North Atlantic analysis of precursors and processors[END_REF]). The two first regimes can be viewed as the negative and positive phases of the NAO (Fig. 1). The third regime is named Atlantic Ridge and the fourth is often referred to as Scandinavian Blocking.

Figure 1: (a-d) Centroids of the four wintertime NAE Z500 weather regimes (m). Each percentage represents the mean frequency occurrence of the regime computed over 1958-2002 from 1 December to 31 March. Contour intervals are 25m. (e-h) Number of days of occurrence of each regime per winter from 1959 to 2002. The NAO index (orange curve) is superimposed on the upper two panels corresponding to the NAO regimes. Correlation (R) between the NAO index and the frequency of occurrence of the NAO regimes is provided. From Cassou et al. (2010).

The day-to-day meteorological fluctuations can be described in terms of temporal transition between regimes. The year-to-year (or longer timescale) climate fluctuations can be interpreted as changes in their frequency of occurrence provided the hypothesis of long term quasi-stationarity climate. This climate-oriented interpretation for weather regimes is shared with the so-called continuum paradigm elaborated for instance in [START_REF] Franske | The continuum and dynamics of the northern hemisphere teleconnection patterns[END_REF] to better understand the low-frequency fluctuations of the northern hemisphere atmospheric teleconnections patterns. The NAO-and NAO+ regimes interannual occurrence (Fig. 1ef) is strongly correlated to the traditional NAO index above discussed. Note however that spatial asymmetry between the two phases are clearly evidenced in winter by clustering techniques. Similar conclusions can de drawn for summertime.

Links between flow regimes and mean conditions over Europe have been documented from daily to decadal timescale as well as for weather [START_REF] Cassou | Tropical Atlantic influence on European heat waves[END_REF] and climate extremes [START_REF] Yiou | Inconsistency between atmospheric dynamics and temperatures during the exceptional 2006-2007 fall/winter and recent warming in Europe[END_REF]. 10-meter wind and air temperature composites derived from regime occurrence are also indicative of strong relationships between daily large-scale atmospheric circulation and ocean surface over the entire Atlantic basin. Evidence has been provided that a large fraction of the low frequency trends in the Atlantic observed at the surface over the last 50 years in winter can be tracked back to changes in the occurrence of weather regimes and their tropical counterpart (Fig. 2 Large-scale atmospheric circulation tendency are characterized by an intensification of the midlatitude westerlies and a concomitant strengthening of the Northeast trades. Northward displaced storm track is associated with prevalent southerlies north of 40 o N. The node for the wind trend corresponds to the mean position of the Azores High and is consistent with the positive trend in the NAO (Fig. 1 and Hurrell et al 2003). Note that updated data until 2010 suggest though that the upward trend towards NAO+ enhanced occurrence is somehow reduced in the 2000's. The largest lowfrequency changes for 2-meter temperature occur in the western part of the basin and strongly project on the North Atlantic tripole known to be forced by NAO dynamics [START_REF] Deser | Atmosphere-ocean interaction on weekly timescales in the North Atlantic and Pacific[END_REF].

The NAO governing processes

That the spatial pattern of the NAO remains largely the same throughout the year does not imply that it also tends to persist in the same phase for long. To the contrary, it is highly variable tending to change its phase from one week to the next, from one month to another, and its longer term behaviour clearly reflects the combined effect of residence time in any given phase and its amplitude therein. There is little evidence for the NAO to vary on any preferred time scale. The spectrum of the wintermean NAO index reveals somewhat enhanced variance at quasi-biennial periods, a deficit of power at 3 to 6 year periods and slightly enhanced power in the 8-10 year band, but no significant peaks [START_REF] Hurrell | An overview of the North Atlantic Oscillation[END_REF]. Note that those statistical characteristics are not stationary throughout the observed period.

It is now well recognized that the NAO type of variability arises from processes internal to the atmosphere in which various scales of motion interact with one another to produce quasi-random (and thus quasi unpredictable) variations. The variability of the extratropical flow can be divided into two broad classes: jet meandering, in which the latitude of the local zonal wind speed maximum in the upper troposphere exhibits pronounced north-south shifts; and jet pulsation, in which the local windspeed maximum exhibits marked strengthening and weakening without shifting latitude. Variability in the Atlantic zonal flow is characterized by both mechanisms because the subtropical jet is relatively weak and the baroclinic eddies tend to organize themselves in the poleward baroclinic zone ; the latter is particularly pronounced in the Atlantic due to the sharp meridional sea surface temperature (SST) gradient along the Gulf Stream/Labrador current, hence driving a purely eddy-driven jet. The NAO is thus associated with anomalous fluxes of zonal momentum of baroclinic waves across ~45 o N and fluctuates on preferred timescales of ~10 days resembling a first-order Markov process [START_REF] Feldstein | The time-scale, power spectra and climate noise properties of teleconnections patterns[END_REF]. Positive feedbacks between the zonal flow and the eddies, as well as interactions with the stationary waves, play a key role in setting the basin-wide shape of the NAO.

An emerging view for explaining the jet variation and its link to the NAO has been recently proposed based on wave-breaking theory (e.g. [START_REF] Benedict | Synoptic view of the North Atlantic Oscillation[END_REF][START_REF] Woollings | A regime view of the North Atlantic Oscillation and its response to anthropogenic forcing[END_REF]. Synoptic wave-breaking on the equatorward side of the jet tends to be anticyclonic, following the ambient background shear. This leads to poleward eddy fluxes of zonal momentum that act to push the jet to the North and to set NAO+ regimes. Similarly, cyclonic wave-breaking dominates on the poleward side of the jet and the associated momentum fluxes push it to the south and tend to excite NAO-regimes.

Finally, the coupling between the lower stratosphere and the troposphere could also play an active role in setting the phase and persistence of the NAO. Large amplitude anomalies in the strength of the zonal flow along ~60 o N frequently seem to originate in the stratosphere and descend into the troposphere. The downward propagating stratospheric circulation anomalies appear to modulate relatively high frequency tropospheric variability for period up to ~60 days following the initiation of the stratospheric signal. A stronger stratospheric polar vortex more frequently leads to NAO+ like anomalies and vice-versa, thus exerting a downward control on surface climate. Even if controversial, the annular shape of the stratospheric mode supports the view that the NAO could be the regional expression of a larger scale hemispheric mode of variability known as the Arctic Oscillation [START_REF] Thompson | Atmospheric processes governing the Northern Hemisphere Annular Mode/North Atlantic Oscillation[END_REF].

The NAO modulation by external factors

Progress in understanding low frequency variability and predictability of the midlatitude atmospheric circulation relies on the identification of specific causes responsible for the favoured occurrence of the associated weather regimes, their persistence and/or transition. In other words, which factors external to pure atmospheric dynamics at the origin of the NAO could modulate its strength and phase? At intraseasonal timescale, recent studies (e.g. [START_REF] Cassou | Intraseasonal interaction between the Madden and Julian Oscillation and the North Atlantic Oscillation[END_REF]) suggest that the main climate oscillation in the tropics -the Madden-Julian Oscillation (MJO)-controls part of the distribution and sequences of the four NAE winter weather regimes (Fig. 1). The NAO regimes are the most affected. NAO+ events mostly respond to a midlatitude low frequency wave train initiated by the MJO in the western-central Pacific and propagating eastward. Precursors for NAO-are found in the eastern tropical Pacific-western Atlantic leading to changes along the North Atlantic Storm-track. Wave breaking diagnostics tend to support the MJO preconditioning and the role of the transient eddies in setting the phase of the NAO. These findings reconcile the oscillatory paradigm of the MJO with the episodic view of the North Atlantic dynamics via the regime approach and open some perspectives in terms of medium range predictability.

From seasonal to decadal timescale, evidence is provided that the ocean plays an active role in determining the evolution of the NAO (e.g. Hurrell et al 2003 for a review). While intrinsic atmospheric variability exhibits temporal incoherence, the ocean tends to respond to it with marked persistence of heat content anomalies that feedback to the local atmosphere. The level of retroaction of the anomalous extratropical SST upon the NAO is however under debate and is likely very dependent on the time scale. Adding to the complexity of local ocean-atmosphere interaction is the possibility of remote forcing of the NAO from the tropical oceans; the latter influence appears to be more robust. Several studies have concluded that NAO variability is closely tied to SST variations over the tropical Atlantic. Those involve changes in the meridional SST gradient across the equator, which affect the strength and location of the tropical convection along the Inter Tropical Convergence Zone, and thus ultimately the North Atlantic midlatitude circulation via the excitation of Rossby waves propagating northeastward from the anomalous diabatic heating source. At decadal timescale, the warming of the Indo-Pacific warmpool from the mid-1970's onwards could be responsible for part of the positive trend of the NAO at the end of the XX th century. At interannual timescale, the impact of El Niño Southern Oscillation (ENSO) on the NAO remains open to debate. The correlation between the two indices is not significant and the relationships might be asymmetrical; La Niña events seems to favour NAO+ phases while El Niño events might not induce systematic remote responses. In addition, the ENSO-NAO link might be indirect via the ENSO impact on tropical North Atlantic SSTs or via the ENSO influence on the stratospheric vortex as suggested by the latest literature (e.g [START_REF] Bell | Stratospheric communication of ENSO teleconnections to European Winter[END_REF].

One of most urgent challenge is to advance our understanding of the interaction between greenhouse gases forcing and the NAO. The response might be felt as a change in the residence frequency of the NAO+ regimes with respect to NAO-ones, leading to a weak positive trend according to the most recent results. However, the locations of both regimes in the NAO phase space may be also altered in warmer climate, moving towards more positive values of the NAO index; in that context, the background change thus contributes to the trend.

  , Cassou et al 2010).

Figure 2 :

 2 Figure 2: Linear trends computed over 1959-2002 for wintertime (a-c) ERA40 10-meter zonal (U10) and meridional (V10) wind and 2-meter temperature (T2) and for (d-f) reconstructed fields from multiple linear regression using NAE weather regime occurrences and tropical wind classes as predictors. Hashing stands for sign disagreement between observed and reconstructed trends. Contour intervals are 0.15 ms-1/decade, 0.075 ms -1 /decade for UV10 and V10 respectively, and 0.1 o C for T2. From Cassou et al. (2010).