Amir Aly

Adriana Tapus

Multimodal Adapted Robot Behavior Synthesis within a Narrative Human-Robot Interaction

. This synthesized multimodal robot behavior is evaluated by the interacting human at the end of each emotion-eliciting experiment. The obtained results validate the positive effect of the generated robot behavior multimodality on interaction.

I. INTRODUCTION

The need for an intelligent robot that can customize the emotional content of its synthesized multimodal behavior to the context of interaction so as to increase the credibility of its communicative intents, is increasing rapidly. Speech, gestures, and facial expressions are used together to convey coordinated and synchronized verbal, paraverbal, and nonverbal information that could enhance the content of interaction. The importance of gestures and facial expressions lies in their ability to clarify the meaning of speech when the signal is deteriorated, in addition to the fact that they can replace or accompany words in a synchronized manner [START_REF] Ekman | About brows: Emotional and conversational signal[END_REF].

The correlation between emotion and speech had been intensively investigated in the literature. Speech prosody can reflect human emotion through changes in basic cues, such as: pitch, intensity, rate, and pauses [START_REF] Cowie | Describing the emotional states that are expressed in speech[END_REF]. The variation in the characteristics of voice prosody for different emotions: anger, disgust, fear, and sadness, was studied in [START_REF] Bachorowski | Vocal expression and perception of emotion[END_REF]. The process of emotion perception and decoding, in addition to the required time to recognize different emotions based on their prosodic cues, was studied in [START_REF] Pell | On the time course of vocal emotion recognition[END_REF]. The evolutionary nature of emotion was considered in [START_REF] Aly | Towards an online fuzzy modeling for human internal states detection[END_REF] and [START_REF] Aly | An online fuzzy-based approach for human emotions detection: An overview on the human cognitive model of understanding and generating multimodal actions[END_REF], while studying the cognitive perception of emotion through a fuzzy model.

adriana.tapus@ensta-paristech.fr On the way towards synthesizing emotional speech that can add more naturalness to human-robot interaction and human-computer interaction, the first emotional speech synthesis system was developed based on the rule-based formant synthesis technique [START_REF] Murray | Implementation and testing of a system for producing emotion-by-rule in synthetic speech[END_REF], but the quality was a bit poor. Another interesting approach based on the diphone concatenation technique that achieved some limited success in expressing specific emotions, was discussed in [START_REF] Edgington | Investigating the limitations of concatenative synthesis[END_REF]. This last technique was later developed to the unit selection technique that tries to avoid interference with the recorded voice during synthesis so as to obtain a better quality, and reported some small success in expressing only three emotions: happiness, anger, and sadness [START_REF] Iida | Speech database design for a concatenative text-to-speech synthesis system for individuals with communication disorders[END_REF]. Generally, the previously discussed techniques are missing some explicit control on the prosodic parameters of speech so as to be able to express emotions on a wider scope. This constraint and the quality of the synthesized voice, constituted our inspiration for using a more controllable and efficient text-to-speech engine, like Mary-TTS [START_REF] Schroder | The German text-to-speech synthesis system Mary: A tool for research, development, and teaching[END_REF], in our work.

On the other hand, the basic definition for gesture was given by Kendon [START_REF] Kendon | Gesticulation and speech: Two aspects of the process of utterance[END_REF] and McNeill [START_REF] Mcneill | Hand and mind: What gestures reveal about thought[END_REF]. They defined a gesture as a synchronized body movement with speech, which is related parallely or complementarily to the meaning of the utterance. The first step towards categorizing gestures, was discussed in Ekman and Friesen [START_REF] Ekman | The repertoire of nonverbal behavior: Categories, origins, usage, and coding[END_REF]. They proposed 5 gesture categories: (1) emblems, (2) illustrators, (3) facial expressions, (4) regulators, and (5) adaptors. However, Kendon [START_REF] Kendon | The study of gesture: Some remarks on its history[END_REF] criticized the proposed gesture categories of Ekman for ignoring the linguistic phenomena. Therefore, he proposed a new classification for gestures of 4 categories: (1) gesticulation, (2) pantomime, (3) emblem, and (4) signs.

McNeill [START_REF] Mcneill | Hand and mind: What gestures reveal about thought[END_REF] presented a more elaborate widely used gesture classification of 4 categories: (1) iconics (i.e., gestures representing images of concrete entities), ( 2) metaphorics (i.e., gestures representing abstract ideas), (3) deictics (i.e., pointing), and (4) beats (i.e., rhythmic finger movements).

Several research studies in the fields of human-robot interaction and human-computer interaction, have focused on synthesizing iconic and metaphoric gestures, which form together (according to McNeill) the major part of the generated nonverbal behavior during human-human interaction. Pelachaud [START_REF] Pelachaud | Multimodal expressive embodied conversational agents[END_REF] developed the 3D agent GRETA, which can synthesize a multimodal synchronized behavior to the human users based on an input text. Generally, GRETA can synthesize gestures of all categories regardless of the domain of interaction, to the contrary of other 3D conversational agents (e.g., the conversational agent MAX) [START_REF] Kopp | Synthesizing multimodal utterances for conversational agents[END_REF]. An interesting framework was discussed in [START_REF] Le | A common gesture and speech production framework for virtual and physical agents[END_REF], which can synthesize a multimodal synchronized behavior for both the 3D agent GRETA and robots. Cassell et al., [START_REF] Cassell | BEAT: The behavior expression animation toolkit[END_REF] presented BEAT toolkit, which is a rule-based gesture generator. It applies the natural language processing (NLP) algorithms on an input text in order to produce an animation script that can animate both of humanoid robots [START_REF] Aly | A model for synthesizing a combined verbal and nonverbal behavior based on personality traits in human-robot interaction[END_REF], [START_REF] Aly | Towards an intelligent system for generating an adapted verbal and nonverbal combined behavior in human-robot interaction[END_REF], and virtual agents (e.g., REA agent) [START_REF] Cassell | Human conversation as a system framework: Designing embodied conversational agents[END_REF]. This toolkit can generate gestures of different kinds (including iconic gestures) except for metaphoric gestures. Generally, the majority of gesture generation approaches are not considering the effect of emotion (expressed through prosody) on body language, which puts a difficulty towards adapting the generated robot behavior to human emotion [START_REF] Busso | Interrelation between speech and facial gestures in emotional utterances: A single subject study[END_REF]. In this paper, we present an extension of our previous work [START_REF] Aly | Prosody-based adaptive metaphoric head and arm gestures synthesis in human robot interaction[END_REF], [START_REF] Aly | Towards an interactive human-robot relationship: Developing a customized robot's behavior to human's profile[END_REF], which proposes a statistical model for synthesizing adapted head-arm metaphoric gestures to the prosodic cues of speech (for this reason, the experimental design in this paper is based on a human-robot interaction scenario in order to use the modeled synthesized speech on the robot (whose content corresponds to the comment of the interacting human on the video) as an input to the gesture generator). This model has been integrated to the system of this paper in order to generate an adapted multimodal robot behavior to the emotional content of interaction.

On the other hand, the correlation between facial expressions and speech had been long recognized in psychological studies [START_REF] Busso | Analysis of emotion recognition using facial expressions, speech, and multimodal information[END_REF]. The movement of face muscles and the prosodic cues of speech can change in a synchronized manner in order to communicate different emotions. The single-modal based perception of human emotion through audio or visual information, was discussed in [START_REF] Silva | Facial emotion recognition using multimodal information[END_REF]. Chen et al., [START_REF] Chen | Multimodal human emotion/expression recognition[END_REF] discussed the complementarity of both modalities, so that the perception of human emotion will be ameliorated when both modalities are considered in the same time. These last findings are considered in the experimental design of our study.

The synthesis and modeling of facial expressions in computer-based applications and 3D agents received more attention than in human-robot interaction. Parke [START_REF] Parke | Computer generated animation of faces[END_REF] developed the first 3D face model that can convey different expressions. Platt and Badler [START_REF] Platt | Animating facial expressions[END_REF] presented the first model that employs FACS (Facial Action Coding system) in controlling the muscular actions corresponding to facial expressions. Spencer-Smith et al., [START_REF] Spencer-Smith | Making faces: Creating three-dimensional parameterized models of facial expression[END_REF] developed a more 3D realistic model that can create a stimuli with 16 different FACS action units and determined intensities. Similarly, robots underwent different studies aiming towards allowing them to generate reasonable facial expressions. An early initiative to model facial expressions on robots was taken by Breazeal [START_REF]Towards sociable robots[END_REF], who developed the robot head Kismet. It uses facial details, like: eyes, mouth, and ears to model facial expressions, such as: anger, happiness, surprise, sadness, and disgust. Breemen et al., [START_REF] Breemen | iCat: An animated userinterface robot with personality[END_REF] developed the research platform iCat, which can render different facial expressions, such as: sadness, anger, happiness, and fear. Beira et al., [START_REF] Beira | Design of the robot-cub (iCub) head[END_REF] developed the complete expressive humanoid robot iCub, which can synthesize a variety of emotions using gestures and facial expressions, including: anger, sadness, surprise, and happiness. In this paper, we use the highly expressive ALICE robot (Section Fig. 1: Overview of the emotionally-adapted narrative system architecture II-C) for the purpose of generating a complete multimodal robot behavior, which was not sufficiently addressed before in the literature. The rest of the paper is structured as following: Section (II) presents a detailed illustration for the system architecture, Section (III) illustrates the design, the hypotheses, and the scenario of interaction, Section (IV) provides a description for the experimental results, and finally, Section (V) concludes the paper.

II. SYSTEM ARCHITECTURE

The proposed system is coordinated through different subsystems: (1) Speech dictation system (HTML5 API multilingual dictation toolkit), (2) Emotion detection phase, in which some defined keywords are parsed from the dictated speech of the human user so as to precise an emotional label for the video's content, (3) Mary-TTS engine, which converts the prepared texts (i.e., robot comments) and the detected emotion in each interaction to a synthesized emotional speech, (4) Metaphoric gestures generator, which maps the synthesized speech to synchronized head-arm metaphoric gestures [START_REF] Aly | Prosody-based adaptive metaphoric head and arm gestures synthesis in human robot interaction[END_REF], (5) Facial expressions modeling and animation stretching phase, and finally (6) ALICE robot as the test-bed platform in the experiments (Section III). An overview of the system architecture is illustrated in Figure [START_REF] Aly | Prosody-based adaptive metaphoric head and arm gestures synthesis in human robot interaction[END_REF].

A. Metaphoric Gestures Generator

The generator uses the Coupled (i.e., 2 chains for speech and gestures) Hidden Markov Models (CHMM) [START_REF] Rabiner | A tutorial on hidden Markov models and selected applications in speech recognition[END_REF] in order to synthesize head-arm metaphoric gestures, as illustrated in details in [START_REF] Aly | Prosody-based adaptive metaphoric head and arm gestures synthesis in human robot interaction[END_REF] and [START_REF] Aly | An integrated model of speech to arm gestures mapping in human-robot interaction[END_REF]. The training of the CHMM requires first segmenting the characteristic curves of speech and gestures. The motion curves of gestures (i.e., the velocity, acceleration, displacement, and position curves) are segmented by calculating the force, momentum, and kinetic energy of body segments (e.g., the up-arm, low-arm, and hand segments), in addition to the total force of the body. The intersection between these descriptors represents the boundary points of gestures in each body segment. Meanwhile, the pitch-intensity curves of speech are segmented in parallel with gestures in terms of the boundary points of each gesture, the frame time, and the sampling frequency [START_REF] Aly | Prosody-based adaptive metaphoric head and arm gestures synthesis in human robot interaction[END_REF]. These segmented patterns of speech and gestures are used to train the CHMM, through which new adapted headarm metaphoric gestures will be synthesized based on the prosodic cues of a speech-test signal.

B. Affective Speech Synthesis

The text-to-speech engine (Mary-TTS) is used for the purpose of adding relevant prosodic and accent cues to a pre-prepared text that summarizes the content of the video under discussion [START_REF] Schroder | The German text-to-speech synthesis system Mary: A tool for research, development, and teaching[END_REF]. This allows the robot to engage in the conversation using -to some extent -adapted emotional speech to the emotional context of the video. The designed vocal patterns are represented using a low-level markup language called MaryXML (which is based on the XML markup language) or using other relatively high-level markup languages, like the SSML (Speech Synthesis Markup Language) [START_REF] Taylor | SSML: A speech synthesis markup language[END_REF]. The SSML representation offers more vocal design features, like imposing a silence period between words, in addition to an easy control on the characteristics of the pitch contour, baseline pitch, and speech rate (Figure 2), which makes it helpful for the emotional vocal patterns' design described in this study. However, the fact that Mary-TTS engine is not prepared yet for efficiently synthesizing emotional speech of different classes in the English language (to the best of our knowledge, no other vocal engine can), makes our proposed vocal design as an approximate step towards conveying -even to some extent -the true meaning of the expressed emotion to a human user. Therefore, the multimodality of the robot behavior is a good solution that could emphasize the meaning of the expressed behavior, so that each modality enhances the other one.

The designed vocal patterns of the target emotions are summarized in Table (I), in which the pitch contours are characterized by sets of parameters inside parentheses (where the first parameter in each set followed by "%" represents a percentage of the text duration, while the second parameter represents the corresponding change in the baseline pitch in semitone, which is half of a tone on the standard diatonic scale). The speech rates of the target emotions vary between the rates of the "sadness" emotion (lowest rate) and the "anger" emotion (highest rate). The inter-sentence break time of each target emotion represents the silence periods between sentences (i.e., story texts), during which the robot's lips/jaw will make certain expressions that could enhance the expressed emotion (Section II-C). Meanwhile, the intrasentence break time represents the short silence periods within a sentence, which are necessary for increasing the credibility of the "sadness" and "fear" emotions.

The indicated experimental parameters in Table (I) give an example to the prosodic patterns of parts of the texts that Mary-TTS engine should convert to speech in different emotions. The other prosodic patterns of the remaining parts of the texts could differ slightly from the contour parameters of Table (I) in order to show some tonal variation through the total of each text.

C. Face Expressivity

The designed facial expressions corresponding to the prescribed target emotions in this study, are based on the Facial Action Coding System (FACS) [START_REF] Ekman | Facial action coding system: A technique for the measurement of facial movement[END_REF]. Table (II) illustrates the FACS coding of each target emotion in this study, in addition to the available equivalent joints in the face of the robot that we used in order to model each expression in the most persuasive manner.

The complexity behind modeling emotions on the face of the robot lies in the absence of the equivalent joints to some FACS descriptors (e.g., cheek raiser, nose wrinkler). Therefore, we imposed experimentally some additional body gestures in order to reduce the negative effect of the missing joints so as to enhance the expressed emotion. These additional gestures do not include -normally-any head gesture (i.e., neck rotation) nor arm-hand gestures, which are being generated by the metaphoric gestures generator explained earlier (Section II-A). 1 However, the combination of the neck rotation (i.e., turning the head aside) and the raising front-bent arms has been helpful for better expressing the "disgust" emotion (consequently, they got considered as additional supportive gestures for this emotion). This will help give the interacting human -even to some extentthe impression that the robot did not like the context of interaction and considered it disgusting. Similarly, the "fear" emotion, the "anger" emotion, and the "sadness" emotion have been attributed additional mouth-guard hand gesture, down head-shaking, and bowing head and covering-eyes hand gesture respectively, in order to help emphasize their meanings, as indicated in Figure [START_REF] Cowie | Describing the emotional states that are expressed in speech[END_REF]. On the other hand, the main role of the additional supportive left smile and right smile face joints of the "fear" emotion, is to depress a little the corners of the open mouth to better reflect the emotion, however they do not have any equivalent FACS descriptor representing the "fear" emotion, as indicated in Table (II).

Generally, the modeling of facial expressions on a humanoid robot (even with the expressive ALICE robot) is not an easy task due to the mechanical limitations that the robot has (unlike the 3D agents). Therefore, the multimodality of the robot behavior is important for interaction, which makes each modality of behavior expression enhances the other modalities so as to emphasize the conveyed meaning of the expressed emotion to a human user.

The temporal alignment between the synthesized emotional speech and the designed facial expressions is controlled by the duration of the generated speech. In case the TABLE I: Approximate design of the vocal pattern and the corresponding contour behavior of each target emotion on the standard diatonic scale. Some emotions have used interjections (with tonal stress) in order to emphasize the desired meaning, like: 'Shit' for the "anger" emotion, 'Ugh' and 'Yuck' for the "disgust" emotion, and 'Oh my God' for the "fear" emotion. duration of the generated speech is longer or shorter than the preliminary duration of the animation, the system calculates easily the new time instant of each control point composing the XML animation script (in which the control points are characterized in terms of position and time) as a function of the new duration of the generated speech, the preliminary duration of the animation, and the last time instant value of each control point. Meanwhile, the position of each control point is kept unchanged during the animation. The segmentation of human speech employs the voice activity detection algorithm in order to label and separate between the speech and silence segments. In case the silence period is related to an inter-sentence break time (Section II-B), the robot's lips/jaw perform certain expressions (e.g., lip corner pulling for the "happiness" emotion) to enhance the conveyed meaning of the expressed emotion, as indicated in Figure (3). This is due to the mechanical limitations of the robot that do not allow for synchronizing the lips with speech, while performing an expression with the lips/jaw in the same time. However, in case the silence period is related to an intra-sentence break time (Section II-B), the robot's jaw is kept opened in the "fear" emotion and closed in the "sadness" emotion, during the duration of the silence period.

On the other hand, the animation of the robot's lips in a synchronized manner with the segmented speech has encountered a big difficulty when using the 3 servo motors controlling the lips motion (2 motors for the corners and 1 motor for the vertical motion), because they can not generate a reasonable homogeneous motion when operating together during continuous speech, in addition to the noise they generate. Alternatively, we used only the motor that controls the vertical motion of the lips. Afterwards, the remote running server of the robot maps the calculated visemes corresponding to the segmented speech to lips motion.

III. EXPERIMENTAL SETUP

In this section, we introduce the database used in inducing emotions in each participant, the experimental hypotheses, the design, and the scenario of interaction between the participant and ALICE robot developed by Hanson Robotics. 2

A. Database

The database used in this research contains 20 videos inducing the following 6 emotions: sadness, disgust, happiness, anger, fear, and neutral. The duration of the videos varies from 29 to 236 seconds, and all of them have been extracted from commercial films. The procedures of validating the efficiency of the database in eliciting the target emotions in

B. Hypotheses

This study aims to test and to validate the following hypotheses:

• H1: The combination of facial expressions, head-arm metaphoric gestures, and synthesized emotional speech will make the emotional content of interaction more clear to the participant than the interaction conditions that employ less affective cues. • H2: Facial expressions will enhance the expressiveness of the robot emotion in contrast to the interaction conditions that do not employ facial expressions. • H3: The dynamic characteristics of the robot head-arm metaphoric gestures will help the participant recognize and distinguish between the target emotions.

C. Experimental Design

Our design contains three robot conditions:

• The robot generates a combined multimodal behavior expressed through synchronized head-arm metaphoric gestures, facial expressions, and speech (i.e., condition C1-SFG).

• The robot generates a combined multimodal behavior expressed through synchronized facial expressions and speech (i.e., condition C2-SF).

• The robot generates a combined multimodal behavior expressed through synchronized head-arm metaphoric gestures and speech (i.e., condition C3-SG).

• The robot generates a single-modal behavior expressed only through speech (i.e., condition C4-S). In order to examine the first hypothesis, the first three conditions were examined (in which the facial expressions are accompanied by the additional supportive gestures illustrated in Table II). In this hypothesis, we excluded the conditions of the robot expressing a single-modal behavior only through head-arm metaphoric gestures or facial expressions without speech, in addition to the condition of the robot expressing combined facial expressions and head-arm metaphoric gestures without speech, because they do not match the context of the non-mute human-human interaction. Consequently, the importance of speech in recognizing emotions is measured directly through the questionnaire. On the other hand, in order to validate the second hypothesis, two conditions were investigated, which are the same as the conditions C2-SF and C4-S. Similarly, in order to validate the third hypothesis, two other conditions were tested, which are the same as the conditions C3-SG and C4-S (the condition C2-SF was excluded from validating the third hypothesis, because facial expressions were accompanied by the additional supportive gestures explained earlier).

Both of the robot and the interacting human follow a series of short videos through 6 experiments that mean to elicit 6 Fig. 4: Two participants interacting with the robot during the "happiness" and "sadness" emotion elicitation experiments emotions (Section III-A) (Figure 4). The different methodologies of emotion induction and assessment were illustrated in [START_REF] Coan | The handbook of emotion elicitation and assessment (Series in affective science)[END_REF]. The idea behind using videos to successfully elicit emotions in the human user, is that they are emotionally convincing and their role is well indicated in the literature [START_REF] Roberts | Learning to influence emotional responses for interactive storytelling[END_REF]. An interesting study about eliciting emotions from films was discussed in [START_REF] Hewig | A revised film set for the induction of basic emotions[END_REF], in which the results proved that the studied target emotions were reasonably recognized. In our study, the scenario of interaction is described as following:

• The robot welcomes the participant and invites him/her to watch some videos so as to have a discussion about. • The robot asks the participant to express his/her opinion about the content of the video. Afterwards, it parses some expected emotional labels from the dictated comment of the participant, such as: This is disgusting!. This helps detect the video's emotional content so as to trigger an adapted robot behavior. • After listening to the comment of the participant on the video, the robot makes itself a comment accompanied by adapted emotional speech, head-arm metaphoric gestures, and/or facial expressions to the video's content. • In case the video rarely elicits a different emotion in the participant from the target emotion, so that the system parses some keywords that belong mainly to another category of non-target-emotion-referring keywords, the robot will comment through a neutral behavior in order to avoid any emotion-biasing effect. Then, it will invite the participant to watch another video, which means to certainly elicit the same emotion that was not successfully induced in the participant with the first video. • The interaction ends for the concerned emotion. Afterwards, the participant starts evaluating the modeled behavior on the robot considering the relevance of its emotional content to the context of interaction, through a Likert questionnaire (in which all questions are presented on a 7-point scale). Whereupon, a new interaction for a new randomly selected emotion starts. • After the experiments end up, the robot and the experimenter thank the participant for his/her cooperation.

IV. EXPERIMENTAL RESULTS

The experimental design was based on the betweensubjects design, and 60 participant were recruited in order to validate our hypotheses. The participants were uniformly distributed between the four experimental conditions (15 participant (6 female and 9 male) /condition). The recruited participants were ENSTA-ParisTech undergraduate and graduate students and employees whose ages were varying between 20-57 years old (M = 29.64, SD = 9.4). The background of the participants was non-technical with an average of 33.3%, and technical with an average of 66.7%. 40% of the participants have interacted before with robots, while 60% of the participants have never interacted with robots beforehand.

For the first hypothesis, a significant difference was found by ANOVA analysis in the clearness of the adapted robot emotional behavior expressed through a combination of head-arm metaphoric gestures, facial expressions, and speech with respect to the robot emotional behavior expressed through facial expressions and speech, and the robot emotional behavior expressed through head-arm metaphoric gestures and speech (F[2, 267] = 9.69, p < 0.001). Tukey's HSD comparisons indicated a significant difference between the robot embodied with combined head-arm metaphoric gestures, facial expressions, and speech (i.e., condition C1-SFG) from one side, and the robot embodied with facial expressions and speech (i.e., condition C2-SF) (p < 0.001), in addition to the robot embodied with head-arm metaphoric gestures and speech (i.e., condition C3-SG) (p < 0.001) from the other side. However, no significant difference was observed between the experimental conditions C2-SF and C3-SG. Moreover, the participants found that the robot behavior was more expressive in the condition C1-SFG than in the condition C3-SG (F[1, 178] = 13.64, p < 0.001). No significant differences were observed in the participants' ratings regarding the naturalness of the robot behavior in the conditions C1-SFG, C2-SF, and C3-SG.

For the second hypothesis, the participants found that the robot behavior expressed though facial expressions and speech, was showing more expressiveness and was more adapted to the content of interaction than the robot behavior expressed only through speech (i.e., condition C4-S) (F[1, 178] = 16.27, p < 0.001). Moreover, the participants considered that facial expressions and speech were synchronized with an average score of M = 5.9, SD = 0.9. Furthermore, they did not find any significant contradiction between the modalities of the robot behavior expressed through facial expressions and speech with an average score of M = 1.8, SD = 1.2. Over and above, they agreed that facial expressions were more expressive than speech with an average score of M = 4.4, SD = 1.5. Table (IV) shows that the facial expressions of the robot have only ameliorated the recognition score of the "anger" emotion in the condition C2-SF with respect to the condition C4-S. This amelioration is related to the encountered difficulties to design a highly persuasive vocal pattern for the "anger" emotion due to the limitations of the Mary-TTS engine (Section II-B). Therefore, the facial expressions of the robot have certainly enhanced the affective meaning of speech so as to give the participant a clear feeling that the robot was expressing the "anger" emotion. To the contrary, the facial expressions of the robot had a negative influence on the recognition score of the "disgust" emotion in the condition C2-SF with respect to the condition C4-S, which is due to the limited expressivity of the robot's face for this emotion (Section II-C). For the third hypothesis, the participants considered that the emotional content of the robot behavior expressed through head-arm metaphoric gestures and speech was more observable than the emotional content of the robot behavior expressed only through speech (F[1, 178] = 17.16, p = 0.0001). Furthermore, the participants found that gestures and speech were synchronized with an average score of M = 6.1, SD = 0.7. At the same time, they agreed that the execution of gestures was fluid with an average score of M = 5.3, SD = 1.01. Moreover, they considered that gestures were slightly more expressive than speech with an average score of M = 4.2, SD = 1.4. The emotional content of the robot head-arm metaphoric gestures was generally recognizable with reasonable scores, as indicated in Table (IV). However, they have only ameliorated the recognition score of the "anger" emotion in the condition C3-SG with respect to the condition C4-S (similarly to the effect of facial expressions), while the other recognition scores were equal in both conditions. Consequently, the dynamic characteristics of the generated gestures in case of the "anger" emotion, like the high velocity and acceleration, have certainly enhanced the expressive meaning of speech and gave the human the feeling that the robot was angry in a more persuasive manner. On the other hand, the emotional expressiveness of the robot behavior was positively perceived in general by the male and female participants in the conditions C2-SF and C3-SG, as indicated in Figures [START_REF] Pell | On the time course of vocal emotion recognition[END_REF] However, the perception of the male participants for the emotional expressiveness of the robot in both conditions, was generally higher than the perception of the female participants. The male participants in the condition C2-SF gave higher ratings for the emotions: sadness, disgust, happiness, and fear, meanwhile the female participants gave higher ratings for the emotions: anger and neutral (Figure 5). Similarly, the male participants in the condition C3-SG gave higher ratings for the emotions: sadness, disgust, anger, and fear, meanwhile the female participants gave higher ratings for the emotions: happiness and neutral (Figure 6). These findings reveal the relatively higher preference of the male participants for the emotional expressiveness of the female ALICE robot, than the female participants. This genderbased evaluation matches the findings of [START_REF] Siegel | Persuasive robotics: The influence of robot gender on human behavior[END_REF], which proved the tendency of the participants to consider the opposite-sex robots as being more credible, engaging, and persuasive.

V. CONCLUSIONS

This paper discusses adapting the multimodal robot behavior to the emotional content of a series of videos eliciting specific emotions in the human user within a narrative human-robot interaction. Each interacting human was exposed to only one of 4 different experimental conditions of multimodal/single-modal behaviors (i.e., conditions: C1-SFG, C2-SF, C3-SG, or C4-S) during the whole 6 experiments of eliciting the 6 target emotions. Our proposed system uses Mary-TTS engine in order to generate emotional speech. The metaphoric gestures generator synthesizes headarm gestures based on the prosodic cues of speech. On the other hand, the designed facial expressions on the robot required some additional supportive gestures to enhance the conveyed meaning of the expressed emotion to the human.

This paper validates the role of the robot behavior multimodality in increasing the clearness of the emotional content of interaction with respect to the interaction conditions that use less affective cues. Besides, it proves the role of facial expressions in enhancing the expressiveness of the robot behavior, and the role of the generated gestures in recognizing the target emotions. For the future work, we are interested in increasing the gestural expressivity of the system by integrating additional gesture generators, which can synthesize gestures of other categories (e.g., iconic gestures). Besides, we are interested in ameliorating the emotional content of the synthesized speech so as to make the generated speech more persuasive and more natural.

Fig. 2 :

 2 Fig. 2: SSML specification of the "sadness" emotion

Fig. 3 :

 3 Fig. 3: Synthesized facial expressions by ALICE robot

Fig. 5 :

 5 Fig. 5: Gender-based evaluation for the emotional expressiveness of the multimodal robot behavior expressed through combined facial expressions and speech (condition C2-SF). The error bars represent the calculated standard errors.

  and (6), respectively.

Fig. 6 :

 6 Fig. 6: Gender-based evaluation for the emotional expressiveness of the multimodal robot behavior expressed through combined head-arm gestures and speech (condition C3-SG). The error bars represent the calculated standard errors.

TABLE II :

 II FACS coding of the target emotions and the corresponding joints in the robot's face, in addition to the other required gestures to emphasize the meaning of facial expression. The bold FACS action units in each emotion represent the observed prototypical units between the subjects in[START_REF] Shichuan | Compound facial expressions of emotion[END_REF], while the other less common non-bold units are observed with different lower percentages between the subjects. The underlined action units represent the units that have approximate corresponding joints in the robot's face.

	Emotion	Baseline Pitch	Pitch Contour	Speech Rate	Start	Contour Features Behavior	End	Break Time
	Sadness	-4st	(0%,+0st)(100%,-0st)	-30%	Negative		Constant	Negative	Inter/Intra-Sentence
	Disgust	+4st	(0%,-5st)(40%,-9st)(75%,-12st)(100%,-12st)	+8%	Negative	Exponential	Negative	Inter-Sentence
	Happiness	+2st	(0%,+8st)(30%,+16st)(50%,+14st)(100%,+11st)	+7%	Positive		Parabola	Positive	Inter-Sentence
	Anger	+5st	(0%,-18st)(50%,-14st)(75%,-10st)(100%,-14st)	+12%	Negative		Parabola	Negative	Inter-Sentence
	Fear	+6st	(0%,+2st)(50%,+5st)(75%,+8st)(100%,+5st)	+7%	Positive		Parabola	Positive	Inter/Intra-Sentence
	Emotion		FACS Coding	Robot's Face Joint		Additional Body Gestures
	Sadness	Brow Lowerer + Lip Corner Depressor + Inner Brow Raiser + Cheek Raiser + Nasolabial Deepener + Chin Raiser	Left Smile + Right Smile + Brows	Covering-Eyes Hand + Bowing Head + Narrowing Eyes + Eyes Blinking + Closing Jaw
	Disgust	Lip Pressor + Brow Lowerer + Nose Wrinkler + Upper Lip Raiser + Chin Raiser	Jaw + Brows		Neck Rotation + Raising Front-Bent Arms + Narrowing Eyes
	Happiness	Lip Corner Puller + Lips Part + Cheek Raiser	Left Smile + Right Smile + Jaw		Eyes Blinking
	Anger	Brow Lowerer + Lid Tightener + Lip Pressor + Lip Tightener + Upper Lip Raiser + Chin Raiser + Nasolabial Deepener	Jaw + Brows + Eyelids	Down Head-Shaking + Short Mouth-Opening
	Fear	Inner Brow Raiser + Brow Lowerer + Lip Stretcher + Lips Part + Outer Brow Raiser + Upper Lid Raiser + Jaw Drop	Left Smile + Right Smile + Jaw + Brows + Eyelids		Mouth-Guard Hand

TABLE III :

 III Target emotions and their corresponding feature films. The main videos used during the experiments were extracted from the bold feature films.

	Target Emotion	Feature Film
	Sadness	The Champ -An Officer and a Gentleman
	Disgust	Pink Flamingos -Maria's Lovers
	Happiness	On Golden Pond -An Officer and a Gentleman
	Anger	My Bodyguard -Cry Freedom
	Fear	Halloween -Silence of the Lambs
	Neutral	Crimes and Misdemeanors -All the President's Men

TABLE IV :

 IV Recognition scores of the target emotions expressed by the robot in 3 different experimental conditions

	Condition	Sadness	Disgust	Emotion Happiness	Anger	Fear	Neutral
	C2-SF	100%	80%	93.3%	92.9%	100%	100%
	C3-SG	100%	93.3%	93.3%	92.3%	100%	100%
	C4-S	100%	93.3%	93.3%	80%	100%	100%

Amir Aly is a postdoctoral research fellow in the Robotics and Computer Vision lab. in ENSTA-ParisTech, 828 Boulevard des Maréchaux, 91120 Palaiseau, France amir.aly@ensta-paristech.fr

Prof. Adriana Tapus is with the Robotics and Computer Vision lab. in ENSTA-ParisTech, 828 Boulevard des Maréchaux, 91120 Palaiseau, France

The metaphoric gestures generator[START_REF] Aly | Prosody-based adaptive metaphoric head and arm gestures synthesis in human robot interaction[END_REF] has the liberty to synthesize the most appropriate gestures based on its own learning algorithm. Therefore, it is probable that the previously mentioned supportive head-arm gestures will not be synthesized by the generator during interaction. Consequently, we imposed them at specific moments during speech with a higher priority than the generator's synthesized gestures to make sure of their presence.

The humanoid ALICE-R50 robot has a full-motion body and an expressive face, with a total of

degrees of freedom. The robot is equipped with two cameras and an array of sensors, including an accelerometer sensor, a torque sensor, a series of touch sensors, in addition to many other different sensors that allow it to precisely perceive its surroundings. The face of the robot composed of synthetic skin, is its main specialty. It can create a full range of credible facial expressions in different emotions (Section II-C).


Didier A Girard

A COMPARISON OF LIKELIHOOD METHODS AND THEIR FAST RANDOMIZED VERSIONS FOR STOCHASTIC PROCESS MODELS WITH MEASUREMENT ERROR

Keywords: asymptotic normality, consistency, Gaussian process, identifiability, infill asymptotics, maximum likelihood estimator, measurement error, nonparametric Bayesian regression, randomized trace

. We show that any consistent root of the randomized version of the likelihood equation has the same asymptotic behavior as for the exact version excepted that the asymptotic variances are increased by the factor 1 + 1/nR. Moreover, to attack the problem of choosing between multiple, possibly non-consistent, roots, we propose a simple randomized version of the whole likelihood, whose maximizer is proved to be consistent even with nR = 1.

Introduction

We consider the classical problem of building up statistical inferences for the model

Y (s) = Z(s) + e(s), s ∈ R d (1.1)
where Z is a zero mean Gaussian stochastic process whose covariance function is known up to a magnitude factor b > 0 and a shape parameter θ > 0, and e is a "measurement error" process independent of Z. These parameters have to be estimated from only n + 1 observations at sites s(0), s(1), . . . , s(n) with i.i.d. Gaussian noise e(s(i)), i = 0, . . . , n; and we consider the classical maximum likelihood (sometimes called "marginal ML" or "type II ML" from an empirical Bayesian point of view) principle for this task.

To simplify the presentation we first assume that the noise variance is known, say equal to 1. Denoting by bK(θ) the assumed model for the covariance matrix of z = Z(s(0)), . . . , Z(s(n)) ′ , the (marginal) law of y = Y (s(0)), . . . , Y (s(n)) ′ is then y ∼ N (0, b 0 K(θ 0 ) + I).

(1.2)

For future references, let us introduce the conditional mean of Z(s) given y, evaluated at s(i), i = 0, . . . , n. This classical optimal prediction of z is well known to be A θ 0 (b 0 )y where (1.5)

A θ (b) = I + b -1 K(θ) -1 -1 (1.3)
Various forms and extensions (e.g. for multidimensional b or θ) of these formulae are almost ubiquitous in the geostatistics field. There, and in many other fields, the size of the data may be very large. So a number of techniques have been developed to implement the likelihood principle.

In recent years, fast randomized versions of the likelihood equation (i.e., here, of solving S θ (b) = 0, which is called the LE or "scoring" method) has been used in many contexts with large data sets (e.g., [START_REF] Wahba | Adaptive tuning of numerical weather prediction models: Part I: randomized GCV and related methods in three and four dimensional data assimilation[END_REF]: in its simplest form, it consists in generating a single vector w ∼ N (0, I), of the same size as y, and replacing the function S θ (b) by

R S θ (b) = - 1 b y ′ A θ (b) [I -A θ (b)] y -w ′ A θ (b)w . (1.6)
Thus, all we need for evaluating such a randomized score, is a (fast) algorithm for the computation of any conditional mean A θ (b)y. We will also denote by R S θ (b) the "averaged" randomized score obtained when

w ′ A θ (b)w is replaced by a simple Monte-Carlo average (1/n R ) n R r=1 w r′ A θ (b)w r .
For example, in the machine learning field, fitting such models is become a popular approach to regression, which was recently widely studied under the name "Gaussian process regression". In the survey by [START_REF] Gibbs | Efficient implementation of Gaussian processes[END_REF] the authors advise to use iterative linear solvers and averages of randomized trace estimates for the computation of any gradient of the log likelihood criterion. [START_REF] Gibbs | Efficient implementation of Gaussian processes[END_REF] noticed that the number (n R is our notation) of randomized trace estimates needed to obtain sufficiently good estimates is surprisingly small. The cost of such inferences is then reduced from order n 3 to order n 2 in general (that is, for full and not structured matrices). Much greater computational gains can actually be obtained in other contexts; see the final section.

The purpose of this paper is to give theoretical explanations of such behaviors when n R is a fixed number (for example equal to 10); and we do this by an asymptotic analysis in the simple context studied in [START_REF] Chen | Infill asymptotics for a stochastic process model with measurement error[END_REF] (abbreviated as CSY henceforth): in (1.1), Z is a classical Ornstein-Uhlenbeck process, that is, a one-dimensional zero-mean Gaussian process whose covariance function is the exponential function:

E(Z(s)Z(t)) = b exp{-θ|t -s|}.
(1.7)

Note that we essentially adopt the same notations as in CSY except that the magnitude of the covariance function is denoted here by b (in place of σ 2 used in CSY). We consider infill (or "fixed domain") asymptotics, that means that the s(i)s becomes dense in a compact interval as n increases. Furthermore, as in CSY we consider equally spaced sites s(i) = i/n, i = 0, . . . , n.

Infill asymptotic frameworks may be more useful than the classical time series frameworks to explain some empirical facts when one is faced with strong

D.A. GIRARD
correlations in the underlying process. This is emphasized in [START_REF] Zhang | Inconsistent estimation and asymptotically equivalent interpolation in model-based geostatistics[END_REF], [START_REF] Zhang | Toward reconciling two asymptotic frameworks in spatial statistics[END_REF]. For example, a clear explanation of the practical difficulties of estimating b 0 and θ 0 simultaneously, is provided by the important remark: it is known (e.g. [START_REF] Ying | Asymptotic properties of a maximum likelihood estimator with data from a Gaussian process[END_REF]) that as long as θb = θ 1 b 1 , the couples (θ, b)

and (θ 1 , b 1 ) are not distinguishable from a single sample path Z(t), t ∈ [0, 1] of the Ornstein-Uhlenbeck process with covariance (1.7).

As was also noted by [START_REF] Zhang | Toward reconciling two asymptotic frameworks in spatial statistics[END_REF] the available results under infill asymptotics are considerably narrower in scope than for increasing domain. This is even more true when the measurements are obtained with errors or there is a so-called nugget effect. In two notable exceptions, analogs of Theorem 2.1 below are proved for the Brownian motion plus white noise model and for its mth order spline generalization, by [START_REF] Stein | A comparison of generalized cross validation and modified maximum likelihood for estimating the parameters of a stochastic process[END_REF] and [START_REF] Kou | On the efficiency of selection criteria in spline regression[END_REF]. Note that, of course, in the model of CSY, randomized-traces are not useful for computing ML estimates since, as is well known, for this model, the exact likelihood (and the score) are calculable in O(n) operations by standard procedures akin to Kalman filtering (e.g. Harvey (1994 section 3.4)). Before going on, let us notice, however, that in the context of a simple additive multidimensional extension of this model, these randomized procedures are particularly suitable: this will be discussed in the final section.

In order to simplify the presentation, we assume that the variance of the white noise is known. Note that, for this variance, one could easily construct a lot of estimators which have the same asymptotic behavior as the ML estimate and are easy to compute. As in CSY, we assume that we know lower (> 0)

and upper bounds for b 0 and θ 0 so that the (approximate) likelihoods can be

maximized over D = [b, b] × [θ, θ] with b > 0, θ > 0.
As in CSY, two cases are distinguished depending on whether θ 0 is known or not. In the second case, in view of the above remark on the non-identifiability of the couple (b 0 , θ 0 ) and the neat result in CSY on the estimation of their product, we focus on the estimation of the single parameter b 0 θ 0 .

One of the main results of this paper is that any consistent estimate obtained by solving the randomized likelihood equation converges to the true parameter with the same rate as the estimate obtained with the exact likelihood, even with n R = 1, and with an asymptotic variance only inflated by a factor 1 + 1/n R .

However, quite often, the likelihood equation, even computed exactly, may have several roots. In such cases, it is indispensable to select a root which globally maximizes the likelihood since consistency is then guaranteed (cf. Theorems 2.2, 2.4 rephrased from CSY and Theorem 2.5).

A second type of results thus concerns a randomized version of the global log likelihood function. From the expression (1.6), it is natural to propose the following definition, in an integral form: having generated w and chosen a "boundary" point b 1 (although b 1 is not necessarily at the boundary of the search domain D), -2 times this randomized log likelihood function is defined by:

R l b 1 (b, θ) = y ′ (I -A θ (b)) y+ b b 1 w ′ A θ (s)w s ds-logdet [I -A θ (b 1 )]+(n+1)log(2π).
(1.8)

As for the score (1.6), this expression is the particular case n R = 1 of an averaged version, also denoted by R l (note that b 1 will often be omitted), directly

obtained with (1/n R ) n R r=1 w r′ A θ (s)w r in place of w ′ A θ (s)w.
Evaluating this criterion is relatively easy as soon we have at hand an efficient algorithm for the computation of any conditional mean A θ (b)y, excepted for the log-determinants at the boundary values (b 1 , θ) (see the final section for comments on the approximation of the integral by discrete sum). Of course, in the case θ 0 known, there is no need to compute logdet [I -A θ 0 (b 1 )] since this is a constant term in the objective function (1.8). This computational simplification is also available in the case where θ is constrained to be a predetermined value θ 1 in the search domain.

In this paper, we show that consistency is still guaranteed (for the product θ 1 times the constrained maximizer in the case θ 0 unknown) by maximizing these randomized likelihoods.

Our results are stated in Section 3. Before that, in order to make easy the comparisons with the non-randomized versions, we essentially recall in Section 2 the main results of CSY. Section 2.1 and Section 3.1 concern the estimation of b 0 assuming θ 0 to be known. Section 2.2 and Section 3.2 concern the estimation of the product b 0 θ 0 . Proofs are given in Section 4. In the final section, we connect the proposal (1.8) with two techniques already used in the literature, and we discuss possible applications and extensions of our results.

D.A. GIRARD

"Asymptotic background" for the exact likelihood principle

First, recall that it is easily seen that, in the case θ 0 known, the standard Fisher information for the magnitude parameter has the following simple expression

E 1 2 ∂ 2 l ∂b 2 (b 0 , θ 0 ) = var 1 2 S θ 0 (b 0 ) = trA 2 θ 0 (b 0 ) 2b 0 2 for the generic model (1.2)-(1.
3); and an asymptotic equivalent for this information, as n → +∞, in the CSY context (1.7) that we study in Sections 2, 3 and 4 under the assumption

s(i) = i/n, i = 0, . . . , n, is c(b 0 , θ 0 )n 1/2 with c(b 0 , θ 0 ) > 0.
So the occurrences of the power n 1/4 in the following statements, in place of the usual n 1/2 of the i.i.d. case (or θ = +∞), are natural.

Let us recall that we assume that the true parameter is in the interior of the

search domain D which is either [b, b] or [b, b] × [θ, θ] with b > 0, θ > 0.
2.1. Case θ 0 known.

From the work of CSY, we easily deduce than their asymptotic normality results can be stated in the following slightly more general form, since they were, there, restricted to the exact ML estimate:

Theorem 2.1.(CSY) Let b be any candidate estimate obtained by solving the likelihood equation at θ 0 , possibly up to o p (n 1/4 ), viz. satisfying

S θ 0 ( b) = o p (n 1/4 ) with S θ 0 (b) defined from (1.5). If b → b 0 in probability then n 1/4 ( b -b 0 ) → D N (0, 4 √ 2θ 0 -1/2 b 3/2 0 ).
Now we rephrase the first consistency result of CSY. Let bML denote any minimizer of -2 times the log likelihood function (i.e. of l(•, θ 0 ) defined from (1.4)).

Theorem 2.2.(CSY) b := bML satisfies the conditions of Theorem 2.1.

2.2. Case θ 0 unknown.

In CSY an asymptotic behavior identical to the behavior of bθ 0 of the case θ 0 known above, is proved for the product of LE estimators, precisely:

Theorem 2.3.(CSY) Let ( b, θ) be any couple of candidate estimates obtained by solving the likelihood equation possibly up to o p (n 1/4 ), viz. satisfying

S θ( b) = o p (n 1/4 ) with S θ (b) defined in (1.5). If bθ → b 0 θ 0 in probability then n 1/4 ( bθ -b 0 θ 0 ) → D N (0, 4 √ 2(b 0 θ 0 ) 3/2 ).
Now, the following theorem is the important result that the ML principle, which was already justified in the time series framework (e.g. Zhang and Zimmerman ( 2005)), is also justified here to estimate the identifiable product b 0 θ 0 .

Let ( bML , θML ) denote any couple minimizer, over D, of l(•, •) defined in (1.4).

Theorem 2.4.(CSY) ( b, θ) := ( bML , θML ) satisfies the conditions of Theorem 2.3.

We now claim a result which was not stated in CSY (however see [START_REF] Ying | Asymptotic properties of a maximum likelihood estimator with data from a Gaussian process[END_REF] for the case of no measurement error) which is useful here for the computational simplification outlined in the Introduction (penultimate paragraph). Let θ 1 > 0 be an arbitrarily fixed constant. Let bML (θ 1 ) denote any minimizer of l defined in (1.4) restricted to θ = θ 1 , i.e.

l( bML (θ 1 ), θ 1 ) = inf b∈[b,b]
l(b, θ 1 ).

Theorem 2.5. Assume that b 0 θ 0 /θ 1 is in the interior of [b, b]. Then ( b, θ) := ( bML (θ 1 ), θ 1 ) satisfies the conditions of Theorem 2.3.

Results

The context is the same as the one of Section 2. Of course the simulated

w ′ A θ (s)w s - trA θ (s) s ds = o p (n 1/4+α )
uniformly in (b, θ) ∈ D.

Remark 3.1. The resulting order of convergence for the global difference R l b 1 -l will be seen sufficient. Of course an extension of this result to the case b 1 = 0 would be very interesting for the practice (since the boundary log-determinants in R l b 1 would then be trivially 0) but we think that the assumption b 1 > 0 is important; this can be seen by analyzing the behavior of the continuous integrand

at the limit value b = 0, noting that b -1 A θ (b) = b + K(θ) -1 -1 approaches K(θ)
; indeed the minimal order of w ′ K(θ)w -trK(θ) is easily seen to be the much larger o p (n 1+α ) instead of o p (n 1/4+α ).

3.1. Case θ 0 known.

We first state an analog of Theorem 2.1 for consistent roots of the randomized score. Note that the same n R vectors ws are used for all (b, θ), in (1.6) or (1.8).

Theorem 3.1. Let R b be any candidate estimate obtained by solving the randomized likelihood equation at θ 0 , possibly up to o p (n 1/4 ), viz. satisfying

R S θ 0 ( R b) = o p (n 1/4 ) with R S θ 0 (b) defined, in the case n R = 1, from (1.6). If R b → b 0 in probability then n 1/4 ( R b -b 0 ) → D N 0, 1 + n -1 R 4 √ 2θ 0 -1/2 b 3/2 0 .
Remark 3.2. In the quite different context of nonparametric estimation of deterministic functions by kernel methods, results which have a roughly similar appearance are given in Girard (1998) for randomized GCV, except that, there, the obtained relative increase in variance was strictly lower than 1 + 1/n R . Now we claim that the consistency property is intact with randomized traces.

Let bRML denote a minimizer of -2 times the randomized log likelihood function (i.e. of R l b 1 (•, θ 0 ) defined from (1.8) with b 1 any fixed "boundary" value > 0).

Theorem 3.2. For any fixed n R ≥ 1, R b := bRML satisfies the conditions of Theorem 3.1.

3.2.

Case θ 0 unknown.

The analog of Theorem 2.3 for any consistent couple of roots for the scalar randomized score is the following:

Theorem 3.3. Let ( R b, R θ) be any couple of candidate estimates obtained by solving the randomized likelihood equation possibly up to o p (n 1/4 ), viz. satisfying

R S R θ( R b) = o p (n 1/4 ) with R S θ (b) defined, in the case n R = 1, in (1.6). If R bR θ → b 0 θ 0 in probability then n 1/4 ( R bR θ -b 0 θ 0 ) → D N 0, 1 + n -1 R 4 √ 2(b 0 θ 0 ) 3/2 .
We now claim two consistency results.

Let b 1 > 0 be an arbitrarily fixed value for the boundary point used in (1.8).

Let ( bRML , θRML ) denote any couple minimizer, over D, of R l := R l b 1 defined in (1.8).

Theorem 3.4. For any fixed n R ≥ 1, ( R b, R θ) := ( bRML , θRML ) satisfies the conditions of Theorem 3.3.

The randomized analog of Theorem 2.5, which is a consistency result allowing a misspecified model, also holds. Let θ 1 > 0 be an arbitrarily fixed constant. Let bRML (θ 1 ) denote any minimizer of R l defined in (1.8

) restricted to θ = θ 1 R l( bRML (θ 1 ), θ 1 ) = inf b∈[b,b]
R l(b, θ 1 ).

Theorem 3.5. Assume that b 0 θ 0 /θ 1 is in the interior of [b, b]. Then, for any fixed n R ≥ 1, ( R b, R θ) := ( bRML (θ 1 ), θ 1 ) satisfies the conditions of Theorem 3.3.

D.A. GIRARD

Proofs

Let us first recall basic results of CSY. These authors exhibit approximations for the exact likelihood and for its derivative w.r.t. b which are much more manageable. Define ρ θ := exp{-θ/n}, λ b,θ := b(1 -ρ 2 θ ) + (1 -ρ θ ) 2 . We also define ρ 0 := ρ θ 0 and λ 0 := λ b 0 ,θ 0 the true parameters. Note that CSY uses the notation λ θ for λ b,θ , λ for λ b,θ 0 and ρ for ρ 0 .

CSY gives, in their Lemma 1, a quasi-diagonalized formula for the exact log likelihood. Note that this was stated in the case θ fixed at θ 0 but it is clear in CSY that this formula holds globally of course with ρ θ , λ b,θ in place of ρ, λ in their Lemma 1 (note that one must also use T θ in place of T θ 0 where T θ denotes the n × (n + 1) bidiagonal matrix satisfying (T θ y) k = y k+1 -ρ θ y k for any vector y). To exploit this as in CSY, we need to recall the definition of

γ k := 2 1 -cos πk n + 1 , k = 1, . . . , n, (4.1)
which are the eigenvalues of the n × n tridiagonal Toeplitz matrix with 2 on the diagonal and -1 on the two neighboring off-diagonals. Let u k , k = 1, . . . , n denote the corresponding eigenvectors.

Proofs of preliminary results

To prove Lemma 3.1, we first state a diagonalized approximation which holds with a largely sufficient accuracy: there exist i.i.d. R W k ∼ N (0, 1), k = 1, . . . , n, such that, for all α > 0, for all b, θ,

w ′ A θ (b)w b - trA θ (b) b = (1 -ρ 2 θ ) n k=1 R W k 2 ρ θ γ k + λ b,θ - n k=1 1 ρ θ γ k + λ b,θ + o p (n α ) (4.2)
Here One of the steps in the proofs of CSY is the claim that, if W k , k = 1, . . . , n, are i.i.d. N (0, 1), then, for any α > 0,

n k=1 W k 2 -1 ρ θ γ k + λ b,θ = o p (n 5/4+α
). (4.3) (generalizing their Lemma 3(i)). So Lemma 3.1 is obtained by applying the mean value theorem and noting that 1 -ρ θ 2 = 2θ/n+O(n -2 ). The proof of (4.3) is not detailed in CSY. We think that the following comments might be useful for the reader : the fact that it holds pointwise is easy to see, by Chebyshev inequality, since the Lemma 3 of SCY (concerning the behavior of trA θ (b) and trA 2 θ (b) and which is stated there for the case θ fixed at θ 0 ) also holds, of course, with ρ 0 (the β in SCY) replaced by ρ θ and λ b,θ 0 (the λ in CSY) replaced by λ b,θ provided appropriate simple changes are made for the constants. Next, one way to prove the required uniformity is to use a particular Lipschitz property of the weights of the (W k 2 -1)'s, as functions of b and θ, precisely sup

(b,θ) =(b ′ ,θ ′ ) |g n,k (b, θ) -g n,k (b ′ , θ ′ )| (b, θ) -(b ′ , θ ′ ) ≤ M sup (b,θ) |g n,k (b, θ)| (4.4)
where g n,k denotes (ρ θ γ k + λ b,θ ) -1 , • is the Euclidean distance, the sup are over D, and M is independent of k, because it is known that this property is sufficient to translate a pointwise convergence in probability to a uniform one (see e.g. [START_REF] Strook | Multidimensional Diffusion Processes[END_REF] or Corollary A of [START_REF] Wu | Asymptotic theory of nonlinear least squares estimation[END_REF]). Indeed the property (4.4) can be checked via appropriate bounds over D for the partial derivatives of g n,k w.r.t. to b and θ; the details are tedious and thus omitted.

Starting from the observations

ρ θ = 1 - θ n + O(n -2 ), λ b,θ = 2bθ n + O(n -2 ), (4.5)
the following comments explain how to easily deduce Theorem 2.5 from the results of SCY. A key intermediate result in the proof of their Theorem 3 is the following:

S θ (b) = -1 -ρ 2 θ n k=1 (ρ 0 γ k + λ 0 )W k 2 (ρ θ γ k + λ b,θ ) 2 + 1 -ρ 2 θ n k=1 1 ρ θ γ k + λ b,θ + o p (n 1/4 ) (4.6)
where W k = u k ′ T θ 0 y does not depend on θ or b. Now with (4.5) and appropriate

D.A. GIRARD
uniform convergence results like (4.3), it can be checked that

S θ (b) = - 2θ n n k=1 γ k + 2b 0 θ 0 n W k 2 γ k + 2bθ n 2 + 2θ n n k=1 1 γ k + 2bθ n + o p (n 1/4 ) (4.7)
(because the score will be seen to be of order n 1/4 at the true parameter, this uniform approximation indicates in passing that the product bθ is a natural choice for reparametrization of the numerical search). Let us denote l(b, θ)

= n k=1 γ k + 2b 0 θ 0 n W k 2 γ k + 2bθ n + n k=1 log γ k + 2bθ n (4.8)
the ideal (not exactly observed) likelihood. The consistency toward b 0 of the minimizer of l(b, θ 0 ) could be obtained exactly as in the detailed proof of CSY for l(b, θ 0 ). A key intermediary result is that

l(b, θ 0 ) -l(b 0 , θ 0 ) ≥ n 1/3 k=1 γ k + 2b 0 θ 0 n γ k + 2bθ n -1 -log γ k + 2b 0 θ 0 n γ k + 2bθ n + o p (n 1/4+α
), (4.9)

and an analysis of the deterministic sum allows CSY to conclude (cf. their proof of Theorem 1). Now by observing that, denoting Sθ (b) := (∂ l/∂b)(b, θ), that is, the sums term at the right side of (4.7)

l(s, θ 1 ) -l(b 0 θ 0 θ 1 , θ 1 ) = l(s, θ 1 ) -l(b 0 θ 0 θ 1 , θ 1 ) + s b 0 θ 0 θ 1 S θ 1 (t) -Sθ 1 (t) dt (4.10) = l(s θ 1 θ 0 , θ 0 ) -l(b 0 , θ 0 ) + o p (n 1/4+α ). (4.11)
Thus from (4.9) applied to b := sθ 1 /θ 0 , we have bML (θ 1 ) → b 0 θ 0 /θ 1 in probability.

The second condition (stationarity of the likelihood l(•, θ 1 ) at bML (θ 1 )) is satisfied with a probability tending to 1 since b 0 θ 0 /θ 1 is assumed in the interior of [b, b].

4.2 Proofs of Theorems 3.2, 3.4 and 3.5

We only consider the general case of global minimization of R l, that is, θ 0 unknown. The proofs for the other cases are similar. To prove Theorem 3.4 the first essential argument is the following: by examining the detailed proof in CSY, we observe that the proof of consistency for RML estimates can be reduced to the proof of

R l(b, θ) -R l(b 0 , θ 0 ) ≥ n 1/3 k=1 ρ 0 γ k + λ 0 ρ θ γ k + λ b,θ -1 -log ρ 0 γ k + λ 0 ρ θ γ k + λ b,θ + o p (n 1/4+α ).
(4.12)

The second essential argument is then Lemma 3.1 whose application gives enough accuracy to deduce (4.12) from its non-randomized analog proved in CSY.

4.3 Proofs of Theorems 3.1 and 3.3

As usual the asymptotic laws of the (approximate) roots are obtained via Taylor approximation. A simplifying feature of the CSY model is that the third derivative can be uniformly bounded in probability. First a simpler approximate form for the randomized score, similar as the approximate form (that we denote by Sθ (b)) stated for S θ (b) in (4.7), can be obtained with exactly the same proof:

there exist i.i.d. R W k ∼ N (0, 1), k = 1, . . . , n, independent of the W k 's such that R Sθ (b) = - 2θ n n k=1 γ k + 2b 0 θ 0 n W k 2 γ k + 2bθ n 2 + 2θ n n k=1 R W k 2 γ k + 2bθ n (4.13) satisfies R S θ (b) -R Sθ (b) = o p (n 1/4
). We consider, to simplify the notation, the case θ 0 known (extension for the product bθ, when θ 0 is unknown will be seen to be immediate) and we drop the index θ 0 in the following. The first condition of Theorem 3.1 is thus also equivalent to R S( R b) = o p (n 1/4 ). By Taylor expansion of R S(•) at b 0 and the second condition of Theorem 3.1, there exists b ⋆ which tends to b 0 in probability, such that

o p (n 1/4 ) = R S(b 0 ) + R b -b 0 R S′ (b 0 ) + 1 2 R b -b 0 2 R S′′ (b ⋆ )
Theorem 3.1 can be classically proved by combining the three following properties: First, R S(b 0 ) satisfies a CLT theorem similar as S(b 0 ) excepted that the variance is increased by a factor 1 +

1/n R . Second, R S′ (b 0 ) is uniformly equiv- alent to the same c(b 0 , θ 0 )n 1/2 as S′ (b 0 ). Third, R b -b 0 R S′′ (b ⋆ ) = o p (n 1/2 ).
The first and the second properties are easily shown to hold true, similarly as for

| R S′′ (b)| ≤ 6 2θ 0 n 3 sup b∈[b,b]    n k=1 γ k + 2b 0 θ 0 n W k 2 γ k + 2bθ 0 n 4 + n k=1 R W k 2 γ k + 2bθ 0 n 3    ≤ M n 3 n k=1 W k 2 + R W k 2 γ k + 2b 0 θ 0 n 3 = O p (n 1/2 )
where the second inequality, with M independent of n, results from Lemma 3(iii) of CSY and the last equality results from n k=1 (γ k + 2bθ n ) -3 ∼ const n 7/2 and Markov inequality.

Discussion

Concerning the randomized likelihood function defined in (1.8), a connection with two other works is in order. This approximation of the log-determinant has some similarities with techniques, today called path sampling, introduced by [START_REF] Ogata | A Monte Carlo method for an objective Bayesian procedure[END_REF] for implementing ML hyperparameter estimation in Gaussian (or Gaussian approximation of) Bayesian models of large size, like those encountered in image analysis. There the integral expression for a log-determinant is approximated by a weighted sum (from e.g. the trapezoidal rule of numerical integration) of simulated averages ((1/n R ) n R r=1 s -1 w r′ A θ (s)w r in our notation) on a grid of s-values, where the n R inner products may be generated by a MCMC technique.

From the symmetry of A θ (s), the expression of its eigenvalues (and those of its powers A k θ (s)) as functions of s, and the identity -b 0 (s + c) -1 ds = -∞ k=0 (1 + b -1 c) -k /k which holds for any b > 0, c > 0, it is direct to deduce that b 0 s -1 w ′ A θ (s)wds = ∞ k=0 w ′ A k θ (b)w/k. This integral can thus be computed by truncating this series, with the attractive property that computing the first k terms requires only k matrix-vector products invoking repeatedly the same A θ (b). Such a technique has already been introduced (without passing by the integral form) by [START_REF] Barry | A Monte Carlo estimator of the log determinant of large sparse matrices[END_REF] in the context of fitting conditional autoregressive (or CAR) models. Bounds for the bias incurred by truncating with few (e.g. 10) terms, and very encouraging numerical experiments analyzing the variability of these log-determinant estimators, can be found in that paper.

Concerning the choice of an algorithm to approximate the integrals by discrete sums or the choice of a truncation index in the power series approach, it seems quite hard to a priori control their impact on the statistical accuracy of the parameter estimates. To circumvent this difficulty, one may look after other methods (e.g. speudo-likelihoods) which would be able, at a reasonable cost, to produce a consistent, if not efficient, estimate, and this initial estimate would then be a starting point for a local search of an approximate root of the randomized likelihood equation (analogs of Theorems 3.1 and 3.3 might then be useful).

This deserves further study.

A simple example of multi-dimensional process is the additive model Y (s 1 , s 2 ) = Z 1 (s 1 ) + Z 2 (s 2 ) + e(s 1 , s 2 )

where Z 1 and Z 2 are two independent Gaussian processes and e represents the measurement error. CSY alluded to this example and wrote that, even if we assume the same Ornstein-Uhlenbeck probability measures (parametered by b and θ) for the two one-dimensional components, the maximum likelihood estimators for this model appear to be more elusive. On the other hand, it is known that any conditional mean A θ (b)y can be efficiently computed, even with very large n, by iterative techniques called backfitting. Thus the proposed randomized likelihood techniques are particularly suitable. It would be thus interesting to establish the asymptotic behaviors of the produced estimates of b and θ in these settings. In view of the arguments used for the proofs in Section 4.3, it is expected that, at least in the case θ 0 known, the increase of variance by the factor 1 + 1/n R should also hold.

  is an example of the so-called influence matrices in the spline literature. Now, simple manipulations of the density function (1.2) shows that -2 times the log likelihood function is l(b, θ) = y ′ [I -A θ (b)] y -logdet [I -A θ (b)] + (n + 1)log(2π) (1.4) and that its derivative w.r.t. the magnitude factor b (a score function) is S θ (b) = -1 b y ′ A θ (b) [I -A θ (b)] y -trA θ (b) .

  vector(s) w(s) is (are) assumed independent of the observation y. We first state a result on the global accuracy of the randomized likelihood function (1.8) as an approximation of the exact one on the whole domain. Since logdet[I -A θ (b)]logdet [I -A θ (b 1 )] = -b b 1 s -1 trA θ (s)ds, it is stated in the following form: Lemma 3.1. For any given b 1 > 0, for all α > 0, we have b b 1

  and in the following the occurrences of o(•), O(•) or o p (•) terms which are function of b or (b, θ) will always denote terms uniformly convergent over, respectively, fixed [b, b] or D = [b, b] × [θ, θ] with b > 0, θ > 0. Starting from their Lemma 1 and an analysis of the term of index k = 0 in the exact quasidiagonalized formula which has n + 1 terms, CSY actually gave a more general result than (4.2), concerning the score and stated in their (3.37), (3.41).


Catherine Letondal
email: catherine.letondal@enac.fr

Julien Castet
email: julien.castet@gmail.com

Pauchet Sylvain

Astrolab

Exploring the Physical Design Space for Situation Awareness and Performance in the Interactive Cockpit

Keywords: Haptics, Tangible interaction, Transformable Surfaces, Gesture-based interaction, Organic User Interfaces, Smart Material Interfaces. ACM Classification H.5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. Human Factors, Design

We explore the design space for interactive instruments in the cockpit of the future. Touch technologies are going to replace current electronic displays for flying and navigating instruments. For safety and performance reasons, interactive instruments should however maximize the perception, action and collaboration spaces of the pilots, and the literature highlights the limits of touch interaction as for these aspects. Our objective is thus to explore how the physicality of interactive technologies could address this issue. Based on a set of elicited requirements for interactive instruments in the cockpit, we explore the literature on tactile, haptic, tangible, gesture-based, organic and smart materialbased interaction along a multi-dimensional design space, based on shape, perception and programmability.

Introduction

In the past century, aircraft cockpits have heavily relied on dials, knobs and switches to provide all the flight information and retrieve the pilot input. LCDs panels have nevertheless been integrated all over the flight deck in the latest decades, making the move from the analogue to the so-called glass cockpit. Real dials are indeed replaced by digital, flexible and reconfigurable displays (Figure 1.a), which are currently not touch capable, as input is generally provided through various physical buttons. For instance, Figure 1.b highlights buttons of the Electronic Flight Instrument System panel for selecting navigation display modes. Today, as different aircraft systems have been aggregated at the data level, ongoing research tries to push this concept further and explores relying solely on large, seamless multi-touch displays, both to provide the flight information and to retrieve the pilot actions. ODICIS or Avionic 2020 (Figure 2) explore large, unique or connected continuous surfaces that will provide more synthetic and interactive views of flight and navigation data to pilots.

Although such options bear obvious advantages such as a dynamically reconfigurable cockpit, getting rid of the physical knobs and switches, it is not without introducing strong drawbacks such as the lack of references or physical feedback. Touch-based interaction 1) provides limited feedback to the user [START_REF] Potter | Improving the accuracy of touch screens: an experimental evaluation of three strategies[END_REF], either regarding the direct feedback of the command or more general feedback from the system and 2) provides a narrower action space in terms of graspability [START_REF] Fitzmaurice | Bricks: laying the foundations for graspable user interfaces[END_REF], bimanuality, 3D manipulations and more generally digital objects manipulation [START_REF] Terrenghi | Affordances for manipulation of physical versus digital media on interactive surfaces[END_REF]. In this paper, we explore how advanced interaction techniques, such as multi-touch, haptic, tangible, gesture-based, organic or smart material-based interaction, by overcoming the limitations of touch-based technologies, could better support pilot flying and navigation activity. To structure this exploration, 1) we elaborate explicit requirements for interactive instruments in the cockpit, from which 2) we draw a set of candidate dimensions for a design space.

Requirements

The relevant requirements for interactive flying and navigation tools are being iteratively elaborated through previous work (e.g. [START_REF] Vinot | Legible, are you sure?: an experimentation-based typographical design in safetycritical context[END_REF]), user studies (2 contextual interviews, 2 workshops) with pilots and instructors. They belong to three categories: 1) cognitive and collaborative concerns, 2) safety, and 3) industrial constraints.

Cognitive and Collaborative Requirements R.C.1. Direct localization perception. Even for experts, as told by a pilot we interviewed, flying feels like being in a "non-usual" universe: you are in the middle of the sky, without any landmark, so you do not immediately know "where you are". For this purpose, pilots need devices providing few and directly available parameters (altitude, speed, heading, artificial horizon).

R.C.2. Situation awareness.

Pilots need to be able to both get information related to the context of the flight (e.g. closest air traffic) and to predict the state of the system with respect to e.g. autopilot, in order to avoid cognitive conflicts and anticipate actions [START_REF] Endsley | Toward a Theory of Situation Awareness in Dynamic Systems Human Factors[END_REF]. R.C.5. Degraded context. In the cockpit, both environmental and cognitive factors can dramatically degrade the performance of human operators. For example, extreme lighting conditions, vibrations or degraded flying conditions (weather, aircraft failure), but also cognitive overload (fatigue, stress, time pressure), might greatly downgrade efficiency. This requires to include adapted interactive solutions, i.e. efficient in degraded contexts, from the start of the design process. Safety Requirements R.S.1. Safety-Critical System. The certification process imposes to comply specific standards, requirements or processes such as RTCA DO-178B on safety of software used in airborne systems [START_REF]B / EUROCAE ED-12B, Software Considerations in Airborne Systems and Equipment Certification[END_REF], and to use formal methods to describe advanced interactions in the cockpit [START_REF] Hamon | Formal description of multi-touch interactions[END_REF].

R.S.2.

Resilience. The robustness of interactive systems is required in relation to various potential breakdowns or vulnerabilities regarding power, light, visibility (smoke), noise, etc. Techniques and modalities used (e.g. for alarms) must support critical context and enable alternative modes.

R.S.3. Availability.

Redefining the cockpit interactive components forces us to re-examine availability and accessibility issues. For instance, interface elements must be dynamically adapted to the phase of the flight. Mobile, or detachable objects raise a challenge both as potential dangerous projectiles and as unreachable elements.

Industrial Requirements

This last category encompasses practical requirements related to industrial and commercial constraints.

Design space

Our aim is to explore the tangible and embodied -the "physical" -design space as structured according to our requirements. From these requirements, a set of design principles might be drawn, that we try here to map to the properties of specific dimensions in this design space. where cost-effective (R.I.2) interface components show a high degree of configurability (R.I.3) and adaptability (R.S.3). So, the three candidate dimensions that emerge from our requirements would be: shape, embodied perception and action, and programmability (see the summarizing table provided as supplemental material).

Device Shape Properties

Although perception is already properly addressed by numeric technologies (e.g. visual perception), sticking with a flat input surface implies missing TPK properties supporting critical information available in former analog systems through eyes-free interaction and graspability (R.C.4, R.S.2). Shapes are also relevant for collaborative and contextual awareness [START_REF] Hornecker | Getting a grip on tangible interaction: a framework on physical space and social interaction[END_REF] (R.C.2, R.C.3), as during the access of distant system: pilots are able to perceive a changed position for a salient device more easily than a change on a distant display area. Non-flat shapes have thus been explored, with a focus on either input or output.

INPUT Cubtile [START_REF] De La Rivière | CubTile: a multi-touch cubic interface[END_REF] is a cubic input device that enables to manipulate 3D models. It strongly relies on multitouch and bimanual gestures to close the gap between digital manipulation and its physical counterpart. Touché [START_REF] Sato | Touché: enhancing touch interaction on humans, screens, liquids, and everyday objects[END_REF] enables tactile input on any object, including for instance the knob of a door. Shape transformation can be used as input as in Gummi [24]. Tangible User Interfaces (TUI) [START_REF] Ullmer | Emerging Frameworks for Tangible User Interfaces[END_REF], use physical objects to interact with digital systems. However, R.S.3 challenges tangible devices as mobile and potentially dangerous projectiles. Even vertical and adhesive devices such as [START_REF] Leitner | Geckos: combining magnets and pressure images to enable new tangibleobject design and interaction[END_REF] or [START_REF] Hennecke | Vertibles: using vacuum self-adhesion to create a tangible user interface for arbitrary interactive surfaces[END_REF] is difficult to certify. OUTPUT Tilt Displays [START_REF] Alexander | Tilt displays: designing display surfaces with multi-axis tilting and actuation[END_REF] are display surfaces with multi-axis tilting and actuation, providing various shapes according to the context. The Emirai cockpit concept of Mitsubishu (Figure 4) provides both non-flat graspable input capabilities through a curved interactive surface and dynamic output through physical non-flat buttons.

Embodied perception and action

Interaction may benefit from a better use of perceptive channels by either being able to rely on non-visual information in case of degraded visibility or extreme lighting conditions (R.C.5, R.S.2), or for performance (R.C.4) and directness (R.C.1), in order to be able to use multimodal information. Fusion of this information is a major factor to obtain an improvement of the perceptive precision [START_REF] Graziano | Coding the location of the arm by sight[END_REF][25] and of the motor control [START_REF] Prablanc | Optimal response of eye and hand motor systems in pointing at a visual target[END_REF] [START_REF] Rossetti | Vectorial coding of movement: vision, proprioception, or both[END_REF].

ARTIFICIAL HAPTICS STIMULATION Haptic stimulations may respectively be achieved through different devices. The force-feedback and the tactile devices address respectively kinesthetic and tactile perceptions. Reverse electrovibration [2] provides haptic feedback on any object. Perceptions are overall differentiated by frequency and force capabilities. Today, one of the challenges associated to these artificial stimulations is to integrate this duality in a single efficient device. This association should ensure a high dexterity to users by using all capabilities of hands [START_REF] Zigelbaum | SpeakCup: Simplicity, BABL, and Shape Change[END_REF].

TANGIBLE INTERACTION

Tangible interaction principles [START_REF] Fitzmaurice | Bricks: laying the foundations for graspable user interfaces[END_REF] [START_REF] Hornecker | Getting a grip on tangible interaction: a framework on physical space and social interaction[END_REF], which rest in the reuse of physical space and objects for interaction, fully apply for the design of the interactive cockpit, as shown in [START_REF] Nomura | The uses of paper in commercial airline flight operations[END_REF]. Analyses such as [START_REF] Klemmer | How bodies matter: five themes for interaction design[END_REF] explain why tangible interaction results in better performance (R.C.4) and [START_REF] Hornecker | Getting a grip on tangible interaction: a framework on physical space and social interaction[END_REF] describes that properties such as non-fragmented visibility and performative actions would be particularly suited for the cooperation between pilots (R.C.3). SEMIOTIC GESTURES Interaction metaphors on touchscreen are generally inspired by physical manipulations. Actually combining these solutions with semiotic gestures could be relevant for awareness (R.C.3) (R.C.4) by interacting without obstructing the view of the other pilot. Gestures may also be used to provide embodied proprioceptive perception: based on [START_REF] Vernier | MultiTouch Widgets[END_REF], we conducted a preliminary exploration for

IMAGINARY INTERFACES

Palm-based interfaces, as demonstrated in [START_REF] Gustafson | Understanding palm-based imaginary interfaces: the role of visual and tactile cues when browsing[END_REF], combine finger tips and palm perception to achieve efficient interactions such as selection (R.C.1, R.C.4), even in blind or poor light condition (R.C.5). Furthermore, they are accessible and non-detachable (R.S.3).

Virtual and Physical Programmability

Software-based components bring industrial benefits related to cost (R.I.2, R.I.3) and provide both rich contextual information and feedback (R.C.2). For instance, SV-PFD display is a synthetic geographical 3D view that merges aircraft and navigation data to reduce perceptual distance (R.C.1). Advanced technologies now let hardware elements show the same degree of contextual dynamicity. [START_REF] Terrenghi | Affordances for manipulation of physical versus digital media on interactive surfaces[END_REF][19] is an emerging paradigm where the feedback of an interaction is extended to the physical device. Actuated surfaces can dynamically change their output shape such as in Pneumatic Displays [START_REF] Harrison | Providing Dynamically Changeable Physical Buttons on a Visual Display[END_REF].

ORGANIC USER INTERFACES Organic User Interfaces

SMART MATERIAL INTERFACES (SMI)

Smart Material Interfaces (SMI) [START_REF] Vyas | Smart material interfaces: a new form of physical interaction[END_REF] take advantage of recent generations of engineered materials that have capabilities of altering physical properties such as shape, texture and colour. SMI explore how to use material properties as programmable features for enhancing interaction. As opposed to tangible interfaces, where coherence might be an issue, especially for (R.C.5), they exhibit a coherent information space [START_REF] Minuto | Smart Material Interfaces: A Vision[END_REF]. SMI can for instance change their shape as in SpeakCup [START_REF] Zigelbaum | SpeakCup: Simplicity, BABL, and Shape Change[END_REF] or Sprout I/O [START_REF] Coelho | Sprout I/O: a texturally rich interface[END_REF].

Conclusion

In this paper, we co-articulate explicit requirements and physical properties of a large set of interactive technologies along three axes that, according to us, best describe the features that we need. From this design space, our aim is to iteratively both design a real prototype for the interactive cockpit -hence I.S. (R.S.1) and industrial constraints (R.I.*) -and refine our requirements. We will start with the participatory design of demonstrators that explore and combine described features in order to produce new interaction techniques, and as a mean to better understand pilot needs. For instance, combining smart materials with gesture-or body-based interaction, or combining shape transformation with haptics, could bring interesting insights for evaluating their potential complementarity.

R.C. 3 .

 3 Collaborative awareness: Major procedures must be collaboratively performed and each individual pilot action needs to be shareable and visible to the other. R.C.4. Operational Performance. Devices and interactors should show a high degree of usability, for instance to enable the selection of discrete values (e.g. button with notches, Figure 3.top) while not requiring too much focus; these devices should provide perceptible feedback (e.g. force feedback of the gas throttle) or on the opposite prevent non reversible action (e.g. guarded buttons, Figure 3.bottom).

Figure 2 :

 2 Figure 2 : Avionic 2020 tactile screens.

Figure 1 .

 1 Figure 1. a) Navigation and flight displays in the cockpit of a TB20; b) buttons of the EFIS (Electronic Flight System) control panel that control the display mode of the navigation display (ND) (2 nd from the left).

R.I. 1 .

 1 Dynamicity and adaptability. Interactive components should dynamically adapt to numeric information flow (hence the progressive introduction of software components). R.I.2. Development. Display and flight components should be less expensive to design, develop and certify. R.I.3. Configurability. To reduce costs, suppliers promote the design of new product lines adaptable to different aircraft programs. Industrials thus seek technologies that are flexible enough to enable inexpensive components reuse and reconfiguration.

First, several

  requirements related to easy access (R.C.1, R.S.3) graspability (R.C.4, R.S.2) and eyes-free interaction (R.C.5, R.S.2, R.S.3), advocates for a dimension on form [11] and device shape. Second, interaction techniques based on embodied perception and action, in particular TPK-based perception (tactile, proprioceptive and kinesthetic), meet requirements related to performance (R.C.1, R.C.4), awareness (R.C.2) and degraded context (R.C.5, R.S.2). Third, programmability, even for physical objects, follows current development toward providing more dynamic software-based contextual information (R.I.1, R.C.2),

Figure 3 :

 3 Figure 3 : (top) Airbus FCU physical buttons (Flight Control Unit); (bottom) overhead panel: guarded buttons.

Figure 4 :

 4 Figure 4 : Mitsubitshu Emirai cockpit concept