Karen Tatarian

Rebecca Stower

Damien Rudaz

Marine Chamoux

Arvid Kappas

Mohamed Chetouani

How does Modality Matter? Investigating the Synthesis and Effects of Multi-modal Robot Behavior on Social Intelligence

Keywords: Multi-modal interaction, Gestures, Gaze, Proxemics, Social Cues, Social Intelligence, Nonverbal Behavior, Human Robot Interaction

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

By merely observing humans, one can directly infer that no social interaction takes place without cues, whether verbal or nonverbal, that allow others to interpret behaviors and reasonably estimate intentions [START_REF] Schegloff | Analyzing single episodes of interaction: an exercise in conversation analysis[END_REF]. Furthermore, those verbal and nonverbal cues have an effect on others by eliciting tangible change in their observable behavior or even internal changes, e.g., awareness of a particular social setting [START_REF] Vinciarelli | New social signals in a new interaction world: the next frontier for social signal processing[END_REF]. Moreover, proper communication and exchange of information is crucial to a human's need to feel connected, promote well-being, and gain acceptance by social groups [START_REF] Sandstrom | Social interactions and well-being: The surprising power of weak ties[END_REF]. However, these powerful social signals and nonverbal behaviors are complex and multi-modal. They are made of different combinations of modalities and cues such as kinesics (e.g., gestures) ( [START_REF] Kong | A coding system with independent annotations of gesture forms and functions during verbal communication: Development of a database of speech and gesture (dosage)[END_REF], [START_REF] Ekman | The repertoire of nonverbal behavior: Categories, origins, usage, and coding[END_REF]), gaze behavior [START_REF] Kendon | Some functions of gaze-direction in social interaction[END_REF], and proxemics (e.g., management of space and environment) [START_REF] Hall | A system for the notation of proxemic behavior1[END_REF]. Similarly, these multi-modal nonverbal behaviors hold several functions, which include the ability to understand and manage others in social interactions and "act wisely in human relations", and as such contribute to one's social intelligence. [START_REF] Thorndike | Intelligence and its use[END_REF].

In today's world, humans not only have to interact with each other, but also with machines, including robots. With robots gaining further presence in a human's everyday life, synthesizing and understanding these multi-modal behaviors is crucial to designing better and more appropriate human-robot interactions. In an attempt to solve this issue, some studies have been inspired by human-human interaction to design rule-based algorithms targeted at investigating individual modalities ( [START_REF] Andrist | Conversational gaze aversion for humanlike robots[END_REF], [START_REF] Shi | Spatial formation model for initiating conversation[END_REF], [START_REF] Liu | Generation of nodding, head tilting and eye gazing for human-robot dialogue interaction[END_REF]). In contrast, other research has focused on data-driven learning methods designed to synthesize multi-modal behavior, however, lacking a clear understanding of the effect of each modality forming the multi-modal behavior ( [START_REF] Kucherenko | Data driven non-verbal behavior generation for humanoid robots[END_REF], [START_REF] Qureshi | Robot gains social intelligence through multimodal deep reinforcement learning[END_REF]). Thus, there is still a need to investigate how such modalities can be combined and the effect and function of each when performed in a multi-modal combination. This will allow for a better understanding of how and when the robot could use combinations of different modalities to appear as a socially intelligent agent and express intentions and information using verbal and nonverbal behavior more naturally.

This paper presents a system of multi-modal behaviors comprised of the following modalities: gaze, kinesics, proxemics, and social dialogue. The system was evaluated objectively by studying the behavioral outcomes. In addition, critical to evaluating peoples interactions with social robots is also the extent to which they like the robot, or form a general positive impression during their interaction. Many studies in HRI capture liking directly, by asking questions such as 'I like [the robot]' ( [START_REF] Torrey | How a robot should give advice[END_REF], [START_REF] Strait | Let me tell you! investigating the effects of robot communication strategies in advice-giving situations based on robot appearance, interaction modality and distance[END_REF], [START_REF] Mumm | Human-robot proxemics: Physical and psychological distancing in human-robot interaction[END_REF]), whereas others take a more indirect approach, assessing statements such as '[the robot] is kind / friendly / warm' ( [START_REF] Leichtmann | How much distance do humans keep toward robots? Literature review, meta-analysis, and theoretical considerations on personal space in humanrobot interaction[END_REF], [START_REF] Salem | To Err is Human(-like): Effects of Robot Gesture on Perceived Anthropomorphism and Likability[END_REF]). Thus, in this study we also included a subjective measure of liking through a self-report "liking" scale 1 . Instead of analyzing the effects of each modality by contrasting them in isolation and thus losing possible coupling effects, this paper compares a version with all implemented modalities together with versions, in which each modality is subtracted in turn.

The remaining paper is organized as follows: Section 2 reviews background work done on each of the modalities to be integrated. In Section 3, the implemented system of multi-modal behavior is discussed. Section 4 presents the hypotheses as well as the design and set-up of the evaluation study. In Sections 5 and 6, the results are presented and analyzed and the findings are discussed in more depth. In Section 7 the paper is concluded.

Background

In the past decades, there has been considerable interest among psychologists and sociologists to investigate non-verbal behaviors observed in humans and used as communication methods and tools. Inspired by those findings, more recently, social roboticists have tried to synthesize these modalities on different robots in order to study their impact on human-robot interaction (HRI). These modalities include gaze, kinesics, and proxemics. This section highlights the human-human studies done in addition to the HRI research for each modality as well as social dialogue.

Gaze

In 1967, Kendon [START_REF] Kendon | Some functions of gaze-direction in social interaction[END_REF] was the first to classify and analyze gaze aversion in human-human interaction, claiming that humans in fact do not spend the majority of their time in a conversation directing gaze straight at another human's face. He concluded that gaze aversion was done for four primary reasons: turn-taking, turn-yielding, floorholding, and intimacy regulation (used to regulate the level of shared emotional arousal) ( [START_REF] Kendon | Conducting Interaction: Patterns of Behavior in Focused Encounters[END_REF], [START_REF] Vertegaal | Eye gaze patterns in conversations: There is more to conversational agents than meets the eyes[END_REF], [START_REF] Turkstra | Interactive behaviors in adolescent conversation dyads[END_REF]). Today, gaze mechanisms, including gaze aversion, are still a study of interest for social roboticists. Research in HRI has involved conducting studies to better understand social gaze ( [START_REF] Mutlu | Conversational gaze mechanisms for humanlike robots[END_REF], [START_REF] Andrist | Conversational gaze aversion for humanlike robots[END_REF]), using gaze to reference an object of conversation by joint attention ( [START_REF] Kirchner | Nonverbal robot-group interaction using an imitated gaze cue[END_REF], [START_REF] Andrist | Designing effective gaze mechanisms for virtual agents[END_REF]), designing gaze cues to modulate group conversation ( [START_REF] Mutlu | Conversational gaze mechanisms for humanlike robots[END_REF], [START_REF] Mutlu | Footing in human-robot conversations: How robots might shape participant roles using gaze cues[END_REF]), and regulating turn-taking in conversations ( [START_REF] Andrist | Conversational gaze aversion for humanlike robots[END_REF], [START_REF] Mutlu | Footing in human-robot conversations: How robots might shape participant roles using gaze cues[END_REF]). In addition, conversational social gaze constructed of gaze aversions to perform role-signaling, turn-taking, and topic-signaling prompted high indices of likeability towards the robot [START_REF] Mutlu | Conversational gaze mechanisms for humanlike robots[END_REF]. Moreover, for robots which lack expressive eyes, head controlled tilts have been designed to convey gaze aversion [START_REF] Andrist | Conversational gaze aversion for humanlike robots[END_REF]. The former concluded that while social gaze aversions did not increase the human's comfort in eliciting more self-disclosure, it did decrease interruption time caused by the user and the robot was perceived as more thoughtful. Additionally, the study analyzed the direction of gaze aversions in human-human interactions with respect to its three primary functions: cognitive, intimacy-regulation, and floor management [START_REF] Andrist | Conversational gaze aversion for humanlike robots[END_REF].

An additional important function of gaze is joint attention. It supplies people with a way of interpreting and predicting each other's actions and focus attention [START_REF] Grosz | Attention, intentions, and the structure of discourse[END_REF]. For instance, speakers tend to use deictic expressions followed by a glance towards the object of reference [START_REF] Clark | Speaking while monitoring addressees for understanding[END_REF]. Thus it is no surprise that joint attention attracted the attention of researchers in the HRI field. For instance, it was shown in [START_REF] Boucher | I reach faster when i see you look: Gaze effects in human-human and human-robot face-to-face cooperation[END_REF] that users reached objects faster when they could follow the gaze of the robot iCub, who was giving instructions while glancing at referenced object. Similarly, [START_REF] Skantze | Turn-taking, feedback and joint attention in situated human-robot interaction[END_REF] showed that users interacting with a robot that had a gaze with a reference function found it easier to complete a task than with a robot that had random gaze. Joint attention has proven to be functional for social robots to shift the human's attention to the spot at which it is looking [START_REF] Yonezawa | Gazecommunicative behavior of stuffed-toy robot with joint attention and eye contact based on ambient gaze-tracking[END_REF]. In addition, for collaboration tasks involving object selection, robot gaze shifts assisting its speech were shown to be advantageous for cooperation specially when the human was required to choose the object being referred to by the robot as fast as possible ( [START_REF] Admoni | Speech and gaze conflicts in collaborative human-robot interactions[END_REF], [START_REF] Boucher | I reach faster when i see you look: Gaze effects in human-human and human-robot face-to-face cooperation[END_REF]). Furthermore, in [START_REF] Moon | Meet me where i'm gazing: How shared attention gaze affects human-robot handover timing[END_REF], during hand over tasks, users started reaching for an item much sooner when a robot consistently gazes at the handover area than when it gazes away from that point. In parallel, when gaze is used as part of a multi-modal behavior, it often has a supportive and enhancing role to other social behaviors, notably speech and gestures [3].

Kinesics

Gestures for humans have been categorized and defined primarily based on their role in communication and their functions as follows ( [START_REF] Mcneill | Hand and Mind: What Gestures Reveal about Thought[END_REF], [START_REF] Kong | A coding system with independent annotations of gesture forms and functions during verbal communication: Development of a database of speech and gesture (dosage)[END_REF], [START_REF] Akbıyık | The relationship between co-speech gesture production and macrolinguistic discourse abilities in people with focal brain injury[END_REF], [START_REF] Ekman | The repertoire of nonverbal behavior: Categories, origins, usage, and coding[END_REF]):

-Iconic gestures for describing physical objects and events mentioned in a conversation; e.g., forming a small circle with the hand to refer to a small ball. Gestures have also been studied and implemented on robots aiming to improve human-robot interaction. While deictic, beat, iconic, and metaphoric gestures were all found to boost the robot's performance as a narrator in a narrative scenario, deictic gesture significantly ameliorated the user's recall of information on the story [START_REF] Huang | Modeling and evaluating narrative gestures for humanlike robots[END_REF]. Additionally, the robot which performed correctly timed nods in a conversation and proper gaze and gesture sharing behaviors was ranked more highly than a robot who did not have such behaviors [START_REF] Kanda | A humanoid robot that pretends to listen to route guidance from a human[END_REF]. Moreover, gestures play a role in portraying emotional expressions. For instance, submissiveness can be expressed by an open hand shape; on the other hand, dominance can be portrayed in a pointing hand shape [START_REF] Kipp | Gesture and emotion: Can basic gestural form features discriminate emotions?[END_REF]. Similarly in social robotics, modulating the robot's body movement by varying its head tilts and body expansiveness influenced perceived dominance [START_REF] Peters | Robots expressing dominance: Effects of behaviours and modulation[END_REF].

Another important aspect of social interaction is alignment, which refers to the convergence of linguistic behavior and/or similarity in mental representation ( [START_REF] Pickering | Toward a mechanistic psychology of dialogue[END_REF], [START_REF] Branigan | Linguistic alignment between people and computers[END_REF]). Alignment is an ubiquitous feature used to measure to which extent interactions shape behavior and their success at communicating shared understanding [START_REF] Hasson | Mirroring and beyond: coupled dynamics as a generalized framework for modelling social interactions[END_REF]. For instance, it was shown that alignment, through mimicry of postures, mannerisms, and facial expressions in dyadic interactions (chameleon effect [START_REF] Chartrand | The chameleon effect: the perception-behavior link and social interaction[END_REF]), increased the rapport between the participants, the pro-social behavior even beyond the interaction and smoothness of interaction [START_REF] Van Baaren | Mimicry and prosocial behavior[END_REF]. Moreover, a study found that users, when retelling a story to a third participant, were more likely to demonstrate the same iconic gestures they witnessed the first time [START_REF] Mol | Alignment in iconic gestures: Does it make sense?[END_REF]. Alignment equally plays an important role in human-computer interaction in enhancing communicative success [START_REF] Branigan | Linguistic alignment between people and computers[END_REF]. In robotics, people have been found to nod more when interacting with a robot that nods along in response to that in comparison to a robot who does not mirror their nodding [START_REF] Sidner | The effect of head-nod recognition in human-robot conversation[END_REF]. In addition, a computational method for evaluating and modeling of interpersonal synchrony in behaviors during interactions offered a perspective for building social interfaces for robots and embodied conversational agents [START_REF] Delaherche | Interpersonal synchrony: A survey of evaluation methods across disciplines[END_REF]. Furthermore, motor resonance, which is the activation of the observer's motor control system during action perception, was used to not only produce more natural interactions for robots with humans but also as an evaluation method to determine quantitatively how the robot is perceived by the human [START_REF] Sciutti | Measuring human-robot interaction through motor resonance[END_REF].

Proxemics

Proxemics, which is the study of space around a person with respect to others, was first defined by Hall [START_REF] Hall | The silent language / Edward Hall[END_REF]. It was found that proxemic zones are shaped by culture and psychophysical features ( [START_REF] Hall | A system for the notation of proxemic behavior1[END_REF], [START_REF] Hall | The Hidden Dimension[END_REF], [START_REF] Hall | for the Anthropology of Visual Communication, S.: Handbook for proxemic research. Studies in the anthropology of visual communication[END_REF]). Schegloff [START_REF] Schegloff | Body torque[END_REF] has shown that, in daily interactions, intentions are derived from the poses of the lower and upper parts of the human body, i.e. whether the involvement of the participant would be dominant or subordinate. Moreover, situational awareness is the ability to understand and perceive the environment around the person in order to plan and execute decisions and such relies on proxemics [START_REF] Endsley | Toward a theory of situation awareness in dynamic systems[END_REF]. Thus, proxemics plays an important role in defining human-human interactions and relationships.

In the field of HRI, proxemics is vastly used for social navigation. For instance, [START_REF] Kruse | Exploiting human cooperation in human-centered robot navigation[END_REF], [START_REF] Dautenhahn | How may i serve you? a robot companion approaching a seated person in a helping context[END_REF] used Hall's theory of proxemics to optimize social navigation of the robot while taking into account the human's safety and visibility. Additionally, proxemics can be used to initiate interactions. Shi et al. [START_REF] Shi | Spatial formation model for initiating conversation[END_REF] proposed a model based on proxemics and navigation to initiate a conversation with a human inspired by the study of human-human interactions. The robot with the implemented proxemics model was ranked higher in a subjective evaluation of appropriateness of initiation. An understanding of proxemics grants the robot a finer tool to perceive, predict, and manipulate the environment around the interaction and provides greater naturalness [START_REF] Shi | Spatial formation model for initiating conversation[END_REF]. Furthermore, proxemics was used to estimate group formations around the robot allowing the robot to adapt its gaze based on the roles users are playing the group being formed, such as active participant, bystander, or overhearer [START_REF] Tatarian | Robot gaze behavior and proxemics to coordinate conversational roles in group interactions[END_REF]. In addition, a recent meta-analysis showed that for mobile robots, robot appearance or whether one is active or passive in the approach has no meaningful effects [START_REF] Leichtmann | How much distance do humans keep toward robots? literature review, meta-analysis, and theoretical considerations on personal space in humanrobot interaction[END_REF]. A pivotal question is whether proper use of this modality, which focuses on interpersonal space, would lead to stronger effects related to the outcomes of an interaction when compared to other modalities such as gaze and gesture.

Dialogue

In addition to the modalities of nonverbal behaviors, dialogue plays an influential role in forming impressions and manipulating social outcomes of the interaction. In human-human interactions, one study has shown that starting a conversation by asking people how they were feeling that day increases the likelihood of their compliance to a request for both charity donations and/or commercial purchases [START_REF] Dolinski | Dialogue involvement as a social influence technique[END_REF]. Moreover, in human-agent interactions, it was shown that having an agent start with a small request increased the chances of having the participant accept a bigger request shortly after [START_REF] Eastwick | Is it a game? evidence for social influence in the virtual world[END_REF]. Furthermore, verbal phrases influence social interactions with agents, e.g., separating emotional expressions targeted at an attitude versus at a person such as"your opinion" versus "you should" [START_REF] Wang | Effects of emotional expressions on persuasion[END_REF]. In HRI, dialogue similarly has an impact on the interactions including facilitating collaborations, managing errors, and personalizing conversations. For instance, it is used to exchange information and assist in human robot collaboration to achieve common goals [START_REF] Fong | Collaboration, dialogue, human-robot interaction[END_REF]. Furthermore, social dialogue was shown to help robots recover from prior errors and gain future influence [START_REF] Lucas | Getting to know each other: The role of social dialogue in recovery from errors in social robots[END_REF]. Additionally, service robots with personalized dialogues reinforced participants' rapport, cooperation, and engagement [START_REF] Lee | Personalization in hri: A longitudinal field experiment[END_REF].

Multi-Modal Social Behaviors

Several studies have also combined two or more of these behaviors to assess the interaction between them. Most commonly, gaze behaviours are combined with gestures [START_REF] Ham | Making Robots Persuasive: The Influence of Combining Persuasive Strategies (Gazing and Gestures) by a Storytelling Robot on Its Persuasive Power[END_REF], proxemics ( [START_REF] Tatarian | Robot gaze behavior and proxemics to coordinate conversational roles in group interactions[END_REF], [START_REF] Travis | Effects of Robotic Social Cues on Interpersonal Attributions and Assessments of Robot Interaction Behaviors[END_REF], [START_REF] Fiore | Toward understanding social cues and signals in human -robot interaction : effects of robot gaze and proxemic behavior 4[END_REF]) or verbal behaviours [15]. Works combining other modalities ( [START_REF] Saad | Welcoming robot behaviors for drawing attention[END_REF], [START_REF] Peters | Robots educate in style: The effect of context and non-verbal behaviour on children's perceptions of warmth and competence[END_REF]), or comparing more than two or three modalities ( [START_REF] Birnbaum | Machines as a source of consolation: Robot responsiveness increases human approach behavior and desire for companionship[END_REF], [START_REF] Kennedy | The robot who tried too hard: Social behaviour of a robot tutor can negatively affect child learning[END_REF]) are rare. There is also a large body of work comparing 'social' and 'non-social' robots (or affective / emotional / personalized robots). However, the manipulations in these cases typically combine multiple modalities and evaluate overall system performance, as opposed to investigating the effect of specific modalities. As such, there is a clear need to develop a more comprehensive perspective on how different combinations of modalities (gaze, gesture, proxemics, and verbal content) contribute to overall perceptions of social intelligence during the course of an interaction.

Implemented System for Multi-Modal Behavior

This section introduces the implemented system to achieve multi-modal behavior on the Pepper robot (Soft-Bank Robotics). However, the system can also be implemented on other social robots. The source code for Fig. 2 Sample of time-line including speech, gaze mechanisms (turn-taking, floor-holding, turn-yielding), and social gestures (deictic gestures: "You" vs "Me" if mentioned in speech, beat gesture: emphasizing the two choices user needs to select from)

the entire system has been made available online 2 . The overall scheme of the multi-modal social cues synthesized are shown in Figure 1. The system is composed of the following modalities: proxemics, gaze mechanism, gestures, and a social dialogue. A sample extract of the system implemented is shown in Figure 2.

Social Gaze Mechanisms

Since several humanoid robots lack expressive eyes that can be controlled, the presented social gaze aversions are achieved using head motion control. The social gaze mechanisms presented here were designed and implemented to fulfill the following functions: joint attention, turn-taking, floor-holding, and turn-yielding. When not performing these gaze aversions, the robot would be gaze tracking the human it is interacting with at all times. For this reason, information about the human can be extracted, notably the 3-dimensional frames of the human's face and the robot's gaze. This allows the robot to carry out all implemented gaze-averted head motions on the robot with respect to the frame of the human's face. Thus the design of each social gaze head movement was a combination of dynamics, magnitude, and duration, all of which are crucial for the social gaze motion to achieve its function naturally. A summary of the gaze mechanisms can be found in Figure 3.

Gaze aversion for turn-taking in human-human interaction as well as human-robot interaction holds a cognitive function; it gives the speaker more time to better plan and address their speech while also avoiding 2 Multi-modal Social Cues System Implementation GitHub Repository https://github.com/KarenTatarian/multimodal_ socialcues possible external distractions [START_REF] Argyle | Gaze and Mutual Gaze[END_REF], [START_REF] Andrist | Conversational gaze aversion for humanlike robots[END_REF]. For this reason, the turn-taking gaze behavior was given the relatively longer duration of 2.5 seconds. As for the floor management and turn-yielding functions, which take place during and at the of the speaking turn, they were assigned a shorter duration of 1.5 and 1.2 seconds, respectively. The longest duration was designed for the joint attention gaze, which has the duration of 3.8 seconds and the function of indicating and referring to an object of discussion. The angle rotations, duration length, and directions of the gaze aversions were selected and designed based on the gaze aversion system, with similar functionalities, implemented on the NAO robot (SoftBank Robotics) in [START_REF] Andrist | Conversational gaze aversion for humanlike robots[END_REF], which are as well based on the findings of Kendon [START_REF] Kendon | Conducting Interaction: Patterns of Behavior in Focused Encounters[END_REF] for gaze aversions in human-human interaction. In turn, these variables were tuned to be suitable for the robot Pepper. Examples of how the duration of each gaze mechanism was synthesized is shown in Figure 2.

Fig. 3 Summary of gaze mechanisms

The goal is to generate head motions, which serve as social gaze cues, and to realize a natural and subtle appearance while still fulfilling their purpose. The directions of the head rotations performed were chosen based on human-human as well as HRI studies done to better understand gaze aversions during interactions ( [START_REF] Andrist | Conversational gaze aversion for humanlike robots[END_REF] [36] [START_REF] Mutlu | Conversational gaze mechanisms for humanlike robots[END_REF]). For the turn-taking and floor-holding cues, which signify that the robot is in a cognitive phase and holding the speaking role, a vertical head movement was implemented. The gaze frame of the robot is the frame it produces as it is tracking the human's face as seen in Figure 3, thus the gaze frame is aligned with the human's face frame. At each instance in which the turn-taking and floor-holding gaze mechanisms were called, the gaze frame performs a rotation of 33.7 degrees in the (y-z) plane with respect to the frame on the human's face resulting in an upward slightly leftsided rotation. As for the turn-yielding gaze, in order to achieve a smoother and more subtle gaze to indicate the shift in speaking roles and after manually fine tuning, Equation 1 represents the translations implemented for the (x, y, z) of the robot's gaze frame respectively

x" = [(1 -cos θ)]x + sin θy -δ 1 y" = -sin θx + (1 -cos θ)y z" = δ 2 (1)
where θ = 12.318°, δ 1 = 1.2, and δ 2 = -0.6 and x, y, &z are the coordinates of the frame of the human's face.This resulted in a downward left-sided rotation as seen in Figure 4, which illustrates an example of the execution. Finally, the head rotation for joint attention was not intended to be subtle but rather draw the attention of the user towards the object it is referring to. The head rotation was done with a head yaw of -46°and a head pitch of -4.65°with respect to the object it is referring to and with a Bézier curve velocity profile for smoothness in the animation. This resulted in an animation that starts slow, speeds up during the main shift, and finally slows down before it ends. For instance, for the gaze mechanisms implemented as head-motions, the head of the robot would start rotating slowly then rapidly towards target frame then slowly returns back to its initial gaze frame as shown in the example of Figure 4.

Gestures

Three of the five categories of gestures classified in [START_REF] Mcneill | Hand and Mind: What Gestures Reveal about Thought[END_REF], [START_REF] Kong | A coding system with independent annotations of gesture forms and functions during verbal communication: Development of a database of speech and gesture (dosage)[END_REF], are implemented in this multi-modal system through five gestural designs. First, in the emblem category, two gestures were implemented: the wave gestures for greeting users and closing interaction and nodding as back-channeling. When the robot enters a listening state, it nods in response to additional information the user provides while speaking. Whenever a new sentence is detected by the robot, it executes a back-channeling nod animation and waits for a next sentence to be heard. The back-channeling nod animation consists of three consecutive nods over a span of 2.5 seconds. Second, in the deictic category, two types were represented: object pointing gesture, which accompanied the joint attention gaze and performed as a right arm extension, and self/other references, which are the "you" vs "me" indicators, as portrayed in Figure 2. In the latter, the "you" gesture was designed as both of the arms open and extended from the robot's body and pointing upwards towards the human with the open palms. The "you" gesture accompanied questions,in which the robot infers about the user. On the other hand, the self-pointing or the "me" gesture has the robot's arms closed and directed towards the body and it is used by the robot to introduce itself, its tablet modality, and state an opinion. Furthermore, beat gestures were used in the question series to emphasize the options. In order to generate smooth behavioral social cues, Bézier curves were used to implement the velocity changes of all gestures.

Proxemics for Approaching Human

According to the findings in ( [START_REF] Sisbot | A human aware mobile robot motion planner[END_REF], [START_REF] Kruse | Exploiting human cooperation in human-centered robot navigation[END_REF]), the most optimal and socially acceptable path to navigate towards a static standing human is in a straight line in the human's sight zone while maintaining a minimum distance of about 0.5 meters (or 1.5 meters for sitting humans). This is due to safety and visibility constraints to minimize human discomfort. In addition, approaching the human for initiating an interaction was ranked as most appropriate when the robot navigated in the human's front zone (120°c one-shaped area in front of a person's head) or his/her gaze zone (30°) [START_REF] Shi | Spatial formation model for initiating conversation[END_REF]. For this system, once the human has been detected at the beginning of the interaction, Fig. 5 Schematics of the experimental room set-up with the robot during the travel agent scenario the robot would first greet the human by speech and a wave gesture before beginning navigation towards the human. The default speed of the robot Pepper is 0.35 m/s. However to avoid a recoil movement by the human seen in some user experience testing, the speed for greeting navigation was slowed down to 0.25 m/s. The robot would then navigate to establish a distance of 0.85 meters between itself and the user. The robot would navigate while maintaining gaze directed towards the participant. The distance of 0.85 meters was chosen for four main reasons. First, this distance allows the robot to continuously track the human's face regardless of their height. Second, the human at this distance is able to clearly see the different gaze and gesture behaviors generated by the robot. Third, this distance eases the access and view of the tablet on the robot. Fourth, 0.85 meters is still within Hall's defined personal distance, in which friendly interactions take place [START_REF] Hall | for the Anthropology of Visual Communication, S.: Handbook for proxemic research. Studies in the anthropology of visual communication[END_REF]. Once the desired distance has been reached, whether by navigation of the robot or chosen distance by human, the start button pops up on the tablet to continue the rest of the interaction.

Design Method and Evaluation

In order to directly compare the effects the different modalities of the robot's behavior have on the user's behavior and attitude, the chosen scenario for the interaction was planning for a hypothetical holiday (Fig. 5). The robot acted as a travel agent helping the user plan their next vacation and it exhibited one of five behavioral conditions:

-Multi-modal Interaction, which is all modalities including social dialogue, (Social Gaze + Gestures + Proxemics + Social Dialogue)

-Minus Proxemics, which is all the modalities excluding proxemics, (Social Gaze + Gestures + Social Dialogue) -Minus Social Dialogue, which is all the modalities excluding social dialogue(Social Gaze + Gestures + Proxemics ) -Minus Gestures, which is all the modalities excluding social gestures, (Social Gaze + Proxemics + Social Dialogue) -Minus social gaze, which is all the modalities excluding social gaze, (Gestures + Proxemics + Social Dialogue)

The flow of interaction went as seen in Figure 6: first in the introductory phase, the robot greeted the user (highlighting the gesture modality), approached the human until the desired distance is established, engaged the user in a short social dialogue and then offered the participant some drinking water, using joint attention gaze and pointing gesture. Second was the travel planning phase, where the robot started asking a series of questions about the travel and vacation preferences, followed by a self-disclosure segment, where an open ended question was asked to know more about the user and the robot entered a listening state and demonstrated backchanneling. The third and final part was the closing phase, where the robot suggested two options based on answers previously provided by participants and recommended its personal preference between the two. Once the user made a final decision on a travel destination, the robot concluded the interaction and waved goodbye.

Social Dialogue

The dialogue throughout all conditions was adaptive: the robot's answers depended on what the user's previous choices were. In addition, since it was a travel planning scenario, all replies from the robot were consistent with the decisions the user made. For instance, if the person selected that they prefer to travel by train then the robot would suggest a destination that can be reached from Paris by train (e.g., Amsterdam) and similarly for the other descriptions, e.g., city or beach, solo trip or with friends and loved ones, culture or activities etc. However, the social dialogue modality differs in the social and friendly openings and replies as summarized in Table 1.

Conditions Multi-modal Interaction, Minus Proxemics, Minus Gestures, and Minus Social Gaze

Minus Social Dialogue

Openings/Closing

Social small talk Formal talk ex: "How are you today?" ... "great" ex: "Is it the first time you come here?" ex: "I am so happy to meet you!" ex: "You are the fourth person today whom I will help plan their vacation"

Replies

Personal preference replies

Non-personal general replies ex: "Excellent choice, I also like this location!" ex: "Many people like this location" ex: "I also find it awesome to travel by train because it's much more comfortable!" ex: "Traveling by train is more comfortable" Table 1 Social dialogue designs per condition

Design & Materials

An independent groups design was used, with the independent variable being multi-modal behavior with 5 levels: first: multi-modal interaction referring to all implemented modalities (proxemics, social gaze, gestures, and social dialogue), second: minus proxemics -referring to all implemented modalities except for proxemics, third: minus social dialogue -referring to all implemented modalities except for social dialogue, fourth: minus gestures -referring to all implemented modalities except for gestures, and fifth: minus social gazereferring to all implemented modalities except for social gaze mechanisms. The dependent variables were extracted using recorded logs from the robot application, data extracted from the recorded videos, and self-report questionnaires. First, the logs provided from the robot application include information on the position of the human relative to the robot extracted every 5 seconds as well as the angular facial frame information, which were extracted before the execution of every social gaze aversion. In addition, the time it took the user to press the buttons on the tablet and to take decisions as well as the decisions made were recorded. Second, the videos recorded were used to annotate and obtain the verbal and nonverbal responses and behaviors of the user throughout the interaction, including if the user accepted the water offer, back-channels performed, verbal responses, total speaking time, amount of information shared, number of audio/voice recognition errors that may have occurred, and gestures performed.

It is critical to note that the back-channels of the users in this set-up refer to both non-lexical back-channels, such as "uhh", "yeah", "mmm", .. etc., phrasal backchannels, such as "wow", "great",.. etc., and gestural back-channels, e.g., nodding. However, facial expressions were not considered. Third, a self-report questionnaire was used to evaluate perceived agency, social trust, competency trust, liking, rapport, acceptance, social presence, and social information processing. This paper will only discuss the results of the Likeability scale of the questionnaire. The subjective measurement for robot's Likeability or liking was comprised of 9 items. The participants were asked to rank how well they agree with the statements in Table 2 on a scale of 1 to 7, with 1 agree with the least and 7 agree with the most. These items were then averaged to form a scale with good reliability α = 0.88. The scale was normally distributed, W = 0.99, p = 0.35. As a compensation for their time each participant received 6 euros. Participants were randomly assigned to one of the five conditions (21 per condition). In addition, the participants were assigned randomly across the hours of the day to make sure half of the users in each condition interacted in the morning before lunch time while the other half in the afternoon after lunch. Upon arrival, the participants filled out the consent forms. Then, one of the experimenters introduced the experiment by explaining that they will interact with a robot that will help them plan their vacation as if it were their real holiday. They were informed the interaction would last 5-7 minutes and then they would have to fill a questionnaire, which included the liking scale as well as scales not relevant to the current study. They were also advised to speak loudly and articulate clearly in order to avoid any audio or voice recognition problems. Participants were then led to a room with the robot, as seen in Figure 5, and were asked to place their belongings on the side and stand wherever they wished. Shortly after the experimenter leaves the room, the interaction began. Besides the front and side cameras set-up in the room, there was a webcam streaming the interaction live. The robot as shown previously is completely autonomous, the videos and webcam are for recording and monitoring. Once the interaction was over, the participants were asked to fill the questionnaire in a different room and then they were debriefed on the study and were given their participation compensation.

Hypotheses

To evaluate the modalities in a multi-modal interaction and their effects on perceived social intelligence, the following hypotheses were formulated. First, the proxemics implemented respected the personal distance established by [START_REF] Hall | for the Anthropology of Visual Communication, S.: Handbook for proxemic research. Studies in the anthropology of visual communication[END_REF] and visibility and safety [START_REF] Kruse | Exploiting human cooperation in human-centered robot navigation[END_REF] while still initiating the interaction by approaching the user within his/her gaze zone [START_REF] Shi | Spatial formation model for initiating conversation[END_REF]. Following this, H1 was suggested:

-H1: Social distances established by the robot would be maintained throughout the interaction in all conditions except Minus Proxemics.

Second, social gaze aversion was shown to play a major role in intimacy regulation during human-human interactions to elevate the comfort of speakers ( [START_REF] Kendon | Some functions of gaze-direction in social interaction[END_REF], [START_REF] Argyle | Eye-contact, distance and affiliation[END_REF]). In addition, gaze aversion for turn-taking functioned as a social cue to hand the conversational floor to the user and thus making him/her the speaker. Furthermore, gaze aversion is practiced by humans specially when listening in order to minimize the negative perception attributed to staring and to promote the comfort of the speaker ( [START_REF] Abele | Functions of gaze in social interaction: Communication and monitoring[END_REF], [START_REF] Chiu | Gaze direction and fluency in conversational speech[END_REF] ). While Andrist et al. [START_REF] Andrist | Conversational gaze aversion for humanlike robots[END_REF] did not find that a social robot with proper timings for gaze aversions increased self-disclosure and comfort in humans more than a social robot with badly timed gaze aversions, we hypothesise that gaze mechanisms supported by multimodal behaviors would elicit more self-disclosure from the participants, such that:

-H2: Time participants spend speaking in the selfdisclosure segment would be the shortest in the Minus Gaze condition relative to the other conditions.

Third, gesture and joint attention through gaze have shown to be modalities used to communicate and point at an object of reference in an interaction as well as asking to grab the object referred to ( [START_REF] Boucher | I reach faster when i see you look: Gaze effects in human-human and human-robot face-to-face cooperation[END_REF], [5] [68]). With H3 formulated as:

-H3: Water suggestions are more likely to be taken when participants interact with a robot performing social gaze mechanisms and gestures.

Fourth, gestural alignment was proposed to measure the extent to which an interaction shapes the behavior of the user and the smoothness of the interaction ([14], [START_REF] Chartrand | The chameleon effect: the perception-behavior link and social interaction[END_REF]). As such, the following hypotheses were formulated looking into gestural alignment at the greeting and termination phases of the interaction in order to understand the possible change in gestural alignment behavior of the users. The use of back-channels, which include nodding and verbal content, throughout the interaction were also analysed as part of gestural alignment:

-H4a: Gestural alignment in the greeting and termination phases would be least present in the Minus Gesture condition. -H4b: the complete multi-modal behaviour condition would have the most participants who at the beginning did not greet the robot but at the end did close the interaction with the robot whether verbally or non-verbally. -H4c: Back-channeling throughout the interaction would be least performed by participants in the Minus Gesture condition.

Fifth, all the modalities combined make up multi-modal social cues designed to facilitate a more natural and friendly interaction. It was hypothesized that that would additionally have an effect on the subjective attitude of the users.

-H5: The condition with all modalities and social dialogue would score higher on the likeability scale questionnaire.

Results

Distances from Pepper

First, looking into the distances established throughout the interaction, one would expect no difference in the initial distance, as no interaction has yet occurred, but rather a difference after the navigation of the robot. We therefore conducted Kruskal-Wallis H-tests for the average distance maintained from the robot at the beginning of the interaction (prior to the social navigation phase), shown in Figure 7, as well as the average distance maintained after the social navigation phase until the end of the interaction, shown in Figure 8. The former model yielded no significant difference for condition, however, the latter revealed a significant effect of condition after the social navigation phase, see Table 3. Follow up pairwise comparisons with Dwass-Steel-Critchlow-Flinger correction revealed participants in the minus proxemics condition stood significantly further away from the robot throughout the interaction than in all other conditions (all W s ≥ 4.68, all p s < 0.008). No other differences between conditions were significant. This denotes that participants maintained the close social distance due to the proxemics established by the robot, thus validating H1.

Self-Disclosure

Second, we examined the effect of modalities on selfdisclosure by looking into how comfortable a human was in sharing information about themselves when asked by the robot. We again conducted a Kruskal-Wallis H-test to evaluate the total speaking time (in seconds) the user spent answering the robot's open-ended question to talk about themselves, shown in Figure 9. The average total speaking time was M = 11.55, (SD = 6.05) seconds. However, there was no significant effect of modality, χ 2 (4) = 4.09, p = .394, 2 = 0.04. As such, H2 was not supported. The number of pieces of information given, which is the number of new facts or opinions revealed and provided by the user about him/herself, was annotated and measured for each condition. The following is an example of how the data was annotated: if a participant after the open-ended self-disclosure question answered "I like hanging out with my friends...I like watching movies", then this was annotated as two pieces of information since two facts and/or opinions were revealed about the participant. The average number of pieces of information given was M = 3.429, (SD = 1.6), but there was again no significant effect of condition, χ 2 (4) = 1.76, p = .780, 2 = 0.02.

Accepting Water Offered

The number of participants who accepted the water offered in each condition is shown in Figure 10. It is important to note that absent in this figure and those that follow refer to the number of times the observed behavior was absent in each condition. For instance, in Figure 10, the grey or absent plots indicate the number times the water suggestion was not taken (and as as such absent). We ran a binomial logistic regression with condition as the predictor variable, and accepting the water offered as an outcome variable. The overall test of condition was marginally significant, χ 2 (4) = 8.18, p = .09, McFadden's Pseudo R 2 = 0.06. Follow up pairwise comparisons with Tukey's correction revealed a marginally significant difference between the complete multimodal condition and the minus proxemics condition, β = 1.89, p = .095. The odds of accepting the water offered in the minus proxemics condition were 0.15 [0.03, 0.62] times less than in the complete multi-modal condition.

Social Behavior

We again constructed binomial logistic regression models assessing the effect of condition on opening and closing waves. For both opening and closing waves, the logistic regression model was significant for condition, χ 2 (4) = 19.20, p < .001, McFadden's Pseudo R 2 = 0.13 and χ 2 (4) = 35.54, p < .001, McFadden's Pseudo R 2 = 0.25, respectively. We conducted follow up pairwise comparisons with Tukey's correction. As the closing wave model exhibited complete separation (i.e., no participants in the minus gesture condition waved goodbye), we further applied Firth's bias reduction method for this model. For both opening and closing waves, participants were significantly less likely to wave in the minus gesture condition than in the complete multi-modal condition and the minus Social Dialogue condition (see Table 4). No other comparisons were significant. See Figures 11 and12 Present refers to appearance of the participant's behavior of waving to the robot at the beginning of the interaction, while Absent refers to lack of this behavior, e.g., the participants not waving to the robot.[*] significant at the p < .05 level. [***] significant at the p < .001 level Beyond waving gestures in greeting and closing the interaction, Table 5 shows the presence and absence of all greeting and/or closing turns made by users, whether verbally or non-verbally. We classified participants behaviour as either consistent-social (both greeting and closing turn). consistent-nonsocial (neither greeting nor closing), inconsistent-social (no greeting turn, but a closing turn) or inconsistent-nonsocial Fig. 12 Number of closing waves performed by participants at the end of the interaction(with 95% CI errors). Present refers to appearance of the participant's behavior of waving to the robot at the end of the interaction, while Absent refers to lack of this behavior, e.g., the participants not waving to the robot.[*] significant at the p < .05 level (greeting turn, but no closing turn). However, a chisquare test comparing participants consistency in social behaviour did not reveal any differences between conditions, χ 2 (16) = 13.11, p = .664, Kramer's V = 0.194 [0.00, 0.33].

In addition, to assess the effect of modalities on behavioral alignment, we analysed the back-channeling performed by the participants in each condition. A binomial logistic regression was conducted to analyze the effect of the modalities on the number of participants who performed back-channeling, as shown in Figure 13. The regression model was statistically significant with χ 2 (4) = 12.90, p = .012, McFadden's Pseudo R 2 = 0.09. Evaluation of the log odds with Tukey's correction revealed participants in the minus gaze condition were less likely to produce back-channels than in the complete multi-modal condition, see Table 6.

Fig. 13 Number of back-channels performed while interacting with the robot in each condition (with 95% CI errors). Present refers to appearance of the back-channels performed by the participants, while Absent refers to lack of back-channels detected during the interaction.[*] significant at the p < .05 level. [**] significant at the p < .01 level

Liking

A Kruskal-Wallis H-test for participants subjective evaluations of their liking of the robot revealed no differences between any of the conditions, shown in Figure 14, χ 2 (4) = 3.84, p = .428, 2 = 0.04; failing to support H5. Further exploratory investigation was done to look into behavioral outcomes that might represent liking. First, Figure 15 shows the number of addressee terms used to address the robot in each condition. In the french language, the pronoun"tu" (referring to"you") is used in informal and/or friendly contexts, whereas, the pronoun "vous" is used for formal and/or acquaintance contexts. See Table 7 for the frequency of each mode of address used by participants. A chi square test comparing participants mode of address towards Pepper in each condition was significant χ 2 (12) = 23.01, p = .028, Kramer's V = 0.27, [0.00, 0.32]. Follow up tests with FDR correction, however, did not reveal specific differences between conditions (Table 8). As this analysis was exploratory, we then relaxed the need for correction with multiple comparisons. Without correction, there was a significant difference between the minus Social Dialogue and minus proxemics conditions.

Second, Figure 16 shows in each condition the number of participants who made utterances while using the tablet (despite it being clear that speech was not needed 5 Number of greeting and/or closing turns, which may be verbal or non-verbal, present and absent in each modality in beginning and end of the interaction. The rows represent the statuses in each condition as follows: 1) participants who greeted the robot and also closed the interaction 2) those who greeted the robot but did not close the interaction 3) those who did not greet the robot but closed the interaction 4) those who did not greet the robot nor did they close the interaction to carry out choice selection on the tablet of the robot).

For instance, listening mode and speech recognition in the robot was activated when the robot asked open ended questions; whereas for making choices regarding the planning of the destination in this scenario, the interface to select the preference was by using the tablet (as was instructed by the robot in the beginning). There was no significant difference in how likely participants were to talk to Pepper in addition to using the tablet, χ 2 = 5.18, p = .270, McFadden's Pseudo R 2 = 0.04.

Voice Recognition Errors

Although we attempted to limit the amount of autonomous voice recognition, the introductory phase included some reciprocal interaction between Pepper and the participant (e.g., asking "how are you"). Thus, there was still some potential for voice recognition errors to occur. Although not initially part of the experimental design, based on observations of instances where voice recognition errors occurred we decided to to explore if these (naturally occurring) errors had any effect on participants behaviour.

A one-way Kruskal Wallis H-test performed on the number of voice recognition errors occurring per condition was non-significant, χ 2 (4) = 2.04, p = .727, η 2 = 0.02, indicating there was no difference in the number of voice recognition errors occurring between conditions.

Significant negative correlations were identified between the number of voice recognition errors, the total time the participant spent talking during the selfdisclosure phase ρ = -0.20, p < .05 and the number of pieces of information they disclosed ρ = -0.28, p < .01.

Discussion

First, H1 was supported, showing the influence of proxemics on the distances maintained by the users throughout the interaction. Prior to the navigation of the robot, participants chose to stand far from the robot at a (M = 1.28, SD = 0.26) meters distance, with no significant difference between conditions. The initial distance chosen was within the social distance defined by Hall [START_REF] Hall | for the Anthropology of Visual Communication, S.: Handbook for proxemic research. Studies in the anthropology of visual communication[END_REF] and did not give accessibility to the robot's tablet. In the conditions where the robot navigated to establish the personal distance of 0.85 meters, distances maintained during the rest of the interaction until the end were much closer. Conversely, in the minus proxemics condition, participants kept a further distance from the robot. Proxemics once again played an influential role on the behavioral outcomes of the interaction and was the main reason users kept a close distance to the robot.

Second, H2 was not supported, social gaze mechanisms did not elicit an increase in self-disclosure speaking time nor the amount of information the user revealed about themselves. Further investigation was held to interpret what might have affected the self-disclosure speaking turn of the users. It was found that voice recognition errors significantly predicted total speaking time of participants and the amount of information shared. While gaze aversions and their respective functions play a guiding role in intimacy regulation and comfort in self-disclosure in human-human interactions, findings in this study seem to show that for human-robot interactions technological voice recognition errors precede gaze aversions in governing behavioral outcomes for such contexts. This shows that getting the robot technologically ready may have a great impact on how naturally a user answers an open ended question about themselves rather than how close a robot's subtle behavior is to a human. Third, while H3 was not fully supported, the results gave an insight into the effect of proxemics modality. The condition in which there was least water suggestions taken was in the minus proxemics condition. Even though, the state of the art has been focused on using deictic gesture and joint attention gaze for pointing at objects to grab for task-oriented scenarios ( [START_REF] Andrist | Designing effective gaze mechanisms for virtual agents[END_REF], [START_REF] Skantze | Turn-taking, feedback and joint attention in situated human-robot interaction[END_REF]), there was no significant difference for these modalities in this study. In addition, the suggestion of object grabbing in this paper was more focused on its social context and implications. It was shown that the participants not only took the object suggested by robot, in this case the water cup, but also drank the water. It may perhaps be linked to the perception of the user to the robot's situational awareness, which is the ability to perceive and infer knowledge from the surrounding environment [START_REF] Bolstad | Situation awareness: Does it change with age[END_REF]. There is a need for future work to better understand the potential of proxemics on object manipulation in the shared environment between the user and robot. Fourth, while H4a was validated, H4b was not validated and H4c was partially supported. The lack of social gestures significantly affected the behavioral alignment for the greeting and closing of the interaction in the minus gestures condition. In the greeting part of the interaction, the minus gestures condition had significantly less wave gestures performed to greet the robot than in the complete multi-modal and minus social dialogue conditions. For closing waves this difference was even more extreme, with, no wave gestures performed in closing the interaction with the robot in the minus gestures condition. Even further, In Table 5, while the number of participants that did not greet the robot but eventually performed a closing turn at the end of the interaction were highest in the multi-modal interaction and minus gestures conditions, there was no significant difference. This may also imply that even though there was no behavioral gesture done in the closing of the interaction in the minus gestures condition, there was a verbal closing turn. On the other hand, H4c was partially supported. While it was not the minus gestures condition that had the least amount of back-channeling alignment performed by users as hypothesized, it was instead the minus social gaze condition. This may indicate that gesture mirroring was not the main cause of back-channeling alignment, but rather how naturally the interaction flowed. Conditions with social gaze mechanisms included turn-taking and floor-holding which hold cognitive functions and were accompanied by very short pauses in speech. The users may have performed more back-channeling during these conditions as it was a natural human behavior and as a way to provide the robot feedback that they were in fact still listening to its speech and aligned in the interaction. Thus, the social gaze plays a role in shaping the human-robot interaction seem more instinctive to the human and in forming alignment.

Fifth, a self-reported questionnaire was used to measure liking or 'likability' of the robot and it was hypothesized that the multi-modal interaction condition would score higher; however H5 was not supported. Further behavioral outcomes were annotated and analyzed that might be related to liking of the robot. First, the way the participants addressed the robot was studied. The experiment took place in French with native french speakers and in the french dialect the "you" pronoun is represented by "vous" for formal set-ups and/or with acquaintances and by "tu" for rather informal set-ups and/or with friends. The minus social dialogue condition was significantly different to the minus proxemics condition. The minus social dialogue condition was the only one to have users address the robot by its name, e.g., here being "Pepper". In addition, the minus proxemics condition had the highest number of participants using informal/friendly pronouns. Further research needs to done to better comprehend what that would signify but it can be concluded at this point that modalities affect the terms participants exercise in the interaction with the robot. Second, during the interaction, the robot is in listening mode at only two phases: at the beginning during the social small talk shown in Table 1 and at the open ended question to measure self-disclosure. At all other times during the interaction, the robot was not in a listening mode and its tablet was required to be used by the participant to answer the questions asked by the robot. However, it was noted that some participants chose to talk while using the tablet and to sometimes justify their choices to the robot and discuss their thought process out loud. While no significant difference was found in the results, the multi-modal interaction condition had the highest number of participants who chose to also talk while using the tablet and the minus gestures condition had the least. This may give an insight into the effect of social gestures on how social participants were with the robot. These findings hint into the type of relationships users formed with the robot based on the multi-modal behaviors they interacted with. There seems to be more to discover and investigate in future works.

Conclusion

Non-verbal behavior plays a key role in human communication not only by reinforcing and enhancing speech in diverse formats of the interaction, but also by carrying fundamental functions in communication that can stand-alone from speech. However, this non-verbal behavior is not made of only one modality but rather of multi-modalities all composed together to serve their purpose. For this reason, while studying each modality separately may lead to improving human-robot interaction, a deeper understanding of the different modalities when performed together and their combinations as well as their interaction outcomes is imperative for effective use of the multi-modalities of robots in maximizing targeted outcomes. This paper presented work attempting to build such an understanding. The process involved implementing a system of multi-modalities including social gaze mechanisms, different types of gestures, proxemics for navigation in initiating conversations, and social dialogue followed by an evaluation study where participants interacted with the robot in a travel agent scenario. The system and methodology presented in this paper can be as well utilized on other robots. The results showed various insights into the contributions of modalities in a multi-modal interaction onto several notable behavioral outcomes of the users, including taking physical suggestions, distances maintained during the interaction, wave gestures performed in greeting and closing, back-channeling, how the robot is addressed, and how socially it is treated. It can be concluded that certain modalities in multi-modal behaviors particularly influence the outcomes of the interaction, and at times not in the same way as seen in the state-of-the-art of the modality on its own. Notably, this paper showed how multiple modalities can be combined in an interaction and how subtracting each modality at a time revealed insights about the effect of that modality. For instance, it is now clear how proxemics influence the distances maintained during an interaction and the probability of the user accepting the robot's offer. In addition, social gestures can predict how humans greet the robot and close interactions with it and the utterances the user makes while using other modalities of the robot such as the tablet. Moreover, social gaze shaped how naturally humans back-channel when interacting with the robot but did not have an effect on how much they disclose to the robot. All these findings may lead to further understanding on human-robot interaction and how multi-modal behavior can be used to increase the perceived social intelligence of the robot.

Fig. 1

 1 Fig. 1 Summary of Multi-Modal Social Cues

Fig. 4

 4 Fig. 4 Turn Yield Gaze Mechanism

Fig. 6

 6 Fig. 6 Summary of flow of interaction

Fig. 7

 7 Fig. 7 Initial Distance (meters) by participants per condition at the beginning of the interaction with the robot before proxemics

Fig. 8

 8 Fig.8Maintained Distance (meters) by participants per condition throughout the interaction with the robot after proxemics.

Fig. 9

 9 Fig. 9 Total speaking time (seconds) of participants per condition during self-disclosure open-ended question

Fig. 10

 10 Fig. 10 Number of times the water offer was accepted by participants per condition. Present refers to appearance of the participant's behavior of accepting the water offered by the robot by grabbing the cup and/or drinking the water, while Absent refers to lack of this behavior and as such not accepting the water offered by the robot.

  .

Fig. 11

 11 Fig. 11 Number of greeting waves performed by participants at the beginning of the interaction (with 95% CI errors).Present refers to appearance of the participant's behavior of waving to the robot at the beginning of the interaction, while Absent refers to lack of this behavior, e.g., the participants not waving to the robot.[*] significant at the p < .05 level. [***] significant at the p < .001 level

Fig. 14

 14 Fig. 14 Mean Liking ranked by participants for the robot in each condition

Fig. 15

 15 Fig. 15 Number of addressee terms used to address the robot in each condition. The addressee terms are placed in four categories: absent, no term was used to directly address the robot, informal/friendly pronoun, formal/acquaintance pronoun, and name of the robot

Fig. 16

 16 Fig.[START_REF] Chartrand | The chameleon effect: the perception-behavior link and social interaction[END_REF] Number of Utterances while using the tablet of the robot in each condition.Present refers to detection of utterances/speech while tablet was used by the participants, while Absent refers to lack of detection.

In:

  Proceedings of the 9th International Conference on Multimodal Interfaces, ICMI '07, p. 140-145. Association for Computing Machinery, New York, NY, USA (2007). DOI 10.1145/1322192.1322218

  Je demanderais volontiers des conseils à Pepper I would ask Pepper for advice J'aimerais avoir Pepper comme collègue I would like Pepper as a colleague J'aimerais avoir Pepper comme colocataire I would like Pepper as a housemate J'aimerais que Pepper et moi soyons amis I would like to be friends with Pepper Pepper et moi sommes similaires Pepper is similar to me

	Questions Used, French	Questions English
	Pepper est gentil	Pepper is friendly
	Pepper est chaleureux	Pepper is warm
	Pepper est aimable	Pepper is likeable
	Pepper est accessible	Pepper is approachable

Table 2

 2 Likeability Scale ranked from 1 to 7

Table 3

 3 Kruskal-Wallis Tests for the Effect of Condition on Initial and Maintained Distances

	Outcome	χ 2	df	p	2
	Initial Distance	2.40	4	.663 0.02
	Distance Maintained 23.92	4 <.001 0.23

Table 4

 4 Comparison of the likelihood of producing opening and closing waves in each condition

	Contrast	Log Odds (SE)	z	p-value
			1.28	.704
	Multi-Modal Interaction vs Minus Social Dialogue 0.56 (0.66)	0.85	.915
	Multi-Modal Interaction vs Minus Proxemics	0.94 (0.66)	1.44	.602
	Minus Gaze vs Minus Social Dialogue	-0.29 (0.63) -0.46 .991
	Minus Gaze vs Minus Proxemics	0.10 (0.63)	0.15	.999
	Minus Social Dialogue vs Minus Proxemics	0.38 (0.62)	0.62	.972
	Closing Wave †
	Minus Gesture vs Multi-Modal Interaction	-4.35 (1.54) -2.83 .038*
	Minus Gesture vs Minus Gaze	-3.95 (1.53) -2.58 .074
	Minus Gesture vs Minus Social Dialogue	-4.63 (1.54) -3.01 .022*
	Minus Gesture vs Minus Proxemics	-3.30 (1.53) -2.15 .197
	Multi-Modal Interaction vs Minus Gaze	0.40 (0.65)	0.61	.973
	Multi-Modal Interaction vs Minus Social Dialogue -0.28 (0.67) -0.42 .994
	Multi-Modal Interaction vs Minus Proxemics	1.05 (0.65)	1.62	.482
	Minus Gaze vs Minus Social Dialogue	-0.68 (0.66) -1.03 .840
	Minus Gaze vs Minus Proxemics	0.65 (0.64)	1.03	.842
	Minus Social Dialogue vs Minus Proxemics	1.33 (0.66)	2.03	.251
	† With Firth's bias reduction method
	* significant at the p < .05 level
	** significant at the p < .01 level

Opening Wave

Minus Gesture vs Multi-Modal Interaction -3.10 (0.89) -3.49 .005** Minus Gesture vs Minus Gaze -2.25 (0.87) -2.60 .071 Minus Gesture vs Minus Social Dialogue -2.54 (0.86) -2.94 .027* Minus Gesture vs Minus Proxemics -2.16 (0.86) -2.50 .090 Multi-Modal Interaction vs Minus Gaze 0.85 (0.66)

Table 6

 6 Comparison of the likelihood of producing backchannels in each condition

	Contrast	Log Odds (SE)	z	p-value
	Minus Gesture vs Multi-Modal Interaction	-0.41 (0.69) -0.59 .977
	Minus Gesture vs Minus Gaze	1.79 (0.69)	2.58	.073
	Minus Gesture vs Minus Social Dialogue	0.00 (0.66)	0.00	1.00
	Minus Gesture vs Minus Proxemics	0.21 (0.65)	0.32	.998
	Multi-Modal Interaction vs Minus Gaze	2.20 (0.73)	3.01	.022*
	Multi-Modal Interaction vs Minus Social Dialogue 0.41 (0.69)	0.59	0.98
	Multi-Modal Interaction vs Minus Proxemics	0.61 (0.69)	0.80	.899
	Minus Gaze vs Minus Social Dialogue	-1.79 (0.69) -2.58 .073
	Minus Gaze vs Minus Proxemics	-1.58 (0.69) -2.31 .140
	Minus Social Dialogue vs Minus Proxemics	0.21 (0.65)	0.32	.998

* significant at the p < .05 level

Table 7

 7 Frequency of modes of address used by participants towards Pepper in each condition

	Condition		Mode of Address
		Tu Vous	Pepper Nothing
	Multi-Modal Interaction	1	1	0	19
	Minus Gaze	1	3	0	17
	Minus Gesture	2	3	0	16
	Minus Social Dialogue	0	1	3	17
	Minus Proxemics	5	1	0	15
	Totals	9	9	3	83

Table 8

 8 Comparison between modes of address used by participants towards Pepper in each condition

	Contrast	Raw p-value	FDR-corrected p-value
	Multi-Modal Interaction vs Minus Gaze	.798	.886
	Multi-Modal Interaction vs Minus Gesture	.593	.740
	Multi-Modal Interaction vs Minus Social Dialogue	.232	.386
	Multi-Modal Interaction vs Minus Proxemics	.229	.386
	Minus Gaze vs Minus Gesture	1.00	1.00
	Minus Gaze vs Minus Social Dialogue	.223	.386
	Minus Gaze vs Minus Proxemics	.207	.386
	Minus Gesture vs Minus Social Dialogue	.122	.386
	Minus Gesture vs Minus Proxemics	.347	.496
	Minus Social Dialogue vs Minus Proxemics	.022*	.222

* significant at the p < .05 level

Further subjective measurements, referring to the comparison between self-reported attitudes and behaviors towards social robots will be examined elsewhere.

Acknowledgment

This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 765955.

The authors would also like to thank the team at INSEAD for assistance with data collection, as well as Hugues Pellerin for advice with the statistical analyses.

Conflict of interest

The authors declare that they have no conflict of interest.