Mohamed Chetouani
email: mohamed.chetouani@upmc.fr

Emilie Delaherche

Guillaume Dumas

David Cohen

Interpersonal

social interaction. Edited Judee K Burgoon

Nadia Magnenat-Thalmann

Interpersonal synchrony: from social perception to social interaction

Keywords:

Introduction

Synchrony refers to individuals' temporal coordination during social interactions [START_REF] Cappella | Coding Mutual Adaptation in Dyadic Nonverbal Interaction[END_REF]. The analysis of this phenomenon is complex, requiring the perception and integration of multimodal communicative signals.

The evaluation of synchrony has received multidisciplinary attention because of its role in early development [START_REF] Feldman | Infant-mother and infant-father synchrony : the coregulation of positive arousal[END_REF], language learning [START_REF] Goldstein | Social interaction shapes babbling: Testing parallels between birdsong and speech[END_REF] and social connection [START_REF] Harrist | Dyadic synchrony: Its structure and function in children's development[END_REF]. Initially, instances of synchrony were directly perceived in the data by trained observers. Several methods have been proposed to evaluate interactional synchrony, ranging from behavior micro-analysis [START_REF] Cappella | Behavioral and judged coordination in adult informal social interactions: vocal and kinesic indicators[END_REF] to global perception of synchrony [START_REF] Bernieri | Synchrony, pseudo synchrony, and dissynchrony: Measuring the entrainment process in mother-infant interactions[END_REF]. Behavioral synchrony has now captured the interest of researchers in such fields as social signal processing, robotics and machine learning [START_REF] Prepin | Shared understanding and Synchrony Emergence: Synchrony as an Indice of the Exchange of Meaning between Dialog Partners[END_REF][START_REF] Kozima | Keepon[END_REF].

In this chapter, we focus especially on description and definition of synchrony for the development of computational models. The chapter begins with a review of evidences of interpersonal synchrony from di↵erent research domains (psychology, clinics, neuroscience and biology). Then, we introduce a working definition of interpersonal synchrony (section 1.4). The chapter surveys evaluation models and methods from the literature of psychology (section 1.5) and social signal processing (section 1.6). Finally, the chapter discusses a number of challenges that need to be addressed (section 1.7).

1.2 Non-verbal evidence of interpersonal synchrony Among social signals, synchrony and coordination have been considered lately [START_REF] Ramseyer | Nonverbal Synchrony or Random Coincidence? How to Tell the Di↵erence[END_REF][START_REF] Delaherche | Interpersonal Synchrony : A Survey Of Evaluation Methods Across Disciplines[END_REF]. [START_REF] Condon | A segmentation of behavior[END_REF] initially proposed a micro-analysis of human behavior (body motion and speech intonation) and evidenced the existence of interactional synchrony, the coordination between listener's and speaker's body movements or between the listener's body movement and the speaker's pitch and stress variations. [START_REF] Bernieri | Synchrony, pseudo synchrony, and dissynchrony: Measuring the entrainment process in mother-infant interactions[END_REF] define coordination as "... degree to which the behaviors in an interaction are non-random, patterned or synchronized in both form and timing". [START_REF] Kendon | Movement coordination in social interaction: some examples described[END_REF] raises fundamental questions about the condition of interactional synchrony arousal and its function in interaction. When he synchronizes with the speaker, the listener demonstrates his ability to anticipate what the speaker is going to say. This way, he gives feedback to the speaker and smoothens the running of the encounter.

In the "double video setting", several teams manipulated the timing of exchanges between mother and baby by alternating live and pre-recorded exchanges [START_REF] Nadel | Expectancies for social contingency in 2-month-olds[END_REF]. They showed that in the pre-recorded sessions the child showed more negative signs (anger or distress manifestations, cries) and that when they came back to the live exchanges, the positive signals (gazes toward the mother, smiles...) were restored. In these experiments, they demonstrated expectancies for synchronized and contingent exchanges with the social partner (here the mother), since two months old. The key role of synchrony was also found at early age in more natural early interaction such as home breast feeding [START_REF] Viaux-Savelon | Prenatal Ultrasound Screening: False Positive Soft Markers May Alter Maternal Representations and Mother-Infant Interaction[END_REF]. In [START_REF] Saint-Georges | Do Parents Recognize Autistic Deviant Behavior Long before Diagnosis? Taking into Account Interaction Using Computational Methods[END_REF], we investigated early signs of autism by modeling child's development with an interpersonal synchrony point of view. Regarding synchrony, the main results show that (i) parents seemed to feel weaker interactive responsiveness and mainly weaker initiative from their infants; and (ii) parents increasingly tried to supply soliciting behaviours and touching.

As part of the social signals, interpersonal coordination is a signal of great importance to evaluate the degree of attention or engagement between two social partners. It is often related to the quality of interaction [START_REF] Chartrand | The chameleon e↵ect: The perceptionbehavior link and social interaction[END_REF], cooperation [START_REF] Wiltermuth | Synchrony and Cooperation[END_REF] or entitativity [START_REF] Lakens | Movement synchrony and perceived entitativity[END_REF]. Finally, its assessment constitutes the first step in prospect of equipping a social robot with the ability to anticipate a human partner reaction and enter in synchrony with him [START_REF] Michalowski | Rhythmic attention in child-robot dance play[END_REF][START_REF] Prepin | How an agent can detect and use synchrony parameter of its own interaction with a human[END_REF][START_REF] Boucenna | Learning of social signatures through imitation game between a robot and a human partner[END_REF].

1.3 Biological evidence of interpersonal synchrony ix 1.3 Biological evidence of interpersonal synchrony Concerning the development of social interaction, it is important to highlight the major role of synchrony of rhythms in bonding. Thus, [START_REF] Guedeney | Infant rhythms versus parental time: Promoting parent-infant synchrony[END_REF] emphasize the importance of synchronization between infant and parental rhythms in very early social interaction and socio-emotional development, from biological rhythms during pregnancy to later exchange between caregiver and child.

Synchrony between partners has been correlated with biological markers. Correlation at biological levels has also been found. In [START_REF] Feldman | Parent-infant synchrony and the construction of shared timing; physiological precursors, developmental outcomes, and risk conditions[END_REF], a biobehavioral synchrony model is introduced on the basis of investigations of synchrony through physiological signals (e.g. ECG, skin conductance) and behaviors during parent-infant interactions.

Naturally occurring variations in maternal behavior are associated with di↵erences in estrogen-inducible central oxytocin receptors, which are involved in pro-social behaviors [START_REF] Champagne | Naturally occurring variations in maternal behavior in the rat are associated with di↵erences in estrogen-inducible central oxytocin receptors[END_REF]). Oxytocin appears to enhance both maternal/paternal as well as a liative behaviors in humans and is considered as the bonding hormone [START_REF] Weisman | Oxytocin Administration to Parent Enhances Infant Physiological and Behavioral Readiness for Social Engagement[END_REF]. [START_REF] Dumas | Inter-Brain Synchronization during Social Interaction[END_REF] use hyper-scanning recordings to examine brain activity, including measures of neural synchronization between distant brain regions of interacting individuals through a free exchange of roles between the imitator and the model. Their study was the first to record dual EEG activity in dyads of subjects during spontaneous nonverbal interaction. Five female-female pairs and 6 male-male pairs were scanned. They showed that interpersonal hand movements were correlated with the emergence of synchronization in the brain's alpha-mu band (an area involved in social interaction [START_REF] Perry | Exploring motor system contributions to the perception of social information: Evidence from EEG activity in the mu/alpha frequency range[END_REF]) between the right centro-parietal regions.

Rhythm, synchrony and emotion are increasingly being viewed by developmental psychologists as key aspects of appropriate early interaction [START_REF] Feldman | Parent-infant synchrony and the construction of shared timing; physiological precursors, developmental outcomes, and risk conditions[END_REF][START_REF] Saint-Georges | Motherese in interaction: at the crossroad of emotion and cognition? (a systematic review)[END_REF][START_REF] Weisman | Oxytocin shapes parental motion during father-infant interaction[END_REF] 1.4 Proposed definition Synchrony is the dynamic and reciprocal adaptation of the temporal structure of behaviors between interactive partners. Unlike mirroring or mimicry, synchrony is dynamic in the sense that the important element is the timing, rather than the nature of the behaviors. As noted in [START_REF] Ramseyer | Synchrony: A Core Concept for A Constructivist Approach to Psychotherapy[END_REF], the distinction between synchrony and mirroring can be unclear; these phenomena are not disjunctive and can often be observed simultaneously.

As described in [START_REF] Harrist | Dyadic synchrony: Its structure and function in children's development[END_REF], synchrony requires a (1) maintained, (2) shared focus of attention, (3) temporal coordination, and (4) contingency. Computational models of synchrony need, if not all, most of theses ingredients. And the main problem is that each of them is ambiguous and requires investigations. However, taking into account advances in various fields: computational linguistics, social signal processing and social robotics or either virtual agents.

1.5 Non-computational methods of synchrony assessment Several non-computational methods have been proposed to evaluate interpersonal synchrony, ranging from behavior micro-analysis to global perception of synchrony. Behavioral coding methods propose evaluating the behavior of each interactional partner on a local scale. These methods require the use of computer-based coding (e.g., Observer or Anvil [START_REF] Kipp | Spatiotemporal Coding in ANVIL[END_REF]) and trained raters. Various category and time scales can be used for coding. Generally, a measure of synchrony is deduced from the covariation of the annotated behaviors. The codes can be either continuous (speed of a gesture) or categorical (type of gesture). [START_REF] Cappella | Coding Mutual Adaptation in Dyadic Nonverbal Interaction[END_REF] synthesized the three crucial questions to be addressed when conducting an interaction study: "what to observe (coding), how to represent observations (data representations) and when and how frequently to make the observations (time)".

Behavioral coding methods are time-consuming and tedious with regard to the training of observers, the number of behaviors coded and the duration of the video files to be coded, particularly for longitudinal studies. [START_REF] Cappella | Behavioral and judged coordination in adult informal social interactions: vocal and kinesic indicators[END_REF] and [START_REF] Bernieri | Synchrony, pseudo synchrony, and dissynchrony: Measuring the entrainment process in mother-infant interactions[END_REF] proposed an alternative to behavior microanalysis: the judgment method. In their studies, they investigated the use of human raters to evaluate video clips of infants interacting with their mothers. Raters judge for simultaneous movement, tempo similarity and coordination and smoothness on a longer time scale using a Likert scale. Cappella showed that untrained judges were consistent with one another and reliably judged the synchrony between partners [START_REF] Cappella | Behavioral and judged coordination in adult informal social interactions: vocal and kinesic indicators[END_REF].

Non-computational methods su↵er serious drawbacks. Within the tedious task of coding, segmenting and annotating behaviors can be confusing: when does a behavior start, when does it end, how should it be labeled? Often, the annotator makes trade-o↵ because no label accurately describes what he observes. The judges' reliability in assessing such a subjective and complex construct is also questionable, and no general framework for synchrony assessment has been accepted to date. A method was recently proposed to convert the judgments of multiple annotators in a study on dominance into 1.6 Fully automatic measures of synchrony xi a machine-learning framework [START_REF] Chittaranjan | Inferring truth from multiple annotators for social interaction analysis[END_REF]. Finally, conversational partners are often studied individually when coding. Thus, it is particularly di cult to recreate the dynamic and interpersonal aspects of social interaction manually and after coding. Nonetheless, annotation and judgment methods are essential in proposing automatic systems for synchrony assessment and testing their performance. Currently, no automatic systems modeling synchrony using real interaction data are free from annotation.

Annotation is mainly used in two di↵erent manners. First, annotation is used to train automatic systems to model and learn communication dynamics (see section 1.6.1). These studies often rely on behavioral coded databases. Second, another set of studies intends to measure the degree of synchrony between dyadic partners with unsupervised methods. In these studies, the measure of synchrony is not validated per se but is judged by its ability to predict an outcome variable that has been manually annotated, often using judgment methods. The outcome variable can be friendship (Altmann, 2011), conflicting situations [START_REF] Altmann | Studying Movement Synchrony Using Time Series and Regression Models[END_REF], success in psychotherapy [START_REF] Ramseyer | Nonverbal Synchrony in Psychotherapy: Coordinated Body Movement Reflects Relationship Quality and Outcome[END_REF], etc.

Fully automatic measures of synchrony

To exploit synchrony cues in human-machine interaction, automatic techniques can be used to capture pertinent social signals and assess movement synchrony in human-human interactions. This studies aim at measuring the degree of similarity between the dynamics of the non-verbal behaviors of dyadic partners. The goals of these studies are generally divisible into two categories: (a) compare the degree of synchrony under di↵erent conditions (e.g., with or without visual feedback) [START_REF] Shockley | Mutual interpersonal postural constraints are involved in cooperative conversation[END_REF][START_REF] Varni | A System for Real-Time Multimodal References Analysis of Nonverbal A↵ective Social Interaction in User-Centric Media. Multimedia[END_REF] and (b) study the correlation between the degree of synchrony and an outcome variable (e.g., friendship, relationship quality) [START_REF] Altmann | Studying Movement Synchrony Using Time Series and Regression Models[END_REF][START_REF] Ramseyer | Nonverbal Synchrony in Psychotherapy: Coordinated Body Movement Reflects Relationship Quality and Outcome[END_REF].

The first step in computing synchrony is to extract the relevant features of the dyad's motion with motion-tracking devices [START_REF] Ashenfelter | Spatiotemporal symmetry and multifractal structure of head movements during dyadic conversation[END_REF], image-processing techniques (tracking algorithms, image di↵erencing) (Delaherche and Chetouani, 2010; [START_REF] Varni | A System for Real-Time Multimodal References Analysis of Nonverbal A↵ective Social Interaction in User-Centric Media. Multimedia[END_REF] or physiological sensors [START_REF] Varni | A System for Real-Time Multimodal References Analysis of Nonverbal A↵ective Social Interaction in User-Centric Media. Multimedia[END_REF]. After extracting the motion features, a measure of similarity is applied. Correlation is the most commonly used method to assess interactional synchrony [START_REF] Altmann | Studying Movement Synchrony Using Time Series and Regression Models[END_REF][START_REF] Ramseyer | Nonverbal Synchrony in Psychotherapy: Coordinated Body Movement Reflects Relationship Quality and Outcome[END_REF]. A time-lagged cross-correlation is applied between the movement time series of the interactional partners using short windows of interaction. Another method to assess the similarity of motion of two partners is recurrence anal-ysis [START_REF] Richardson | Synchrony and swing in conversation: Coordination, temporal dynamics, and communication[END_REF]. Recurrence analysis assesses the points in time that two systems show similar patterns of change or movement, called "recurrence points". Spectral methods constitute an interesting alternative to temporal methods when dealing with rhythmic tasks. Spectral methods measure the evolution of the relative phase between the two partners as an indication of a stable time-lag between them [START_REF] Oullier | Social coordination dynamics: Measuring human bonding[END_REF][START_REF] Richardson | Rocking together: Dynamics of intentional and unintentional interpersonal coordination[END_REF]. Spectral methods also measure the overlap between the movement frequencies of the partners, called cross-spectral coherence [START_REF] Richardson | Looking To Understand: The Coupling Between Speakers' and Listeners' Eye Movements and Its Relationship to Discourse Comprehension[END_REF][START_REF] Richardson | Rocking together: Dynamics of intentional and unintentional interpersonal coordination[END_REF][START_REF] Delaherche | Multimodal coordination: exploring relevant features and measures[END_REF] or power spectrum overlap [START_REF] Oullier | Social coordination dynamics: Measuring human bonding[END_REF].

A critical question when attempting to detect dependence relationships between features is where the boundary between scores indicating significant and insignificant synchrony should be. A well-spread method consists of applying surrogate statistical testing [START_REF] Richardson | Looking To Understand: The Coupling Between Speakers' and Listeners' Eye Movements and Its Relationship to Discourse Comprehension[END_REF][START_REF] Ashenfelter | Spatiotemporal symmetry and multifractal structure of head movements during dyadic conversation[END_REF]Sun et al., 2011;[START_REF] Delaherche | Multimodal coordination: exploring relevant features and measures[END_REF]. Video images of dyadic partners are isolated and re-combined in a random order to synthesize surrogate data (pseudo-interactions). Synchrony scores are assessed using the original and surrogate datasets. The synchrony scores on the surrogate dataset constitute a baseline for judging for the dyad's coordination. Fully automatic measures of movement synchrony are subject to several criticisms in the context of studying naturalistic interaction data. First, the measures provided by these methods are mostly global and do not shed light on what happened locally during the interaction; they do not provide a local model of the communication dynamics. Second, the importance of speech and multimodality is often concealed in these methods.

Machine understanding of interpersonal synchrony

Given these criticisms, many in the field adopted the alternative practice of modeling the timing and occurrence of higher-level behavioral events such as smiles, head gestures, gazes and speaker changes. These behavioral events can be either extracted from a human-annotated database or predicted from low-level signals automatically extracted from data. These methods arise from a great interest in identifying the dynamical patterns of interaction and characterizing recurrent interpersonal behaviors.

Machine learning methods o↵er an interesting framework for the exploration of interactive behaviors. A key challenge is proposing models with the content and temporal structure of dyadic interactions. Various sequential learning models, such as Hidden Markov Models (HMMs) or Conditional Random Fields (CRFs), are usually used to characterize the temporal 1.7 Conclusions and main challenges xiii structure of social interactions. Messinger et al. employ related techniques for the understanding of communicative development, which is characterized by mutual influences during interaction: infants and parents influence and respond to one another during communication [START_REF] Messinger | Applying machine learning to infant interaction: The development is in the details[END_REF]. In [START_REF] Mahdhaoui | Understanding parent-infant behaviors using non-negative matrix factorization[END_REF], an integrative approach is proposed to explicitly consider the interaction synchrony of behaviors. The model is applied to the characterization of parent-infant interactions for di↵erential diagnosis: autism (AD), intellectual disability (ID) and typical development (TD). The authors estimate transitions between behaviors of the infant and the parent by analyzing behaviors co-occurring in a 3s window.

Among interpersonal behaviors, the prediction of turn-taking and backchannels has been largely studied in the perspective of building fluent dialog systems. The central idea is to develop "predictive models of communication dynamics that integrate previous and current actions from all interlocutors to anticipate the most likely next actions of one or all interlocutors" [START_REF] Ozkan | Latent Mixture of Discriminative Experts for Multimodal Prediction Modeling[END_REF]. The purpose of the turn-taking prediction is to accurately predict the timing between speaker transitions and the upcoming type of utterance (speaker holding the floor, speaker changes) as it occurs in humanhuman interactions [START_REF] Ward | Dialog prediction for a general model of turn-taking[END_REF]. Back-channel behavior assures the speaker that the listener is paying attention and is in the same state in the conversation [START_REF] Thorisson | Natural Turn-Taking Needs No Manual: Computational Theory And Model, From Perception to Action[END_REF]. Several teams have investigated how the speaker behavior triggered listeners' back-channels [START_REF] Morency | Predicting Listener Backchannels: A Probabilistic Multimodal Approach[END_REF][START_REF] Huang | A multimodal end-of-turn prediction model: learning from parasocial consensus sampling[END_REF][START_REF] Gravano | Backchannel-inviting cues in task-oriented dialogue[END_REF][START_REF] Al Moubayed | Generating Robot/Agent backchannels during a storytelling experiment[END_REF].

Conclusions and main challenges

Several questions regarding the dimension and perception of synchrony remain to be explored. These questions are fundamental to the development of an automatic model to assess synchrony.

The first issue relates to the nature of synchrony: is synchrony an all-ornone condition (synchronous vs. non-synchronous)? Is synchrony a continuous or a discrete notion? Or can dyadic interaction can approach or move away from synchrony [START_REF] Harrist | Dyadic synchrony: Its structure and function in children's development[END_REF]? Most current sources suggest that synchrony varies over the course of interaction, being stronger at the beginning and the ending of an exchange [START_REF] Kendon | Movement coordination in social interaction: some examples described[END_REF] or at moments of particular engagement [START_REF] Campbell | An audio-visual approach to measuring discourse synchrony in multimodal conversation data[END_REF]. Feldman operationalizes synchrony as the degree to which the partners change their a↵ective behavior in reference to one another and obtains a number ranging between 0 and 1 [START_REF] Feldman | Infant-mother and infant-father synchrony : the coregulation of positive arousal[END_REF]. When addressing the matter of movement synchrony and its relation to perceived entitativity, Lakens observed that objective di↵erences in movement rhythms were linearly related to ratings of perceived entitativity [START_REF] Lakens | Movement synchrony and perceived entitativity[END_REF]. A recent study showed that the perception of coordination was more unanimous when coordination was very high or very low. However, judges were not reliable when judging dyads with "medium" coordination [START_REF] Delaherche | Characterization of coordination in an imitation task : human evaluation and automatically computable cues[END_REF].

The second issue relates to the multiple scales of interpersonal synchrony. As previously described, there are evidences of interpersonal synchrony at di↵erent levels: behavioral, neural and physiological. One of the major challenge is to propose frameworks dealing with these di↵erent levels [START_REF] Kelso | Outline of a general theory of behavior and brain coordination[END_REF][START_REF] Chatel-Goldman | Nonlocal mind from the perspective of social cognition[END_REF]. This will require specific tools and protocols in order to acquire, process and model various signals. In addition, interpersonal synchrony has been found at di↵erent timescales: ranging from milli-seconds to minutes. Social signal processing approaches should now deal with multi-scale situations using various sources of information. In [START_REF] Weisman | Oxytocin shapes parental motion during father-infant interaction[END_REF] it is described a first approach to analyze the e↵ect of oxytocin during parent-infant interaction. Understanding these mechanisms will help to propose objective evaluation of interpersonal synchrony and more generally be of great benefit for social signal processing in terms of low-resolution brain scanning [START_REF] Pentland | Using reality mining to improve public health and medicine[END_REF].

The third issue is related to acquisition and annotation of databases. Indeed, the definition of coordination is wide and di↵erent dimensions of coordination can be analyzed. Several works have shown that the similarity measures do not always predict the degree of coordination perceived by annotators. Which begs the question : what are the signals received by the annotators annotate when partners are coordinated? These questions relate to definitions and dimensions of interpersonal synchrony. In response, a collaboration with psychologists seems essential. The question of the corpus is also crucial. Like in other related domains, i.e. a↵ective computing, real-life, annotated and publicly distributed databases were a breakthrough that allowed researchers to propose new relevant models (e.g. continuous models of emotions). Indeed, define a research protocol, collect interaction data and annotate them is a long process. In addition, these baselines would compare the performance of di↵erent systems. Until (Sun et al., 2011) recent contribution of their mimicry database, no publicly available annotated corpus were dedicated to the detection of synchrony. We can hope that this e↵ort will benefit the field, aiding engineers in their work to develop new algorithms, skipping the data collection and annotation phases.

The fourth issue is related to machine understanding of interpersonal synchrony. Most of studies investigate interpersonal synchrony through similar-1.7 Conclusions and main challenges xv ity measures (ranging from correlation to recurrence analysis) in relation to variables such as pathological groups, success of interaction... Very few studies are proposing predictive approaches: evaluation against ground truth by using traditional machine learning metrics [START_REF] Petridis | Audiovisual Detection of Laughter In Human Machine Interaction[END_REF][START_REF] Michelet | Automatic Imitation Assessment in Interaction[END_REF][START_REF] Delaherche | of: Multimodal Pattern Recognition of Social Signals in Human-Computer-Interaction[END_REF]. Reasons are multiple and include obviously lack of databases. Definitions may also help to propose relevant models. For instance, in [START_REF] Delaherche | of: Multimodal Pattern Recognition of Social Signals in Human-Computer-Interaction[END_REF], imitation is considered as unsupervised action recognition problem, where the idea is to detect similar actions independently on the nature of actions performed by the partner.

The last issue relates to the identification of applications. Automatic characterization of interpersonal synchrony might be of great interest in psychology. Such methods could provide automatic and objective tools to study interactive abilities in several psychiatric conditions, such as depression and autism. Although few studies are currently available in this specific field, they appear to be very promising: couple therapy [START_REF] Lee | An Analysis of PCA-based Vocal Entrainment Measures in Married Couples, A↵ective Spoken Interactions[END_REF], success in psychotherapy [START_REF] Ramseyer | Nonverbal Synchrony in Psychotherapy: Coordinated Body Movement Reflects Relationship Quality and Outcome[END_REF], mother-infant interaction [START_REF] Cohn | Advances in Behavioral Science Using Automated Facial Image Analysis and Synthesis[END_REF]. Another great potential lies in the opportunity to build robots or virtual agents with interactive abilities [START_REF] Gratch | Creating Rapport with Virtual Agents[END_REF][START_REF] Al Moubayed | Generating Robot/Agent backchannels during a storytelling experiment[END_REF][START_REF] Prepin | Shared understanding and Synchrony Emergence: Synchrony as an Indice of the Exchange of Meaning between Dialog Partners[END_REF][START_REF] Boucenna | Learning of social signatures through imitation game between a robot and a human partner[END_REF].

Acknowledgments

This work was supported by the UPMC "Emergence 2009" program, the European Union Seventh Framework Programme under grant agreement n288241, the the Agence Nationale de la Recherche (SAMENTA program: SYNED-PSY). This work was performed within the Labex SMART supported by French state funds managed by the ANR within the Investissements d'Avenir programme under reference ANR-11-IDEX-0004-02.


Salvatore M Anzalone
email: sanzalone@univ-paris8.fr

Antoine Tanet

Olivier Pallanca

David Cohen

Mohamed Chetouani

A humanoid robot controlled by neurofeedback to reinforce attention in autism spectrum disorder

Keywords: Social robots, Neurofeedback, Autism Spectrum Disorder, EEG, Attention, Joint activities

Children with Autism Spectrum Disorder can find very difficult focusing their attention towards activities they find less attractive than the ones they like. In this paper we will introduce a prototype designed to reinforce their mental skills through neurofeedback using EEG data and a small humanoid robot to stimulate attention towards a joint activity.

Introduction

As autism Spectrum Disorder is characterized by restricted and repetitive patterns of behaviors, interests and activities [START_REF] Xavier | Autism spectrum disorders: An historical synthesis and a multidimensional assessment toward a tailored therapeutic program[END_REF], children with ASD may be able to maintain firm and durable attention over activities they like and motivates them. At the same time, in can be really hard for them to keep their attention on other, less attractive, activities. Children with ASD can develop their skills through joint activities with an expert clinician that can guide them on improving their attention span [START_REF] Dawson | Randomized, controlled trial of an intervention for toddlers with autism: the early start denver model[END_REF]. However, it is always difficult also for the most expert clinicians to infer mental states from the behavior of the children. The EEG technology, in this case, can help them by providing in real-time neuro-physiological measures resuming particular mental states, as the attention of the child towards the current joint activity [START_REF] Coben | Eeg power and coherence in autistic spectrum disorder[END_REF]. A companion robot can be an helpful tool in these scenarios, with the role of conveying such neuro-physiological measures to clinicians while acting like a playfellow with the child. At the same time, robots endowed with some kind of intelligent, social skills, can autonomously act according to particular mental states, providing feedbacks or rewards to the child [START_REF] Scassellati | Robots for use in autism research[END_REF][START_REF] Diehl | The clinical use of robots for individuals with autism spectrum disorders: A critical review[END_REF][START_REF] Boucenna | Interactive technologies for autistic children: a review[END_REF].

Past studies highlighted the possibility of employing a small humanoid robot to elicit joint attention [START_REF] Anzalone | How children with autism spectrum disorder behave and explore the 4-dimensional (spatial 3d+ time) environment during a joint attention induction task with a robot[END_REF]. In this paper we exploit this ability by introducing a robotic system designed to provide feedback to a child with ASD in accord to his mental state, to improve his attention skills toward joint activities. The presented system is still a prototype: the paper will introduce the details on its conception and a the results of a pilot study involving an adolescent with ASD.

Methodology

The system presented in this paper is a prototype designed to reinforce the attention of a child towards a particular activity, thanks to the feedbacks offered by a small humanoid robot. Among the possible choices of joint activities, paying attention towards a movie or a cartoon has been selected as a simple, stereotyped, joint activity that the child can be reasonably able to perform. A measure of attention is retrieved through the real-time exploitation of EEG signals. The small humanoid robot, as a proactive companion of the child, would return feedbacks in case of attention breakdown, with the goal of re-inducing attention towards the joint activity.

More in detail, the experimental protocol is defined as in Figure 1:

1. A movie or a cartoon, is used as audio-visual stimulus for the child. At the same time, his EEG activity is captured. 2. EEG data is exploited to extract a measure able to represent the attention the child is giving to the current task. The selected metric is able to highlight the child's attention breakdowns. 3. In case of attention breakdown, the robot produces a feedback to trigger back the attention towards the task. The feedback is defined upon a set of well pre-defined mix of verbal and non-verbal behaviours. As shown in Figure 2, the system integrates different technologies:

-An EEG cap, Enobio from NeuroElectrics; -A screen for the audio-visual stimulation;

-A small humanoid robot, Nao from Aldebaran Robotics;

Two computer are also employed: one to capture and exploit the EEG data; one to control the robot and store logs and sensors data. All the systems are synchronized through Network Time Protocol. The session is composed by two main stages:

-Training stage: The child watches an audio-visual content that assures his attention. This data is used to build the baseline for the child attention measure, to train the system how to recognize attention breakdowns. -Test stage: The child watches an audio-visual content that does not ensure his attention. When the system recognizes an attention breakdown, a feedback behavior of the robot is triggered.

At the end of each session, the child behavior is evaluated in terms of information retained, of number of attention breakdowns, and through the average of the attention measure.

Multimodal stimulus selection

The audio-visual content employed to stimulate the child is one of the most important component of the system. A stronger stimulus or a weaker one in terms of attention induced or cognitive load required to understand it, can compromise the capabilities of the system on recognizing the attention breakdowns. Several contents have been selected, distinguishing, in particular, the ones for the training stage of the system. In this case, the stimulus employed should be particularly effective on inducing attention in the children because the data captured will be used as baseline of the attention. Consequently, the stimulus should be very personalized according to the particular preferences of the child. On the contrary, the stimulus employed during the Test stage will be less attractive and engaging.

The developed prototype employed documentaries (Length: 3min) and storytelling videos (Length: 5min), as in Figure 3. Fig. 3: Two contents employed in the proposed setup to induce attention: a documentary and a storytelling video.

Attention breakdown identification

The EEG signal captured is exploited to identify attention breakdowns. Attention has been characterized through the Theta -Beta ratio of the signals captured by the central, frontal and temporal EEG sensors (c3, c5, f3, f4, t7, t8) [START_REF] Arns | A decade of eeg theta/beta ratio research in adhd: a meta-analysis[END_REF]. In particular, the average ratio between the channel has been selected as attention feature. The attention model of the child is learned at the beginning of the experimental protocol, during the Training stage, by exploiting the data obtained while an audio-visual content strongly engages the child. More in detail, in the Training stage it is supposed that the child will be attentive towards the stimulus. To model attention, then, an inhibit threshold tor the Theta -Beta ratio activity defining the attentive state was set at a power level that the training activity fell over its range for the 50% of time [START_REF] Steven | Neurofeedback in adolescents and adults with attention deficit hyperactivity disorder[END_REF]. This model will return a measure between 0 and 1, as attentive -not attentive, and will be employed by the system as convenient measure to evaluate the attention given by the child to the current activity.

Attention breakdown are identified in a sliding window of 5sec of the real-time attention measure captured. The breakdown is defined by a prevalence ( 50% of the window) of measures lower than 0.5 (not attentive).

Robot feedback

During the experiment, the robot is placed at the side of the child, attentive towards the screen, to stimulate on him imitation behaviors and joint attention towards the audio-visual stimulus. When the attention breakdown is identified, the system triggers a feedback by executing one of some specific, predefined behaviors, able to re-induce attention towards the joint activity. This set of behaviours mixes gestures, pointing in particular, and voice, as: Ah!, Look!, I love it!, Ah! I love this!, as in Figure 4.

In the presented pilot, only 8 behaviors have been implemented: 4 coupling gestures and voice; 4 proposing just non-verbal behaviors. The selection of the behavior after the attention breakdown detection is randomly chosen among the developed behaviors. Fig. 4: Two multimodal feedbacks from the robot: "ah" and "I love this"! without and with pointing gesture.

Preliminary results

To evaluate this pilot system, an exploratory study has been conducted involving an adolescent (17 years old) from the Ecole Georges Heuyer, part of the Child and Adolescent Psychiatry Service of Piti-Salptrire Medical Hospital in Paris, France. While not statistically significants, the results obtained in this pilot study are extremely important to evaluate the feasibility of the system, the possible protocols that would be employed, the adverse events that could happen and an appropriate sample size before a more exhaustive experiment. The child has has been diagnosed with autism spectrum disorder, with a particular evidence of attentional troubles. The child has been selected because he does not show any motor disability, or any other serious behavioral trouble.

The experiments involving the child have been carried on in 3 sessions during a single week: each session has been divided in an experimental condition and in a control condition.

After a common training stage, in which the system learns the attention model, according to the previous introduced methodology, the child faces the two conditions. In the experimental condition, the child is in front of the screen, side by side with the robot, that eventually intervene by giving feedbacks according to the attentional state revealed by the EEG-based measure. In the control condition, the robot does not intervene: it just stand on the side of the child, watching carefully the audio-visual stimulus. The two condition are opportunely randomized among the days of the week.

In both conditions, a trained clinician ask questions about the content proposed to the child, in order to obtain a rough measure of the retained information. This measure is coupled with the number of attention breakdowns captured by the system and with an attention score, defined as the mean of the attention measure among the whole condition.

Figure 5 shows that the amount of information retained by the child among the different sessions is higher in the case of proactive robot rather than in the case of static robot.

Table 1 shows that the number of attention breakdown events increase among the time. It shows also that these events are more frequent in the control condition than the experimental condition. Table 2 shows how the score of attention

Conclusions and Future Works

In this paper we introduced a prototype neurofeedback system designed to stimulate attention of a child towards an audio-visual stimulus. The system exploits data from EEG to estimate the attention, while employs a robot as convenient and natural interface to stimulate attention in the child. Results obtained seem to highlight a perturbing effect of the robot on the attention of the child. Such effect could be a result of the eventual presence of false positives on the attention breakdown detection system. In any case, due to the limited samples collected in this pilot study, it is difficult to obtain statistically reliable conclusion. Those results encourage the development of more precise measures of attention, mixing together EEG data and behavioral information, as the psycho-motor agitation [START_REF] Salvatore M Anzalone | Evaluating the engagement with social robots[END_REF][START_REF] Avril | Social signal processing for studying parent-infant interaction[END_REF]. In future experiments, several captures would be deployed in the environment, offering the possibility of exploiting the verbal and non-verbal behaviors of the child. These measures would be be coupled with the neurological measures obtained by the exploitation of the EEG data, to produce a more accurate and comprehensive description of the attention state of the child and, consequently, of his attention breakdowns. The protocol here introduced, with tailored multimodal stimuli, will be finally introduced in a small classroom of a special school for autistic children to statistically evaluate its effects.

Fig. 1 :

 1 Fig. 1: An overview of the experimental protocol proposed.

Fig. 2 :

 2 Fig. 2: The experimental setup.

Fig. 5 :

 5 Fig. 5: The amount of information retained by the child in the two conditions among the sessions.

Table 1 :

 1 Number of attention breakdowns in the two conditions among the sessions.among the days is almost constant in the control condition while it highlights fluctuations in the experimental condition.

	Experiment Day Control Condition Experimental Condition
	0	19	11
	1	6	3
	2	24	29
	Experiment Day Control Condition Experimental Condition
	0	0.456512	0.605021
	1	0.403154	0.267962
	2	0.477982	0.33084

Table 2 :

 2 Attention score in the two conditions among the sessions.