Youssef Hmamouche

Magalie Ochs

Laurent Prévot

Thierry Chaminade
email: thierry.chaminade@univ-amu.fr

Interpretable Prediction of Brain Activity during Natural Social Interactions using Multimodal Behavioral Signals

We present an analytical framework aimed at predicting the local brain activity of individuals during a conversation with another human or a humanoid robot based on multimodal recordings of their behavior. In this framework, we first extract high-level features from the raw behavioral recordings of both interlocutors. Then, classifiers are applied to predict binarized brain activity from these features using a dynamic prediction model. Here, we focus on brain regions involved in social interactions, both speech processing involved areas in conversations and information integration areas, in order to validate our framework. This framework not only predicts local brain activity significantly better than random, but it also identifies the behavioral features required for this prediction depending on the brain area under investigation and on the nature of the conversational partner. In the left Superior Temporal Sulcus, perceived speech is the most important behavioral feature for predicting brain activity, regardless of the agent, while multiple features, which differ between the human and robot interlocutors, contribute to prediction in regions involved in social signal integration, such as the TemporoParietal Junction. This framework allows us to study how multiple behavioral signals from different modalities are integrated in individual brain regions during inherently complex unconstrained natural social interactions.

1. A methodology for predicting discretized fMRI responses from behavioral signals

Introduction

Investigating the causes of brain activity during natural social interactions is a difficult problem given that multiple cognitive processes are at play in such complex behavior. Meanwhile, brain activity follows non-linear dynamics potentially influenced by factors that are difficult to measure, such as internal thoughts and other psychological factors.

However, it is important to evaluate which behavioral event, and more importantly combinations of such events, significantly influence the activity in local brain areas to better understand brain-behavior relationships in natural interactions.

In this article, we tackle the problem of finding dependencies between neurophysiological time series in Regions Of Interest (or ROIs, which are intermediate functional units between neurons and the whole brain, corresponding to a patch of cortical surface measuring several mm 3 ). The analysis framework can be described with three main steps. We (i) use a non-invasive neuroimaging technique (here, fMRI) to record human brain activity during a natural social interaction as well as synchronized behavior of the interacting agents across multiple modalities, such as audio or video; then (ii) high-level features are extracted from the raw recordings captured during the interaction using knowledge from cognitive neuroscience that informs us about the behaviors that can be extracted; finally (iii) machine learning methods are used to find relationships between the recorded fMRI signal and the behavioral features. To validate the analysis framework proposed, we focus in this paper on conversation as the natural social interaction, on brain areas involved in speech perception on the one hand, and on areas known to respond to several modalities of behaviors related to social cognition in controlled experimental settings on the other hand.

Testing causal relationships between time series can be performed using causality tests. Those tests are generally used to test the causality of one or multiple variables on a target variable. Many of those tests are based on prediction models, such as the Granger causality test and its alternatives [START_REF] Hlavácková-Schindler | Equivalence of granger causality and transfer entropy: A generalization[END_REF]. The problem here is that we have a large set of multimodal predictive variables, and we don't know which subset of variables we have to include in our models given a large number of possibilities. The challenge is then, first, to find the subset of behavioral features that has an impact on a given neurophysiological time series, and then to test their prediction. Therefore, we propose to first build a dynamic prediction model with a feature selection and identify the most relevant features in terms of their prediction scores. An advantage of this approach is that the model can be used not only to detect relationships between variables but also to make predictions.

The majority of existing approaches test a priori hypotheses between brain activity and behavior [START_REF] Mitchell | Predicting Human Brain Activity Associated with the Meanings of Nouns[END_REF][START_REF] Yarkoni | BOLD Correlates of Trial-by-Trial Reaction Time Variability in Gray and White Matter: A Multi-Study fMRI Analysis[END_REF][START_REF] Chen | A Gaussian mixture regression approach toward modeling the affective dynamics between acoustically-derived vocal arousal score (VC-AS) and internal brain fMRI bold signal response[END_REF][START_REF] Huth | Natural speech reveals the semantic maps that tile human cerebral cortex[END_REF], and rely on multivariate regressions to handle the problem of brain activity prediction based on behavior. Here, we propose to use advanced machine learning algorithms to predict the neurophysiological response based on recorded behaviors without relying on strong a priori hypotheses, allowing us to identify new relationships between specific aspects of complex behaviors on the one hand, and neurophysiology on the other hand. In practice, we build a framework that first extracts several multimodal features from raw behavioral signals and then automatically finds the smallest set of features that have a significant impact on the prediction of localized brain activity recorded with fMRI.

For (i), we use an existing corpus of natural conversations recorded during fMRI experiments with 24 participants, providing synchronized neurophysiological and behavioral signals [START_REF] Birgit | Brain activity during reciprocal social interaction investigated using conversational robots as control condition[END_REF]. This corpus is unique in the sense that participants' behavior is unconstrained and therefore different from classical fMRI datasets generally acquired in highly controlled conditions. This corpus includes human-human and human-robot conversations, which allows the analysis of the variability of relations between local activity and behavior not only in terms of the brain area under investigation, but also of the social context of the interaction operationalized by the nature on the conversational agent, human or artificial [START_REF] Chaminade | An experimental approach to study the physiology of natural social interactions[END_REF]. With regards to (ii), a number of recent publications explain how high-level behavioral features have been extracted from raw recordings, either conversational features extracted from raw audio recordings after transcription and annotation [START_REF] Hallart | Comparaison linguistique et neuro-physiologique de conversations humain humain et humain robot[END_REF] or visual features extracted from the video recordings of the interlocutor [START_REF] Chaminade | Perceived facial happiness during conversation correlates with insular and hypothalamus activity for humans, not robots[END_REF]. Other features used here rely on the recording of the eye movements of the participant and an exhaustive list of all raw recordings can be found in [START_REF] Rauchbauer | Multimodal Corpus of Bidirectional Conversation of Human-human and Human-robot Interaction during fMRI Scanning[END_REF]. Here we mainly focus on the machine learning aspect of the framework (iii) that has been implemented for predicting fMRI brain activity based on multimodal raw signals of human-human or human-robot conversation. It brings two major contributions: of bidirectional conversation is presented in sections Analysis Framework and Implementation.

The identification of dependencies between behavior and brain activity in specific

brain areas is presented in section Results and discussed in Discussion.

To summarize our contributions, the problem presented in this paper is specific in the sense that the data are unique and have never been used in related work for a prediction task similar to ours. The prediction methodology is also different since we are dealing with time series with the same frequency as the fMRI signals, and we have not found any related work that is similar to our research in terms of prediction approach.

Related work

Prediction using multiple modalities is a challenging task due to the diversity of the available signals to process. Classical prediction models are more simple in the sense that the input features belong to the same modality, and thus have similar structures. Concerning multimodal prediction, many problems arise, such as how to synchronize signals of different types and frequencies, how to represent data from each modality, and the fusion methodology to use for including all signals into one prediction model.

Multimodal approaches

There are several real applications that involve multimodal signals to predict a given feature. Multimodal data are very useful in emotion recognition. In [START_REF] Gupta | Multimodal prediction of affective dimensions and depression in human-computer interactions[END_REF], a system is presented to predict depression using audio, visual, and linguistic features. Similarly, in [START_REF] Brady | Multi-modal audio, video and physiological sensor learning for continuous emotion prediction[END_REF], the goal was to classify emotional states based on multimodal signals including audio, video, and physiological sensor signals. In both papers, the approaches used are based on extracting multiple features from each modality, then fusing them in one model that performs a classification task based on the extracted features. This approach seems very logical since it enables explaining the results from the variables extracted so as to make interpretation possible. On the other hand, the approach that consists in building a one-step prediction model using the raw data as input may be efficient in terms of prediction accuracy but lacks interpretability. Multimodal data are also very common in the field of human social interaction. In [START_REF] Morency | A probabilistic multimodal approach for predicting listener backchannels[END_REF], a multimodal approach is proposed to predict back-channel feedback related to bidirectional human-human interaction. These back-channel represent signs indicating the continuity of the interaction. The used model is probabilistic, and it is based on Hidden Markov Model or Conditional Random Fields. The model takes as input three types of features, the prosody, the spoken words, and the eye gaze coordinates. In [START_REF] Mckeown | The SEMAINE corpus of emotionally coloured character interactions[END_REF], the SEMAINE multimodal corpus is provided in the context of human social interaction. It consists of several bidirectional conversations with 20 participants containing both visual and speech recordings. This corpus was recorded to provide researchers with a multimodal data with two main aims: first, to analyse the interactions from a social cognition point of view; and second, to teach machines to interact with humans. For more details about multimodal approaches, an important survey about multimodal approaches has been presented in [START_REF] Baltrušaitis | Multimodal machine learning: A survey and taxonomy[END_REF]. It contains a general discussion of multimodal strategies for learning prediction models from information from different sources. The authors discuss the main and general steps for multimodal machine learning and present several real applications involving multimodal data. The authors describe a set of fundamental steps for building a machine learning multimodal system, where the most important step is data representation, which is related to extracting and summarizing useful features from the different modalities. Other steps are also discussed about coordinating between the modalities, aligning and fusing them in order to make predictions. Those steps are very important, but each application will require specific processing depending on the underlying modalities and the predictive variables.

Multimodal approaches for brain activity prediction

Regarding the issue of predicting brain activity based on behavior, several approaches have been proposed in the literature. In [START_REF] Okada | An fMRI Study of Audiovisual Speech Perception Reveals Multisensory Interactions in Auditory Cortex[END_REF], the authors investigate the effect of adding visual information to auditory speech signals on the activity of auditory cortex areas. The results show a significant increase in the activation of the studied regions of interest (ROIs) based on ANOVA analysis. In [START_REF] Mitchell | Predicting Human Brain Activity Associated with the Meanings of Nouns[END_REF], the fMRI neural activation associated with the semantic is predicted based on a large text dataset. The brain regions studied are in the sensory-motor cortex. The model used consists of transforming the text into semantic features, and then building a regression model that expresses the fMRI brain activity as a linear combination of semantic features. The authors show a prediction accuracy of 0.62 or higher, but on each participant independently. This issue has also been addressed with a multi-subject approach, by concatenating data of multiple participants. For example, in [START_REF] Huth | Natural speech reveals the semantic maps that tile human cerebral cortex[END_REF], the goal was to predict voxel activity from cortical areas, measured via the BOLD (Blood Oxygen Level Dependent) signal based on the speech signal. The data used has been collected from an fMRI experiment on 7 subjects. The methodology adopted is based, first, on constructing semantic features from natural language, then, a dimension reduction using PCA (Principal Component Analysis) is applied to reduce the number of predictive variables, and a model is learned based on multiple linear regression with regularization in order to predict the BOLD signal. Finally, the obtained prediction results and the principal components of the predictive variables are both combined to classify brain areas according to the semantic features categories. Other types of behavioral signals have been investigated by evaluating the effect of a single feature on brain activity. For example, the speech reaction time has been used to predict activity in specific brain regions [START_REF] Yarkoni | BOLD Correlates of Trial-by-Trial Reaction Time Variability in Gray and White Matter: A Multi-Study fMRI Analysis[END_REF]. In [START_REF] Chen | A Gaussian mixture regression approach toward modeling the affective dynamics between acoustically-derived vocal arousal score (VC-AS) and internal brain fMRI bold signal response[END_REF], the acoustically-derived vocal arousal score [START_REF] Bone | Robust Unsupervised Arousal Rating: A Rule-Based Framework with Knowledge-Inspired Vocal Features[END_REF] is used to predict the BOLD signal using the Gaussian mixture regression model. In [START_REF] Knops | Recruitment of an Area Involved in Eye Movements During Mental Arithmetic[END_REF], the authors predict the BOLD signal in the posterior parietal cortex based on eye movement data using a multivariate regression model. More general approaches have been tried to predict the brain activity of various areas using different types of signals at the same time. For example, in [START_REF] Desouza | Correlating Behavioral Responses to fMRI Signals from Human Prefrontal Cortex: Examining Cognitive Processes Using Task Analysis[END_REF], correlations are analyzed using linear regression between the BOLD signal and behavioral features computed from observed facial expressions, speech reaction time, and eye-tracking data.

Discussion

In the related works presented above, dependencies between behavior and specific functional brain areas have been investigated. However, only one or a few modalities have been taken into account. In addition, the methods used are generally based on correlation analysis or multiple regression. However, finding relevant predictive features using feature selection techniques with machine learning methods, such as prediction models based on artificial neural networks, can be particularly relevant to this research question. In this article, we propose a framework that consists in extracting highlevel features from raw multimodal behavioral data consisting of audios, videos and eye-tracking recordings, then applying feature selection and prediction with different classifiers to predict discretized neuro-physiological signals in circumscribed brain regions from multimodal behavioral signals.

.

Analysis Framework Overview

The analysis framework presented is rooted in a meta-model using machine learning tools which allows to 1) predict brain activity from multimodal behavioral signals recorded in complex natural social interactions, and 2) to identify dependencies between brain activity and the high-level features extracted from the raw signals. The proposed framework is illustrated in Fig 1 . In this section, we present an overview of its main steps. The methodology presented may be used with other input signals and other types of brain activity, such as EEG and MEG, but would require additional assumptions and processes, taking into account for example the non-independence of successive data points.

The output of the framework consists of the predictions of regional brain activity and the features selected for the prediction, and in particular the importance score of the selected behavioral features. The importance of the features is generally missing in most of the existing works that merely identify features that trigger the activation of a brain area [START_REF] Yarkoni | BOLD Correlates of Trial-by-Trial Reaction Time Variability in Gray and White Matter: A Multi-Study fMRI Analysis[END_REF][START_REF] Chen | A Gaussian mixture regression approach toward modeling the affective dynamics between acoustically-derived vocal arousal score (VC-AS) and internal brain fMRI bold signal response[END_REF][START_REF] Bone | Robust Unsupervised Arousal Rating: A Rule-Based Framework with Knowledge-Inspired Vocal Features[END_REF][START_REF] Knops | Recruitment of an Area Involved in Eye Movements During Mental Arithmetic[END_REF]. An additional output is also when the regional activity can't be predicted, given the brain region of interest, the experimental condition scrutinized and the features used for the prediction.

The first step is to extract features from raw input signals for each modality separately in order to construct time series that will be used as predictive variables. But they require resampling in order to obtain time series with the same number of observations as the input signals have different recording frequencies. Finally, the predictive variables also need to be restructured as sequences, since we are trying to predict the next brain activity based on the past values of the behavioral features. Then, feature selection methods and prediction models are applied to predict the discretized BOLD signals, and also to find the smallest subset of features leading to the best possible predictions for each brain area. Thus, the system does not require a priori hypotheses about the relationship between brain activity and behavior. Instead, it allows the finding of new causal relationships between conversational behavior and brain activity, which can be interpreted from a neuroscience and social cognition point of view in order to identify complex multimodal cognitive mechanisms. Our approach consists in using a new prediction model based on a specific temporal function to predict the discretized brain activity time series based on the behavioral features.

Feature extraction (ii)

Integrating multimodal signals within the same deep learning architecture is a delicate task compared to classical unimodal architectures. With multimodal signals, different networks must coordinate to have efficient and robust predictions. Our idea is to transform all the input modalities into interpretable sequences, i.e., features that describe behaviors that can be described verbally, such as a binary time series describing whether one person is speaking or not. Such a strategy allows us to have a unified classification network that not only works for all modalities together but can also work with any single one.

The aim of feature extraction in our case is to compute high-level features from raw recordings as the latter are not easy to introduce in machine learning models and not easily interpretable.

The importance of these features is demonstrated by the fact that they are more effective to explain brain activity in higher-order integrative brain areas, that are of major interest when investigating complex social interactions. While raw recordings that consist of audios and videos are likely to affect in priority primary brain areas in the auditory and visual cortex.

A similar approach was used for the emotions extracted from voice signals, showing a shift from early feed-forward processing of stimulus categories to later processing of the salience of the stimuli [START_REF] Giordano | The representational dynamics of perceived voice emotions evolve from categories to dimensions[END_REF]. This step, based on domain knowledge, provides the input behavioral features that are known to affect brain activity and are amenable to improvements as knowledge itself progresses, providing new features that can be extracted from raw signals, used in the prediction model and allowing to evaluate their predictive power. Iteratively, such an approach could help refine the types of representations used in the brain. As a result, an obvious limitation, but also a strength, of the approach, is that the current results only provide a snapshot constrained by the raw signals used and the features extracted in the current implementation, but that can later be compared when new features become available and added to this extraction step.

Machine learning (iii)

Feature selection

Feature selection is performed on the variables representing the temporal sequences of the extracted recorded features. The goal here is to identify the most relevant set of variables in terms of the prediction accuracy for each variable that will be predicted. In the following, we detail three different methods that we have used for feature selection.

Wrapper feature selection method

This method uses the prediction model itself to perform the selection of the appropriate variables. First, the prediction model is executed with all features. Then, the top-k features are selected based on their weights provided by the model. This method is simple in terms of implementation, but its main drawback is that the selected features are specific to the prediction model used, which means that if we change the model, the selected features can also change.

Ranking based on mutual information

The filter method selects variables without the need of a prediction model. It ranks variables based on their Shannon mutual information (MI) with the target variable. The Shannon MI works only for discrete variables, for example, the bivariate MI between two variables X, Y can be expressed as follows:

I (X; Y ) = x,y P (x, y) log P (x, y) P (x) P (y) (1)
In our case, we can't use formulation [START_REF] Hlavácková-Schindler | Equivalence of granger causality and transfer entropy: A generalization[END_REF] directly because some features are continuous. Two solutions are possible to overcome this problematic, the first and most common one consists in discretizing the continuous features, and then to use the Shannon MI. The second one is to use a continuous estimation of the Shannon MI, as proposed in [START_REF] Kraskov | Estimating mutual information[END_REF]. The authors present an estimation of MI based on k-nearest neighbors approach, which relies an idea similar to the one for estimating the continuous entropy [START_REF] Kozachenko | Sample estimate of the entropy of a random vector[END_REF]. This MI estimator can be expressed as follows:

Î(X; Y ) = Γ(k) + Γ(n) - 1 n n i=1 (Γ(n x (i) + 1) + Γ(n y (i) + 1)), ( 2
)
where Γ is the Gamma function, n is the number of observations of X and Y , k is the number of neighbors to consider, n x (i) is the number of points where the distance from x i is strictly less than d i /2 where d i is the distance from X i to its k th neighbor.

Clustering based method

This method is another filter-based feature selection method. It corresponds to an extension of the previous method in the sense that it considers the mutual information between predictive variables in addition to their mutual information with the target variable. It first groups close predictive variables into clusters using the k-medoids algorithm based on the principle of maximizing mutual information within clusters and minimizing mutual information between clusters. Then, it selects one variable from each cluster based on the mutual information with the target variable. This method is designed to work with multimodal data, since the fact of grouping variables before ranking them seems a relative solution to the problem of dependencies between variables that belong to the same group.

Predictions

Our approach is based on a multimodal fusion. This step creates a shared representation of the features irrespective of their modality. The sequenced behavioral features obtained previously are fed into deep network classifiers and then fully connected layers. The classifiers used to belong to two types. The first one includes the Random Forest (RF), Support vector Machine (SVM), and Logistic Regression (LReg).. The second type includes models based on neural networks including a fully connected network and an LSTM (Long Short Term Memory) network. For both networks, the backpropagation (through time for LSTM) is applied to train the network over 50 iterations using the stochastic gradient descent (SGD) algorithm. The multimodal architecture with the LSTM network is illustrated in Figure 4.

To measure the robustness of our predictions, we added a baseline classifier that generates random predictions (named Rand in the rest of this article). We trained this classifier after tuning a parameter representing the strategy of prediction generation. Three strategies are considered: A stratified way by generating predictions regarding the distribution of the training data, a uniform way by generating predictions uniformly, and Finally, Student's t-tests are performed to test the equality (null hypothesis) of the average F-scores between the best and the baseline classifier obtained in the training step. Student's t-tests are the most recommended and used statistical tests to compare machine learning models [START_REF] Yu | Blocked 3× 2 cross-validated t-test for comparing supervised classification learning algorithms[END_REF].

Implementation

Here, we describe how the general framework described in the previous section was used to analyse a specific dataset. More precisely, the proposed analysis framework is used to 1) predict brain activity from multimodal signals of bidirectional human-human and human-robot conversations, and 2) to identify dependencies between brain activity and high-level features extracted from raw behavioral signals. The dataset is fully relevant to the issues the analysis framework is supposed to address, namely, physiological and multi-modal behavioral recordings performed during complex behaviors, between which we want to identify dependencies. The complex behaviors under scrutiny are unconstrained conversations between participants and a fellow human and a robot. Inputs of the framework are, on the one hand, brain activity measured in fMRI via the BOLD signal, and on the other hand, behavioral signals of bidirectional conversations between a participant whose brain activity is scanned inside the fMRI machine and an interlocutor, either a human or a robot, located outside the fMRI machine (here, three types of conversational signals: speech, video, and eyetracking).

Corpus (i)

The data used in this work were collected with fMRI recordings described in previous work [START_REF] Birgit | Brain activity during reciprocal social interaction investigated using conversational robots as control condition[END_REF], and are available at [24]. It consists of four sessions for 24 participants, each containing six conversations of 60 seconds, three with a human and three with a conversational robot in alternating order. An advertising campaign provides a cover story to make sure that the participants are unaware that the actual focus of the experiment is to record a corpus of natural social interactions: participants are told that they should guess what is the message carried by images in which fruits appear either as superheroes or rotten. Each conversation between the participant and either a confederate of the experimenter or a Furhat conversational robot [START_REF] Moubayed | Furhat: A Back-Projected Human-Like Robot Head for Multiparty Human-Machine Interaction[END_REF] (controlled by the confederate in a Wizard-of-Oz mode, unknown to the participant), is about one single image of the purported advertising campaign. The project received ethical approval from the Comité de Protection des Personnes (CPP) Sud-Marseille 1 (approval number 2016-A01008-43). Written consent was obtained from all participants. The input raw data consist of 3 type of signals: video of the interlocutor (human or robot), speech (raw audio recordings and manual transcriptions) of both the participant and the interlocutor, and eye-tracking recordings of the participant. Note that the videos of the participants are not recorded, as they were inside the scanner during the fMRI experiment.

fMRI data preparation

The fMRI data requires processing. First, whole brain recording is processed to remove as much of the noise it contains as possible. It is then parcellated in regions of interest to summarize the activity of functionally homogeneous areas. The mean data is extracted in each region of interest, single trials extracted from the continuous recordings of the session, and finally binarized. In addition, the BOLD response lags from the behavioral sources with a delay modelled by the hemodynamic response function that needs to be taken into account.

Processing fMRI signals

Standard fMRI acquisition procedures were used, described in detail in [START_REF] Birgit | Brain activity during reciprocal social interaction investigated using conversational robots as control condition[END_REF]. BOLD signal 3-dimensional images are recorded in the whole brain every 1.205 second (repetition time). Standard SPM12 preprocessing procedures are used [START_REF] Penny | Statistical Parametric Mapping: The Analysis of Functional Brain Images[END_REF], including correction for time delays in slice acquisition ("slice timing"), image realignment, magnetic field inhomogeneities correction, normalization to the standard MNI space using the DARTEL [START_REF] Ashburner | A fast diffeomorphic image registration algorithm[END_REF] procedure for coregistration of individual participants' anatomy, and finally spatial smoothing with a 5-mm full-width half-maximum 3-dimensional Gaussian kernel. Extraction of the BOLD signal in regions of interest is performed using the conn toolbox [START_REF] Whitfield-Gabrieli | Conn: A Functional Connectivity Toolbox for Correlated and Anticorrelated Brain Networks[END_REF], and includes several denoising procedures, firstly a linear detrending using a high-pass filter with a threshold of 128 seconds, secondly using realignment parameters to calculate nuisance regressors related to participants' movement during scanning, thirdly taking heartbeat and breathing recordings to remove physiological artefacts with the PhysIO toolbox [START_REF] Kasper | The PhysIO Toolbox for Modeling Physiological Noise in fMRI Data[END_REF], and finally extracting BOLD signal in the white matter and cerebrospinal fluid and using the 5 first eigenvariates of the time series as nuisance representing signal fluctuations in non-cortical brain tissues.

Brain Regions Of Interest (ROIs)

A 275-area parcellation based on functional and anatomical connectivity patterns ( [START_REF] Fan | The Human Brainnetome Atlas: A New Brain Atlas Based on Connectional Architecture[END_REF], https://atlas.brainnetome.org/bnatlas.html) defines ROIs for the whole brain. Continuous time series (385 time points) are extracted for each ROI and each session and participant represents the mean activity within the ROI after denoising. In this paper, we focus on specific regions (Fig 2 andTable 1) involved in social cognition (TemporoParietal Junction, Precuneus, ventral and dorsal medial PreFrontal Cortex) including speech perception (in the caudal superior temporal region) and emotional processing (Amygdala). Two different types of control ROIs were also included, one corresponding to the white matter, where signal fluctuations are not supposed to reflect neuronal information processing and shouldn't be predictable, and another in the Primary Visual Cortex where predictions should rely on visual information instead of auditory or social information.

Binarization

Importantly, the raw BOLD is a continuous measure. A binarization of the signal into 0 (inactive) and 1 (active) is required as our approach consists in predicting whether a brain region is active or not. Such binarization of the BOLD signal has been used repeatedly when machine learning approaches are applied to fMRI signals, e.g. [START_REF] Ezaki | Energy landscape analysis of neuroimaging data[END_REF][START_REF] Deco | Hierarchy of Information Processing in the Brain: A Novel 'Intrinsic Ignition' Framework[END_REF][START_REF] Watanabe | A pairwise maximum entropy model accurately describes resting-state human brain networks[END_REF][START_REF] Weistuch | Metabolism modulates network synchrony in the aging brain[END_REF][START_REF] Ezaki | Closer to critical resting-state neural dynamics in individuals with higher fluid intelligence[END_REF].

Here, we use a binarization method proposed by Ostu et al. [START_REF] Otsu | A threshold selection method from gray-level histograms[END_REF] reproducing previous approaches based on filtering the BOLD signal of each brain area into two states (active, or non-active) based on its average.

Compensating for the hemodynamic response delay

The BOLD signal follows the Hemodynamic Response Function (HRF), which characterizes the Blood-oxygen-level-dependent (BOLD) signal to a single behavioral event that is recorded [START_REF] Gössl | Bayesian modeling of the hemodynamic response function in BOLD fMRI[END_REF]. This function peaks with a delay of around five seconds, but that can vary somehow around this value depending on the brain area, the participant, as well as other factors that are not well known. To handle this variability, we express the discretized bold signal at time t based on a sequence of 4 previous consecutive observations of behavioral features between, which span the duration between t -τ 1 and t -τ 2 , where τ 1 = 7.2s and τ 2 = 3.6s (Fig 3).

January 11, 2023 10/24 t-7τ t-6τ t-5τ t-4τ t-3τ t-2τ t-τ t t+τ

Behavioral events Hemodynamic response Figure 3. The time delay corresponding to the hemodynamic response to behavioral events is taken into account in the model by considering four consecutive behavioral time points happening between 7.2 and 3.6 seconds before the time t where the hemodynamic response is evaluated.

The number of observations in each sequence depends on the re-sampling rate of behavioral features with respect to the target variable. In the current study, we resample them with the same frequency of the BOLD signal, that is, one observation each τ = 1.2s. We also tried to re-sample the BOLD signal based on interpolation to have an observation of each 0.6s to double the number of temporal features. The performance results show that this re-sampling does not improve the predictions while requiring more computational time. Therefore, we focus in this work on the same re-sampling rate for both BOLD and behavioral features. Let y t be the discretized time series associated to the BOLD signal, and x t = {x 1,t , x 2,t , . . . , x k,t } are k behavioral time series, representing the predictive features, where x i,t is the i th variable at time t. We use the following notations to represent the sequences of behavioral variables in a concise way:

x t-τ1:t-τ2 i = (x i,t-6τ , x i,t-5τ , x i,t-4τ , x i,t-3τ ) . (3)
As mentioned before, we aim at predicting y at time t based on a sequence of x between t -8s and t -4s, with a time-step τ = 1.2s. This is a temporal classification problem, and with the previous notations, our dynamic model can be expressed as follows:

y t = f x t-τ1:t-τ2 1 , . . . , x t-τ1:t-τ2 k + e t , ( 4
)
where f is function of the model, and e t represents its error vector.

Behavioral features processing (ii)

The aim of feature extraction is to compute high-level features from multimodal raw behavioral data, that may describe specific social and conversational factors involved in a conversation. Our feature extraction approach is based on interpretable features that are computed automatically from raw multimodal signals. From the raw recordings, we extract high-level features described in Table 2. They are also described with additional details in [START_REF] Birgit | Brain activity during reciprocal social interaction investigated using conversational robots as control condition[END_REF]. The extraction process is in itself performed using deep learning models that rely on multiple types of networks:

• Computer vision-based networks for emotion recognition, face and eye movements, etc.

• Classical and recurrent neural networks for audio and text data: sentiment analysis, semantic and structural features from the text, spectral features, etc.

• Classical Time series analysis for eyetracking signals.

Facial features are directly extracted from the video of the interlocutor, speech features are extracted from manual transcriptions of the participant and interlocutor's recorded conversations, and eye-tracking features are extracted from the participant's gaze recorded inside the fMRI scanner. The features are constructed as time series for each conversation. For example, in the case of speech, we analyzed Inter-Pausal Units (IPU1 ) by IPU (for example to compute the lexical richness) or word by word (e.g., to compute Feedback and Discourse Markers). For videos, Openface 2.0 toolkit [START_REF] Baltrusaitis | OpenFace 2.0: Facial Behavior Analysis Toolkit[END_REF] is used to detect facial action units, landmarks, head pose estimation and gaze coordinates. Eyetracking coordinates of the participant are recorded using the Eyelink1000 system. We added other features characterizing where the participant is looking (Face, Eyes, Mouth), by combining the detected landmarks points of the conversant and the gaze coordinates.

In total, more than 40 features are extracted. Table 2 contains the names of the extracted features per modality with a description of how each feature is extracted. For more details, all features are available online2 , and are also described in [START_REF] Birgit | Brain activity during reciprocal social interaction investigated using conversational robots as control condition[END_REF].

To make the results more interpretable, we create higher-order features by grouping individual features (see Table 2) into meta-features that each represent one aspect of behavior that is relevant to social cognition (see Table 3). This underlying assumption is that it is not the importance score of each individual feature that is relevant, but the importance scores of all features pertaining to similar aspects of social cognition. For example, all individual features measuring the head movements of the interlocutor (translations, rotations, energy) are pooled together as "Head-movements of the interlocutor" or Head-Movement-I. Table 3. Regrouping the extracted features into social meta-features (suffix "-P" for the participant, and "-I" for the interlocutor).

Meta-features

Original features The extracted features require further processing in order to homogenize their structure. This processing includes resampling the obtained features with respect to the frequency of the fMRI signal and a concatenation of the time series of all trials and participants.

Head

Predictions (iii)

The framework proposed in the previous section Analysis Framework is a general metamodel that can be used in a variety of ways depending on the dataset and under investigation. Now, we describe how it was applied to the dataset described above.

Human-human and human-robot data are evaluated separately in order to compare results from the two conditions. For each condition, the obtained data consist of 13248 observations. We fix the training set to 18 participants from 24, and we apply the ADASYN algorithm [START_REF] He | ADASYN: Adaptive synthetic sampling approach for imbalanced learning[END_REF] on this set to address the problem of imbalanced data. This algorithm generates new observations by considering the distribution of the data. Then, we performed a 9-fold-cross-validation on training data to find the appropriate parameters of the classifier based on the F-score measure. We choose 9 folds instead of the classical 10 folds because we use data from 18 participants out of 24 in the training set and we want to evaluate the models on data of participants unseen by the models in order to avoid over-fitting. The models are then tested on data of 6 participants, that is, 25% of all data. Student's t-tests are performed to test the equality (null hypothesis) of the average F-scores between the different models and the baseline classifier obtained in the training step via the 9-fold cross-validation, with a significance threshold of p≤0.05.

Results

In this section, we present the obtained prediction results. They include performance scores of the used classifiers, as well as interpretable results about the most relevant features for each ROI. The results include recall score, and the set of the best predictive meta-features and their respective scores, to reach the prediction. Note that a threshold of 10% is used for reporting meta-features to focus on the most important ones. Tables 5 and6 contain these results for human-human and human-robot interactions respectively.

Discussion

Results indicate that the proposed analysis framework is able to predict brain activity higher than chance in many brain areas but not in the white matter, as expected given that the white matter is not known to contain large BOLD responses. Even if a such response can be found with very specific settings, it is unlikely to be associated with the subtle and complex behavioral features we are investigating here. In addition, we used a whole-brain white matter mask to extract the white matter signal, so that any localized response is lost by the averaging of signal coming from different fibres, and it was also regressed out during the denoising of the fMRI data. This confirms the validity of the significant results. Importantly, predictions are significantly greater than chance for both human and robot conditions (HHI and HRI respectively) in the Superior Temporal Sulcus (STS) and TemporoParietal Junction (TPJ) bilaterally. The former was selected as being an area strongly devoted to language perception, as this region of the cortex also contains "voice patches", regions responding specifically to the perception of voices [START_REF] Pernet | The human voice areas: Spatial organization and inter-individual variability in temporal and extra-temporal cortices[END_REF]. Therefore, perceiving speech from others is likely to activate these patches in a simple on/off fashion. Indeed, the highest F-scores are obtained in this region in both hemispheres and both interaction conditions. Furthermore, on the left side, dominant for language, the interlocutor's speech (SpeechActivity-I) is the only predictor (with an importance score of 1) used to reach a recall score of 0.69 in the HRI condition (see Table 6). Yet, there is ongoing speculation about a more general involvement of this part of the cortex in social cognition, for example in visual perception [START_REF] Allison | Social perception from visual cues: role of the STS region[END_REF]. The second meta-feature used for prediction on the right STS region in HRI is the speech produced by the participant (with an importance score of 0.25), indicating a complex interaction between producing and perceiving speech in the non-dominant hemisphere. But in HHI, other meta-features are identified , pertaining to language in the left hemisphere (Interpersonal-I combines linguistic features associated with social aspects of the interaction) but also related to visual social cues (social gaze is related to the scanned participants' visual exploration of the interlocutor, including mutual gaze). In other words, while in both HRI and HHI, the second predictor for the right STS is the speech of the participant, these two speech-related meta-features are completed with social ones bilaterally for the human interlocutor, but not for the robot. Altogether, this results derived from a natural discussion with a natural and artificial agent fit largely with expectations, with specialized left STS for speech perception, right STS being also involved in speech production, and non-verbal social behavioral meta-features when the interlocutor is a human.

A second series of observations can be made for areas associated with social cognition in which we obtained significant F-scores. These areas include rTPJ, lTPJ, and rPre for both HHI and HRI, and VMPFC, Amy and rDMPFC for HHI only. First, their predictions score are smaller than those of the left and right STS. Then, in a number of these areas, notably the left TemporoParietal Junction (TPJ) for both HHI and HRI and the left and right Precuneus (lPre & rPre), right Amygdala (rAmy) for HHI only, a combination of several meta-features, combining different types of signals, including social ones, are combined to provide significant predictions. These two remarks support the hypothesis that these areas are associating more complex multimodal signals than the STS. Given that we acknowledge that defining all features (and meta-features) relevant for social cognition is exploratory, we do expect activity in these areas to be less predictable. Interestingly, the speech activity of the participant is the only predictor for the right TPJ in HRI, while the speech activity for the interlocutor was the only meta-feature for right STS despite the closeness of these two areas on the cortical surface (see figure 2. This result rules out the possibility that non-specific relations could be identified due to the proximity of our regions of interest. It should also be noted that a number of meta-features unrelated to speech are used to significantly predict activity in the left TPJ for both human and robot interlocutors, including visual signals (head movements in HHI, facial movements in HRI), highlighting a complex function of combining information from different modalities while experimental paradigms that focus on individual sources of information are usually preferentially associated with right hemisphere responses. It is possible that complex associations of multiple modalities are not correctly captured by classical experimental paradigms focusing on more controlled, and unimodal, aspects of social cognition.

The right Primary Visual Cortex (rV1) corresponds to the input of retinal information in the cortex. Indeed, its response is explained by the head and facial movements of the interlocutor as well as the participant's eye movements, all related to the visual input. It was not predicted in HRI, possibly because the robot's movements were very limited compared to the human's. The medial PreFrontal Cortex region is only associated with speech production in HHI, which is more consistent with its role in motor control language [START_REF] Picard | Motor areas of the medial wall: a review of their location and functional activation[END_REF] than in mentalizing [START_REF] Bird | The impact of extensive medial frontal lobe damage on 'Theory of Mind' and cognition[END_REF]. Its absence in HRI could also be explained by the fact that speech was also lessened in HRI compared to HRI. Predictions in areas involved in emotional aspects of social cognition, VentroMedial PreFrontal Cortex and Amygdala, yield very similar results: activity can only be predicted for human-human interaction and in the right hemisphere. Several meta-features are used for predictions, that include Interpersonal-I for the amygdala and eye and head movements from the participant and interlocutor respectively for the VentroMedial PreFrontal Cortex. Altogether, these results concerning higher-order associative areas (that are shared to a large extend by the Precuneus too, known to have a more complex response in HHI than HRI in this experiment [START_REF] Spatola | Precuneus brain response changes differently during human-robot and human-human dyadic social interaction[END_REF])-increased predictions for the human interactions, limited to the right hemisphere, and relying on complex associations of meta-features -all indicate that the analysis approach used here is valid to identify relations between complex multimodal recordings of natural behaviors and local brain activity, but it is less statistically powerful when relations between multimodal behavioral features and brain activity are limited -either by the limited association between the behavior and cognition in the case of emotion processing in the left hemisphere or by the poverty of the interaction in the case of the robot. This final remark is confirmed by the p-values of the statistical test performed of the mean of F-scores of the best classifier and the baseline (random predictions generator) using the Students t-test; the results show that the p-values obtained for HRI are greater than those of HHI for all brain areas investigated. This implies that the brain activity in HRI are less predictable than in HHI, as was predicted given that interacting with a human is more natural than interacting with a robot

Conclusion

In this paper, we present a new framework for predicting fMRI brain activity from behavioral signals and identifying dependencies between them during natural conversations between two agents. Evaluations are made on a corpus containing several human-human and human-robot conversations of 24 participants. We focused on brain areas involved in speech perception and in social interaction. We obtained predictions significantly better than those obtained with a random classifier in many of the brain regions investigated. The obtained dependencies confirm existing hypotheses about the relationship between circumscribed aspects of behaviors and functional brain areas but also allows to address new questions about the multimodal integration of behaviors in brain areas involved in higher-order aspects of cognition. The framework's output provides new and precise results, by finding the best possible model, the associated subset of relevant features and a quantification of their impact for each brain area. In addition, they allow comparing the difference between brain activation in the cases of human-human and human-robot

  the third way is based on the most frequent class. The strategy that provides the best prediction results in the training stage is the one that is kept for testing the baseline classifier. The classifications metrics used are the F-score and the Recall. Recall is the percentage of examples classified as positive, among the total number of positive examples, while Precision is the percentage of true positive examples among the examples classified as positive. The F-score is more balanced since it considers both false positives and false negatives. Note that Precision can be inferred from the results since the F-score is the weighted average of Precision and Recall.

Figure 2 .

 2 Figure 2. Regions of interest (ROIs) under investigation superimposed to sagittal sections of the average of the participants' brains normalized to the Montreal Neurological Institute (MNI) template space. Sections' location (in mm from the midbrain section, negative for the left hemisphere) are indicated by numbers above the sections and on the reference three-dimensional render seen from the front on the right panel. ROI order as in Table 2 for clarity: Primary Visual Cortex (V1): brown; Superior Temporal Sulcus (STS): Blue; TemporoParietal Junction (TPJ): Yellow; Precuneus (Pre): Pink; Amygdala (Amy): Red; VentroMedial PreFrontal Cortex (VMPFC): Green; DorsoMedial PreFrontal Cortex (DMPFC): cyan.

  -movement-I Head-Rx-I, Head-Ry-I, Head-Rz-I, Head-Tx-I, Head-Ty-I, Head-Tz-I, Head-translation-energy-I, Head-rotationenergy-I Facial-movement-I AU-all-I, AUs-mouth-I, AUs-eyes-I, Neutral-I Emotions-I Angry-I, Disgust-I, Fear-I, Happy-I, Sad-I, Surprise-I, Smiles-I Eyetracking-P Gaze-speed-P, Gaze-movement-energy-P, Saccades-P Social-gaze Face-looks-P, Mouth-looks-P, Eyes-looks-P, Direct-gaze-I Interpersonal-I FilledBreaks-I, Feedbacks-I, Discourses-I, Particles-I, Laughters-I, Interpersonal-I, Polarity-I, Subjectivity-I, Turn-Latency-I Interpersonal-P FilledBreaks-P, Feedbacks-P, Discourses-P, Particles-P, Laughters-P, Interpersonal-P, Polarity-P, Subjectivity--P LexicalRichness-P, TypeToken-Ratio-P

Figure 4 .

 4 Figure 4. Illustration of our multimodal prediction approach. Feature extraction is performed based on specific methods and pre-trained models. Then the obtained time series are re-sampled and concatenated, then they are fed to a network composed of LReg and fully connected layers.

  An illustration of the analysis framework. It is composed of three main steps. The first one (i) includes input signal acquisition and processing. The second

	(i) Behavioral signals	(ii) Feature Extraction	(iii) Machine learning
	Raw recordings	Feature	Extraction	Resampling		Sequence	Representation	Feature	selection	Prediction	1) Brain activity predictions 2) Depen-dencies
	Brain activity	fMRI Processing				Supervised
	fMRI recordings	Denoising	Parcellation	Time series	extraction	Binarization			learning
	Figure 1.

step (ii) is for feature extraction and resampling. In this step, high-level and interpretable features are extracted from input signals, and since the signals are multi-modal, they are resampled and sequenced. The final step (iii) consists in training/testing machine learning models to predict brain activity time series in order to find dependencies between the extracted features and the target variables.

Table 1 .

 1 The regions of interest (ROIs).

	Abbreviations Brain areas	Brainnetome atlas
	l,r V1	left and right Primary Visual Cortex	191,192
	l,r STS	left and right Superior Temporal Sulcus	75,76
	l,r TPJ	left and right TemporoParietal Junction	143,144
	l,r Pre	left and right Precuneus	153,154
	l,r Amy	left and right Amygdalae	213,214
	l,r VMPFC	left and right VentroMedial PreFrontal Cortex 41, 42
	r DMPFC	right DorsoMedial PreFrontal Cortex	12
	WM	White Matter

Table 5 .

 5 Meta-features (with importance score ≥ 0.10) used to significantly predict ROIs activity in conditions of human-human interaction (HHI). Recall score is obtained from the best model classifier (see Table8).

	ROI	Recall score	Meta-features	Importance scores
	rV1	0.63	Facial-movement-I	0.45
			Eyetracking-P	0.38
			Head-movement-I	0.16
	lSTS	0.71	SpeechActivity-I	0.85
			Interpersonal-I	0.12
	rSTS	0.70	SpeechActivity-I	0.52
			SpeechActivity-P	0.21
			Social-gaze	0.15
	lTPJ	0.62	SpeechActivity-P	0.33
			SpeechActivity-I	0.25
			Head-movement-I	0.21
			Social-gaze	0.11
	rTPJ	0.70	SpeechActivity-P	0.67
			Interpersonal-P	0.33
	lPre	0.63	Eyetracking-P	0.58
			Head-movement-I	0.16
			Facial-movement-I	0.15
			SpeechActivity-P	0.11
	rPre	0.63	Head-movement-I	0.38
			SpeechActivity-P	0.17
			Facial-movement-I	0.15
			Social-gaze	0.11
	rAmy	0.63	SpeechActivity-P	0.26
			SpeechActivity-I	0.23
			Interpersonal-I	0.15
			Linguistic-Complexity-I	0.15
	rVMPFC	0.61	Eyetracking-P	0.45
			Head-movement-I	0.34
	rDMPFC	0.72	SpeechActivity-P	1.00

Table 6 .

 6 Meta-features (with importance score ≥ 0.10) used to significantly predict ROIs activity in conditions of human-robot interaction (HRI). Other details as in Table5.

	ROI	Recall score	Meta-features	Importance scores
	lSTS	0.69	SpeechActivity-I	1.00
	rSTS	0.70	SpeechActivity-I	0.75
			SpeechActivity-P	0.25
	lTPJ	0.62	Eyetracking-P	0.19
			SpeechActivity-I	0.16
			Facial-movement-I	0.15
			SpeechActivity-P	0.14
			Interpersonal-I	0.13
	rTPJ	0.67	SpeechActivity-P	1.00
	rPre	0.62	SpeechActivity-P	0.88
			Eyetracking-P	0.11

Table 7 .

 7 The F-scores obtained by each model.

	ROIs			F-scores HHI					F-scores HRI
		LReg LSTM FNN RF	SVM Rand LReg LSTM FNN RF	SVM Rand
	lV1	0.58 0.57	0.56	0.57	0.57	0.50	0.52	0.53	0.53	0.56 0.55	0.50
	rV1	0.59	0.59	0.55	0.60 0.60 0.53	0.56	0.57	0.56	0.58 0.58 0.51
	lSTS	0.71 0.70	0.70	0.71 0.68	0.50	0.68	0.68	0.66	0.69 0.68	0.51
	rSTS	0.70	0.69	0.69	0.71 0.68	0.49	0.70	0.71 0.70	0.67	0.63	0.52
	lTPJ	0.58	0.58	0.59	0.61 0.60	0.50	0.59 0.58	0.55	0.58	0.56	0.51
	rTPJ	0.63	0.63	0.61	0.64 0.64 0.50	0.62	0.62	0.59	0.63 0.63 0.54
	lPre	0.55	0.58	0.55	0.60	0.59	0.51	0.52	0.54	0.53	0.57 0.56	0.50
	rPre	0.57	0.58	0.58	0.62 0.60	0.50	0.49	0.49	0.50	0.56	0.57 0.50
	lVMPFC	0.47	0.50	0.55	0.54	0.55	0.49	0.47	0.52	0.54 0.54 0.54 0.50
	rVMPFC	0.54	0.55	0.57	0.57	0.58 0.50	0.47	0.45	0.49	0.54 0.54 0.51
	rDMPFC	0.66 0.60	0.57	0.65	0.65	0.54	0.66 0.64	0.58	0.62	0.64	0.54
	lAmy	0.58	0.55	0.54	0.59 0.53	0.49	0.55	0.58	0.55	0.58	0.61 0.51
	rAmy	0.53	0.57	0.58 0.57	0.54	0.48	0.64 0.61	0.58	0.61	0.61	0.51
	WM	0.47	0.46	0.49	0.54 0.53	0.51	0.46	0.46	0.46	0.51 0.38	0.51

Table 8 .

 8 The Recall scores obtained by each model.

	ROIs			Recalls HHI					Recalls HRI
		LReg LSTM FNN RF	SVM Rand LReg LSTM FNN RF	SVM Rand
	lV1	0.62 0.62 0.57	0.58	0.61	0.49	0.59 0.59 0.59 0.56	0.55	0.49
	rV1	0.63 0.59	0.55	0.60	0.62	0.52	0.63 0.61	0.60	0.60	0.58	0.50
	lSTS	0.71 0.71 0.71 0.70	0.68	0.49	0.69 0.68	0.67	0.69 0.68	0.50
	rSTS	0.70 0.70	0.69	0.71 0.67	0.48	0.70	0.71 0.69	0.66	0.63	0.51
	lTPJ	0.62 0.62 0.60	0.61	0.60	0.49	0.62 0.59	0.56	0.58	0.55	0.50
	rTPJ	0.66 0.64	0.63	0.64	0.65	0.49	0.67 0.67 0.63	0.63	0.65	0.52
	lPre	0.63 0.62	0.60	0.62	0.59	0.50	0.62 0.55	0.54	0.57	0.59	0.49
	rPre	0.63 0.61	0.59	0.63 0.62	0.49	0.61	0.62 0.62 0.58	0.58	0.49
	lVMPFC	0.57	0.56	0.58 0.54	0.54	0.49	0.59	0.52	0.56	0.54	0.57 0.50
	rVMPFC	0.61 0.58	0.59	0.58	0.58	0.49	0.57	0.57	0.61 0.54	0.57	0.50
	rDMPFC	0.72 0.68	0.57	0.65	0.65	0.51	0.69	0.71 0.61	0.61	0.66	0.51
	lAmy	0.65 0.54	0.55	0.59	0.52	0.48	0.66 0.63	0.59	0.63	0.62	0.50
	rAmy	0.61 0.59	0.59	0.57	0.59	0.48	0.64 0.60	0.58	0.61	0.61	0.50
	WM	0.61 0.61 0.60	0.54	0.54	0.51	0.61 0.61 0.61 0.51	0.43	0.50

January 11, 2023

An IPU is a speech block of a single speaker bounded by pauses[START_REF] Levitan | Measuring acoustic-prosodic entrainment with respect to multiple levels and dimensions[END_REF].

https://github.com/Hmamouche/NeuroTSConvers January 11, 2023

January 11, 2023 20/24

January 11, 2023 24/24

Table 2. Description of the extracted multimodal features.

Features names Description Details

Linguistic features

Speech Activity

The interlocutor speaking?

Based on time-aligned IPU transcript.

Overlap

Both interlocutors speaking?

idem.

Laughter Laughter occurrences

Based on word-level time-aligned transcripts.

Filled-pauses Filled-Pauses occurrences

Based on word-level time-aligned transcripts : 'euh', 'heu', 'hum', 'mh'.

Feedback Conversational Feedback occurrences

Based on word-level time-aligned transcripts : 'oui' (yes), 'ouais' (yeah), 'non' (no), 'ah', 'd'accord' (right), 'ok' + Laughters.

Discourse-Markers

Occurrence of words used to keep speech organized

Based on word-level time-aligned transcripts : 'alors' (so), 'mais' (but), 'donc' (therefore), 'et' (and), 'puis' (then), 'enfin' (finally), 'parce que' (because), 'ensuite' (after).

Spoken-Particles

Occurrence of (final) spoken particle items Lexical-Richness Lexical richness measure [START_REF] Ochs | Toward an automatic prediction of the sense of presence in virtual reality environment[END_REF].

Based on time-aligned transcript: (number of adjectives + number of adverbs) / (total number of tokens).

Polarity & Subjectivity

Sentiment analysis metrics [START_REF] Smedt | Pattern for python[END_REF].

Based on time-aligned transcript, and a pretrained KNN classifier.

Facial features

Head-Tx, Head-Ty, Head-Tz

Head translation

Based on head pose estimated using Openface.

Head-Rx,Head-Ry, Head-Rz Head rotation idem.

Head-T-energy Kinetic energy of head translation

idem.

Head-R-energy Kinetic energy of head rotation

idem.

AU-mouth

Sum of facial movements related to mouth.

Based on Facial Action Units (AUs) existence detected by Openface library.

AU-eyes

Sum facial movements related to eyes.

idem.

AU-all

Sum of all action units. idem.

Direct-gaze

Percentage of direct gaze direction of the conversant idem.

Emotions ('Neutral', 'Happiness','Sadness', 'Surprise', 'Fear', 'Anger', 'Disgust') Based on a pre-trained CNN classifier.

Smiles

Smile's detection.

Based on a CNN classifier from Opencv library.

Eyetracking

Saccades Occurrence of Saccades

Based on gaze coordinates of the participant, recorded using the Eyelink1000 system.

Gaze-speed Gaze Speed. idem.

Gaze-movementenergy

Gaze movements energy idem.

Face-looks,Eyeslooks,Mouthlooks Number of looks in face, eyes and mouths respectively.

Based on participant's gaze coordinates and interlocutor's detected landmarks. F-score and recall measures are calculated for each condition (human-human and humanrobot interactions), each region of interest as well as the different models used are given in supplementary material (Tables 7 and8 for the F-score and the Recall respectively).

Statistically significant predictions

Here, we focus on the models that yielded the best F-scores, reporting in Table 4 the recall score obtained for both conditions in all ROIs. Prediction of ROI activity of human-human interactions are significant for all ROIS except the lV1, lAmy and lVMPFC, while for human-robot interactions, there are only 5 areas with significant prediction F-scores (lSTS, rSTS, rPre, lTPJ, and rTPJ).

Interpretable prediction results

In this part, we present detailed results concerning the selected features that lead to the best predictions for each brain area. We also show their importance scores in order to discuss the impact of each social modality on the studied brain areas.

interaction in terms of the predictive behavioral features for each brain area. The next step of this work is to test the framework on all brain areas. Understanding why certain regions cannot be predicted is a puzzling finding requiring further investigation and in particular the addition of new behavioral features in the future.

Supplementary information

Detailed prediction results

The detailed prediction F-scores and recalls with all the models tested in this analysis tested on human-human and human-robot datasets are provided in Tables 7 and8.