Jieyeon Woo
email: woo@isir.upmc.fr

Mireille Fares
email: fares@isir.upmc.fr

Catherine Pelachaud
email: pelachaud@isir.upmc.fr

Catherine Achard
email: achard@isir.upmc.fr

AMII: ADAPTIVE MULTIMODAL INTER-PERSONAL AND INTRA-PERSONAL MODEL FOR ADAPTED BEHAVIOR SYNTHESIS

Keywords: Adapted Behavior Synthesis, Reciprocal Adaptation, Conversational AI, Multimodal modeling

Socially Interactive Agents (SIAs) are physical or virtual embodied agents that display similar behavior as human multimodal behavior. Modeling SIAs' non-verbal behavior, such as speech and facial gestures, has always been a challenging task, given that a SIA can take the role of a speaker or a listener. A SIA must emit appropriate behavior adapted to its own speech, its previous behaviors (intra-personal), and the User's behaviors (inter-personal) for both roles. We propose AMII, a novel approach to synthesize adaptive facial gestures for SIAs while interacting with Users and acting interchangeably as a speaker or as a listener. AMII is characterized by modality memory encoding schema -where modality corresponds to either speech or facial gestures -and makes use of attention mechanisms to capture the intra-personal and inter-personal relationships. We validate our approach by conducting objective evaluations and comparing it with the state-of-the-art approaches.

Introduction

Socially Interactive Agents (SIAs) are embodied agents emitting human-like non-verbal behavior such as facial, body, and hand gesturing during speech [START_REF] Cassell | Human conversation as a system framework: Designing embodied conversational agents[END_REF]. In many applications such as medical therapy [START_REF] Shidara | Automatic thoughts and facial expressions in cognitive restructuring with virtual agents[END_REF] and educational assistance [START_REF] Kim | An embodied agent helps anxious students in mathematics learning[END_REF], they can carry out interactive and natural conversations with humans (Users) by sending and receiving multimodal nonverbal signals [START_REF] Joseph | Mutual adaptation and relativity of measurement[END_REF][START_REF] Judee K Burgoon | Nonverbal signals. The SAGE handbook of interpersonal communication[END_REF] in addition to the verbal message. To render SIA-User communication effective and engaging, SIAs must play active roles of speakers and listeners. For instance, in successful and engaging human-human conversations, speaker's and listener's behaviors are constantly coordinated and adapted to each other [START_REF] Judee K Burgoon | Interpersonal adaptation: Dyadic interaction patterns[END_REF], through the perception and response of the multimodal emitted signals such as linguistic [START_REF] Martin | Toward a mechanistic psychology of dialogue[END_REF], facial [START_REF] Judee K Burgoon | Nonverbal signals. The SAGE handbook of interpersonal communication[END_REF], and body [START_REF] Mark L Knapp | Nonverbal communication in human interaction[END_REF] features. The behavior adaptation can be: [START_REF] Cassell | Human conversation as a system framework: Designing embodied conversational agents[END_REF] intra-personal which is between the multimodal signals of a single person or [START_REF] Shidara | Automatic thoughts and facial expressions in cognitive restructuring with virtual agents[END_REF] inter-personal which is between listener's and speaker's signals adapting to the other's verbal and non-verbal behavior. This adaptation is continuous, reciprocal and dynamical and is referred to as reciprocal adaptation [START_REF] Woo | Reciprocal adaptation measures for human-agent interaction evaluation[END_REF]. The display of such adaptation capability can enable SIAs to be perceived as social and engaging [START_REF] Biancardi | Adaptation mechanisms in human-agent interaction: Effects on user's impressions and engagement[END_REF].

Various works have generated SIAs' non-verbal behavior focusing on only the modeling of intra-personal relationship for a single person [START_REF] Simon Alexanderson | Style-controllable speech-driven gesture synthesis using normalising flows[END_REF][START_REF] Bhattacharya | Text2gestures: A transformer-based network for generating emotive body gestures for virtual agents** this work has been supported in part by aro grants w911nf1910069 and w911nf1910315, and intel[END_REF][START_REF] Fares | Zero-shot style transfer for multimodal data-driven gesture synthesis[END_REF]. Other works looked into capturing multimodal inter-personal information in dyadic settings. In the work of Feng et al. [START_REF] Feng | Learn2smile: Learning non-verbal interaction through observation[END_REF] and Dermouche et al. [START_REF] Dermouche | Generative model of agent's behaviors in human-agent interaction[END_REF], SIA facial gestures are synthesized based on past gestures of both SIA and User without considering the existing relation with audio modality [START_REF] Marc | Prosody-face interactions in emotional processing as revealed by the facial affect decision task[END_REF][START_REF] Hani C Yehia | Linking facial animation, head motion and speech acoustics[END_REF], and do not ensure arXiv:2305.11310v1 [cs.HC] 18 May 2023 the motion continuity. Grafsgaard et al. [START_REF] Grafsgaard | Generative multimodal models of nonverbal synchrony in close relationships[END_REF] synthesize interlocutor's gestures based on both the interlocutors' audio and their facial modalities. In the work of Jonell et al. [START_REF] Jonell | Let's face it: Probabilistic multi-modal interlocutor-aware generation of facial gestures in dyadic settings[END_REF], SIA's facial gestures are generated based on SIA's speech and the User's speech and facial gestures. However, these models [START_REF] Grafsgaard | Generative multimodal models of nonverbal synchrony in close relationships[END_REF][START_REF] Jonell | Let's face it: Probabilistic multi-modal interlocutor-aware generation of facial gestures in dyadic settings[END_REF] are prone to produce non-continuous gestures. The works presented in Woo et al. [START_REF] Woo | Creating an interactive human/agent loop using multimodal recurrent neural networks[END_REF][START_REF] Woo | Asap: Endowing adaptation capability to agent in human-agent interaction[END_REF] and Ng et al. [START_REF] Ng | Learning to listen: Modeling non-deterministic dyadic facial motion[END_REF] ensure SIA's behavior continuity by employing autoregressive online inference while modeling SIA's and User's multimodal features. Only the listener's behavior is modeled in [START_REF] Ng | Learning to listen: Modeling non-deterministic dyadic facial motion[END_REF]. [START_REF] Woo | Creating an interactive human/agent loop using multimodal recurrent neural networks[END_REF][START_REF] Woo | Asap: Endowing adaptation capability to agent in human-agent interaction[END_REF] and all the aforementioned models for dyadic setting do not explicitly model the intra-personal relationship.

Our overall aim is to create a social and engaging SIA by modeling its behavior adaptation. Our hypothesis is that behavior adaptation is captured by intra-personal and inter-personal synchronizations. We propose Adaptive Multimodal Inter-personal and Intra-personal (AMII) model, a novel method to synthesize adaptive and engaging facial gesturing for SIAs. We explicitly model the intra-personal relationship by encoding the prior emitted multimodal signals (modality memory) while ensuring motion continuity. We model the inter-personal relationship to generate SIA behavior for both roles of speaker and listener. We explore the best way to capture the reciprocal adaptation to generate adaptive non-verbal SIA behavior within a dyadic setting. Intra-personal and inter-personal relations are learned through attention mechanisms. We demonstrate that our AMII model outperforms the state-of-the-art models, in terms of both behavior appropriateness and reciprocal adaptation resemblance, by conducting objective evaluations. We perform ablation studies and show the influence of AMII's key encoders.

Our contributions are as follows:

• We propose AMII, an approach to capture the reciprocal adaptation of SIA behaviors.

• We explicitly model the intra-personal and inter-personal relationships by encoding prior emitted multimodal signals via our modality memory encoders.

The paper is organized as follows. Section 2 describe the proposed AMII model architecture. The dataset is presented in Section 3 and we detail the training regime in Section 4. Then, we report our experiments in Section 5 and discuss our results in Section 6. We finally conclude and talk about our future works in Section 7.

2 Adaptive Multimodal Inter-personal and Intra-personal (AMII) Model Architecture

We focus on generating adaptive non-verbal SIA behavior as both a speaker and listener. For this purpose, we propose an approach trained on real human-human interactions, to learn human-human inter-personal and intra-personal relations for SIA and simulate our predictions on a SIA. We propose a new model architecture, the Adaptive Multimodal Inter-personal and Intra-personal model (AMII), to synthesize adaptive and engaging facial gestures for SIAs. It takes as input speech and facial gestures of both SIA (A) and User (U ), corresponding to their past behavior, and predicts SIA's and User's facial gestures at the next time step.

Facial gestures are represented by:

• Gaze movements: represented by G x and G y which are the gaze angles w.r.t. the x and y axis.

• Head movements: represented by R x , R y , and R z which are the Euler head rotations w.r.t. the x, y and z axis. • Facial movements: represented by facial Action Units (AUs) [START_REF] Ekman | Measuring facial movement[END_REF] which are facial muscle movements defined by the Facial Action Coding Systems (FACS) [START_REF] Ekman | Facial action coding system[END_REF]. We use AU1 (inner brow raiser), AU2 (outer brow raiser), AU4 (brow lowerer), AU6 (cheek raiser), and AU12 (lip corner puller for smile).

Speech and facial gestures are highly tied together [START_REF] Marc | Prosody-face interactions in emotional processing as revealed by the facial affect decision task[END_REF][START_REF] Hani C Yehia | Linking facial animation, head motion and speech acoustics[END_REF]. With this relation, we decide to use speech information to drive SIA's facial gestures. Speech features are listed as follows:

• Pitch: represented by the fundamental frequency f 0 , which is the main prosodic feature correlated with facial gestures [START_REF] Bolinger | A theory of pitch accent in english[END_REF]. • Loudness: speech intensity from the auditory spectra.

• Voicing probability: speech presence probability expressed as a probability score in the range of 0 to 1.

• Mel-Frequency-Cepstral Coefficients (MFCC): represented by 13 MFCC features (0-12).

AMII model operates as follows. It takes as input the 100 past frames (t -99 : t), where t is the current frame, of the:

1. Speech features of A (X A speech ) and those of U (X U speech ), 2. Facial features of A (X A f ace ) and those of U (X U f ace ). For each prediction of the next frame (t + 1), the model predicts:

1. A's facial gestures ( Y A f ace ), 2. U 's facial gestures ( Y U f ace ).
AMII consists of three main components, as illustrated in Figure 1. The first component is the intra-personal encoder E intra , which explicitly encodes the intra-personal relations via a modality memory schema. This schema consists of encoding each modalityspeech features and facial features -corresponding to the past 100 frames. The second component is the inter-personal encoder E inter , which encodes the inter-personal relations by applying cross-attentions between A's and U's features' embeddings. The last component is the behavior generator which generates A's and U's facial gestures of the next frame. These components are detailed in the following.

Intra-personal Encoder (E intra )

As shown in Figure 1, E intra takes as input X speech and X f ace of either A or U and generates the corresponding intra-personal embedding Z intra . It consists of two sub-encoders. The first is the modality memory encoder (E speech or E f ace ). The second is the dual-modality encoder E dual .

Modality Memory Encoder (E speech or E f ace ) Both E speech and E f ace takes its corresponding modality -X speech or X f ace respectively -as input and renders the modality memory embedding Z mem speech and Z mem f ace representing the past 100 frames. Each corresponding modality memory encoder firstly learns the modality specific information by applying self-attention, with a h = 2 (where h is the head size), preceded and followed by dense layers (E d ) with c = 16 (where c is the cell size). Then, it embeds the memory sequence of the chosen modality via a LSTM layer (E m ) with c = 16, as depicted in Figure 1. It takes X mod -where mod represents either speech or facial gestures -and outputs Z mem mod , and can be expressed as:

Z mem mod = E m (E d (SA (E d (X mod )))) (1)
where SA(•) denotes self-attention layer.

Dual-modality Encoder (E dual ) E dual captures the relationship between the multimodal signals by applying crossattention mechanisms on the corresponding modalities: CA speech and CA f ace with h = 2 followed by E d with c = 16, as shown in Figure 1. CA speech has a query Q equals to Z mem speech with key K and value V equal to Z mem f ace . CA f ace has a query Q equals to Z mem f ace with key K and value V equal to Z mem speech . E dual takes Z mem speech and Z mem f ace as inputs and generates Z intra . It can be written as:

Z intra = E d ([ CA speech (Q speech , K f ace , V f ace ) , CA f ace (Q f ace , K speech , V speech ) ]) (2)
where CA(Q, K, V ) denotes cross-attention layer and [•] denotes concatenation layer.

Inter-personal Encoder (E inter )

As illustrated in Figure 1, E inter takes as input Z A intra and Z U intra , which are the intra-personal representations of A and U respectively. It renders Z inter , a representation of inter-personal relation between A and U . E inter applies cross-attention mechanisms on the both intra-personal representations: CA A and CA U with h = 2 followed by E d with c = 16. CA A has a query Q equals to Z A intra with key K and value V equal to Z U intra . CA U has a query Q equals to Z U intra with key K and value V equal to Z A intra . It can be written as:

Z inter = E d CA A Q A , K U , V U , CA U Q U , K A , V A (3)
where CA(Q, K, V ) denotes cross-attention layer and [•] denotes concatenation layer.

Behavior Generator (G f ace )

G f ace takes as input the:

1. A's intra-personal representation (Z A intra ), 2. U 's intra-personal representation (Z U intra ), 3. Inter-personal representation of A and U (Z inter ).

It generates the corresponding facial gestures Y A

f ace and Y U f ace by decoding with a dense layer (D d ) with c = 20, as depicted in Figure 1. The final outputs Y P f ace can be written as:

Y P f ace = D d Z P intra , Z inter (4)
where P represents either A or U. At training time, AMII synthesizes Y U f ace to better learn inter-personal and intrapersonal relations. Y U f ace is disregarded at inference time since the aim is to predict only A.

Training and Inference Modes

We train our model on real human-human (U 1 -U 2 ) interactions. Our model learns to synthesize adapted gestures of U 1 and U 2 . During inference, AMII synthesizes the behavior of A and U . Y A f ace is inferred using the previous prediction of A and the ground truth of U . We apply adaptive online prediction to generate continuous A's behavior in an autoregressive fashion.

NoXi Dataset

For our experiments, we choose the French NoXi dataset [START_REF] Cafaro | The noxi database: Multimodal recordings of mediated novice-expert interactions[END_REF] consisting of 21 human-human dyadic interactions performed by 28 participants with a total duration of 7h22min. We extract non-verbal behavior features for each time-step: openSMILE [START_REF] Eyben | Opensmile: the munich versatile and fast open-source audio feature extractor[END_REF] for speech and OpenFace [START_REF] Baltrušaitis | Openface: an open source facial behavior analysis toolkit[END_REF] for facial gestures. Data preprocessing -median filter and linear interpolation -is applied on both extracted features and they are adjusted to 25f ps. We split our dataset into 3 sets: training (70%), validation (10%), and test (20%). The test set does not include data of speakers and listeners that are seen during training. The aim is to test AMII's capacity to extrapolate on new unseen speakers and listeners and therefore its capability to generalize.

Training Regime

To train our model, we use the Mean Squared Error (MSE) as our loss function and the Adam optimizer [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF] with Cyclical Learning Rate (CLR) [START_REF] Leslie | Cyclical learning rates for training neural networks[END_REF] (triangular learning rate policy, base_lr of 1e -7, max_lr of 1e -3, and step size factor of 10). The training was done for 300 epochs (with an average runtime of 125h) on a 2.2GHz Intel Xeon Linux server with NVIDIA GeForce GTX TITAN X and 64GB RAM with a batch size of 32. The best set of hyperparameters is chosen after manual optimization based on the validation set.

Experiments

To assess our model, we conduct objective evaluation to check its performance against the state-of-the-art approaches, which we select as our baselines, and to verify the effectiveness of each AMII's key components through ablation studies.

Objective Metrics

We want to assess whether the generated behavior is appropriate (intra-personal) and reciprocally adaptive (interpersonal). To do so, we employ the metrics used in previous works [START_REF] Feng | Learn2smile: Learning non-verbal interaction through observation[END_REF][START_REF] Dermouche | Generative model of agent's behaviors in human-agent interaction[END_REF][START_REF] Grafsgaard | Generative multimodal models of nonverbal synchrony in close relationships[END_REF][START_REF] Woo | Creating an interactive human/agent loop using multimodal recurrent neural networks[END_REF][START_REF] Woo | Asap: Endowing adaptation capability to agent in human-agent interaction[END_REF][START_REF] Ng | Learning to listen: Modeling non-deterministic dyadic facial motion[END_REF]. We measure behavior appropriateness of A's predictions ( A) against its ground truth (GT) behavior (A). The metrics are as follows:

• MAE and RMSE/L2: distance between the predictions and GT to measure the generated error, which are used in [START_REF] Feng | Learn2smile: Learning non-verbal interaction through observation[END_REF][START_REF] Dermouche | Generative model of agent's behaviors in human-agent interaction[END_REF][START_REF] Woo | Creating an interactive human/agent loop using multimodal recurrent neural networks[END_REF][START_REF] Woo | Asap: Endowing adaptation capability to agent in human-agent interaction[END_REF][START_REF] Ng | Learning to listen: Modeling non-deterministic dyadic facial motion[END_REF].

• Kolmogorov-Smirnov two-sample test (KS test) [START_REF] Frank | The kolmogorov-smirnov test for goodness of fit[END_REF]: density probability difference between A's corresponding predictions and GT to check for the distribution similarities between them, which is used in [START_REF] Woo | Asap: Endowing adaptation capability to agent in human-agent interaction[END_REF].

For reciprocal adaptation resemblance, we measure the resemblance between A and U 's GT data (U ). We choose to assess only smile (AU12) as it is a key socio-emotional signal [START_REF] Mark L Knapp | Nonverbal communication in human interaction[END_REF]. The metrics are as follows:

• Time lagged cross correlation coefficient (TLCC) [START_REF] Steven M Boker | Windowed cross-correlation and peak picking for the analysis of variability in the association between behavioral time series[END_REF]: linear relationship invariant to speed, which is used to quantify global synchrony like PCC. TLCC is computed in chunks of 8sec with a time lag of 2sec as in [START_REF] Ng | Learning to listen: Modeling non-deterministic dyadic facial motion[END_REF].

• DTW [START_REF] Müller | Dynamic time warping. Information retrieval for music and motion[END_REF]: proximity/resemblance check of A&U against that of the GT interaction (between two humans) to evaluate reciprocal adaptation. DTW is computed in chunks of 1min with a stride of 30sec as in [START_REF] Woo | Asap: Endowing adaptation capability to agent in human-agent interaction[END_REF].

• Synchrony (Sync) and Entrainment Loop (EL) [START_REF] Woo | Reciprocal adaptation measures for human-agent interaction evaluation[END_REF]: synchrony and entrainment loop measures proposed in [START_REF] Woo | Reciprocal adaptation measures for human-agent interaction evaluation[END_REF] to evaluate the reciprocal adaptation between A and U.

Lower values denote better performance for MAE, RMSE, and KS test. For the resemblance metrics (PCC, TLCC, DTW, Sync, and EL), the closer the value of the metric is to the GT, the better the model performs in generating adaptive A's behaviors.

Baseline Models and Ablation Studies

We consider the following baseline models (base):

• IL-LSTM [START_REF] Dermouche | Generative model of agent's behaviors in human-agent interaction[END_REF]: models only the inter-personal based on a single modality of facial gestures of A and U,

• Symmetrized IL-LSTM with online LSTM (sym-IL-LSTM) [START_REF] Woo | Creating an interactive human/agent loop using multimodal recurrent neural networks[END_REF] and ASAP [START_REF] Woo | Asap: Endowing adaptation capability to agent in human-agent interaction[END_REF]: model the inter-personal relation of A and U, multimodality of speech and facial gestures, and assure motion continuity.

To check for the effectiveness and influence of each of AMII's key encoders, we conducted additional ablation studies. We performed the ablations of:

• Modality Memory Encoder E m (noE m ),

• Dual-modality Encoder E dual (noE dual ),

• Inter-personal Encoder E inter (noE inter ).

Table 1: Objective evaluation of AMII against the baselines along with ablations using the selected metrics. GT denotes ground truth interaction. Best results are highlighted in bold. ∆ base represents the change in performance over the best performing baseline approach of each metric. ∆ base entries in green when AMII outperforms best baseline, in red when it is not the case. 6

Results and Discussion

The evaluation results are listed in Table 1. ∆ base represents the change in performance over the best performing baseline approach for each metric.

Comparing with Baselines We remark that AMII outperforms the baselines in terms of behavior appropriateness. This is reflected through low errors of MAE and RMSE represented by ∆ base (↓ 0.024 and ↓ 0.020 respectively). For the density distribution, via the KS test, we observe that AMII performs comparatively less than the baselines indicating that AMII possesses the least similar density distribution compared to that of the GT. In detail, ASAP performs the best in terms of having the most similar density w.r.t. GT (0.282) and AMII the worst (0.437) with ∆ base of ↑ 0.155. This low performance of AMII does not imply that it generates wrong SIA behavior but that it has either a smaller or a wider range of behavior variety than that of the GT. The focus of this study is not to produce a variety of behaviors but to generate SIA behaviors that are adaptive to its interlocutor. Thus, this weak performance of the KS test metric is not critical for our aim. Moreover, AMII performs the best in terms of the reciprocal adaptation resemblance metrics as seen in the Table 1. DTW, synchrony, and entrainment loop of A&U show that AMII resembles the GT the most with ∆ base of ↓ 79.7, ↓ 4.6, and ↓ 534.9 respectively. With TLCC, we remark that the sym-IL-LSTM is the closest to the GT while AMII is the farthest one with ∆ base of ↑ 0.042. As DTW considers the variation of sequence length while being invariant to speed unlike TLCC, it represents better the global correlation. Thus, for the interpretation, we can put more emphasis on the DTW results compared to that of TLCC. This comparative study shows that the inclusion of explicit modeling of intra-personal relation leverages the quality of produced gestures in terms of both behavior appropriateness and reciprocal adaptation resemblance.

Ablation studies The ablation of each of AMII key encoder -E m , E dual , and E inter -results in the improvement of the reciprocal adaptation resemblance. This was seen by an increase in DTW (87.0, 33.0, and 187.6 respectively), synchrony (48.3, 124.9, and 45.1 respectively), and entrainment loop resemblance (385.2, 900.2, and 402.7 respectively). TLCC shows that the insertion of E m improves the AMII by 0.020 along with E inter by 0.030. However, E dual slightly deteriorates the performance by 0.009. As in the baseline comparison study, it is better to concentrate on the other reciprocal adaptation resemblance metrics as DTW is a more dynamic measure of synchrony than TLCC. However, this enhancement of reciprocal adaptation resemblance is at the expense of lowering its behavior appropriateness performance. This is observed via MAE (0.057, 0.013, 0.020 respectively) and RMSE (0.065, 0.011, 0.019 respectively). The same conclusion can be drawn by looking at KS test result. The fall of performance is seen for the additions of E dual (0.041) and E inter (0.031) while E m improves (0.078). This compromise of losing behavior appropriateness to gain an adaptive one may be a good exchange. It is more valuable to generate SIA behaviors with adaptation capacity than to reproduce the same GT behavior. In fact, in a human-human interaction there could be multiple possible behaviors and generation timings facing a same interacting partner's behavior. This might vary depending on the various factors such as the context, situation, and interlocutor's personality and mood.

We can conclude that it is important to model the intra-personal relation with the encodings of E m and E dual , and the inter-personal relation with E inter to synthesize adaptive non-verbal facial gestures for both roles as speaker and listener.

In this paper, we propose a new approach to generate adaptive SIA behavior as both speaker and listener by encoding the multimodality and intra-personal and inter-personal relationships. We conclude that AMII model achieves state-of-the art performance notably in terms of reciprocal adaptation resemblance. Our approach still has some limitations. We do not model the inter-personal relation memory which could improve our results. As the next step, we plan to add this component. We also plan to integrate and assess AMII within a real-time SIA-User interaction. Our overall aim is to model SIA that can socially engage users in an interaction through its behaviors that capture intra-personal and inter-personal relations. In the next future, we will subjectively evaluate the dimensions of social attitudes and engagement to validate this hypothesis.

Figure 1 :

 1 Figure 1: AMII (Adaptive Multimodal Intra-personal and Inter-personal) model architecture. The intra-personal encoder (E intra ) takes the speech X speech and the facial gestures X f ace of the previous 100 frames of either the SIA (A) or the User (U ) to encode the corresponding intra-personal relationship Z intra . The inter-personal encoder (E inter ) learns from intra-personal relationships Z A intra and Z U intra to encode the inter-personal relationship between them Z inter . The behavior generator (G f ace ) takes Z A intra , Z U intra , and Z inter to generate the sequence of facial gestures for the next frame t+1 Y A f ace and Y U f ace . At training time, AMII is trained with human-human (U 1 -U 2 ) interactions (U 1 for A and U 2 for U ) and predicts both of humans' facial gestures ( Y U1 f ace and Y U2 f ace ). At inference time, AMII renders the facial gestures of A and U . To infer the next A's behavior, we feed back the predicted A's behavior and the ground truth of U .