Lucie Galland

Catherine Pelachaud

Florian Pecune

Modality Attentive Late Embracenet Fusion with Interpretable Modality Contribution (MALEFIC) Seeing and hearing what has not been said A multimodal client behavior classifier in Motivational Interviewing with interpretable fusion

Motivational Interviewing (MI) is an approach to therapy that emphasizes collaboration and encourages behavioral change. To evaluate the quality of an MI conversation, client utterances can be classified using the MISC code as either change talk, sustain talk, or follow/neutral talk. The proportion of change talk in a MI conversation is positively correlated with therapy outcomes, making accurate classification of client utterances essential.

In this paper, we present a classifier that accurately distinguishes between the three MISC classes (change talk, sustain talk, and follow/neutral talk) leveraging multimodal features such as text, prosody, facial expressivity, and body expressivity. To train our model, we perform annotations on the publicly available AnnoMI dataset to collect multimodal information, including text, audio, facial expressivity, and body expressivity. Furthermore, we identify the most important modalities in the decision-making process, providing valuable insights into the interplay of different modalities during a MI conversation.

INTRODUCTION

Motivational Interviewing (MI) is an approach to therapy that emphasizes collaboration and encourages behavioral change. During Motivational Interviews, therapists rely on a set of strategies to guide clients toward expressing motivation toward change [START_REF] William | Motivational interviewing: Helping people change[END_REF]. Assessment of the quality of the therapy interaction is classically done by annotating therapist's and client's behaviors. To this intent, various annotations schema have been developed such as the Motivational Interviewing Skill Code (MISC) [START_REF] William R Miller | Albuquerque: Center on Alcoholism, Substance Abuse and Addictions[END_REF] that classifies both therapist and client behaviors into three relevant categories:

• Change talk (CT): reflecting actions toward behavior change • Sustain talk (ST): reflecting actions away from behavior change • Follow/Neutral (F/N): unrelated to the target behavior This classification of client language is of interest as it is a predictor of the therapy outcome. Indeed, [START_REF] Magill | The technical hypothesis of motivational interviewing: A meta-analysis of MI's key causal model[END_REF] revealed that sustain-talk was associated with poorer treatment results. Furthermore, [START_REF] Magill | A meta-analysis of motivational interviewing process: Technical, relational, and conditional process models of change[END_REF] showed that change talk was linked to reductions in risk behavior during follow-up assessments. This correlation makes MISC a promising tool for studying the efficacy of Motivational Interviewing (MI).

The labeling of client utterances is usually done by training coders to manually encode utterances into these three categories. However, this process of annotation can be resource-intensive, as it requires trained annotators to carefully review videos. Furthermore, it can not be done in real-time and can not be used in the context of a human-agent dialogue for instance. As a result, there has been growing interest in developing automatic annotation methods for MISC using various modalities and approaches. These efforts aim to streamline the annotation process and reduce the time and resources required for the analysis.

In this paper, we continue these efforts by presenting a classifier that can distinguish automatically between the three MISC classes. This classifier is based on multimodal features of face-to-face conversations, including (spoken) text, prosody, facial expressivity, and body expressivity. Our classifier is designed to be interpretable, meaning that it is possible to identify the modality that was most important in its decision-making process.

In the remaining of the paper, we first present the data we used to train our MISC classifier, then we present our modality attentive fusion architecture. We explore the performance of different models and compare our results with existing work. Finally, we present a way to interpret the results of the classification to shed a light on the contribution of modalities in the classification process.

RELATED WORK

The correlation between MISC codes and therapy outcomes has motivated several studies to develop their own classification systems for client language, categorizing it as change talk, sustain talk, or follow neutral. These studies use various modalities as inputs.

Text-based modalities have been widely investigated in the context of MISC annotation on different temporal levels. For example, [13] used topic modeling to predict therapy outcomes at the session level, while [START_REF] Huang | Modeling temporality of human intentions by domain adaptation[END_REF] incorporated topic angles and session timing (beginning or end) to predict MISC codes at the utterance level. In their work, an utterance represents a turn by either the client or the therapist. More recent advances have been made using deep learning-based approaches, such as those presented in [11], which leveraged word-level features, and in [START_REF] Chen | Feature fusion strategies for end-to-end evaluation of cognitive behavior therapy sessions[END_REF], which incorporated additional utterance-level features like Linguistic Inquiry and Word Count (LIWC) for improved annotation accuracy. In the latter work, utterances were segmented after a pause of at least two seconds. While these advancements highlight the ongoing exploration of various feature sets and modalities in the automatic annotation of MISC codes, they also expose a variety of ways to decide the level used for coding as well as the specification of an utterance.

Text is not the only modality that can convey the nuances of change talk. Several studies have incorporated prosody or acoustic features to improve MISC classification. For instance, [START_REF] Aswamenakul | Multimodal Analysis of Client Behavioral Change Coding in Motivational Interviewing[END_REF] combined acoustic features with linguistic features to slightly improve the accuracy of change talk detection. Deep learning methods such as Long Short-Term Memory (LSTM) [START_REF] Singla | Using prosodic and lexical information for learning utterance-level behaviors in psychotherapy[END_REF] has also been employed to predict change talk using both text and audio modalities. In this work, the addition of the audio modality improves the prediction score. More recently, such classification was performed using Transformers [START_REF] Tavabi | Multimodal automatic coding of client behavior in motivational interviewing[END_REF]. The use of audio generates a loss in performance that can be explained by the low quality of the recordings.

In addition to acoustic cues, other social signals such as laughter have been explored.

[12] demonstrated that adding laughter as input improved the accuracy of change talk prediction compared to text alone. Furthermore, non-verbal cues such as facial Action Units have been utilized as predictors for change talk, as shown in [START_REF] Nakano | Detecting Change Talk in Motivational Interviewing using Verbal and Facial Information[END_REF] which resulted in improving the prediction.

While the text remains a commonly studied modality, incorporating prosody, non-verbal, and other multimodal information alongside text has shown promising potential for improving the accuracy and robustness of MISC annotation and prediction tasks.

DATA

Motivational interviewing data that could be used to train a MISC classifier is difficult to find due to the sensitive nature of the discussed topics. Most of the existing corpora are either private for medical reasons [START_REF] Borsari | Addressing alcohol use and problems in mandated college students: a randomized clinical trial using stepped care[END_REF][START_REF] Carey | Computer versus in-person intervention for students violating campus alcohol policy[END_REF] or owned privately and payable. Because of this, most studies need to collect a new dataset first and models can not be compared. For instance, [START_REF] Nakano | Detecting Change Talk in Motivational Interviewing using Verbal and Facial Information[END_REF] collected their own non public corpus over Zoom and developed a classifier on the resulting corpus. However, Two corpora of MI conversations have recently been published and are publicly available. The High Low-quality MI dataset [START_REF] Pérez-Rosas | What makes a good counselor? learning to distinguish between high-quality and low-quality counseling conversations[END_REF] is composed of 249 videos of MI annotations available on YouTube. Some errors remain in the automatic transcription of the videos and even though MISC annotations have been performed, they are not currently available. The second public corpus is AnnoMI [START_REF] Wu | Anno-MI: A Dataset of Expert-Annotated Counselling Dialogues[END_REF], a corpus of MI conversations transcribed and annotated with MISC with publicly available annotations. These datasets do not provide multimodal annotations.

AnnoMI corpus

In our work, we rely on the AnnoMI dataset [START_REF] Wu | Anno-MI: A Dataset of Expert-Annotated Counselling Dialogues[END_REF] to train ou MISC classifier. AnnoMI is a publicly available dataset of MI videos of 7 minutes on average that have been annotated by 133 experts. The videos are designed as a demonstration of either high or low-quality therapy. Each video is transcribed and each utterance is annotated in term of primary therapist behavior (question, reflection, therapist input, and others) and client talk type (neutral, change, sustain) using MISC. In this work, we are interested in the client side of MISC. A client utterance can be annotated into three categories: Change Talk (CT), Sustain Talk (ST), or Follow/Neutral (F/N). An utterance classified as CT conveys movement towards the behavior of change while ST conveys a movement away from the behavior of change. A F/N utterance does not indicate a preference towards or against change. The data is annotated by MI practitioners into these 3 classes with 0.9 inter-annotators agreement.

From this corpus we use 121 videos: 3 videos were removed because of outdated URLs and 9 were removed for the poor quality of the video stream. The original transcriptions of the AnnoMI dataset are separated into utterances where a new utterance starts every time a new interlocutor is speaking, only the timestamp of the start of each of these utterances is provided.

Dataset preprocessing

In this paper, we take advantage of the publicly available videos of AnnoMI to train a classifier that predicts client's MISC category relying on multimodal behavior. Multimodality gives valuable insights for various tasks such as sentiment analysis [START_REF] Zadeh | Tensor fusion network for multimodal sentiment analysis[END_REF]. Moreover [START_REF] Pampouchidou | Automatic assessment of depression based on visual cues: A systematic review[END_REF] shows that visual cues such as facial Action Unit occurrences, head pose, eye gaze, and body gestures can be a sign of depression. Therefore in this paper, we study multiple modalities such as (spoken) text, audio (prosody), and facial and body expressivity.

Text. In the original AnnoMI transcriptions, sentences were cut into two utterances whenever a listener's backchannel occurred during their production. However, backchannels are not aimed to take the speaking turn. In our model, backchannels are removed from the original transcript and utterances are reorganized to recreate sentences corresponding to speaking turns. We updated the MISC coding whenever utterances of the same sentence received different labels in the original AnnoMI annotation. The only conflicts involved utterances annotated as Facial expressivity. The facial expressivity is extracted using OpenFace [START_REF] Baltrusaitis | Openface 2.0: Facial behavior analysis toolkit[END_REF]. As the performance of the OpenFace model is significantly better on videos containing only one face, we produce two new videos from the original ones: one with the therapist only, and one with the patient only. In most cases, the camera focuses mainly on the person talking, leaving out of focus the other interlocutor. Yet, speaking makes the detection of mouth-related action units by OpenFace noisy. Therefore, we extract the action units of the upper face (AU 1 2 4 5 6 7 9 and 45). OpenFace is also applied to extract gaze angles and head positions and rotations. The action units are smoothed using a median filter with a kernel of size 5 and missing data are interpolated.

Body expressivity. Body expressivity can convey information on one's affective state [START_REF] Castellano | Recognising human emotions from body movement and gesture dynamics[END_REF]. Two interesting measures of body expressivity are Amplitude of movement [START_REF] Castellano | Recognising human emotions from body movement and gesture dynamics[END_REF] and Quantity of motion [START_REF] Castellano | Automated analysis of body movement in emotionally expressive piano performances[END_REF]. Amplitude is defined as the width of a movement and Quantity of motion is defined as an approximation of the amount of detected movement.

Raw body joints position data are extracted using OpenPose [START_REF] Cao | Realtime multi-person 2d pose estimation using part affinity fields[END_REF]. From these raw skeleton data of the client and the therapist, we compute the Amplitude and Quantity of motion for each frame.

The Amplitude is defined as the bounding box around the speaker for a given time frame. It is computed by dividing the length between the two wrists by the height 𝐻 of the bust in the current framing. Dividing by 𝐻 accounts for the different sizes in framing.

The quantity of motion QoM is computed following a simplified version of the method described in [START_REF] Castellano | Automated analysis of body movement in emotionally expressive piano performances[END_REF]. Given a silhouette 𝑡 that moves over 𝑛 frames, QoM is defined as: ' We define 𝐴𝑟𝑒𝑎(𝑆𝑖𝑙ℎ𝑜𝑢𝑒𝑡𝑡𝑒 (𝑡)) as the bounding box used for the Amplitude and we set n=10 frames. This simplification is chosen as the interlocutors are seated and the motion is mainly focused on the arms. As the bounding box only takes into account the upper body, the simplification is acceptable.

𝑄𝑜𝑀 = 𝐴𝑟𝑒𝑎(𝑆𝑖𝑙ℎ𝑜𝑢𝑒𝑡𝑡𝑒 (𝑡 + 𝑛)) -𝐴𝑟𝑒𝑎(𝑆𝑖𝑙ℎ𝑜𝑢𝑒𝑡𝑡𝑒 (𝑡)) (1
On both Amplitude and Quantity of motion, missing data are interpolated and a Median filter of size 5 is applied to reduce detection errors from OpenPose.

Data distribution

Similar to other MI datasets [START_REF] Nakano | Detecting Change Talk in Motivational Interviewing using Verbal and Facial Information[END_REF][START_REF] Tavabi | Multimodal automatic coding of client behavior in motivational interviewing[END_REF], our corpus is unbalanced: the Follow/Neutral class is significantly more prevalent than the Change Talk or Sustain Talk classes (see Table 1). However, our data are more balanced than some previous studies, since we considered speakers' sentences and removed listeners' backchannels.

The proportion of each class in the corpus is similar for all modalities, which means that the available modalities are independent of the classes and therefore will not affect the model.

ARCHITECTURE

Our MISC classifier relies on the following architecture: each modality of the client input is first prepossessed individually by an adapted network. These encoding networks represent each of the modalities as an embedding vector. The different modalities represented are merged using a modified version of Embracenet [START_REF] Choi | Embracenet for activity: A deep multimodal fusion architecture for activity recognition[END_REF], a fusion architecture that allows missing modalities. We modify Embracenet by adding attention to modalities and call this new architecture MALEFIC (see Section 4.2) The optimal sizes of the models are determined using a grid search.

Modalities pre processing

Text preprocessing. The text is preprocessed using a frozen Bert pre-trained model from the HuggingFace library (bert-base-uncased) followed by two linear layers of size 30 interposed with dropout layers, Leaky-Relu activations and one skip connection. We choose to use a frozen Bert model to avoid overfitting.

Text and context preprocessing. According to the findings of previous works [START_REF] Nakano | Detecting Change Talk in Motivational Interviewing using Verbal and Facial Information[END_REF][START_REF] Tavabi | Multimodal automatic coding of client behavior in motivational interviewing[END_REF], we take into account both the therapist's and the client's behaviors. We take as input the previous turn of the therapist, the previous sentences that make up the turn of the client, and the actual client sentence to classify. Each of these sentences is processed sequentially through an un-frozen Bert, and the embeddings obtained from average pooling are concatenated.

Audio preprocessing. The Audio modality is preprocessed using the pre-trained Beats model [? ]. It takes as input the Mel filter bank of the audio and outputs an embedding of size 758.

Facial expressivity preprocessing. Action Units and head pose values are preprocessed using an encoder composed of two 2-dimensional convolutional layers with 16 filters and a 1-layer Transformer encoder. The encoding of the transformer is then combined to compute an embedding for the entire sequence of size 256.

Body expressivity preprocessing. Amplitude and Quantity of motion are preprocessed using an encoder composed of 2 convolutional layers and a 1 layer transformer encoder. The encoding of the transformer is then combined to compute an embedding for the entire sequence of size 8.

Fusion

The fusion of modalities is achieved using a modified version of Embracenet. This method is useful for handling missing modalities. First, each preprocessing network's output is reduced to the size of the final embedding by a linear layer. Then, Embracenet combines the embeddings by randomly selecting one modality per embedding dimension. In addition, dropout of modality is used during training to prevent over fitting on modalities. During training, modality dropout involves randomly removing available modalities. This approach enables each preprocessing network to efficiently learn the data structure while also taking advantage of multimodality. Furthermore, it enables us to address missing data in our corpus (namely, the face and body information that are not available for every sentence). In fact, as a result of this training, any missing modality can be easily ignored.

We improve the EmbraceNet architecture by incorporating self-attention. Self-attention is used to determine the significance of a given modality. If a modality is deemed important by the self-attention module, then this modality will be more likely to be selected (see Fig. 1).

The output of the self-attention layer gives the weight of each modality for each embedding dimension. During training, the output of the self-attention layer for a given embedding dimension is used as the probability of selecting each modality. During the evaluation, the selected modality for a given embedding dimension is the modality with the highest probability. We choose to use probabilistic selection during training to avoid over fitting.

We enhance the Embracenet framework with self-attention, as some of the modalities in our problem contribute more to the classification. (for instance, the Text modality has a more substantial classification power than the nonverbal modality, see Tab.2).

The resulting architecture also estimates the usefulness of each modality, which allows for interpretation (see Section 6) In the following, we use this architecture that we call : Modality Attentive Late Embracenet Fusion with Interpretable Modality Contribution (MALEFIC), with diffrerent combinations of modalities : Facial and body expressivity; Text and context; Text, context and audio; Text, context and facial expressivity; and Text, context, audio and facial expressivity. For Text and context, we previously took the context into account by concatenating the Bert embeddings of the surrounding sentences. Here, we take advantage of our fusion architecture and treat the context as another modality. A self-attention layer will decide whether in this case the client-therapist context is relevant.

CLASSIFICATION RESULTS

To explore the performance of our architecture to predict the MISC classes, we train and evaluate different models using the data described in Section 3. The unbalanced data set is handled using a weighted random sampler. First, we evaluate the performance of each modality regarding the classification by training different unimodal classifiers. Then, we investigate whether multimodality improves the performance of our best unimodal model. Finally, we compare our results to existing multimodal MISC classification models.

Single modality models

Our first objective is to evaluate which modality allows for the best MISC classification score. To that extent, we train different models that take as input a single modality. These models are composed of the preprocessing networks described above, followed by a linear classifier. The results summarized in Table 2 show that the text + context modality appears to be the most efficient. On the other hand, body expressivity has low prediction power. Confidence intervals are calculated using the bootstrap method [START_REF] Efron | An introduction to the bootstrap[END_REF]. Training details are provided below.

Text based model. The text preprocessing model is trained for 150 epochs with an AdamW optimizer [START_REF] Loshchilov | Decoupled weight decay regularization[END_REF] and a Cosine Aligned scheduler [START_REF] Loshchilov | Sgdr: Stochastic gradient descent with warm restarts[END_REF] with a maximum learning rate of 2 * 10 -4 .

Text and context based model. The text and context preprocessing model is trained for 25 epochs with an AdamW optimizer [START_REF] Loshchilov | Decoupled weight decay regularization[END_REF] and a learning rate of 2 * 10 -5 . Audio based model. The audio preprocessing model is trained for 25 epochs with an AdamW optimizer [START_REF] Loshchilov | Decoupled weight decay regularization[END_REF] and a learning rate of 10 -5 .

Facial expressivity based model. The facial expressivity preprocessing model is trained for 150 epochs with an AdamW optimizer [START_REF] Loshchilov | Decoupled weight decay regularization[END_REF] and a One Cycle LR scheduler [START_REF] Leslie | Super-convergence: Very fast training of neural networks using large learning rates[END_REF]with a maximum learning rate of 10 -4 .

Body expressivity based model. The body expressivity preprocessing model is trained for 1500 epochs with an AdamW optimizer [START_REF] Loshchilov | Decoupled weight decay regularization[END_REF] and a learning rate of 5 * 10 -5 .

Multimodal models

Now that we learned more about our unimodal models performance, we investigate whether multimodality could improve the performance of our MISC classification model. Using the fusion architecture described above, we train several multimodal models. We use a frozen Bert and Beats models to improve training time and avoid over fitting. As a mean of comparaison, we also train the model using text and context linearly from the previous section with a frozen-Bert transformer. These multimodal models are trained for 150 epochs with AdamW optimizer [START_REF] Loshchilov | Decoupled weight decay regularization[END_REF] and Cosine Aligned scheduler [START_REF] Loshchilov | Sgdr: Stochastic gradient descent with warm restarts[END_REF] with a maximum learning rate of 2 * 10 -4 . The results are displayed in Table 4. Because of the low diversity of body expressivity (clients are seated in the videos and do not move much) and the large number of missing data (a quarter of sentences are provided with body expressivity information), the addition of body expressivity decreases the accuracy of change talk detection, which is the most important classe. Therefore, in the following, we decide not to use body expressivity in the model.

In all cases, using the MALEFIC architecture improves classification results over the most performant preprocessing network (Text + context linear) Particularly, combining text, context, audio, and facial expressivity outperforms all models with frozen Bert and Beats embeddings. Meaning that the combination of visual, vocal, and verbal modalities improves the classification performance. MALEFIC is able to take advantage of the new modalities and to select relevant multimodal information. For a MISC classifier, we especially want to be able to classify change talk and avoid classifying change talk as sustain talk and vice versa. The confusion matrix in Tab.3 shows that our model makes few change talk/sustain talk mistakes.

Comparison with existing studies

We compare our results with three existing studies [START_REF] Nakano | Detecting Change Talk in Motivational Interviewing using Verbal and Facial Information[END_REF][START_REF] Tavabi | Multimodal automatic coding of client behavior in motivational interviewing[END_REF][START_REF] Wu | Creation, Analysis and Evaluation of AnnoMI, a Dataset of Expert-Annotated Counselling Dialogues[END_REF]. However, the data set used in these studies is not available, so the conclusion of the comparison should be made with care. The Table 5

Text based model.

In [START_REF] Wu | Anno-MI: A Dataset of Expert-Annotated Counselling Dialogues[END_REF], a Bert model is trained on AnnoMI to predict MISC classes only on the current utterance (text without context). This model is similar to the one we described in section 5.1 and is trained on the same dataset. The only difference with our work is the reorganization of the transcripts performed in Section 3.2. The model in [START_REF] Wu | Anno-MI: A Dataset of Expert-Annotated Counselling Dialogues[END_REF] reaches a 0.55 F1 macro score, which is significantly lower than the score achieved by our approach (0.68), which uses a similar architecture.

One factor that may explain the performance gap is the preprocessing of the text performed in our approach, as discussed in Section 3.2. By providing full sentences with semantic meaning, our approach is able to capture more nuanced linguistic features, enabling a more accurate classification of MISC classes. These results provide a validation of the effectiveness of our text preprocessing.

Text and audio-based model.

In [START_REF] Tavabi | Multimodal automatic coding of client behavior in motivational interviewing[END_REF], audio and text are used to classify utterances into the 3 MISC classes, change talk, sustain talk, and follow / neutral. Our approach achieves a significantly higher F1 micro score of 0.62 compared to their score of 0.53, based solely on audio input (see Table 5). However, this accuracy gap may be attributed to the poor quality of audio recordings in their corpus, which is not the case in ours.

Moreover, in their approach, adding the audio modality results in a small drop in precision, where, using our fusion method, we are able to slightly improve accuracy by adding the audio modality.

Text and

Facial expressivity based model. In [START_REF] Nakano | Detecting Change Talk in Motivational Interviewing using Verbal and Facial Information[END_REF], text and facial expressivity (action units, head positions, and eye direction) are used to predict whether an utterance displays change talk or not. They looked at a two-label classification problem when we classify utterances into 3 categories. Their corpus was collected using Zoom, meaning that participants are always facing the camera, whereas our corpus shows a greater variety of body orientations and, therefore, noisier OpenFace outputs. However, we are able to classify change talk significantly better.

In their approach, adding facial expressivity improves the F1 scores on the not change talk class, but does not change the change talk F1 score. Our approach allows us to slightly improve the F1 score on change talk and to produce a higher overall F1 score despite the variety of positions of the clients in the videos and the missing data (when the camera does not show the client's face).

INTERPRETATION

The ability to quantify the contribution of each modality in the classification process is a key advantage of our approach. By utilizing multiple modalities, such as text, prosody and facial expressivity, we can gain a more comprehensive understanding of the client's communication and behavior during an MI conversation.

Identifying which modality is relevant to the classification of a given sentence can offer valuable insights into the client's state of mind. For example, if facial expressivity or prosody are found to be more influential in the classification process, it may suggest that the client is trying to conceal their true thoughts. Several elements of our model offer the bases to draw explanations of the model outputs. We can name the use of dropout and random selection of embeddings during training allows the final embeddings of each modality to be computed in the same embedding space as the fusion embedding. This ensures that all modalities are represented consistently.

Furthermore, the self-attention layers included in our approach allow the model to dynamically weigh the importance of each modality for each sentence. These layers give a sense of the relevance of each modality not only for each embedding but also for each sentence to be classified.

In this section, we take advantage of these properties to visualize and quantify the contribution of each modality. All the following statistics are computed on the part of the validation set where all modalities are available.

Overall modality contributions

To quantify the contribution of each modality within the corpus, we examine the average number of times a modality is selected by the self-attention module over all embedding dimensions. Our analysis reveals the following overall contribution: text (26%), audio (16%), face (26%), previous client sentence in the turn (16%), and previous therapist turn (16%). This distribution shows that all modalities are considered by the model with more weight given to the Text and Facial expressivity. These results demonstrate that the model considers all modalities, with a greater weight placed on text and facial expressivity. This aligns with our finding that text is the strongest predictor when taken as a single input (see Table 2). The fact that facial expressivity has a strong weight despite its low predictive powers can be explained in the following sections (see Section 6.3).

Embedding specialization

To understand the role of each embedding dimension, we examine the average number of times a modality was selected for a given embedding dimension. Figure 3 shows the distributions of the modality contribution averaged over each embedding dimension.

This figure shows that some embedding dimensions have a modality contribution of 1 for the text and facial expressivity modalities. This means that this dimension has specialized into a certain modality. This modality will be systematically selected if available. The two modalities that have the greater weight in the overall corpus (text and facial expressivity) are the two modalities with specialized embeddings. The fact that the dimensions are specialized in the text modality aligns with our finding that the text is the strongest predictor when taken as a single input (see Table 2).

On the other hand, there are, for every modality, some dimensions with a contribution of 0 meaning that this modality is never selected for this dimension.

Quantification of modality contribution for each sentence

To quantify the contribution of each modality to the classification of a given sentence, we examine the number of dimensions of the fusion embedding that have been selected from this modality for a particular sentence. This provides insights, for a given instance of the client's speech (a sentence), of the amount of information of a modality that is used to make a decision. Figure 4 shows the distribution of the modality contribution averaged over each sentence. Our analysis indicates that the contribution of each modality is highly dependent on sentences. Specifically, we observed that the distribution of text, audio, and context from both the client and the therapist can be characterized by two Gaussian distributions, indicating that these modalities are more informative for some sentences than for others. In contrast, only one Gaussian distribution is visible for facial expressivity, suggesting that this modality is used more consistently across the dataset. This may be because facial expressivity is not a strong predictor for classifying MISC classes. Indeed, because of the use of modality dropout, the model is not able to completely ignore a modality. Therefore, in case of weak predictor, the model has a harder time determining when the modality is useful and takes it into account consistently across the corpus. This can also explain why facial expressivity has a weight as large as the text modality in the overall contribution and why some embeddings are specialized in this modality. Indeed, the face modality is always selected as the model is not able to detect the sentences where it is really useful and the other modalities are selected only when they are relevant.

To better understand the differences in the sentences that lead to the above results, we perform a clustering of the contribution of each of the considered modalities using the elbow method and K-means and find five clusters with a silhouette score of 0.96.

Sentences can be clustered into groups where the contributions of the modalities are different (see Fig. 5). The five clusters can be interpreted as five types of sentences:

• Cluster 1: The text and the context of both, the client and the therapist are relevant: 57%

• Cluster 2: The previous speaking turn of the therapist is relevant: 16%

• Cluster 3: The previous sentences of the client in the speaking turn are relevant: 12%

• Cluster 4: The current sentence is relevant: 9%

• Cluster 5: The audio is relevant: 6% Table 6 shows an example of sentences for each group. These clusters confirm that facial expressivity contributes consistently across the dataset. Additionally, they demonstrate the importance of considering multiple modalities. By revealing which modality is most relevant for a given sentence, this analysis provides a valuable tool for validating decisions and could be used by the therapist to provide feedback to the client in real-time. It could also be used by a virtual agent acting as the therapist to detect change talk and to use this information for its next dialog move. For example, the agent could explain its decisions by saying something like "From your tone of voice, it sounds like you are not ready to change.". As foreseen, the cluster distributions display that text and context are the most important features in most cases.

Embedding visualization

The embedding space is visualized using UMAP [START_REF] Mcinnes | Umap: Uniform manifold approximation and projection for dimension reduction[END_REF], a framework used for dimensionality reduction that is reversible. Due to its reversible quality, we are able to create a map of the embedding space showing how each embedding point would be classified. This visualization visible in Figure 6 allows us to determine how confident the classification is for every modality. The text is indeed the most expressive modality (see Fig. 6b) and that most of the other modalities are pertinent to accurately classify only in some cases, as seen in the previous sections. This visualization illustrates also which modalities contributed and in which direction to the classification of each sentence. Figure 7 shows example of sentences where the text embedding alone does not classify accurately but is improved by other modalities (Figure 7a). On the left, the text alone classifies as change, on the right as sustain, when the true classification is neutral. It also shows an example where only text alone classifies the sentence correctly as change, and the model is not misled by other modalities (see Figure 7b).

CONCLUSION AND FUTURE WORK

In this paper, we present a multimodal classifier for the three MISC classes of client behavior: change talk, sustain talk, and follow neutral. Our classifier is based on AnnoMI, an open access Motivational Interviewing database that is annotated in MISC classes and has been transcribed. We reorganized the transcript into sentences with lexical meaning and performed multimodal annotations of facial and body expressivity. Taking advantage of these multimodal inputs, we train a classifier that achieves greater accuracy than a unimodal approach and outperforms the existing approaches. We also use self-attention layers to determine the contribution of each modality, allowing us to interpret the results of our classifier and identify the most informative modality for a given sentence.

In future work, we plan to improve the model's performance by fine-tuning the Bert and Beats transformers. In addition, we envision endowing a virtual therapist agent with this model to enable it to detect whether the client is responding to therapy and is producing change talk. The agent could also provide feedback to the user regarding why it detected that the client may not be ready to change (e.g., tone of voice). Finally, we aim to make the model publicly available to facilitate the annotation of new MI videos and serve as a baseline for future work. Overall, our approach demonstrates the value of multimodal input in improving the accuracy of MISC classification while providing interpretable features.

Figure 2 :

 2 Figure 2: Example of transcript reorganization

Figure 3 :

 3 Figure 3: Distribution of modalities contribution for each embedding dimension

Figure 4 :

 4 Figure 4: Distribution of modalities contribution for each sentence

Figure 5 :

 5 Figure 5: Proportion of modalities contribution within each cluster

Figure 6 :

 6 Figure 6: Visualization of modalities embeddings with UMAP projection

( a )

 a Classification improved by multimodality (b) Better classification with only text

Figure 7 :

 7 Figure 7: Examples of sentences representation

Table 1 :

 1 AnnoMI distribution

	)

Table 2 :

 2 F1 score of single-modality models

		Predicted
		ST F/N CT
	Actual	ST 0.65 0.29 0.06 F/N 0.07 0.79 0.14 CT 0.04 0.27 0.69

Table 3 :

 3 Confusion matrix of the model Text+Audio+Face

  summarizes our comparisons.

	modalities:	Text + context (linear)	Text + context (MALEFIC)	Face + Body	Text + Face	Text + Audio	Text + Audio + Face
	F1 -CT	0.61[0.54,0.66] 0.63[0.57,0.68] 0.24[0.18,0.31] 0.64[0.58,0.69] 0.65[0.59,0.70] 0.65[0.59,0.71]
	F1 -ST	0.58[0.53,0.63] 0.63[0.58,0.68] 0.41[0.35,0.47] 0.60[0.55,0.66] 0.66[0.62,0.70] 0.66[0.61,0.71]
	F1 -F/N	0.78[0.75,0.80] 0.80[0.77,0.82] 0.63[0.60,0.67] 0.80[0.78,0.83] 0.81[0.78,0.83] 0.81[0.77,0.82]
	F1 -micro 0.71[0.68,0.73] 0.73[0.70,0.76] 0.51[0.47,0.54] 0.74[0.71,0.76] 0.74[0.72,0.77] 0.76[0.72,0.77]
	F1 -macro 0.65[0.62,0.69] 0.69[0.65,0.72] 0.43[0.40,0.46] 0.68[0.65,0.71] 0.71[0.67,0.73] 0.70[0.67,0.73]

Table 4 :

 4 F1 score of models trained with frozen Bert and Beats models

	modalities	Text				Audio Text + Audio	Facial expressivity	Text + Facial expressivity
	metric	CT	ST	F/N	Micro	Macro Micro CT ST	F/N Micro CT	ST F/N Macro CT	ST F/N Macro
	MALEFIC*

Table 5 :

 5 Comparison with other studies (* = trained using the same corpus, u = without context, c = with context)

Table 6 :

 6 Example of transcript for each cluster


Introduction

The label Central Sudanic traditionally applies to a group of some 60 languages that are spoken in the center of the African continent and cover parts of Chad, Sudan, C.A.R., D.R.C., and Uganda (see Map). With the exception of the Sara languages of Chad and Moru of Sudan/D.R.C. (more than 1 million each)1 most languages have less than 100.000 speakers and many of them, just a few thousands. Several languages are declining, namely under the influence of Chadian or Sudanese Arabic.

[Map 'The Central Sudanic Languages' somewhere here] Typological features include short word shapes (V, CV, VCV, and CVCV), frequent ±ATR vowel feature (in the south-eastern part of the area), widespread glottalized consonants (see section 4), and tone systems that may include up to four constrastive levels. Word order is diverse (see section 6). [START_REF] Tucker | The Non-Bantu Languages of North-Eastern Africa[END_REF][START_REF] Santandrea | The Birri Language. Brief Elementary Notes[END_REF] divided these languages in two 'larger units' (1. Moru-Mangbetu, 2. Bongo-Bagirmi) despite some similarities in lexicon and morphology (1956: 141-143). [START_REF] Greenberg | Languages of Africa[END_REF] coined the term Central Sudanic (henceforth CSD) and brought all the languages together -first as a part of Chari-Nile, then directly [START_REF] Greenberg | Nilo-Saharan and Meroitic[END_REF])-within his Nilo-Saharan family. Finally [START_REF] Bender | Central Sudanic segmental and lexical reconstructions[END_REF] divided Central Sudanic (renamed 'Family F') in two subgroups ('Peripheral' and 'Central') that roughly correspond to Tucker & Bryan's 'larger units'. These classifications are summarised in Table 1.

[Table 1 somewhere here] The aim of this paper is to evaluate the situation of these languages in the light of several criteria. I will argue that they probably constitute a genetic, historical unit, made of five, more or less equidistant subgroups, with the exception of some indeterminate languages, the situation of which remains uncertain for the time being. I will not address the relation of CSD with other Nilo-Saharan languages.

Inventory

CSD includes the following subgroups that basically correspond to those identified by earlier scholars (some representative languages are indicated in brackets):

1. [MMD] Moru-Madi (Moru, Avokaya, Logo, Lugbara, Ma'di) 2. [LND] Lendu (Lendu, Ngiti) 3. [MAS] Mangbetu(-Asua) (Mangbetu, Meje, Lombi, Asua) 4. [MEF] Mangbutu-Efe (Mangbutu, Ndo, Mamvu, Lese, Efe) 5.
[SBB] Sara-Bongo-Bagirmi (Modo, Baka, Bongo, Yulu, Gula dialects, Ndoka, Bagiro, Na, Kenga, Bagirmi, Sara languages) Every subgroup is relatively well documented for at least two languages. Given its linguistic coherence, each one can be assumed to constitute a historical unit. SBB, that represents the largest and most diversified subset, is also the best established genetic unit (Boyeldieu 2000b;[START_REF] Boyeldieu | Lexique comparatif historique des langues sara-bongo-baguirmiennes[END_REF]. The tricky question that is addressed in this paper then concerns the historical unity of Central Sudanic as a whole. Lastly, and although they are usually considered as Central Sudanic, I provisionally regard the following languages as indeterminate outsiders. Despite obvious similarities, Kresh, Dongo, Woro, Aja, and Birri [START_REF] Santandrea | The Birri Language. Brief Elementary Notes[END_REF](Santandrea /66, 1976) ) do not show lexical and sound correspondences that are significant enough with the above mentioned subgroups. On the basis of limited data [START_REF] Doornbos | Languages in Wadai-Darfur[END_REF], Sinyar has been counted among the SBB subgroup. However, recent fieldwork of my own shows that Sinyar also differs from SBB in many important features, notably the existence of a case marking system. The specific development of Sinyar therefore remains an open question.

Lexical evidence

Two types of estimates have been made regarding the lexicon. Tables 2a-b provide the results of a lexicostatistical study for a sample of 29 languages (based on a 100-word Swadesh list; Branch Average).2 MAS shows an internal similarity rate of 70%, MMD, LND, and MEF a rate of 60%, while the internal rate of SBB is markedly lower (30%). The similarity rates between subgroups vary from 20% (MMD with LND) to 05% (all subgroups). While the general figure clearly indicates five coherent subgroups, the higher connections between the latter rely on rates that are much less distinctive. As for the indeterminate languages, Sinyar is close to SBB (25%). Kresh, Dongo, Aja, and Birri also stand closer to SBB, although with a low rate of 10%. [Tables 2a and2b somewhere here] Table 3a displays the distribution patterns of 267 likely cognates between the five subgroups.3 Two remarks should be made here. First, the number of likely cognates shared by all the five subgroups (20) or even by four of them ( 51) is noticeably low. Second, the number of cognates shared by two subgroups (Table 3b) vary from 71=26,6% (MEF with LND, and SBB with LND) to 109=40,8% (MAS with LND), thus revealing no clear-cut predominance in their distribution. Here again the overall figure does not bring out clearly any marked proximity between some of the subgroups, that rather appear as more or less equidistant.

[Tables 3a and3b somewhere here] 4. Sound correspondances Establishing regular sound correspondences proves to be a difficult task given the rstricted number of general cognates, the variety of likely reflexes (especially for vowels and tones), and the shortness of the word shapes to be reconstructed (mostly *VCV). [START_REF] Bender | Central Sudanic segmental and lexical reconstructions[END_REF] fairly tried his hand at it, although his results are not quite convincing in my opinion. 4 In a very limited way [START_REF] Boyeldieu | Reflexes of a Labiovelar Series in Central Sudanic[END_REF] establishes regular correspondences for labial-velar consonants (intermediate reflexes are summarised in Table 4). Glottalized consonants also show persuasive series of regular reflexes (Boyeldieu, in progress). LND languages display an uncommon contrast of voiceless/voiced glottalized consonants [START_REF] Dimmendaal | Language typology, comparative linguistics and injective consonants in Lendu[END_REF][START_REF] Lojenga | Lendu: A new perspective on implosives and glottalized consonants[END_REF] that must be assigned to the likely CSD sound system. This contrast was lost and reflexes have merged in all other subgroups (intermediate reflexes are summarised in Table 5).

[Tables 4 and5 somewhere here] 5. Morphology [START_REF] Tucker | The Non-Bantu Languages of North-Eastern Africa[END_REF][START_REF] Santandrea | The Birri Language. Brief Elementary Notes[END_REF] pertinently pointed up morphological similarities that can be observed in most CSD subgroups. This regards the three following verbal prefixes: T(V)-: Verbal intensive (Tucker & Bryan 1956: 142) [MMD] e.g. Logo tV- [START_REF] Vallaeys | Dictionnaire logo-français, suivi d'un index français-logo[END_REF] (Tucker & Bryan 1966: 34-35) [MMD] e.g. Moru (Tucker, 1940: 205-211, 360- (Larochette, 1958: 27-28, 183-196) a.

-ɔku 'repartir' -ɛkú 'revenir'

b.

-owú 'égoutter' -ewú 'laisser égoutter' [MEF] e.g. Mamvu (Vorbichler, 1971: 191, 199) a.

ɔsi 'gehen' ɛsí 'kommen' b. òfe 'zu Ende gehen' ìfe 'beenden'

[SBB] No evidence

Although the distribution of these similarities between the subgroups reveal some gaps (especially in LND, see Table 6), there is no clear indication that a specific subgroupespecially not SBB-should be left out on this point.

[Table 6 somewhere here] 6. Word order Tucker & Bryan (1956: 141-143) further justified their distinction of two larger units, Bongo-Bagirmi (SBB) and Moru-Mangbetu (other subgroups), by a contrast in word order. However, things are not so straightforward. Table 7 displays a selection of typological features as characterized by Dryer & WALS (2011) for Central Sudanic languages. Notwithstanding clear contrasts of MMD and LND with SBB (features 81B and 85A), Mangbetu (MAS) and Mamvu (MEF) show the same SVO order like SBB (81A), while Moru (MMD) shows the same Noun-Genitive order like SBB (86A). Most probably SBB, that extends into the north-western part of the area, reinforced innovations in this respect but, here again, there is no evidence for a clear-cut contrast between SBB and the other subgroups all together.

[Table 7 somewhere here]

Conclusion

The various observations made in the preceding sections lead to the following concluding remarks:

-there is no clear evidence that SBB should be contrasted with the other subgroups as a whole; CSD rather comprises five, more or less equidistant subsets; -there is some evidence that the five subgroups may well result from the historical splitting of a single language (or dialect group); however, considerable work is still necessary to confirm this assumption and to explain the diversity of the present-day languages, a diversity that seems somewhat contradictory with their geographical proximity; -lastly the affiliation of some indeterminate languages remains uncertain; their similarities with CSD could turn out to be the result of different types of contact.

-┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └─┬┘ │ │ │ │ │ 90 -│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 80 -│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └─┬┘ │ │ │ │ └─┬┘ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 70 -│ └──┬─┘ │ │ │ │ └─┬─┘ └─┬┘ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └──┬──┘ └─┬┘ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └─┬┘ │ │ 60 - └─────┬────┘ └─┬┘ │ └─┬─┘ │ │ │ │ │ │ │ │ │ └──┬─┘ │ │ │ │ │ │ └─┬┘ │ │ │ │ │ │ └──────┴─┬───┘ │ 50 - │ │ │ │ │ │ │ │ └─┬┘ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 40 - │ │ │ │ │ │ │ └──┬─┘ └──────┬──────┘ │ │ │ │ │ │ │ │ │ │ │ 30 - │ │ │ │ │ │ │ └──────┬─────┘ │ │ │ │ │ │ │ │ └─────────┬─────────┘ 20 - └─────┬────┘ │ │ └─┬─┘ │ │ └──────┬─────┘ │ └──┬─┘ │ 10 - └───────┬──────┘ └────────────┬───────────┘ └───────────────────────────────┘ 0 -

Table 1 .

 1 Previous groupings of Central Sudanic languages ka lr ma ni lu as mt li mv ls ef kr dn bi aj md bk bn yu gk gz bo na sr ke si 100

	Pascal Boyeldieu
	CNRS (UMR 8135, LLACAN)
	France
	boyeldieu@vjf.cnrs.fr

Table 2a .

 2a Lexicostatistical distances between 29 CSD languages (based on a 100-word Swadesh list; Branch Average)

Estimations fromBender (2000: 51).

I am indebted to Thilo Schadeberg for his LEXISTAT software.

The counting is based on a personal database. An occurrence in one language is considered enough to represent a subgroup. Indeterminate languages are not taken into account.

For instance, out of his four series illustrating a *kp formula, only one is valid for me[START_REF] Boyeldieu | Reflexes of a Labiovelar Series in Central Sudanic[END_REF].As for the glottalized consonants, the author disregards the important voiceless/voiced contrast that characterizes the LND languages (see below).

ngb / *ndz *m ? MAS *VCV *kw-a/ɛ *g ? *ngw-a/ɛ *ŋw-a/ɛ ? MEF *(C)VCV *kp *? / *g-u ? *ngb ? *? SBB *(C)VCV *kp *gb / *g-u ? (*ngb ?) *ŋm Table 4. Central Sudanic correspondence formulas for labial-velar consonants CSD *VCV *ƥ *ɓ *ƭ *ɗ *ƈ *ʄ *ʔ MMD *(V)CV *ɓ *ɗ (/ *ɗw) *ʄ *ʔ LND *VCV *ƥ *ɓ *ƭ *ɗ *ƈ *ʄ *ʔ MAS *VCV *ɓ / *ʙ *ɗ *ʄ ? *ʔ MEF *(C)VCV *ɓ *l *j *VØV SBB *(C)VCV *ɓ *ɗ *ʄ *ʔ Table 5. Central Sudanic correspondence formulas for glottalized consonants

 (Dryer and WALS 2011)