Verena Rieser

Conversational AI has received a growing interest in recent years from both the research community and the industry. Products have started to emerge (e.g. Amazon's Alexa, Google's Home, Apple's Siri) but performances of such systems are still far from human-likeness communication. As an example, conversation with the aforementioned systems is often limited to basic question-response interactions. Among all the reasons why people communicate, the exchange of information and the strengthening of social bound appeared to be the main ones. In dialogue research, the two aforementioned problems are well known and addressed using dialogue act classification and emotion/sentiment recognition. Those problems are made even more challenging as they involve spoken dialogues in contrast to written text. A spoken conversation is a complex and collective activity that has a specific dynamic and structure. Thus, there is a need to adapt both natural language processing and natural language understanding techniques which have been tailored for written texts as it does not share the same characteristics. This thesis focuses on methods for spoken dialogue understanding and specifically tackles the problem of spoken dialogues classification with a particular focus on dialogue act and emotion/sentiment labels. Our contributions can be divided into two parts: in the first part, we address the problem of automatically labelling English spoken dialogues. In this part, we start by formulating this problem as a translation problem which leads us to propose a seq2seq model for dialogue act classification. Then, our second contribution focuses on a scenario relying on small annotated datasets and involves both pre-training a hierarchical transformer encoder and proposing a new benchmark for evaluation. This first part addresses the problem of spoken language classification in monolingual (i.e. English) and monomodal (i.e. text) settings. However, spoken dialogue involves phenomena such as codeswitching (when a speaker switch languages within a conversation) and relies on multiple channels to communicate (e.g. audio or visual). Hence, the second part is dedicated to two extensions of the previous contributions in two settings: multilingual and multimodal. We first address the problem of dialogue act classification when multiple languages are involved and thus, we extend the two previous contributions to a multilingual scenario. In our last contribution, we explore a multimodal scenario and focus on the representation and fusion of modalities in the scope of emotion prediction.

En premier lieu, j'aimerais remercier mes directeurs de thèse Chloé Clavel et Matthieu Labeau. D'abord Chloé, je te suis reconnaissant de m'avoir donné l'opportunité de faire cette thèse ainsi que d'avoir veillé à ce qu'elle se déroule dans les meilleures conditions possibles du début à la fin. Matthieu, merci à toi pour tous nos échanges, pour les retours que tu as pris le temps de me donner dès que j'avais besoin d'un avis.

Merci à mes rapporteurs de thèse, François Yvon et Benoit Favre, d'avoir pris le temps de relire attentivement mon manuscrit. I sincerely thank

Résumé

L'intelligence artificielle conversationnelle a suscité un intérêt croissant ces dernières années, tant dans la communauté des chercheurs que dans l'industrie. Des applications grand publique ont commencé à voir le jour (par exemple, Alexa d'Amazon, Home de Google, Siri d'Apple), mais les performances de ces systèmes sont encore loin d'une communication semblable à celle des humains. Par exemple, la conversation avec les systèmes susmentionnés se limite souvent à des interactions de base de type question-réponse. Parmi toutes les raisons pour lesquelles les gens communiquent, l'échange d'informations et le renforcement des liens sociaux semblent être les principales. Dans la recherche sur le dialogue, ces deux problèmes sont bien connus et abordés à l'aide de la classification des actes de dialogue et de la reconnaissance des émotions/sentiments. Ces problèmes sont d'autant plus difficiles à résoudre qu'ils concernent des dialogues parlés, contrairement aux textes écrits. Une conversation parlée est une activité complexe et collective qui possède une dynamique et une structure spécifiques. Il est donc nécessaire d'adapter les techniques de traitement et de compréhension du langage naturel qui ont été conçues pour les textes écrits car elles ne partagent pas les mêmes caractéristiques. Cette thèse se concentre sur les méthodes de compréhension des dialogues parlés et aborde spécifiquement le problème de la classification des dialogues parlés avec un accent particulier sur les étiquettes des actes de dialogue et des émotions/sentiments. Nos contributions peuvent être divisées en deux parties : dans la première partie, nous abordons le problème de l'étiquetage automatique des dialogues parlés en anglais. Dans cette partie, nous commençons par formuler ce problème comme un problème de traduction, ce qui nous amène à proposer un modèle seq2seq pour la classification des actes de dialogue. Ensuite, notre deuxième contribution se concentre sur un scénario reposant sur de petits ensembles de données annotées et implique à la fois le pré-entraînement d'un encodeur transformateur hiérarchique et la proposition d'un nouveau benchmark pour l'évaluation. Cette première partie aborde le problème de la classification du langage parlé dans des contextes monolingues (i.e. anglais) et monomodaux (i.e. texte). Cependant, les dialogues parlés impliquent des phénomènes tels que le code-switching (lorsqu'un locuteur change de langue au cours d'une conversation) et s'appuient sur plusieurs canaux pour communiquer (par exemple, audio ou visuel). La deuxième partie est donc consacrée à deux extensions des contributions précédentes dans deux contextes: multilingue et multimodal. Nous abordons d'abord le problème de la classification des actes de dialogue lorsque plusieurs langues sont impliquées et nous étendons donc les deux contributions précédentes à un scénario multilingue. Dans notre dernière contribution, nous explorons un scénario multimodal et nous nous concentrons sur la représentation et la fusion des modalités dans le cadre de la prédiction des émotions. aspects négatifs, après tout. Merci pour toutes ces soirées à refaire le monde. Merci Marine, ainsi que toutes les petites mains qui s'affairent autour d'elles, pour tous ces cageots de Rubis des jardins qui nous ont régalé tout le printemps. Merci également à mon taquin d'oncle Jean-Yves avec qui j'ai tant de plaisir à me chamailler ! Je remercie ma grand-mère, André, pour sa joie de vivre, son amour du jeu et de la poésie. Je tiens à remercier ma grand-mère, Mamick, qui encore à l'âge de 89 ans commence ses phrases par un "quand je serai vieille...". Merci pour tout ce que tu nous as donné, pour toute ton énergie que j'admire tant.

Je remercie mes parents pour tout. Pour m'avoir donné une si belle enfance. Pour m'avoir permis de faire des études longues. Pour m'avoir soutenu à chaque instant. J'ai beaucoup de chance de vous avoir. Je remercie mes frères Yves et Pierre ainsi que ma soeur Anne, pour tous ces moments passés ensemble, je lève mon verre à nos gamineries et nos blagues atroces ! Anne n'oublie pas que ta connaissance de l'art ancestrale de la Bavaroise te confère désormais un statut que tu te dois d'honorer ! List of Tables For MRDA, Wilcoxon test has been performed (10 runs with different seeds) for B train = 5 and B inf = 1 against all model with B train = 2. .... for the mHT. mHT u (µ spoken ) stands for the model pre-trained with the utterance level loss mL u on spoken data and mHT(µ written ) stands for a hierarchical encoder where sentence embeddings is computed using a pre-trained BERT encoder. ..................121 6.7 Results on the II task with monolingual input context. On this task the accuracy is reported. ..............................122

List of Figures

6.8 Accuracy of pre-trained and baseline encoders on MIAM. Models are divided in three groups: hierarchical transformer encoders pre-trained using our custom losses, baselines (see subsection 6.3.4) using either multilingual or language specific tokenizer. Toke. stands for the type of tokenizer: multi and lang denotes a pre-trained tokenizer on multilingual and language specific data respectively. When using lang tokenizer, MUG pre-training and finetuning are performed on the same language. .. Chapter 1

Introduction and Overview

Introduction

Spoken language is the most natural way of communication by which we transmit information or orders but also create social interactions. Its complexity is now a challenge for science and industry, leading to the rise of conversational AI with popular applications including Amazon's Alexa, Apple's Siri and Google's Home. Even if progress has been made, it has often been noted that such virtual agents are limited to a simple question-answer paradigm. Hence, we wonder whether it is possible to go beyond and develop an AI that will take into account the flow of the conversation, creating a more engaging experience for the user. Conversational AI relies on dialogue systems or more precisely, in spoken dialogue systems to manage the agent interaction. Such a system gathered two main components: a Natural Language Understanding (NLU) block which understands the intent of the user and a Natural Language Generation (NLG) whose aim is to deliver the agent's responses. In this thesis, we focus on the NLU part as it will condition the quality of the agent's future response.

NLU is a sub-domain of Natural Language Processing (NLP) and deals with machine comprehension. NLU covers a wide range of applications such as question-answering, text categorisation, machine translation, and slot filling. We focus on the applications that we believe to have the most impact on the agent's response which is the same as asking what are the main reasons humans communicate ?. We have identified two main reasons: exchanging information and strengthening social bonds. To exchange and share information (i.e stories, thoughts, ideas), we often communicate with others following a certain dialogue flow. Generally, human dialogues are not limited to a basic question/answer sequence which is, as previously said, one of the most common patterns in current dialogue systems. Instead, humans often first respond to previous context and then suggest their own answer (e.g questions, assertions). In this way, people display their interest and involvement in the conversation. The second reason for human communication is to strengthen their social bonding with others. For this reason, daily conversations are full of emotions. By expressing emotions, people communicate information to others about their feelings, intentions, relationship with the target of the emotions, and the environment. Understanding and empathising with this emotional state will influence the speech of those involved in the conversation and thus improve their relationship. The example of conversation in Table 1.1 provides a good illustration of the two phenomena. In the second speaker turn, speaker B understands the statement expressed by A and then asks The communication purposes mentioned above are two well known problems addressed in NLU as dialogue act (DA) classification and emotion/sentiment (E/S) recognition. Formally, DAs are semantic labels associated with each utterance in a conversational dialogue that indicate the speaker's intention, e.g., question, backchannel, statement-non-opinion, statement opinion. A key step to model dialogue is to detect the intent of the speaker. Indeed, correctly identifying a question gives an important clue to produce an appropriate response. Thus, the identification of both Dialogue Acts (DA) and Emotion/Sentiment (E/S) in spoken language is an important step toward improving models performances on spontaneous dialogue tasks and it is essential to avoid the generic response problem, i.e., having an automatic dialogue system that generates an unspecific response -that can be an answer to a very large number of user utterances.

Hence, this thesis focuses on methods for spoken dialogue understanding and specifically tackles the problem of spoken dialogue utterances classification, DA and E/S labels in particular. Formally, we introduce new methods relying on deep neural architectures to address the problem of dialogue utterances classification. This problem is first addressed in a monolingual and mono-modal setting (i.e. when the input conversations are composed of text in English). In this first setting, we propose new methods for two different situations: (1) when a high amount of English-only labelled dialogues are available, (2) when the amount of labelled data is limited but a large amount of unlabelled conversations are available. We believe this second situation is of great interest due to the high annotation costs and time of the first CHAPTER 1. INTRODUCTION AND OVERVIEW situation. Then, we aim at generalizing our approach and work on extended settings of dialogues classification. We first propose an extension of our systems to handle several languages even in the presence of code-switched dialogue. Last, we study the addition of multimodal signals when dealing on E/S classification and focus on their representation and fusion. This final chapter is an introductory study of a broader project where we investigate the representation of visual and acoustic signals into a textual modality which is well leveraged by current neural architectures. Notably, punctuation marks have been studied as a textual representation of prosodical cues, hence we propose to quantitatively study their impact on current neural network models in the scope of E/S prediction.

These different settings are studied through a series of research questions that we precise below.

Research questions

Over the last decade, a lot of progress has been made in neural network architectures which has had a tremendous impact on multiple fields including NLP (e.g text classification, dialogue modelling, text summarizing). One of the key features of deep neural networks is their ability to handle different types of data (e.g. written texts, dialogues, news) coming from various data sources. Handcrafted feature systems have been replaced by neural architectures allowing to automatically learn rich representations. The present work leverages deep neural architectures to handle spoken dialogue data for classifications tasks. Therefore, we aim at proposing and studying new methods for spoken dialogue understanding. This thesis is organised along two groups of research questions. The first group of questions are dedicated to English data and aims at answering the following general research question: RQ1: How to leverage recent advances in deep learning to build a system that can automatically label English spoken dialogue utterances? We start by addressing this question on a direct supervised approach when large labelled datasets are available. Thus RQ1.1 boils down to how to leverage recent advances in deep learning to build a system that can automatically label English spoken dialogue utterances when a large corpus of labelled data is available? As previously mentioned, there is a multilevel dependency: between emitted utterances within the conversation, utterances and their corresponding labels and between labels. Hence, RQ1.1 can be divided into the following sub-questions :

1. How to adapt the encoder architecture to the hierarchical aspect of dialogue?

How to better model long range dependency between labels in the decoder?

3. How to leverage the utterance/label dependency?

However, the proposed solution requires large annotated corpora which are not always available for both DA and E/S. Thus, to alleviate this problem we put ourselves in a situation where data is scarce and when large annotated datasets are not available. One of the current solutions to the problem of lack of annotated data is to pre-train neural architectures in a self-supervised manner. Pre-trained language models such as BERT or RoBERTa have drastically changed the shape of modern NLP systems and allowed to make an efficient transfer learning. While it has been shown that the aforementioned models achieved competitive results in a variety of NLU tasks, they may CHAPTER 1. INTRODUCTION AND OVERVIEW not be optimal when dealing with spoken dialogue data due to (1) the structure of the dialogue which is not a flattened text and (2) the discrepancy that exists between the written text these models are trained on and spoken dialogue data. Hence, within the context of data scarcity RQ1.2 boils down to how to leverage recent advances in neural networks to build a system that can automatically label English spoken dialogue utterances when there is a lack of labelled data, especially in the case of small corpora or in low-resource setting? This question can then be decomposed into the following sub-questions:

1. Does an evaluation benchmark suited to low-resource sequence labelling for DA and E/S exist? If no, how to gather such an evaluation benchmark?

2. How to gather a large pre-training data that is suited for dialogue? How does the nature of the pre-training corpus influence the final performance of the whole system?

3. How to adapt transformer based architecture to the hierarchical aspect of dialogue?

4. Spoken dialogues have a hierarchical structure. How to take into account the hierarchy during pre-training ? How does the hierarchy impact pre-training objectives?

The second group of questions is dedicated to the study spoken dialogue understanding in two different settings: multilingual and multimodal. Thus, RQ2 can be phrased as How to modify spoken dialogue understanding systems to handle multilingual and multimodal data? What are the specificities of multilingual and multimodal data that the system needs to handle? Henceforth, we further precise RQ2 in the two aforementioned settings.

In the first part of this thesis we focused on English data as most of the available resources are English-centric. However, as AI increasingly blends into everyday life across the globe, researchers are investigating the development of models that can handle multiple languages. Similarly, spoken dialogue systems must be able to handle languages other than English and as well as multilinguality within the same conversations. Hence, we formulate the following research question: RQ2.1 How to build a system that can label multilingual conversational data and thus handle code-switching ? This research question can be further split into the following subquestions:

1. Does an evaluation benchmark for spoken dialogue classification in a multilingual setting exist? If no, how to gather such an evaluation benchmark? How to ensure that our representations are robust to different languages as well as code switching?

2. How to gather a large pre-training data that is suited for multilingual dialog? What are the specifics of the multilinguality that can influence the gathering of the pre-training corpus?

3. How to adapt the pre-training procedure of the previously introduced neural architectures to handle multilingual conversations?

CHAPTER 1. INTRODUCTION AND OVERVIEW

Another possible extension of monolingual spoken dialogue classifiers is to include multimodality. Multimodality is a setting of high interest when dealing with emotion recognition as an emitted sentence can be perceived in a different manner according to the tone of the speaker or his/her visual expression and thus multi-modal cues can be used to better classify the input utterance. Current multimodal architectures join information from two or more modalities into one synthetic representation.

They often rely on a fusion mechanism performed on monomodal data coming from different sources: language is a symbolic and discrete signal while audio and visual modalities are represented by continuous signals. Hence, building a common representation is a challenging task due to the different nature of modalities. It has also been shown that the textual modality is well leveraged by current architectures and carries most of the useful information. From this point of view, we investigate the possibility of translating other modalities, i.e visual and audio, into the textual modality.

As a preliminary study, we are interested in the use of punctuation marks to indicate prosodical information, such as intonations or pauses, and thus convey emotions in textual communication as in chats or in theatre plays. Linguists have showcased the prosodic aspect of punctuation marks thereby acknowledging their importance in human communication. However, punctuation marks are often overlooked, considered as noise when working with spoken language. Thus, it is interesting, in the case of emotion recognition for spoken dialogues to study the link between punctuation and multimodality. Hence, we aim at answering the following research questions: RQ2.2

What is the link between punctuation and multi-modality? which we decompose into subquestions:

1. What multimodal information is stored into the punctuation?

How much the use of punctuation marks influence the prediction quality?

3. What is the link between punctuation marks and emotions? Which punctuation marks convey most of the information for emotions prediction?

Contributions and detailed thesis organisation

In this thesis, we explore and propose new methods for spoken dialogue understanding following the latest trends in NLU with a particular focus on deep neural networks.

In [Chapter 2] In this chapter, we briefly introduce the spoken dialogue system and its component. Then we recall the particular characteristics of spoken dialogue such CHAPTER 1. INTRODUCTION AND OVERVIEW as turns, utterances, disfluencies, dialogue act. We finally present the related work relevant for sequence labelling with a particular focus on DA and E/S classification in dialogue which are the main problems we address in this work.

[Chapter 3]: Provide an overview of deep learning concepts useful for the present work. We start by introducing the supervised approach highlighting its defects. Then, we introduce the self-supervised learning that allows to overcome the shortcomings of supervised learning. This approach is the first step of a broader framework called transfer learning for which we showcase the characteristics. Finally, we recall the neural architectures used throughout this thesis.

[Part II] The second part gathers the contributions related to the RQ1. We present two approaches to label English spoken dialogue (1) in large annotated corpora and (2) in a data scarcity setting.

[Chapter 4]: This first model targets only DA classification problems as it requires large corpora (e.g. SwDA and MRDA) that are only labelled in dialogue acts. In this approach, we cast the DA classification problem in spoken dialogue as a Natural Machine Translation NMT problem. We successfully leveraged the well established Seq2Seq framework to model global dependency whereas models relying on the CRF based decoder are better suited for local dependencies. We introduced a hierarchical encoder tailored for the dialogue structure, a novel guided attention mechanism and beam search applied at both training and inference time. Compared to former models, our approach does not require any handcrafted features and is trained endto-end. We show such an approach achieves competitive results on MRDA and SwDA.

[Chapter 5]: The methodology in Chapter 4 presents several shortcomings: 1) the architecture is based on non-contextual word embeddings 2) encoder and decoder are based on GRU cells making them hard to train 3) the model requires a huge amount of labelled data that are not always available. Hence, in this chapter, we propose a pretrained hierarchical encoder based on a transformer architecture. We adapt existing pretraining objectives, such as Masked Language Model MLM or to this new architecture. Acknowledging the discrepancy between written and spoken dialogue data, we also chose a pretraining dataset better suited for the latter. Our representations are tested on a new benchmark called Sequence labelling evaLuatIon benChmark fOr spoken laNguagE benchmark (SILICONE) which gathered 10 broadly used datasets labelled in E/S or DA. We demonstrate that such an approach allows achieving competitive results while reducing the number of parameters.

[Part III] In this third part, we gather the contributions related to the RQ2.W e address problems of spoken dialogue classification exposed in Part II within two extended settings: multilingual and multimodal.

[Chapter 6]: So far, only english data have been considered. However, spoken dialogue systems should be able to handle multiple languages and multilinguality within a conversation. In this chapter, based on the architecture presented in Chapter 5, we present new pretraining losses tailored to learn multilingual spoken dialogue representations. These losses aim at exposing the model to code switched language. We automatically build a multilingual pretraining corpus composed of multilingual conversations in 5 European languages (English, French, Spanish, Ital-CHAPTER 1. INTRODUCTION AND OVERVIEW ian and German). We then test our representations on a new benchmark called MIAM for Multilingual dIalogAct benchMark which gathered one dataset labelled in DA for each aforementioned language. Additionally, we provide two new evaluation tasks: contextual inconsistency detection and next utterance retrieval with both monolingual and multilingual input context.

[Chapter 7]: In the final chapter, we focus on emotion recognition in spoken data. So far our work has relied mostly on the textual modality. However several works have acknowledged the multimodal aspect of communication building complex fusion mechanisms to leverage these different canals of information. Yet, latest mutlimodal SOTA architectures have showcased the dominant aspect of textual modality in performance results. We investigate the possibility to translate non textual modalities, i.e. audio and visual, into the textual modality. As a starting point, we focus on punctuation marks as it is a simple and easily accessible textual representation of prosodical cues. Indeed, when the communication modality is the text, such as in chats or in theatre plays, the writers use punctuation marks to indicate intonations or pauses and therefore to convey emotions. However, punctuation marks are often reduced to their semantic aspect and removed in most NLP tasks. In this chapter, we investigate the role played by punctuation marks in an emotion recognition task with current neural network architectures.

The following references summarize the published contributions of this thesis:

Conferences: • E.

Part I

Background

Part I Abstract

This part gathers the related work useful to understand our contributions related to RQ1 and RQ2. The background is divided into two chapters and is structured as follows:

• in Chapter 2, we provide an informal introduction to fundamental concepts linked to spoken dialogue. This chapter presents the particular characteristics of spoken dialogue (e.g disfluencies, dialogue act, grounding) with a particular focus on the aspect that we will leverage in the next part of the thesis. The last part of this chapter is dedicated to the related work relevant to the sequence labelling problems which is one of the core problems we address in this thesis.

• in Chapter 3, we offer an introduction to deep learning concepts useful for this thesis. Although described pre-training methods have been tested in other settings (e.g written text, chat), we will adapt them in Part II so they can be used to better handle spoken dialogue. This chapter also includes related work dedicated to multilingual and multimodal learning that will be useful to understand Part III.

Chapter 2

Spoken Dialogue Understanding Chapter 2 Abstract

This thesis focuses on spoken dialogue understanding. Specifically we tackle the problem of spoken dialogues classification, DA and E/S labels in particular. Hence, this chapter aims at presenting the most important concept and methods employed in the field of spoken dialogue understanding. We start by giving an overview of spoken dialogue systems. We review some of the core linguistics concepts relevant to understand the specificity of a conversation compared to an ordinary text: turn taking, grounding, dialogue act, disfluency, code-switching. Lastly we recall the related works needed to address DA and E/S classification problems presented in RQ1 as well as background related to the two extension scenarios presented in RQ2.

CHAPTER 2. SPOKEN DIALOGUE UNDERSTANDING

Dialogue Systems Overview

Conversational AI aims at developing dialogue systems, that are not only capable of conversing with human but also answering a diverse range of questions or realize complex tasks such as travel planning. Conversational AI is at the junction of multiple areas of research e.g. Natural language processing (NLP), Automatic speech recognition (ASR), Machine Learning ML, reinforcement learning (RL), Linguistic,

Psychology [ARORA and collab., 2013;GAO and collab., 2019]. Due to its promising potential commercial values and the great advances in deep learning and reinforcement learning, it has received a lot of attention in recent years. Dialogue systems are often divided into two categories [CHEN and collab., 2017;GAO and collab., 2019]: (1) task-oriented systems and (2) non-task-oriented systems (also known as chatbots).

The first helps the user to complete a certain task (i.e. purchasing products, booking a reservation) hence the interaction is limited by the task itself, whereas non-task oriented systems communicate with human users in order to provide an engaging conversation and entertainment. This work is in line with the latter category. Indeed, we aim at providing neural systems that are able to label emitted utterances during the conversation in DA and/or E/S. These are key information that will allow the dialogue system to generate adequate responses that retain the user's interest. Typically, dialogue systems follow the structure presented in Figure 2. 1 [ARORA and collab., 2013].

• Input Encoder This block converts the input into textual data. With a textual dialogue system, the textual content is directly provided by the user and this component is therefore not needed. In the case of spoken dialogue systems, the input encoder turns a speech signal into its corresponding transcript.

• Natural Language Understanding It maps emitted utterances with semantic labels which are then used by the dialogue manager. In our case, the NLU block provides dialogue act and emotion/sentiment labels associated with each emitted utterance.

• Dialogue Manager This component tracks the state and flow of the conversation and decides the next action to be taken by the system.

• Natural Language Generation This component generates the best response according to information provided by the dialogue manager.

• Output Renderer This final component renders the generated utterance.

In this thesis, we focus in particular on the NLU component. We also notice that in recent years fully data-driven and end-to-end approaches have been studied to conversational response generation [START_REF] Olabiyi | Multi-turn dialogue response generation with autoregressive transformer models[END_REF]SHANG and collab., 2015;SORDONI and collab., 2015;VINYALS and LE, 2015;ZHANG and collab., 2019b]. Such systems follow the seq2seq framework [SUTSKEVER and collab., 2014] and directly provide a response without relying on the aforementioned components of the traditional dialogue systems.

Characteristics of Spoken Dialogues

As we aim at developing dialogue systems that emulate the human conversation, it is essential to understand how humans dialogue with each other. In this section, we introduce the key concepts employed in the field of spoken dialogue analysis that showcase the dynamics and structure of a dialogue.

Dialogue

Overall, a dialogue is a complex and collective activity and is often defined as follows:

Definition 2.2.1 (Dialogue). talk between two (dyadic) or more people (multiparty) in which thoughts, feelings, and ideas are expressed, questions are asked and answered, or news and information are exchanged.

A dialogue can also be presented as a restricted form of conversation limited to two participants with an aim and a central topic. Alternatively, a conversation gathers a group of people that communicate without any specific goal and are somewhat more tentative. Moreover, a conversation may include change of topics. Still, since there is no consensus on the distinction between dialogue and conversation, we will use them indifferently. Turn A dialogue is structured by turns which are single contributions by one speaker to the dialogue. A change of turn occurs when the roles of the speaker and listener are exchanged. A turn can be either a whole sentence, several sentences or simply a word. For instance, Table 2.1 presents a dialogue between the two characters of the Samuel Becket piece Waiting for Godo. Since the example comes from a written play, the dialogue is segmented by turns that gather one speaker's whole speech. However, in spoken dialogue analysis, a turn is often divided into smaller units of speech that we present in the following. Utterances Utterances are often referred as "the smallest unit of speech. It is a continuous span of speech beginning and ending with a clear pause". Over the years, many different definitions of utterances have been proposed and make use of one or more of the following factors [TRAUM and HEEMAN, 1997]:

• Speech by a single speaker, speaking without interruption by speech of the other, constituting a single Turn.

• Defines a single dialogue act.

• Has syntactic and/or semantic completion.

• Is an intonational phrase.

• Separated by a clear pause.

In spoken dialogue systems, utterances are not given and come from ASR system.

Finding the beginning and the end of an utterance is a challenging problem [ANG and collab., 2005;ZIMMERMANN and collab., 2006]. Efficient and effective segmentation of spoken language remains an open problem in spoken dialogue. As an example, in Table 2.2 we can see (1) several consecutive utterances from the same speaker (2) the variety of utterances (e.g. "so", ".", "Uh-huh", "I'm not real"). Dialogue Act: dialogue acts (DA) or speech acts are a key component to analyse dialogue. [START_REF] Austin | How to do things with words[END_REF][START_REF] Searle | Speech Acts: An Essay in the Philosophy of Language[END_REF] explained that utterances emitted in a dialogue are not purposeless and rather described them as actions perform by the speakers. These actions are called dialogue acts and it becomes possible to characterise and classify these actions. [START_REF] Searle | A taxonomy of illocutionary acts[END_REF] established a taxonomy of DA divided into five categories :

1. Assertives: Committing the speaker to the truth of a proposition.

2. Directives: Attempts by the speaker to get the addressee to do something.

Commissives:

Committing the speaker to some future course of action.

Expressives:

Expressing the psychological state of the speaker about a state of affairs.

5.

Declaratives: Bringing about a different state of the world by the utterance.

Other annotation schemes are available with more detailed categories such as DAMSL (Dialogue Act Markup in Several Layers) introduced by [START_REF] Allen | Draft of DAMSL: Dialog act markup in several layers[END_REF].

Moreover there are numerous datasets annotated in dialogue acts such as Switchboard [GODFREY and collab., 1992], MRDA [SHRIBERG and collab., 2004], Dailydialog [LI and collab., 2017], each one with their own annotation scheme but, nevertheless, based on the DAMSL scheme.

Disfluency: as previously mentioned, in spoken dialogue systems the inputs comes from the user's speech. Spoken language is rarely fluent; the speaker's flow is often interrupted with disfluencies such as pause silence, self-correcting, repeating words or fillers (e.g. "um" or "uh") that fill pauses within an utterance during a conversation [START_REF] Shriberg | Preliminaries to a theory of speech disfluencies, thèse de doctorat[END_REF]. [DINKAR and collab., 2020] shows that representing fillers words helps to predict a speaker's stance and expressed confidence.

Code switching (CS): is a linguistic phenomena that has been observed with bilingual speakers, it is defined as the capacity of a speaker to alternate languages, effortlessly, within a conversation or speech. For instance, a well-known example from [POPLACK, 1980] of CS between English and Spanish: CS is considered as an important communication strategy and has been studied extensively in linguistics, especially as a speech phenomenon [AUER, 2013;[START_REF] Gardner-Chloros | Code-switching[END_REF][START_REF] Myers-Scotton | Code-switching[END_REF]POPLACK, 1980].

[ POPLACK, 1980] has shown that CS may occur in different ways which are divided into three categories :

• Extra-sentencial: also called tag-switching is the inserting of tag elements from one language into a monolingual discourse, e.g. "you don't know how to speak Spanish, ¿verdad? (right ?)"

• Intra-sentencial: the switch from one language to the other occurs outside a sentence, e.g : "Le dije que no queria comprar el carro (I told him I didn't want to buy the car). He got really mad"

• Inter-sentencial: the switch from one language to the other take place within a single utterance e.g. "¡ Ay, qué cute se ve !".

In recent years CS has also drawn the attention of the in [PAREKH and collab., 2020] authors propose a bilingual collaborative dialogue system that outputs code-switching conversations in a controlled manner. Benchmarks have also been built in order to provide a unified platform to evaluate CS data, such as Linguistic Codeswitching Evaluation (LinCE) [AGUILAR and collab., 2020] and

GLUECoS [KHANUJA and collab., 2020]. In Chapter 6 we present a multilingual system where pre-training losses and evaluation tasks are inspired from the CS phenomena.

In addition, spoken language presents a variety of specific phenomena that differentiate it from written communication such as surface formality change, lexical diversity and grammatical complexity and accuracy [START_REF] Chafe | The relation between written and spoken language[END_REF][START_REF] Redeker | On differences between spoken and written language[END_REF]. This discrepancy between written and spoken language is a core motivation of our work presented Chapter 5. Furthermore, we explore this difference in Chapter 7 through the lens of punctuation marks.

Related Work on Spoken Dialogue Understanding

In this section, we gather the related work related to spoken dialogue understanding. This section is organized as follows: we start by describing previous systems used for DA classification, then we gather the relevant related work for E/S classification.

DA classification

Several approaches have been proposed to tackle the DA classification problem.

These methods can be divided into two different categories. The first class of methods relies on the independent classification of each utterance using various techniques, such as HMM [STOLCKE and collab., 2000], SVM [SURENDRAN and LEVOW, 2006] and Bayesian Network [KEIZER and collab., 2002]. The second class, which achieves better performance, leverages the context, that relies on the sequence of previously emitted utterances, to improve the classifier performance by using deep learning approaches to capture contextual dependencies between input sentences [BOTHE and collab., 2018;KHANPOUR and collab., 2016]. Another refinement of input contextbased classification is the modelling of inter-tag dependencies. This task is tackled as sequence-based classification where output tags are considered as a DA sequence [START_REF] Chen | Emotionlines: An emotion corpus of multi-party conversations[END_REF]KUMAR and collab., 2018;LI and collab., 2018;RAHEJA and TETREAULT, 2019;STOLCKE and collab., 2000]. In the latter works the tags sequence is decoded using Conditional Random Features we present in the Chapter 3.I n Chapter 4 we showcase a Seq2Seq based model...

Emotion/Sentiment recognition

The MELD [PORIA and collab., 2018], DailyDialog [LI and collab., 2017], SEMAINE[MCKEOWN and collab., 2012]) ERC has gained a lot of attention from the NLP community [MA-JUMDER and collab., 2018;ZHOU and collab., 2017]. As in current DA tagging systems, recent ERC models benefit from the advances in deep learning and aim at leveraging the contextual dependencies of the target utterance [HAZARIKA and collab., 2018;JIAO and collab., 2019;MAJUMDER and collab., 2018]. Efforts have been made to carefully model the dependency graph of utterances [GHOSAL and collab., 2019;SHEN andcollab., 2020, 2021]. Other works also benefit from finer modelling of the speaker personality [LI and collab., 2020;MAJUMDER and collab., 2018;SHEN and collab., 2020;ZHANG and collab., 2019a].

However, these works often rely on written based pre-training models (e.g. BERT, RoBERTa) as a feature extractor at the utterance level which is a defect we address in Chapter 5.

Multimodal Emotion Recognition Humans employ three different modalities to communicate in a coordinated manner: the language modality with the use of words and sentences, the visual modality with gestures, poses and facial expressions and the acoustic modality through changes in vocal tones. Multimodal representation learning has shown great progress in a large variety of tasks including emotion recognition, sentiment analysis [SOLEYMANI and collab., 2017], speaker trait analysis [PARK and collab., 2014] and fine-grained opinion mining [GARCIA and collab., 2019a].

Learning from different modalities is an efficient way to improve performance on the target tasks [XU and collab., 2013]. Nevertheless, heterogeneities across modalities increase the difficulty of learning multimodal representations and raise specific challenges. BALTRUŠAITIS and collab. [2018] identifies five core challenges for multimodal learning:

• Fusion: join information from two or more modalities to perform a prediction.

• Representation: learn how to represent and summarize multimodal data.

• Modality alignment: identify the direct relations between (sub)elements from two or more different modalities.

• Translation: map data from one modality to another.

• Co-learning: transfer knowledge between modalities, their representation, and their predictive models.

In Chapter 7 we will have a particular look at the fusion challenge. Multimodal fusion can be divided into early and late fusion techniques: early fusion takes place at the feature level [YE and collab., 2017], while late fusion takes place at the decision or scoring level [KHAN and collab., 2012]. A plethora of architecture with new fusion mechanisms relying on deep architectures have been proposed to tackle the problem of multimodal learning. Perhaps the most known includes TFN [ZADEH and collab., 2017], LFN [LIU and collab., 2018], MARN [ZADEH and collab., 2018a], MISA [HAZARIKA and collab., 2020], MCTN [PHAM and collab., 2019], HFNN [MAI and collab., 2019], ICCN [SUN and collab., 2020]). These models are evaluated on several multimodal sentiment analysis benchmarks such as IEMOCAP [BUSSO and collab., 2008], MOSI [WÖLLMER and collab., 2013], MOSEI [ZADEH and collab., 2018b] and POM [GARCIA and collab., 2019b;PARK and collab., 2014]. [RAHMAN and collab., 2020]. DAI and collab. [2021] pointed out that multimodal systems rely on handcrafted features for visual and audio and propose a end-to-end architecture to process all the modalities. Lastly, it has been showcased that the textual modality carry most of the performances [DAI and collab., 2021;RAHMAN and collab., 2019]. The symbolic nature of language is well processed by neural model especially with the rise of distributed representation while audio and visual modalities, represented as signals, seems less well handle (especially with handcrafted features) in the scope of emotion recognition. In Chapter 7,we examine the possibility to translate the audio signal into a symbolic representation via punctuation marks and provide a quantitative study.

Chapter 2 Conclusion

In this chapter, we introduced the characteristics of spoken dialogue and presented the two problems of NLU we focus on this thesis: DA and E/S classification. We presented the related work associated with RQ1 and RQ2. Chapter 3

Deep Learning for NLU

Chapter 3 Abstract

This thesis focuses on distributed representation learning of spoken dialogue transcripts. Modern NLP models rely on deep neural architectures pre-trained in an unsupervised manner then fine-tuned on final tasks also called downstream tasks. In this chapter, we present the different components of the aforementioned framework. We start by defining the supervised approach, followed by a description of the classification task in spoken dialogue. We recall definitions of all neural architectures used throughout this thesis. Finally, we present the pretraning stage focusing on the self-supervised approach and its use in NLP.

Supervised Learning

In the supervised learning setting the goal is to build a prediction function f that maps input and output pairs (x, y) by optimizing a well chosen criterion. In our case, an example of supervised learning problem is the emotion classification as shown in Figure 3.1: from a dataset composed of utterances with their respective labels, for instance, (Joy,Sur prise,Neutral,Fear,Anger) the task is to build a system that labels such utterances with minimum error. More formally, let X be the space of input features and Y the space of outputs. Depending on the nature of the output space Y the prediction problem be either a classification task (8y 2 Y , y 2 {1, ..., K} where K 2 N) or a regression task (8y 2 Y , y 2 R). Each sample (x, y) 2 X £ Y comes from an unknown distribution P . We denote H as the family of mapping functions from X to Y . Since this thesis makes use of neural networks we can limit H to the set of functions parameterised by a vector µ 2 R d where d 2 N. Then the risk of the predictor is defined as follows:

R( f ) = E (x,y)ªP L ( f (x), y) (3.1)
Where L is a non-negative loss function that measures the error made predicting ŷ = f (x) when the correct label is y. Hence the goal is to find f § that minimizes R. However this function cannot be computed as the input/output pairs distribution is unknown. In practice we have only access to a dataset D

= {(x i , y i ) 2 X £ Y } N i =1
called training set where each (x, y) ª P . Then we define the empirical risk:

R( f ) = 1 N N X i =0 `( f (x i ), y i ) (3.2)
For classification tasks, we choose the cross-entropy loss as a function `which is a combination of negative log-likelihood and softmax. We denote ŷi 2 R K the predicted probabilities for the K output classes for the input x i . The f function maps the input x i to a vector where each element is a score associated to one of the K classes, i.e f : X 7 ! R K . Hence the probability associated with the class j is calculated as follows:

ŷi,j = Sof tmax( f (x i )) j = e f (x i ) j P K k=1 e f (x i ) k (3.3)
hence the log-likelihood L between ŷi and y i (being the one-hot encoding vector of the output label associated to x i ) is defined as follow:

L( ŷi , y i ) = K X i =1 y i log( ŷi ) (3.4)
It often happens that f will correspond closely to a particular set of data D meaning that f will be low for R but high for R. This situation is called overfitting. Regularisation techniques are used to prevent this phenomenon such as adding a non-negative function (e.g L 1 or L 2 norm) called penalty term to R enforcing smooth predictors. In addition, other techniques are employed with neural networks to prevent overfitting such as adding noise, dropout [SRIVASTAVA and collab., 2014], early stopping [CARUANA and collab., 2000].

Sequence Labelling Formalisation

We start by formally defining the Sequence Labelling Problem. At the highest level, we have a set D of conversations composed of utterances, i.e., D = (C 1 ,C 2 ,...,C |D| ) with Y = (Y 1 ,Y 2 ,...,Y |D| ) being the corresponding set of labels (e.g., DA, E/S). At a lower

level each conversation C i is composed of utterances u, i.e C i = (u 1 , u 2 ,...,u |C i | ) with Y i = (y 1 , y 2 ,...,y |C i | )
being the corresponding sequence of labels: each u i is associated with a unique label y i . At the lowest level, each utterance u i can be seen as a sequence of words, i.e

u i = (! i 1 , ! i 2 ,...,! i |u i | ).
Concrete examples of labelled utterances in a conversation can be found in Figure 3.1.

In this thesis, we aim at building functions f µ that map consecutive utterances (u 1 ,...,u |C i | ) with their corresponding labels (y 1 ,...,y |C i | ), namely:

f µ (u 1 ,...,u |C i | ) = y 1 ,...,y |C i | (3.5)
The sequence u 1 ,...,u |C i | is encoded using neural networks presented in Chapter 4 and Chapter 5. To decode the sequence of labels, we rely either on Conditional

Random Fields method CRF or on a neural based decoder, both are presented and compared in Chapter 4.

Throughout this thesis, we use a simpler approach where we perform one prediction of the current utterances using the history of the conversation, namely:

f µ (u 1 ,...,u |C i | ) = y |C i | (3.6)
The full sequence is then decoded in a sliding window fashion.

In the previous section, we presented the SL framework that aims at minimising CHAPTER 3. DEEP LEARNING FOR NLU an objective over a family of function parameterised by a vector µ. In the following section, we review neural network architectures which are a specific class of function used broadly in machine learning areas and specifically in this thesis.

Neural Network for text classification

In this section, we start by presenting core neural architectures that are part of larger models, i.e. multi-layer perceptron, convolutional neural network and recurrent neural network. Then, we review methods and neural architectures that combine the aforementioned blocks in order to handle textual data.

Multi-Layer Perceptron

The multi-Layer Perceptron MLP or a fully connected Feed-Forward Neural Network FFNN is one of the simplest example of neural network. It consists of an input layer, one or several hidden layers and an output layer. Each layer l is composed of neurons or weights ! l ij and bias b l j and performs an affine transformation followed by an activation function f , namely:

h l +1 = f ( X i ! l ij h l i + b 1 j ) (3.7)
And h 0 = x, i.e. the input of the network. As activation functions, we commonly choose among:

• ReLU: f (x) = max(0, x)[AGARAP, 2018]. • GeLU: f (x) = x 2 (1 + erf( x p 2
)) [HENDRYCKS and GIMPEL, 2016].

• Sigmoid:

f (x) = ae(x) = 1 1+e °x .
• Hyperbolic tangent.

The final output vector is obtained via a different activation function. For classification problems, we typically use a softmax operation as presented earlier. Throughout this thesis, we use the MLP architecture as a decoder to perform the final decision in classification problems.

Convolutional Neural Network

The convolutional Neural Network [LECUN and collab., 1990] is an important neural architecture that has been a turning point in computer vision research [KRIZHEVSKY and collab., 2012]. In this architecture, each layer output unit is computed as a combination of local input units. We define the convolutional operator § between a 3D tensor I (e.g. an image as it is organised in a 2D grid with an additional channel dimension for the different colours) and a 3D tensor K called kernel or convolutional filter, by the following equation:

(I § K) x,y = n H X i =1 n W X j =1 n C X k=1 I x+i °1,y+ j °1,k K i , j ,k (3.8)
In CNN, a layer l is composed of weights W gathering D l kernel, namely W = (K i ) D l i =1 . Each layer output is computed by a local convolutional operator between the input and the layer weights, followed by a pooling operation f in order to perform a dimension reduction. Hence, units of the d thm output of the layer l , h l are calculated with the following equation:

h l x,y,d = f ((h l °1 § K d ) x,y ) (3.9)
CNN are mostly employed in computer vision but have been also used in other ar- eas such as speech recognition [ABDEL-HAMID and collab., 2013], NLP [COLLOBERT and collab., 2011b;GEHRING and collab., 2017]. In Chapter 7 we use CNN as sentence embedding following [KIM, 2014].

Recurrent Neural Network

Previous architectures, i.e MLP and CNN make the hypothesis of the independence of the data. After each processed input, the network's state is lost which is not desirable when the data are related in space or time as it is the case with words in a sentence or sentences in a conversation. Furthermore, the aforementioned architectures are limited to fixed-size vector examples, whereas there are a variety of applications with variable-length inputs and outputs. Recurrent Neural Network RNN [RUMELHART and collab., 1988] is a set of neural architectures that extend previous models and allow to process such sequential data x 1 ,...,x n . RNN can be viewed as a dynamical system where at each time t states are represented by a hidden state h t . At time step t the computation of a new state depends on the previous state h t °1, the input x t and sharing parameters µ following the equation:

h t = f (h t °1, x t ; µ) (3.10)
The simplest example of this equation would be:

h t = tanh(Wh t °1 + Vx t + b) (3.11)
where µ = (W, V, b) with W and V weights of the RNN and b the bias. For each time step tx t 2 R D i and h t 2 R D h . Finally, a RNN is defined by a initial state h 0 which may be set to a zero vector, a random vector or be trained as a parameter.

RNN aims at providing states h t that act as a memory and is trained to contain in- formation of the current sequence input x 1 ,...,x t . As a consequence, the last state may represent the whole sequence and can be used further, for example through a MLP classifier. This version describes the forward RNN where each current state is computed with the knowledge of the past. However, there exists cases where future states are accessible and would be helpful to process: for instance, when processing the utterance's tokens. Similarly, a backward RNN can be then defined reading sequential data starting from the end. The combination of forward and backward [START_REF] Schuster | Bidirectional recurrent neural networks[END_REF]. In that configuration each hidden state H t is the concatenation of the forward and backward RNN hidden states, namely:

RNN is called BiRNN [SCHUSTER
H t = [ °!h t ; √°h t ].
It appears that such models are hard to train as they suffer from gradient vanishing as shown in [START_REF] Hochreiter | The vanishing gradient problem during learning recurrent neural nets and problem solutions[END_REF] or gradient blow up. To overcome these gradients shortcomings [HOCHREITER and SCHMIDHUBER, 1997] introduced a more complex neural architecture based on multiplicative gate unit called Long Short Term Memory LSTM. CHUNG and collab.

[2014] reduces the number of parameters of the LSTM with a new recurrent unit GRU:

r t = ae(W r h t °1 + V r x t + b r ), (3.12) z t = ae(W z h t °1 + V z x t + b z ), (3.13) h t = z t Ø h t °1 + (1 °zt )tanh(W(r t Ø h t °1) + Vv t + b) (3.14)
LSTM and GRU have intensively been used in NLP. Because of their recursive nature, states are processed one after the other making these architectures time consuming.

GEHRING and collab.

[2017] has provided a seq2seq architecture only based on CNN allowing the computation of the whole sequence to be done at once and be fully parallelized during training. Then VASWANI and collab.

[2017b] has introduced the

Transformer architecture pushing a step forward computational efficiency and performances on numerous NLP tasks. We have used Transformer architectures in Chapter 5, Chapter 6 and we present the main components of the following in this section.

Neural Network for language computation

In this section we have a specific focus on neural architectures that have been specifically designed for natural language computation.

Word embeddings

By representing words with continuous vectors, word embeddings allow breaking the local barrier inherent of the discrete aspect of language [HINTON and collab., 1986] and becomes the corner stone of all neural architectures in NLP. Formally each token w in a vocabulary V is represented by a one-hot encoding vector, this vector is then multiplied by a continuous matrix E 2 R D£|V| where D is the dimension of the word embedding space. Thus, each column of E represents a token.

Encoder-Decoder

The encoder-decoder is a general paradigm that covers a wide range of neural models CHO and collab. [2014]; KALCHBRENNER and BLUNSOM [2013]; SUTSKEVER and collab.

[2014]. The encoder embeds the input sequence into a hidden state vector which is then decoded to obtain the target outputs. A well-known application of this framework is the sequence-to-sequence model or Seq2Seq[SUTSKEVER and collab., 2014] originally used in machine translation systems. The encoder extracts a fixed-length vector from an input sequence of variable length (sentence in a source language A) and then is fed to a decoder to generate a output of variable length (sentence translation in a target language B). In that model, a sequence of tokens w 1 ,...,w L i is fed to a LSTM then the last hidden state h L i is used as the initial state of the LSTM decoder to produce the corresponding outputs y 1 ,...,y L o , as follows:

p(y 1 ,...,y L o |x 1 ,...,x L i ) = L o Y i =1 p(y i |h L i , y 1 ,...,y i °1) (3.15)
One shortcoming of the aforementioned model is that it compresses all the length variable input information into one fixed sized hidden state which makes it difficult when coping with long range sequences. [BAHDANAU and collab., 2014] addresses CHAPTER 3. DEEP LEARNING FOR NLU this issue introducing attention mechanism allowing the decoder to softly search among the source position containing the most relevant information. Formally, s i denotes the i th state of the decoder LSTM following s i = f (s i °1, y i °1, c i ) where c i is a context vector computed as a weighted sum of the encoder hidden state :

c i = L i X i =1 AE ij h i (3.16)
where AE ij denotes the attention weights calculated as follows:

AE ij = e score(h i ,s i °1) P L i k=1 e score(h k ,s i °1)
(3.17)

In this first form, the score function was a bi-linear transformation namely:

score(h k , s i ) = h k T W a s i (3.18)
Due to its success, other attention mechanisms have been proposed [CHENG and collab., 2016] as well as in Chapter 4. As mentioned previously, the computational inefficiency of these neural models have led to the introduction of the Transformer. This architecture avoids recurrence by relying only on Feed Forward Network and scaled dot product attention. This attention mechanism connects all positions at constant time complexity, unlike former attention mechanisms, allowing to parallelize the computation, speeding up the training and inference times. This attention mechanism is decomposed between a set of key-value pairs (K,V) and a set of queries Q which are matrices of size d k , d v and d k respectively, defined as follows:

Attention(Q, K, V) = so f tmax( QK T p d k )V (3.19)
In the encoder, all the keys, values and queries come from the same place i.e. outputs of the previous layer of the encoder. In this part of the network, the attention is in fact a self-attention mechanism [CHENG and collab., 2016]. The decoder makes use of a similar self-attention mechanism and a second one also based on 3.1.6 where the queries come from the previous decoder layer and the keys and values come from the outputs of the encoder. Doing so allows the decoder to attend over every position of the input sequence.

In recent years, we have seen the number of parameters of these architectures rapidly skyrocket which strongly prevents the use of a direct SL approach. Indeed, for instance, in classification tasks the number of classes can go from 2 to several thousands meaning that it will not exceed 10 bits of information. Therefore, at each step of the training phase, the classifier only predicts a small amount of information. As a result, a huge amount of training samples is needed. For example, ResNet [HE and collab., 2015], a 60 million parameters deep convolutional network, has been trained on ImageNet a dataset gathering 15 million manually labelled images. Such a dataset is hard to find in the wild as high-quality human annotation campaigns are time consuming and costly, especially for text based datasets.

In this thesis, we directly train a seq2seq model in Chapter 4 on two large corpus annotated in DA. This has prevented us from using E/S labelled corpora that are much smaller and would have led to sub-optimal results. Meanwhile, learning strategies have been developed to overcome the direct and costly SL approach. In the next section, we recall these techniques that we adapt in Chapter 5 and Chapter 6.

Pre-training for Textual Data

Pre-training is now the first step in the modern deep learning pipeline. It is an effective strategy to learn the parameters of deep neural networks which are then used/fine-tuned in further tasks (aka downstream task, probing task). This approach is much efficient than random initialization since it aims to learn generic representation of the data.

In NLP, the pertaining stage has been firstly used to provide distributed word embeddings. COLLOBERT and collab. [2011a] first demonstrated that learning word embeddings on large unlabelled text data could improve the generalisation on a variety of tasks. collections of 570k human-written English sentence pairs. However, the current generation of neural models have widely adopted the unsupervised pre-training and especially the self-supervised learning approach. To summarise, the pre-training tasks can be grouped into three categories:

• Supervised learning (SL): aims at learning a function that maps an input to an output based on training data consisting of input-output pairs.

• Unsupervised learning (UL) is to find some intrinsic knowledge from unlabeled data, such as clusters, densities, latent representations.

• Self-Supervised learning (SSL) is a mix of supervised learning and unsupervised learning. SSL paradigm is the same as SL, but the labels of training data are generated.

In NLP, the SSL framework has attracted the most attention and has been successfully used in the learning process of a variety of well-known architectures: BERT In summary, SSL pre-training gives access to very large datasets allowing to increase the numbers of parameters and capacity of models. In addition, the pre-training shows some advantages over a direct approach explaining the performances boost:

• By alleviating the inputs data manifold, SSL allows to provide universal representations that help with a wide variety of downstream tasks, especially with small datasets WANG and collab.

[2020].

• Provide a better initialisation point for the neural network parameters. This allows a better generalisation and speed up the fine-tuning on downstream tasks DEVLIN and collab. [2018].

• ERHAN and collab. [2010] shows that pre-training acts as a regularisation avoiding overfitting on small datasets.

The aforementioned models were designed to encode unstructured text whereas, as we have shown previously, a dialogue has a specific structure that make it unique. Moreover, it has been shown that transfer learning using sentence embeddings tends to outperform word-level transfer [CER and collab., 2018]. This observation can be applied once again at the conversational level. In Chapter 5 we propose to pre-train an encoder tailored for conversations and adapt standard SSL objectives we recall in the following section.

Self Supervised Learning

Self Supervised Learning (SLL) leverages the label dependency as the labels are obtained from the data itself by using a "semi-automatic" process. Then, part of the data is predicted using the other parts. Specifically, the "other part" could be incomplete, transformed, distorted, or corrupted (i.e., data augmentation technique).

SLL has been successfully applied in a variety of machine learning domains such as graph Learning [HU and collab., 2020], speech recognition [RAVANELLI and collab., 2020], computer vision [DOERSCH and collab., 2015], NLP [DEVLIN and collab., 2018;[START_REF] Radford | Improving language understanding by generative pre-training[END_REF]RADFORD and collab., 2019]. This section covers general concepts of SSL with a particular focus on methods applied to NLP used and adapted in this manuscript. The challenge of SLL is to define proper objectives for unlabelled data (aka pretext task or pre-training objectives). These tasks can be divided into three categories :

• Generative: train an encoder to encode input x in to an explicit vector z and a decoder to reconstruct x from z (e.g. the cloze test [TAYLOR, 1953]).

• Contrastive: train an encoder to encode input x into an explicit vector z to measure similarity (e.g., mutual information maximization, instance discrimination).

• Generative/Contrastive (Adversarial): train an encoder-decoder to generate fake samples and a discriminator to distinguish them from real samples.

In this thesis, we focused on the generative approach. Contrastive and Generative/Contrastive are also two interesting frameworks, especially with NLU. They have been successfully applied to provide sentence embeddings [CLARK and collab., 2020;GIORGI and collab., 2020;[START_REF] Logeswaran | An efficient framework for learning sentence representations[END_REF]. We leave their adaptation for the dialogue structure as future work.

Generative SSL Auto-Regressive Model

In the auto-regressif AR model, giving a sequence of input S = (w 1 ,...,w N ), the prediction of each element w i dependent on its predecessors. Hence the goal is to maximize the joint distribution with the following factorization:

p(S) = N Y i =1 p(w i |w 1:i °1). (3.20)
This is a general framework that has been successfully applied for learning continuous word representation [BENGIO and collab., 2003] or in text generation with RNN [START_REF] Graves | Generating sequences with recurrent neural networks[END_REF]. In NLP, a Language Model LM plays a key role in a variety of tasks.

Their function is to assign probabilities to a sentence, i.e to determine how likely a sentence is in a given language. As a sentence can be naturally described as a sequence of tokens language model can be leveraged maximising the likelihood under the AR factorisation. This quantity is effectively computed in [VASWANI and collab., 2017a] with a self-attention based architecture called Transformers. GPT1 [RAD-FORD and NARASIMHAN, 2018] boards a more flexible approach with the use of a k length sliding window, namely:

p(S) = N Y i =1 p(w i |w i °k:i °1).
(3.21)

The AR formulation 3.2.2 is often referred as forward LM in contrast to the backward LM where the i th element is predicted using elements ahead, namely:

p(S) = N°1 Y i =0 p(w i |w i +1:N ). (3.22)
ELMo word embedding [PETERS and collab., 2018] are trained using both forward and backward AR objectives. As these objectives are unidirectional it prevents from learning deep bidirectional context. Hence there is a discrepancy between AR language model and most of the downstream tasks in NLU since they require bidirectional mobilisation.

Auto-Encoding Model

In the auto-encoding (AE) model, the goal is to reconstruct an input S from a corrupted input S. Doing so, the model does not try to estimate the joint distribution as in AR model and allows bidirectional context for reconstruction. Moreover, AE is an intuitive and flexible framework, broadly used with numerous variants.

In its simplest AE form, the input is not corrupted: S is projected in a latent space with an encoder function f e then reconstructed with a decoder function f d into an output S, namely:

f e (S) = z. (3.23) f d (z) = S. (3.24)
The goal is to minimize the error of reconstruction of S, i.e having S close to the original input S. For instance, if S 2 R p the objective would be to minimize the square distance between S and ~S. However, as such model learns the data with the output being equal to the output, the extreme form would be the identity function, thus no useful information would be learnt.

Denoising Autoencoder Model

Denoising Autoencoder (DAE) models solve this problem by corrupting the inputs adding noise or masking some parts. In recent years one of the most spectacular and broadly use of this paradigm is the Masked Language Model MLM introduced by [DEVLIN and collab., 2018] for training BERT. Inspired by the well-known cloze task [TAYLOR, 1953], they propose to replace part of the inputs tokens with a special token:

[MASK]. As this token is only present during the pre-training stage, authors do not systematically replace the chosen tokens with [MASK] but also in a small proportion with random words or without any modification. The MLM loss is defined as follow:

L MLM =°X i 2M S log(p(w i | Ŝ). (3.25)
where M S denotes the set of masked indices for the sentences S and Ŝ its corrupted form. Since then, several models have been derived from the classic MLM such as SpanBERT [JOSHI and collab., 2019] where continuous spans of tokens i.e consecutive tokens are masked. The masked span's start indices and length are chosen randomly.

Authors show improvement over BERT in coreference resolution. Usually, MLM is solved as a classification task: an encoder embeds the corrupted sentence into a vector that is then fed to a softmax classifier to predict the masked tokens. Another option is to use a seq2seq approach [LEWIS and collab., 2019;RAFFEL and collab., 2019;SONG and collab., 2019]. In this setting, the encoder output is fed to a retrogressive decoder that aims at reconstructing the whole sentence. These seq2seq MLM achieved better results on generative downstream tasks such as questionanswering, summarization or machine translation.

Finally there exists models that combine both AR and AE. As an example, BERT also uses another objective called the Next Sentence Prediction NSP where two consecutive sentences S 1 and S 2 are concatenate. All S 2 tokens are masked and should be predicted with the knowledge of S 1 . Similarly, XLNet [YANG and collab., 2019] brings the best of the two worlds introducing the Permutation Language Model PLM a generalization of the AR objective.

L PLM = E zªZ T T X t =1 logp µ (w z t |w z <t ). (3.26)
where z is a random permutation of the sentences tokens.

In this section, we have seen how SSL leverages the gigantic need for labelled data from raw data to build representation at the utterance level. In Chapter 5 we demonstrate how to adapt these objectives at the conversational level. In the following, we recall elements from the transfer learning paradigm relevant to this work.

Fine-tuning on Downstream Tasks

As previously mentioned the classical NLP pipeline can be divided into two stages as shown on Figure 3.2: 1) self-supervised pre-training stage on a large dataset of a general domain D 2) a fine-tuning stage on a specific task T

= {(X i ,Y i ) 2 D 0 £ Y },
where D 0 is the domain of the target task and Y a specific annotation scheme. Hence several parameters need to be considered when transferring a source model to a

Pre-training task vs fine-tuning task

Currently the MLM pre-training task is ubiquitous and has been useful in a wide range of NLP tasks [RADFORD and collab., 2019]. However, some pretaining tasks can be more suited than others toward specific downstream tasks, for instance, Next Sentence Prediction [DEVLIN and collab., 2018] forces the model to catch the relationship between two sentences. Such a task is useful in a context of Question-Anwsering QA, Natural Language Inferance NLI. Conversely, the MLM of BERT focused mostly on the encoder, making it more suited for NLU tasks than generation.

Source Domain vs Target Domain

The discrepancy between the source domain D, i.e. domain used during pre-training and the target domain D 0 , i.e. the domain used during fine-tuning, is another thing to consider. A domain is characterised by several aspects such as style, topic and genre [GURURANGAN and collab., 2020]. Hence, two texts can deal with the same subject but with two different styles. These aspects will have an impact on the vocabulary used and in fine on the data distribution. Intuitively, one could consider that the closer the data distributions of the source and target domain, the easier the transfer learning, which leads to better results for the considered tasks. Current SOTA models are pre-trained on very large corpus such as Wikipedia or book corpus.

While achieving good performance on most on NLP tasks, they do not seem to be optimal for tasks involving spoken dialogue. As we showcased in Chapter 2, there is a discrepancy between written text and spoken dialogue. This motivates the choice of

OpenSubtitles [LISON and TIEDEMANN, 2016] as pre-training corpus in Chapter 5. Domain adaptation is a whole area of ML and NLP in particular, while considered in this thesis, a deeper study is out of the scope of this work. It is worth noticing GURURANGAN and collab. [2020] showcases that domain shift can be leveraged by continuing the pre-training on large corpus which domain is closest to the target domain.

Layer choice

For multi-layers RNN-LSTM encoders, it has been shown [BELINKOV and collab., 2017] that layers capture different information that can benefit different tasks (POS tagging, parsing, semantic role, co-reference). The same analysis has also been made for pretrained models such as BERT [TENNEY and collab., 2019]: basic syntactic information is captured in early layers while high-level semantic information appears at the top layers. Hence there are several ways to select the representation:

• Word embedding: This approach was the first generation of pre-train models and used to be very popular in the NLP community. However, they are contextindependent and are used by shallow models that still need to be trained from scratch.

• All layers: this strategy is adopted in ELMo [PETERS and collab., 2018] where the final representation is computed as the weighted sum of all layers.

• Last layer: This is a classical approach adopted by a lot of current systems, especially those based on transformers. For example, in Chapter 5 and Chapter 6, an utterance representations is achieved by taking the representation of the

[CLS] token of the top layer.

Chapter 3 Conclusion

In this chapter, we recalled the supervised learning framework and formally introduced both DA and E/S classification tasks in spoken dialogue. As we aim at learning fully neural based classifiers, we presented all neural architectures used throughout this thesis. We showcased the transfer learning strategy as well as pre-training objectives used in NLP. We presented the limitations of existing approaches that we will address in Chapter 5 and Chapter 6. In the next chapter, we will present our contributions related to RQ1.

Introduction

In natural language processing research, the dialogue act (DA) concept plays an important role. DAs are semantic labels associated with each utterance in a conversational dialogue that indicate the speaker's intention, e.g., question, backchannel, statement-non-opinion, statement opinion. A key to model dialogue is to detect the intent of the speaker: correctly identifying a question gives an important clue to produce an appropriate response. As can be observed in tion relies on its conversational aspect, i.e., predicting an utterance's DA requires the knowledge of previous sentences and their associated act labels. For example, if a speaker asks a question, the interlocutor will answer with a response, analogously, a "Greeting" or a "Farwell" will be followed by a similar dialogue act. This means that in a conversation there is a sequential structure in the emitted dialogue acts.

This poses the basis for the adoption of a novel perspective on the DA classification problem, i.e., from a multi-classification task to a sequence labeling one.

Limitations of current models: Current state-of-the-art models rely on the use of linear Conditional Random Field (CRF) combined with a recurrent neural network based encoder [START_REF] Chen | Emotionlines: An emotion corpus of multi-party conversations[END_REF]LI and collab., 2018a;RAHEJA and TETREAULT, 2019] to model DA sequential dependencies. Unfortunately such approaches only capture local dependencies between two adjacent dialogue acts. For instance, if we consider the example in Table 4.1 we can see that the last statement "I know Nancy" is a response to the first question "Is there anyone who doesn't know Nancy" and the knowledge of the previous backchannel does not help the prediction of the last dialogue act. Therefore, we must consider dependencies between labels with a scope that is wider than two successive utterances. In Neural Machine Translation (NMT), the problem of global dependencies has been addressed using seq2seq models [SUTSKEVER and collab., 2014] that follow the encoder-decoder framework. The encoder embeds an input sentence into a single hidden vector which contains both global and local dependencies, and the hidden vector is then decoded to produce an output sequence. In this work, we propose a seq2seq architecture tailored towards DA classification paving the way for further innovations inspired by advances in NMT research.

Contributions:

In this work 1. We formalise the Dialogue Act Prediction problem in a way that emphasises the relations between DA classification and NMT.

2. We demonstrate that the seq2seq architecture suits better to the DA classification task.

3. We present a seq2seq model leveraging NMT techniques that reaches an accuracy of 85%, outperforming the state of the art by a margin of around 2%, on the Switchboard Dialogue Act Corpus (SwDA) [START_REF] Stolcke | Dialog act modeling for conversational speech[END_REF]] and a state-of-the-art accuracy score of 91,6% on the Meeting Recorder Dialogue Act (MRDA). This seq2seq model exploits a hierarchical encoder with a novel guided attention mechanism that fits with our setting without any handcrafted features. We finetune our seq2seq using a sequence level training objective making use of the beam search algorithm. To our knowledge, this is among the first seq2seq model proposed for DA classification.

Background

DA classification

Several approaches have been proposed to tackle the DA classification problem.

These methods can be divided into two different categories. The first class of methods relies on the independent classification of each utterance using various techniques, such as HMM [STOLCKE and collab., 2000], SVM [SURENDRAN and LEVOW, 2006] and Bayesian Network [KEIZER and collab., 2002]. The second class, which achieves better performance, leverages the context, to improve the classifier performance by using deep learning approaches to capture contextual dependencies between input sentences [BOTHE and collab., 2018;KHANPOUR and collab., 2016]. Another refinement of input context-based classification is the modelling of inter-tag dependencies. This task is tackled as sequence-based classification where output tags are considered as a DA sequence [START_REF] Chen | Emotionlines: An emotion corpus of multi-party conversations[END_REF]KUMAR and collab., 2018;LI and collab., 2018a;RAHEJA and TETREAULT, 2019;STOLCKE and collab., 2000].

Two classical benchmarks are adopted to evaluate DA classification systems: the Switchboard Dialogue Act Corpus (SwDA) [START_REF] Stolcke | Dialog act modeling for conversational speech[END_REF]] and the Meeting Recorder Dialogue Act (MRDA) [START_REF] Janin | The icsi meeting corpus[END_REF]. State-of-the-art techniques achieve an accuracy of 82.9% [LI and collab., 2018a;RAHEJA and TETREAULT, 2019].

To capture input contextual dependencies they adopt a hierarchical encoder and a CRF to model inter-tag dependencies. The main limitation of the aforementioned architecture is that a linear-CRF model is able to only capture dependencies at a local level and fails to capture non local dependencies. In this chapter, we tackle this issue with a sequence-to-sequence using a guided attention mechanism.

Seq2seq models

Seq2seq models have been successfully applied to NMT, where modeling non local dependencies is a crucial challenge. DA classification can be seen as a problem where the goal is to map a sequence of utterances to a sequence of DA.

Problem Statement

DA classification as an NMT problem

For the formalisation of the DA classification problem we rely on the mathematical notations introduced in subsection 3.1.1.I nNMT, the goal is to associate for any sentence X l 1 = (x

l 1 1 , ..., x l 1 |X l 1 | ) in language l 1 a sentence X l 2 = (x l 2 1 , ..., x l 2 |X l 2 |
) in language l 2 where x l k i is the i word in the sentence in language l k . Using this formalism, it is straightforward to notice two main similarities (S 1 , S 2 ) between DA classification and NMT.(S 1 )InNMT and DA classification, the goal is to maximise the likelihood of the output sequence given the input sequence (P(X l 2 |X l 1 ) versus P(Y i |C i )). (S 2 ) For the two tasks, there are strong dependencies between units composing both the input and output sequences. In NMT, those units are words (x i and y i ), inDA classification those units are utterances and DA labels (u i and y i ).

Specifics of DA classification

While NMT and DA classification are similar under some point of views, three differences are immediately apparent (D i ). (D 1 )InNMT, the input units x i represent words, in DA classification u i are input sequences composed with words. Considering the set of all possible sequences as input (context consideration leads to superior performance) implies that the dimension of the input space several order of magnitude larger than compared to a standard NMT.(D 2 )I nDA, we have a perfect alignment between input and output sequences (hence T = T0). Some languages, e.g., French, English, Italian share a partial alignment, but in DA classification we have a strong mapping between y i and x i .(D 3 )InNMT, the input space (number of words in l 1 ) is approximately the same size of the output space (number of words in l 2 ). In our case the output space (number of DA tags |Y |<100 has a limited size, with a dimension that is many order of magnitude smaller than the input space one.

In the following, we propose an end-to-end seq2seq architecture for DA classification that leverages (D 1 ) using a hierarchical encoder, (D 2 ) through a guided attention mechanism and (D 3 ) using beam search during both training and inference, taking advantage of the limited dimension of the output space.

Models

In Seq2seq, the encoder takes a sequence of sentences and represents it as a single vector H i 2 R d and then pass it to the decoder for tag generations.

Encoders

In this section we introduce the different encoders we consider in our experiments.

We exploit the hierarchical structure of the dialogue to reduce the input space size (D 1 ) and to preserve word/sentence structure. During both training and inference, the context size is fixed to T. Formally, an encoder takes as input a fixed number of utterances (u i °T, .., u i ) and outputs a vector H i 2 R d which will serve to initialize the hidden state of the decoder. The first level of the encoder computes E u t , an embedding of u t based on the words composing the utterance, and the next levels compute H i based on E u t .

Vanilla RNN encoder: The vanilla RNN encoder (VGRU E ), introduced by SUTSKEVER and collab. [2014], is considered as a baseline encoder. In the vanilla encoder

E u i = 1 |u i | P |u i | k=1 E w i k where E w i k is an embedding of w i k .
To better model dependencies between consecutive utterances, we use a bidirectional GRU [CHO and collab., 2014]:

°°°! h s i °T = √°°°h s i °T = °!0 °! h s t = °°°! GRU(E u t ), t 2 [i °T, i ] √°h s t = √°°°G RU(E u t ), t 2 [i , i °T] H i = [ √°h s i , °! h s i ] (4.1)
Hierarchical encoders: The vanilla encoder can be improved by computing E u i using bi-GRU. This hierarchical encoder (HGRU) is in line with the one introduced by SORDONI and collab. [2015]. Formally E u i is defined as it follows:

°! h w 0 = √°h w 0 = °!0 °! h w t = °°°! GRU(E w i t ), t 2 [1, |u i |] √°h w t = √°°°G RU(E w i t ), t 2 [|u i |,1] E u i = [ √°°h w |u i | , °°! h w |u i | ] (4.2)
H i is then computed using Equation 4.1. Intuitively, the first GRU layer (Equation 4.2) models dependencies between words (the hidden state of the word-level GRU is reset at each new utterance), and the second layer models dependencies between utterances.

Persona hierarchical encoders:

In SwDA, a speaker turn can be splitted in several utterances. For example, if speaker A is interacting with speaker B we might encounter the sequence (AAABBBAA) 1 . We propose a novel Persona Hierarchical encoder (Per-soHGRU) to better model speaker-utterance dependencies. We introduce a persona layer between the word and the sentence levels, see Figure 4.1:

°! h p t = ( °!0 if t and t °1 have different speakers °°°! GRU(E u t °1 ) √°h p t = ( °!0 if t and t + 1 have different speakers √°°°G RU(E u t +1 ) E p u k = [ °! h p k , √°h p k ] 8k 2 [i °T, i ] (4.
3) H i is then obtained following Equation 4.1 where

E u i is replaced by E p u i .

Decoders

In this section, we introduce the different decoders we compare in our experiments. We introduce a novel form of attention that we name guided attention. Guided attention leverages the perfect alignment between input and output sequences (D 2 ).

The decoder computes the probability of the sequence of output tags based on:

p(y i °T,...,y i |u i °T,...,u i ) = i Y k=i °T p(y k |H i , y k°1 ,...,y i °T) (4.4)
Vanilla decoder: The vanilla decoder (VGRU D ) is similar to the one introduced by SUTSKEVER and collab. [2014].

Decoders with attention:

In NMT, the attention mechanism forces the seq2seq model to learn to focus on specific parts of the sequence each time a new word is generated and let the decoder correctly align the input sequence with output sequence. In our case, we follow the approach described by BAHDANAU and collab.

[2014] and we define the context vector as:

c k = i X j =i °T AE j ,k h s j (4.5)
where AE j ,k scores how well the inputs around position k and the output at position j match. Since we have a perfect alignment (D 2 ), we know a priori on which sequence the decoder needs to focus more at each time step. Taking into account this aspect of the problem, we propose three different attention mechanisms.

Vanilla attention: This attention represents our baseline attention mechanism and it is the one proposed by BAHDANAU and collab. [2014], where:

AE j ,k = so f tmax(a(h Dec k°1 , h s j )) (4.6)
and a is parametrized as a feedforward neural network.

Hard guided attention:

The hard guided attention forces the decoder to focus only on the u i while predicting y i :

AE j ,k = ( 0, if k 6 = j 1, otherwise (4.7)
Soft guided attention: The soft guided attention guides the decoder to mainly focus on the u i while predicting y i , but allows it to have a limited focus on other parts of the input sequence.

e AE j ,k = ( a(h Dec k°1 , h s j ), if k 6 = j 1 + a(h Dec k°1 , h s j ), otherwise (4.8) AE j ,k = so f tmax(e AE j ,k ) (4.9)
where a is parametrised as a feedforward neural network.

Training and inference

In this section, we describe the training and the inference strategies used for our models. A seq2seq model aims to find the best sentence for a given source sentence. This poses a computational challenge when the output vocabulary size is large, since even by using beam search it's expensive to explore multiple paths. Since our output vocabulary size is limited (D 3 ), we do not incur in this problem and we can use beam search during both training and inference.

Beam search:

In our work we measure the sequence likelihood based on the following formula:

s( ỹk , u i ) = log P( ỹk |u i ) lp( ỹk ) (4.10)
where u i = (u i °T,...,u i ) and ỹk = ( ỹi°T ,..., ỹi°T+k ) is the current target, and lp( ỹ) = Training objective: For training we follow [EDUNOV and collab., 2017] and train our model until convergence with a token level loss and fine tune it by minimising the expected risk L RISK defined as:

(5+| ỹ|) AE (
L RISK = X ỹ2U(C i ) cost( ỹ, y i )p( ỹ|u i ) X ỹ0 2U(u i ) p( ỹ|u i ) (4.11)
where U(u i ) is the set of the sequences generated by the model using a beam search algorithm for the input u i , and cost( ỹ, y i ) is defined, for a given a candidate sequence ỹ and a target y i , as:

cost( ỹ, y i ) = ( 1 if ỹi = y i 0 otherwise ( 4
p(y|u) = 1 Z µ (u) exp ( T X t =1 √(y t , y t °1, E u t ) ) (4.13)
Where y = y 1 ,...,y T , u = u 1 ,...,u T and Z µ is a normalisation function defined as follow:

Z µ (x) = X y2Y exp ( T X t =1 √(y t , y t °1, E u t ) ) (4.14)
Finally we define the feature function √ as the sum of two scores potentials, namely:

√(y t , y t °1, E u t ) = ¡(E u t )[y t ] + T y t ,y t °1 (4.15)
Where ¡ : R H ! R K is a linear transformation from the embedding space of utterances to an output space of dimension K =|Y | and T 2 R K£K is the transition score matrix between labels.

Experiment Protocol

In this section we describe the experimental protocols adopted for the evaluation of our approach.

Datasets

We consider two classical datasets for Dialogue Act Classification: The Switchboard Dialogue Act Corpus and the MRDA. Since our models explicitly generate a sequence of tags we compute the accuracy on the last generated tag. Both datasets are already segmented in utterances and each utterance is segmented in words. For each dataset, we split each conversation C i in sequence of utterances of length T = 52 .

SwDA: The Switchboard-1 corpus is a telephone speech corpus [START_REF] Stolcke | Dialog act modeling for conversational speech[END_REF], consisting of about 2.400 two-sided telephone conversation among 543 speakers with about 70 provided conversation topics. The dataset includes information about the speakers and the topics and has 42 different tags. In this dataset global dependency plays a key role due to the large amount of backchannel (19%), abandoned or turn-exit (5%), uninterpretable acts (1%). In this context, any models that only take into account local dependencies will fail at extracting information to distinguish between ambiguous tags. For the confusion matrix, we follow LI and collab.

[2018a] and present it for 10 tags only: statement-non-opinion (sd), backchannel (b), statement-opinion (sv), conventional-closing (fc), wh-question (qw), response acknowledgement (bk), hedge (h), open-question (qo), other answers (no), thanking (ft).

MRDA:

The ICSI Meeting Recorder Dialogue Act corpus [SHRIBERG and collab., 2004] contains 72 hours of naturally occurring multi-party meetings that were first converted into 75 word level conversations, and then hand-annotated with DAs using the Meeting Recorder Dialogue Act Tagset. In this work we use 5 DAs, i.e., statements (s), questions (q), floorgrabber (f), backchannel (b), disruption (d).

Train/Dev/Test Splits: For both SwDA and RMDA we follow the official split introduced by STOLCKE and collab. [2000]. Thus, our model can directly be compared to [START_REF] Chen | Emotionlines: An emotion corpus of multi-party conversations[END_REF]KUMAR and collab., 2018;LI and collab., 2018a;RAHEJA and TETREAULT, 2019].

Dataset

Training details

All the hyper-parameters have been optimised on the validation set using accuracy computed on the last tag of the sequence. The embedding layer is initialised with pre-trained fastText word vectors of size 300 [BOJANOWSKI and collab., 2017] 3 , trained with subword information (on Wikipedia 2017, UMBC webbase corpus and statmt.org news dataset), and updated during training. Hyperparameter selection has been done using a random search on a fixed grid. Models have been implemented in PyTorch and trained on a single NVIDIA P100.

Parameters for SwDA: We used Adam optimizer [KINGMA and BA, 2014] with a learning rate of 0.01, which is updated using a scheduler with a patience of 20 epochs and a decrease rate of 0.5. The gradient norm is clipped to 5.0, weight decay is set to 1e°5, and dropout [LECUN and collab., 2015] is set to 0.2. The maximum sequence length is set to 20. Best performing model is an encoder with size of 128 and a decoder of size 48.

For VGRU E , we use two layers for the BiGRU layer. For hierarchical models, we use BiGRU with a single layer.

Parameters for MRDA: We used AdamW optimizer [LOSHCHILOV and HUTTER, 2017] with a learning rate of 0.001, which is updated using a scheduler with a patience of 15 epochs and a decrease rate of 0.5. The gradient norm is clipped to 5.0, weight decay is set to 5e°5, and dropout [LECUN and collab., 2015] is set to 0.3. The maximum sequence length is set to 30. Best performing model is an encoder with size of 40 and a decoder with size 400. For VGRU E we use two layers for the BiGRU layer, for hierarchical models we use BiGRU with a single layer.

Experiments

In this section we propose a set of experiments in order to investigate the performance of our model compared to existing approaches with respect to the difficulties highlighted in the introduction.

Experiment 1: Are Seq2seq better suited to DA prediction than CRF ?

Current state of the art are built on CRF models. In this first section, we aim at comparing a seq2seq with a CRF based model. To provide a fair comparison we perform the same grid search for all models on a fixed grid. At this step, we do not use attention neither use beam search during training or inference. As shown in Table 4.3, with a vanilla RNN encoder the seq2seq significantly outperforms the CRF on SwDa and MRDA. With an HGRU the seq2seq exhibit significantly higher results on SwDA and reaches comparable performances on MRDA. This behaviour suggests that a model based on a seq2seq architecture tends to be achieve higher score on DA classification than a CRF based model.

Global dependencies analysis:

In Table 6 the surrounding context (two sentences before) to disambiguate and assign the sv label . In the second example, the correct tag assigned to "Oh, okay" is a response acknowledgement (bk) and not backchannel (b). The key difference between bk and b is that an utterance labelled with bk has to be produced within a questionanswer context, whereas b is a continuer4 . In our example, the global context this is a question/reply situation: the first speaker asks a question ("What school is it"), the second replies then, the first speaker answers to the reply. This observation reflects the fact CRF models only handle local dependencies where seq2seq models consider global ones as well.

Utterances G. seq2seq CRF How long does that take you to get to work?

qw qw qw

Uh, about forty-five, fifty minutes.

sd sd sd

How does that work, work out with, uh, storing your bike and showering and all that?

qw qw qw

Yeah ,

bb b

It can be a pain .

sd sd sv

It's, it's nice riding to school because it's all along a canal path, uh,

sd sd sd

Because it's just, it's along the Erie Canal up here.

sd sd sd

So, what school is it?

qw qw qw

Uh, University of Rochester.

sd sd sd

Oh, okay. on SwDA. However on MRDA, adding a personna layer harms the accuracy. This suggests either that the information related to the speaker is irrelevant for our task (no improvement observed while adding persona information)5 , or that the considered hierarchy is not the optimal structure to leverage this information.

bk bk b

Our final model makes use of the HGRU encoder since in most of the settings it exhibits superior performance.

Experiment 3: Which attention mechanism to use?

The seq2seq encodes a source sentence into a fixed-length vector from which a decoder generates a sequence of tags. Attention forces the decoder to strengthen its focus on the source sentences that are relevant to predicting a label.

In NMT [LUONG and collab., 2015], complementing a seq2seq with attention contributes to generate better sentences. In Table 4.5 we see that in most the case, the use of a simple attention mechanism provides a rather small improvement with VGRU and harms a bit the performances with a HGRU encoder. In case of a seq2seq composed with a PersoHGRU and a decoder without attention the learning fails: the decrease of the training loss is relatively small and seq2seq fails to generalise. It appears that inDA classification where sequences are short (5 tags), Vanilla attention does not have as much as impact as in NMT (that have longer sequences with more complex global dependencies).

If we consider an HGRU encoder, we observe that our proposed guided attention mechanisms improves dev accuracy which demonstrates the importance of easing the task by using prior knowledge on the alignment between the utterances and the tags. Indeed, while decoding there is a direct correspondence between labels and utterances meaning that y i is associated with u i . The soft guided attention will mainly focus on the current utterance with a small additional focus on the context where hard guided attention will only consider the current utterance. Improvement due to guided attention demonstrates that the alignment between input/output is a key prior to include in our model. Attention analysis: Figure 4.2 shows a representative example of the attention weights of the three different mechanisms. The seq2seq with a normal attention mechanism is characterised by a weight matrix far from the identity (especially the lower right part). While decoding the last tags, this lack of focus leads to a wrongly predicted label for a simple utterance: "Uh-Huh" (backchannel). Both guided attention mechanisms focus more on the sentence associated with the tag, at each time step, and predict successfully the lastDA.

Experiment 4: How to leverage beam search to improve the performance?

Beam Search allows the seq2seq model to consider alternative paths in the decoding phase.

Beam Search during inference:

Using beam search provides a low improvement (maximum absolute improvement of 0.2%) 6 . Compared to NMT, output size is drastically smaller (Y SwDA = 42 while Y MRDA = 5) forDA classification. When considering alternative paths with small output space in imbalanced datasets the beam search is more likely to consider very unlikely sequences as alternatives (eg. "sssss").

Fine tuning with a sequence loss: As previously mentioned, using beam search during inference only leads to a limited improvement in accuracy. We finetune a seq2seq composed with a HGRU encoder and a decoder with hard guided attention (this model has been selected in the previous steps) with the introduced sequence level loss describes in Section 4.4.3.

Experiment 5: Comparison with state-of-the-art models

In this section, we compare the performances of seq2seq BEST with other state of the art models and analyse the performances of the models.

Error analysis

The confusion matrix on SwDA (see Figure 4.3) illustrates that our model faces same difficulties as human annotator: sd is often confused with sv, bk with b, qo with qw.

Due to high imbalance of SwDA, our system fails to recognise underrepresented labels (e.g. no and ft).

The confusion Matrix on MRDA shows that, here, the DA classification is easier compared to SwDA with fewer tags and classes that are more easily distinguished.

seq2seq BEST reaches a perfect score at recognising questions. One of the reasons for the mislabelling between backchannel (b) and statement (s) is that the MRDA dataset is highly imbalanced, with more than 50% of the utterances labelled as class s. True label 0.88 0.01 0.10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.36 0.00 0.64 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.10 0.05 0.00 0.82 0.00 0.00 0.00 0.00 0.00 0.00 0.12 0.00 0.00 0.00 0.82 0.00 0.00 0.00 0.00 0.00 0.00 0.47 0.00 0.00 0.00 0.53 0.00 0.00 0.00 0.00 0.10 0.00 0.20 0.00 0.00 0.00 0.70 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 0.00 0.00 0.89 0.00 0.00 0.67 0.00 0.00 0.00 0.00 0.00 0.33 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 In the previous chapter, we saw that a direct supervised approach requires a huge amount of labelled data which are not always available. In this chapter, we address the data scarcity problem through the light of pre-trained models. We propose a new approach to learn generic representations adapted to spoken dialogue, which we evaluate on a new benchmark we call Sequence la-bellIng evaLuatIon benChmark fOr spoken laNguagE benchmark (SILICONE).

SILICONE a is model-agnostic and contains 10 different datasets of various sizes annotated in DA or E/S. We obtain our representations with a hierarchical encoder based on transformer architectures, for which we extend two wellknown pre-training objectives. Pre-training is performed on OpenSubtitles: a large corpus of spoken dialogue containing over 2.3 billion of tokens. We demonstrate how hierarchical encoders achieve competitive results with consistently fewer parameters compared to state-of-the-art models and we show their importance for both pre-training and fine-tuning. This work has been presented in E. CHAPUIS &P . COLOMBO and collab. [2020].

a Benchmark can be found in the dataset library from HuggingFace [WOLF and collab., 2020] at https://huggingface.co/datasets/silicone

Introduction

As showcased in Chapter 1, it is beneficial for dialogue systems to analyse user's utterances in terms of DA and E/S.InChapter 4, we saw that DA classification is done through sequence labelling systems that are usually trained on large corpora (with over 100k labelled utterances) such as Switchboard [GODFREY and collab., 1992] or MRDA [SHRIBERG and collab., 2004] which may not be always available. Moreover, even though large corpora enable learning complex models from scratch (e.g., seq2seq Chapter 4), those models are very specific to the labelling scheme employed.

Adapting them to different sets of emotions or dialogue acts would require more annotated data. Meanwhile, generic representations [DEVLIN and collab., 2018;LIU and collab., 2019;MIKOLOV and collab., 2013;PENNINGTON and collab., 2014;PETERS and collab., 2018;YANG and collab., 2019] have been shown to be an effective way to adapt models across different sets of labels. Those representations are usually trained on large written corpora such as OSCAR [SUÁREZ and collab., 2019], Book Corpus [ZHU and collab., 2015] or Wikipedia [DENOYER and GALLINARI, 2006]. Although achieving state-of-the-art (SOTA) results on written benchmarks [WANG and collab., 2018], they are not tailored to spoken dialogue (SD). Indeed, [TRAN and collab., 2019] have suggested that training a parser on conversational speech data can improve results, due to the discrepancy between spoken and written language (e.g., disfluencies [START_REF] Stolcke | Statistical language modeling for speech disfluencies[END_REF], fillers [DINKAR and collab., 2020;[START_REF] Shriberg | Phonetic consequences of speech disfluency[END_REF], different data distribution). Furthermore, capturing discourse-level features, which distinguish dialogue from other types of text [START_REF] Thornbury | Conversation: From description to pedagogy[END_REF]], e.g., capturing multi-utterance dependencies, is key to embed dialogue that is not explicitly present in pre-training objectives [DEVLIN and collab., 2018;LIU and collab., 2019;YANG and collab., 2019], as they often treat sentences as a simple stream of tokens.

The goal of this work is to train on SD data a generic dialogue encoder capturing discourse-level features that produce representations adapted to spoken dialogue.

We evaluate these representations on both DA and E/S labelling through a new benchmark SILICONE (Sequence labellIng evaLuatIon benChmark fOr spoken laNguagE) composed of datasets of varying sizes using different sets of labels. We place ourselves in the general trend of using smaller models to obtain lightweight representations [JIAO and collab., 2019;LAN and collab., 2019] that can be trained without a costly computation infrastructure while achieving good performance on several downstream tasks [HENDERSON and collab., 2020] [START_REF] Lee | Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs[END_REF]LOWE and collab., 2015] used in previous works [HAZARIKA and collab., 2019;MEHRI and collab., 2019]. Lastly, we evaluate our encoder along with other baselines on SILICONE, which lets us draw finer conclusions of the generalisation capability of our models 1 .

Method

Pre-training Objectives

In the following, we rely on the notations defined in subsection 3.1.1. Our work builds upon existing objectives designed to pre-train encoders: the Masked Language Model (MLM) from [DEVLIN and collab., 2018;LAN and collab., 2019;LIU and collab., 2019;ZHANG and collab., 2019a] and the Generalized Autoregressive Pre-training (GAP) from [YANG and collab., 2019].

MLM Loss: The MLM loss corrupts sequences (or in our case, utterances) by masking a proportion p ! of tokens. The model learns bidirectional representations by predicting the original identities of the masked-out tokens. Formally, for an utterance u i , a random set of indexed positions m u i is selected and the associated tokens are replaced by a masked token [MASK] to obtain a corrupted utterance u masked i . The set of parameters µ is learnt by maximizing :

L u MLM (µ, u i ) = E " X t 2m u i log(p µ (! i t | ũi )) # (5.1)
where ũi is the corrupted utterance, m u i j ª unif{1, |u i |} 8 j 2 [1, p ! ] and p ! is the proportion of masked tokens.

GAP Loss: the GAP loss consists in computing a classic language modelling loss across different factorisation orders of the tokens. In this way, the model will learn to gather information across all possible positions from both directions. The set of parameters µ is learnt by maximising:

L u GAP (µ, u i ) = E ∑ E zªZ |u i | h X t log p µ (! i z t |u z<t i ) i ∏ (5.2)
where Z |u i | is the set of permutations of length |u i | and u z<t i represent the first t tokens of u i when permuting the sequence according to z 2 Z |u i | .

Hierarchical Encoding

Capturing dependencies at different granularity levels is key for dialogue embedding. Thus, we choose a hierarchical encoder [CHEN and collab., 2018b;LI and collab., 2018a]. It is composed of two functions f u and f c , satisfying:

E u i = f u µ (! 1 ,...,! |u i | ) (5.
3)

E C j = f d µ (E u 1 ,...,E C j ) (5.4)
where

E u i 2 R d u is the embedding of u i and E C j 2 R d d the embedding of C j .
The structure of the hierarchical encoder is depicted in Figure 5.1.

Hierarchical Pre-training General Motivation

Current self-supervised pre-training objectives such as MLM and GAP are trained at the sequence level, which for us translates to only learning f u µ . In this section, we extend both the MLM and GAP losses at the dialogue level in order to pre-train f d µ . Following previous work on both multi-task learning [ARGYRIOU and collab., 2007;[START_REF] Ruder | An overview of multi-task learning in deep neural networks[END_REF] and hierarchical supervision [GARCIA and collab., 2019;SANH and collab., 2019], we argue that optimising simultaneously at both levels rather than separately improves the quality of the resulting embeddings. Thus, we write our global hierarchical loss as:

L (µ) = ∏ u § L u (µ) + ∏ d § L d (µ) (5.5)
where L u (µ) is either the MLM or GAP loss at the utterance level and L d (µ) is its generalisation at the dialogue level.

MLM Loss

The MLM loss at the utterance level is defined in Equation 6.5. Our generalisation at the dialogue level masks a proportion p C of utterances and generates the sequences of masked tokens. Thus, at the dialogue level the MLM loss is defined as:

L d MLM (µ,C k ) = E 2 4 X j 2m C k |u j | X i =1 log(p µ (! j i | Ck )) 3 5 (5.6)
where m

C k j ª unif{1, |C k |} 8 j 2 [1, p C ] is

GAP Loss

The GAP loss at the utterance level is defined in Equation 5.2. A possible generalisation of the GAP at the dialogue level is to compute the loss of the generated utterance across all factorization orders of the context utterances. Formally, the GAP loss is defined at the dialogue level as:

L d GAP (µ,C k ) = E " E zªZ T h|C k | X t =1 |u z t | X i =1 log p µ (! z t i |C z<t k ) i # (5.7)
where ! z t i denotes the first i -th tokens of the permuted t -th utterance when permuting the context according to z 2 Z T and C z<t k the first t utterances of C k when permuting the context according to z.

Architecture

Commonly, the functions f u µ and f d µ are either modelled with recurrent cells (Chapter 4) and Chapter 4 or Transformer blocks [VASWANI and collab., 2017]. Transformer blocks are more parallelizable, offering shorter paths for the forward and backward signals and requiring significantly less time to train compared to recurrent layers. To the best of our knowledge this is the first attempt to pre-train a hierarchical encoder based only on transformers 2 . The structure of the model can be found in Figure 5.1. In order to optimize dialogue level losses as described in Equation 6.1, we generate (through g dec µ ) the sequence with a Transformer Decoder (T dec ). For downstream tasks, the context embedding E C k is fed to a simple MLP (simple classification), or to a CRF/GRU/LSTM (sequential prediction). In the remainder, we will name our hierarchical transformer-based encoder HT and the hierarchical RNN-based encoder HR.W euseµ x y to refer to the set of model parameters learnt using the pre-training objective y (either MLM or GAP) at the level x 3 .

Pre-training Datasets

Datasets used to pre-train dialogue encoders [HAZARIKA and collab., 2019;MEHRI and collab., 2019] [START_REF] Lee | Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs[END_REF], it is a multi-party dataset; and (3) OpenSubtitles is an order of magnitude larger than any other spoken language dataset used in previous work. We segment OpenSubtitles by considering the duration of the silence between two consecutive utterances. Two consecutive utterances belong to the same conversation if the silence is shorter than ± T 5 . Conversations shorter than the context size T are dropped 6 . After preprocessing, Opensubtitles contains subtitles from 446520 movies or series which represent 54642424 conversations and over 2.3 billion of words.

Baseline Encoder

We compare the different methods we presented with two different types of baseline encoders: pre-trained encoders, and hierarchical encoders based on recurrent cells. The latter, achieve current SOTA performance in many sequence labelling tasks [COLOMBO and collab., 2020;LI and collab., 2018a;LIN and collab., 2017].

Pre-trained Encoder Models. We use BERT [DEVLIN and collab., 2018] through the pytorch implementation provided by the Hugging Face transformers library [WOLF and collab., 2019]. The pre-trained model is fed with a concatenation of the utterances. Formally given an input context C k = (u 1 ,...u T ) the concatenation [u 1 ,...,u T ] is fed to BERT.

2 Although it is possible to relax the fixed size imposed by transformers [DAI and collab., 2019] in this work we follow COLOMBO and collab. [2020] and fix the context size to 5 and the max utterance length to 50 -these choices are made to work with OpenSubtitles, since the number of available dialogues drops when considering a number of utterances greater than 5. A representation for all the baselines can be found in Figure 5.3. For all models, both hidden dimension and embedding dimension is set to 768 to ensure fair comparison with the proposed model. The MLP used for decoding contains 3 layers of sizes (768,348,192). We use RELU [START_REF] Agarap | Deep learning using rectified linear units (relu)[END_REF] to introduce non linearity inside our architecture.

Evaluation of Sequence Labelling

Related Work

As stated previously, sequence labelling tasks for spoken dialog mainly involve two different types of labels: DA and E/S. Early work has tackled the sequence labelling problem as an independent classification of each utterance. Deep neural network models that currently achieve the best results [KEIZER and collab., 2002;STOLCKE and collab., 2000;SURENDRAN and LEVOW, 2006] model both contextual dependencies between utterances [COLOMBO and collab., 2020; LI and collab., 2018b] and labels [CHEN and collab., 2018b;KUMAR and collab., 2018;LI and collab., 2018c].

The aforementioned methods require large corpora to train models from scratch, such as: Switchboard Dialog Act (SwDA) [GODFREY and collab., 1992], Meeting Recorder Dialog Act (MRDA) [SHRIBERG and collab., 2004], Daily Dialog Act [LI and collab., 2017], HCRC Map Task Corpus (MT) [THOMPSON and collab., 1993]. This makes harder their adoption to smaller datasets, such as: Loqui human-human dialogue corpus (Loqui) [START_REF] Passonneau | Loqui human-human dialogue corpus (transcriptions and annotations)[END_REF], BT Oasis Corpus (Oasis) [START_REF] Leech | Generic speech act annotation for task-oriented dialogues[END_REF], Multimodal Multi-Party Dataset (MELD) [PORIA and collab., 2018a], Interactive emotional dyadic motion capture database (IEMO), SEMAINE database (SEM) [MCKEOWN and collab., 2013].

Presentation of SILICONE

Despite the similarity between methods usually employed to tackle DA and E/S sequential classification, studies usually rely on a single type of label. Moreover, despite the variety of small or medium-sized labelled datasets, evaluation is usually done on the largest available corpora (e.g., SwDA, MRDA). We introduce SILICONE, a collection of sequence labelling tasks, gathering both DA and E/S annotated datasets. SILICONE is built upon preexisting datasets which have been considered by the community as challenging and interesting. Any model that is able to process multiple sequences as inputs and predict the corresponding labels can be evaluated on SILICONE. We especially include small-sized datasets, as we believe it will ensure that well-performing models are able to both distil substantial knowledge and adapt to different sets of labels without relying on a large number of examples. The description of the datasets composing the benchmark can be found in the following sections, while corpora statistics are gathered in Table 5.1.

DA Datasets

In addition to SwDA and MRDA presented in subsection 4.5.1 we have collected 3 datasets annotated in DA:

DailyDialog Act Corpus (DyDA a ) has been produced by [LI and collab., 2017]. It contains multi-turn dialogues, supposed to reflect daily communication by covering topics about daily life. The dataset is manually labelled with dialog act and emotions.

It is the third biggest corpus of SILICONE with 102k utterances. The SOTA model reports an accuracy of 88.1% [LI and collab., 2018a], using Bi-LSTMs with attention as well as additional features. We follow the official split introduced by the authors.

HCRC MapTask Corpus (MT) has been introduced by THOMPSON and collab. [1993].

To build this corpus, participants were asked to collaborate verbally by describing a route from a first participant's map by using the map of another participant. This corpus is small (36k utterances). As there is no standard train/dev/test split7 performances depends on the split. [TRAN and collab., 2017] make use of a Hierarchical LSTM encoder with a GRU decoder layer and achieves an accuracy of 65.9%.

Bt Oasis Corpus (Oasis) contains the transcripts of live calls made to the BT and operator services. This corpus has been introduced by LEECH and WEISSER [2003] and is rather small (15k utterances). There is no standard train/dev/test split8 and few studies use this dataset.

S/E Datasets

In S/E recognition for spoken language, there is no consensus on the choice the evaluation metric (e.g., [GHOSAL and collab., 2019;PORIA and collab., 2018b] use a weighted F-score while [ZHANG and collab., 2019b] report accuracy). For SILICONE, we choose to stay consistent with the DA research and thus follow [ZHANG and collab., 2019b] by reporting the accuracy. Additionally, emotion/sentiment labels are neither merged nor prepossessed9 . DailyDialog Emotion Corpus (DyDA e ) has been previously introduced and contains eleven emotional labels. The SOTA model [DE BRUYNE and collab., 2019] is based on BERT with additional Valence Arousal and Dominance features and reaches an accuracy of 85% on the official split. Multimodal EmotionLines Dataset (MELD) has been created by enhancing and extending EmotionLines dataset [CHEN and collab., 2018a] where multiple speakers participated in the dialogues. There are two types of annotations MELD s and MELD e : three sentiments (positive, negative and neutral) and seven emotions (anger, disgust, fear, joy,neutral, sadness and surprise). The SOTA model with text only is proposed by [ZHANG and collab., 2019b] and is inspired by quantum physics. On the official split, it is compared with a hierarchical bi-LSTM, which it beats with an accuracy of 61.9% (MELD s ) and 67.9% (MELD e ) against 60.8% and 65.2. IEMOCAP database (IEMO) is a multimodal database of ten speakers. It consists of dyadic sessions where actors perform improvisations or scripted scenarios. Emotion categories are: anger, happiness, sadness, neutral, excitement, frustration, fear, surprise, and other. There is no official split on this dataset. One proposed model is built with bi-LSTMs and achieves 35.1%, with text only [ZHANG and collab., 2019b]. SEMAINE database (SEM) comes from the Sustained Emotionally coloured Machine human Interaction using Nonverbal Expression project [MCKEOWN and collab., 2013]. This dataset has been annotated on three sentiments labels: positive, negative and neutral by [BARRIERE and collab., 2018]. It is built on Multimodal Wizard of Oz experiment where participants held conversations with an operator who adopted various roles designed to evoke emotional reactions. There is no official split on this dataset.

Diversity of SILICONE

We illustrate the diversity of the dataset composing SILICONE.I nFigure 5.4,w e plot two histograms representing the different utterance lengths for DA and E/S. As expected, for spoken dialog, lengths are shorter than for written benchmarks (e.g., GLUE). in Table 5.3, we found that in the case of E/S prediction there is no clear difference between CRFs and MLPs, while GRU decoders exhibit poor performance, probably due to a lack of training data. It is also important to notice, that training a sequential decoder usually requires thorough hyper-parameter fine-tuning. As our goal is to learn and evaluate general representations that are decoder agnostic, in the following, we will use a plain MLP decoder for all the models compared.

CHAPTER 5. A PRE-TRAINED MODEL FOR LOW

General Performance Analysis

Importance of Pre-training for SILICONE

Results reported in Table 5.4 and Table 5.3 show that pre-trained transformer-based encoders achieve consistently higher accuracy on SILICONE, even when they are not explicitly considering the hierarchical structure. This difference can be observed both in small-sized datasets (e.g. MELD and SEM) and in medium/large size datasets (e.g SwDA and MRDA).

Negative Results on GAP

We briefly describe few ideas we tried to make GAP works at both the utterance and dialogue level. We hypothesise that:

• giving the same weight to the utterance level and the dialogue level (see Equation 6.2) was responsible of the observed plateau. Different combinations lead to fairly poor improvements.

• the limited model capacity was part of the issue. Larger models does not give the expected results.

Model Analysis

In this section, we dissect our hierarchical pre-trained models in order to better understand the relative importance of each component. We show how a hierarchical encoder allows us to obtain a light and efficient model.

Pre-training on Spoken vs Written Data

First, we explore the differences in training representations on spoken and written corpora. Experimentally, we compare the predictions on SILICONE made by HT(µ u MLM ) and the one made by HT(µ BERT°2layers ). The latter is a hierarchical encoder where utterance embeddings are obtained with the hidden vector representing the first token [CLS] (see [DEVLIN and collab., 2018]) of the second layer of BERT. In both cases, predictions are performed using an MLP10 . Results in Table 5.5 show higher accuracy when the pre-training is performed on spoken data. Since SILICONE is a spoken language benchmark, this result might be due to the specific features of colloquial speech (e.g. disfluencies, sentence length, vocabulary, word frequencies).

Hierarchy and Multi-Level Supervision

We study the relative importance of three aspects of our hierarchical pre-training with multi-level supervision. We first show that accounting for the hierarchy increases the performance of fine-tuned encoders, even without our specific pre-training procedure. We then compare our two proposed hierarchical pre-training procedures based on the GAP or MLM loss. Lastly, we look at the contribution of the possible levels of supervision on reduced training data from SEM.

Importance of hierarchical fine-tuning

We compare the performance of BERT-4layers with the HT(µ BERT°2layer ) previously described. Results reported in Table 5.5 demonstrate that fine-tuning on downstream tasks with a hierarchical encoder yields to higher accuracy, with fewer parameters, even when using already pre-trained representations.

MLM vs GAP

In this experiment, we compare the different pre-training objectives at utterance and dialogue level. As a reminder HT(µ u MLM ) and HT(µ u GAP ) are respectively trained using the standard MLM loss [DEVLIN and collab., 2018] and the standard GAP loss [YANG and collab., 2019]. In Table 5.6 we report the different pre-training objective results. We observe that pre-training at the dialogue level achieves comparable results to the utterance level pre-training for MLM and slightly worse for GAP. Interestingly, we observe that HT(µ u GAP ) compared to HT(µ u MLM ) achieves worse results, which is not consistent with the performance observed on other benchmarks, such as GLUE [WANG and collab., 2018]. The lower accuracy of the models trained using a GAP-based loss could be due to several factors (e.g., model size, pre-training using 2.9 2.8 28.7 (SMALL) 10.6 10. 6 45 Table 5.7 -Number of parameters for the encoders. Sizes are given in million of parameters.

Multi level Supervision for pre-training MELD

In this experiment we report results of the experiment mentioned in section 5.5.2.

In this experiment we see that the training process seems to be noisier for fractions lower than 40%. For larger percentages, we observe that including higher supervision (at the dialogue level) during pre-training leads to a consistent improvement.

Other advantages of hierarchy

Introducing a hierarchical design in the encoder allows to break dialogue into utterances and to consider inputs of size T instead of size 512. First, it allows parameters sharing, reducing the number of model parameters. The different model sizes are reported in Table 5.7.OurTINY model contains half the parameters of BERT (4-layers). Furthermore, modelling long-range dependencies hierarchically makes learning faster and allows to get rid of learning tricks (e.g., partial order prediction [YANG and collab., 2019], two-stage pre-training based on sequence length [DEVLIN and collab., 2018]) required for non-hierarchical encoders. Lastly, original BERT and XLNET are pre-trained using respectively 16 and 512 TPUs. Pre-training lasts several days with over 500K iterations. Our TINY hierarchical models are pre-trained during 180K iterations (1.5 days) on 4 NVIDIA V100.

Chapter 5 conclusion

In this chapter, we propose a hierarchical transformer-based encoder tailored for spoken dialog. We extend two well-known pre-training objectives to adapt them to a hierarchical setting and use OpenSubtitles, the largest spoken language dataset available, for encoder pre-training. Additionally, we provide an evaluation benchmark dedicated to comparing sequence labelling systems for the NLP community, SILICONE, on which we compare our models and pre-training procedures with previous approaches. By conducting ablation studies, we demonstrate the importance of using a hierarchical structure for the encoder, both for pre-training and fine-tuning. Finally, we find that our approach is a powerful method to learn generic representations on spoken dialog, with less parameters than state-of-the-art transformer models. We hope that the SILICONE benchmark, will encourage further research to build stronger sequence labelling systems for NLP. This work could be then extended to a multilingual setting as we will show in Chapter 6.

Part II Conclusions

In this part, we addressed the RQ1 and presented models that can automatically label English spoken dialogue utterances. First in Chapter 4, when large annotated corpora are available, we trained directly a seq2seq architecture composed of a hierarchical encoder and relied on specifically designed guided attention mechanisms. We showcased that our method allows to reach competitive results on DA classification. However, such an approach is difficult to apply when there is a lack of annotated data as it is the case of E/S corpora for example. In Chapter 5, we address the data scarcity problematic by relying on pre-trained architectures. We devise new pre-training approaches to learn generic representations tailored for spoken dialogue. We evaluate our representations on a new benchmark SILICONE that gathers DA and E/S corpora of different sizes.

In the next part, we address RQ2 and extend the work developed in Part II with two different settings : multilingual in Chapter 6 and multimodal in Chapter 7.

Part III Towards Multilingual and Multi-modal Spoken Dialogue Understanding Part III Abstract

This part gathers our contributions related to RQ2 "How to modify spoken dialogue understanding systems to handle multilingual and multimodal data? What are the specifics of multilingual and multimodal data that the system needs to handle?". This part is further divided into two different chapters:

• Chapter 6 is dedicated to the multilingual scenario where several languages are present within the conversation. We propose to adapt the neural architecture presented in Chapter 5 to handle multilingual data.

We introduce new pre-training losses tailored for multilingual spoken dialogue and for the code-switching phenomena occurring within a conversation. We leverage the Opensubtitles corpus to automatically build multilingual conversations corpus. In addition, we evaluate our representations on a new benchmark called MIAM and composed of datasets in five different languages (French, Italian, English, German and Spanish) annotated in DA and two novel multilingual downstream tasks (i.e multilingual mask utterance retrieval and multilingual inconsistency identification).

• Chapter 7 is dedicated to the multimodal scenario. In this chapter, we focus on the multimodal aspect of conversations in the scope of emotion recognition. Textual modality is particularly well exploited by current neural architectures. Therefore, we seek to study ways to translate non-textual modality into textual modality. We primarily focus on punctuation marks, as they are easily accessible. We choose to study the potential of punctuation marks of conveying prosodical cues. In this manner, punctuation marks can be viewed as the most basic fusion scheme for multimodal representations. We provide a quantitative analysis of the impact of punctuation marks on emotion prediction with current SOTA architectures.

Chapter 6

Cross-Lingual Pre-training Methods for Spoken Dialog

Chapter 6 Abstract

Spoken dialogue systems need to be able to handle both multiple languages and multilinguality inside a conversation (e.g in case of code-switching). In this work, we introduce new pre-training losses tailored to learn multilingual spoken dialogue representations. The goal of these losses is to expose the model to code-switched language. To scale up training, we automatically build a pre-training corpus composed of multilingual conversations in five different languages (French, Italian, English, German and Spanish) from Open-Subtitles, a huge multilingual corpus composed of 24.3G tokens. We test the generic representations on MIAM, a new benchmark composed of five dialogue act corpora on the same aforementioned languages as well as on two novel multilingual downstream tasks (i.e multilingual mask utterance retrieval and multilingual inconsistency identification). Our experiments show that our new code switched-inspired losses achieve a better performance in both monolingual and multilingual settings.

Introduction

An additional difficulty to the previously mentioned modeling problem (Chapter 4 and Chapter 5) is that most people in the world are bilingual [GROSJEAN and LI, 2013]: therefore, progress on these systems is limited by their inability to process more than one language (English being the most frequent). For example, many people use English as a "workplace" language but seamlessly switch to their native language when the conditions are favorable [START_REF] Heredia | Bilingual language mixing: Why do bilinguals code-switch?[END_REF]. Thus, there is a growing need for understanding dialogues in a multilingual fashion [IPSIC and collab., 1999;JOSHI and collab., 2020;RUDER and collab., 2019]. Additionally, when speakers share more than one language, they inevitably will engage in code-switching [AUER, 2013;[START_REF] Gumperz | Discourse strategies[END_REF][START_REF] Milroy | One speaker, two languages: Cross-disciplinary perspectives on code-switching[END_REF]PAREKH and collab., 2020;[START_REF] Sankoff | A formal grammar for code-switching[END_REF]: switching between two different languages. Thus, spoken dialogue systems need to be cross lingual (i.e able to handle different languages) but also need to model multilinguality inside a conversation [AHN and collab., 2020].

In this chapter, we focus on building generic representations for dialogue systems that satisfy the aforementioned requirements. Generic representations have led to strong improvements on numerous natural language understanding tasks, and can be fine-tuned when only small labelled datasets are available for the desired downstream task [DEVLIN and collab., 2018;LAN and collab., 2019;LIU and collab., 2019;MIKOLOV and collab., 2013;YANG and collab., 2019]. While there has been a growing interest in pre-training for dialogue [MEHRI and collab., 2019;ZHANG and collab., 2019d], the focus has mainly been on English datasets. Thus, these works can not be directly applied to our multilingual setting. Additionally, available multilingual pre-training objectives [START_REF] Lample | Cross-lingual language model pretraining[END_REF]LIU and collab., 2020;[START_REF] Qi | Prophetnet-x: Large-scale pre-training models for english, chinese, multi-lingual, dialog, and code generation[END_REF]XUE and collab., 2020] face two main limitations when applied to dialogue modeling: (1) they are a generalization of monolingual objectives that use flat input text, whereas hierarchy has been shown to be a powerful prior for dialogue modeling. This is a reflection of a dialogue itself, for example, context plays an essential role in the labeling of dialogue acts. (2) The pre-training objectives are applied separately to each language considered, which does not expose the (possible) multilinguality inside a conversation (as it is the case for code-switching) [WINATA and collab., 2021]1 .

Our main contributions are as follows: 1. We introduce a set of code-switched inspired losses as well as a new method to automatically obtain several million of conversations with multilingual input context in different languages. There has been limited work on proposing corpora with a sufficient amount of conversations that have multilingual input context. Most of this work focuses on social media, or on corpora of limited size. Hence, to test our new losses and scale up our pre-training, we automatically build a pre-training corpus of multilingual conversations, each of which comprises several languages, by leveraging the alignments available in OpenSubtitles (OPS). 2. We showcase the relevance of the aforementioned losses and demonstrate that it leads to better performances on downstream tasks, that involve both monolingual conversations and multilingual input conversations. For monolingual evaluation, we introduce the Multilingual dIalogAct benchMark (MIAM): composed of five datasets in five different languages annotated with dialogue acts. Following [LOWE and collab., 2016;MEHRI and collab., 2019], we complete this task with both contextual inconsistency detection and next utterance retrieval in these five languages. For multilingual evaluation, due to the lack of code-switching corpora for spoken dialogue, we create two new tasks: contextual inconsistency detection and next utterance retrieval with multilingual input context. The datasets used for these tasks are unseen during training and automatically built from OPS.

In this work, we follow the recent trend [JIAO and collab., 2019;LAN and collab., 2019] in the NLP community that aims at using models of limited size that can both be pre-trained with limited computational power and achieve good performance on multiple downstream tasks. The languages we choose to work on are English, Spanish, German, French and Italian. 2 . Our implementation will be available on github.com and data will be available on Datasets [WOLF and collab., 2020].

Model and Training Objectives

Notations We start by introducing the notations. We have a set D of contexts (i.e truncated conversations), i.e., D =

(C 1 ,C 2 ,...,C |D| ). Each context C i is composed of utterances u, i.e C i = (u L 1 1 , u L 2 2 ,...,u L |C i | |C i |
) where L i is the language of utterance u i3 . At the lowest level, each utterance u i can be seen as a sequence of tokens, i.e u

L i i = (! i 1 , ! i 2 ,...,! i |u i | ).
For DA classification y i is the unique dialogue act tag associated to u i . In our setting, we work with a shared vocabulary V thus ! i j 2 V and V is language independent.

Related work

Multilingual pre-training. Over the last few years, there has been a move towards pre-training objectives, allowing models to produce general multilingual representations that are useful for many tasks. However, they focus on the word level [START_REF] Faruqui | Improving vector space word representations using multilingual correlation[END_REF]GOUWS and collab., 2015;MIKOLOV and collab., 2013] or the utterance level [DEVLIN and collab., 2018;ERIGUCHI and collab., 2018;[START_REF] Lample | Cross-lingual language model pretraining[END_REF]. [WINATA and collab., 2021] shows that these models obtain poor performances in presence of code-switched data. Pre-training to learn dialogue representation. As previously mentioned in Chapter 5, current research efforts made towards learning dialogue representation are mainly limited to the English language [CHAPUIS and collab., 2020;HENDERSON and collab., 2019;MEHRI and collab., 2019] and introduce objectives at the dialogue level such as next-utterance retrieval, next-utterance generation, masked-utterance retrieval, inconsistency identification or generalisation of the cloze task [TAYLOR, 1953]. To the best of our knowledge, this is the first work to pre-train representations for spoken dialogue in a multilingual setting. Hierarchical pre-training As we are interested in capturing information at different granularities, we follow the hierarchical approach of Chapter 5 and decompose the pre-training objective in two terms, namely:

L (µ) = ∏ u £ L u (µ) | {z } utterance level + ∏ d £ L d (µ) | {z } dialogue level . ( 6.1)
As in Chapter 5 these losses rely on a hierarchical encoder composed of two functions f u and f d :

E u L i i = f u µ (! i 1 ,...,! i |u i | ), (6.2)
E C j = f d µ (E u L 1 1 ,...,E u L |C j | |C j | ), (6.3)
where

E u L i i 2 R d u is the embedding of u L i i and E C j 2 R d d the embedding of C j .
The encoder is built on transformer layers.

Utterance level pre-training

To train the first level of hierarchy (i.e f u µ ), we use a Masked Utterance Modelling (MUM) loss [DEVLIN and collab., 2018] rupted version, obtained after masking a proportion p ! of tokens, the set of masked indices is denoted M ! . The set of masked tokens is denoted ≠. The probability of the masked token given ũL i i is given by:

p(≠| ũL i i ) = Y t 2M ! p µ (! i t | ũL i i ). (6.4)

Dialogue level pre-training

The goal of the dialogue level pre-training is to ensure that the model learns dialogue level dependencies (through f d µ ), i.e the ability to handle multi-lingual input context. Generic framework Given C k an input context, a proportion p C of utterances is masked to obtained the corrupted version Ck . The set of masked utterances is denoted U and the set of corresponding masked indices M u . The probability of U given Ck is:

p(U | Ck ) = Y t 2M U |u t |°1 Y j =0 p µ (! t j |! t 1: j °1, Ck ). (6.5)
As shown in Equation 6.5, a masked sequence is predicted one word per step. As an example, at the j-th step, the prediction of ! t j is made given (! t 1: j °1, Ck ) where

! t 1: j °1 = (! t 1 , ••• , ! t 1: j °1).
In the following, we describe different procedures to build M U and Ck used in Equation 6.5.

Masked utterance generation (MUG)

The MUG loss aims at predicting the masked utterance from a monolingual input context. As the vocabulary is shared, this loss will improve the alignment of conversations at the dialogue level. This loss ensures that the model will be able to handle monolingual conversations in different languages. Training Loss We rely on Equation 6.5 for MUG. The input context is composed of utterances in the same language, i.e 8k,C k = (u

L k 1 , ••• , u L k |C k |
). The mask is randomly chosen among all the positions. Example Given the monolingual input context given in Table 6.1, a random mask (e.g [0, 3]) is chosen among the positions [0,1,2,3,[START_REF] Aguilar | Lince: A centralized benchmark for linguistic code-switching evaluation[END_REF]. The masked utterances are replaced by [MASK] tokens to obtain Ck and a decoder attempts to generate them.

Translation masked utterance generation (TMUG)

The previous objectives are self-supervised and cannot be employed with parallel data when available. In addition, these losses do not expose the model to multilinguality inside the conversation. The TMUG loss addresses this limitation using a translation mechanism: the model learns to translate the masked utterance in a new language.

Training Loss We use Equation 6.5 for TMUG with a bilingual input context C k . C k contains two different languages (i.e L and L 0 ) 8k,C k = (u

L 1 1 , ••• , u L k |C k | ) with L i 2 {L, L 0 }.
The masked positions M u are all the utterances in language L 0 . Thus Ck is a monolingual context. Example Given the multilingual input context given in Table 6.1, the positions [3,[START_REF] Aguilar | Lince: A centralized benchmark for linguistic code-switching evaluation[END_REF] are masked with sequences of [MASK] and the decoder will generate them in French.

Multilingual masked utterance generation (MMUG)

In the previous objectives, the model is exposed to monolingual input only. MMUG aims at relaxing this constraint by considering multilingual input context and generating the set of masked utterances in any possible target language.

Training Loss Given a multi-lingual input context C k = (u

L 1 1 , ••• , u L |C k | |C k | ).
A random set of indexes is chosen and the associated utterances are masked. The goal remains to generate the masked utterances. Example In Table 6.1, the positions [2, 3] are randomly selected from the available positions [0,1,2,3,[START_REF] Aguilar | Lince: A centralized benchmark for linguistic code-switching evaluation[END_REF]. Given these masked utterances the model will generate 2 in Italien and 3 in Spanish. MMUG is closely related to code-switching as it exposes the model to multilingual context and the generation can be carried out in any language.

In Figure 6.1 we provides graphical examples for each monolingual and multilingual losses used. Choice of scaling factor in Equation 6.1. In the case of multi-task setting, different losses may have different scales, making the optimization perform poorly. In that case, scaling factors or more advanced techniques [SENER and KOLTUN, 2018] can be applied. As we did not observe such phenomena, all scaling factors are set to 1.

Pre-training with generation

For both TMUG and MMUG, the model needs to be aware of the target language. Thus, the first token fed to the decoder indicates the target language (e.g in English the corresponding id is 99, in Spanish 98). To avoid creating a discrepancy between pre-training objectives we also add this token for MUG.

Choice of the multilingual encoder

The two dominant approaches for multilingual systems involve either using a languagespecific encoder [ESCOLANO and collab., 2020] or one shared encoder across languages [START_REF] Artetxe | Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond[END_REF]FENG and collab., 2020]. To reduce the number of learnt parameters, we rely on the second approach.

Pre-training details

Our model is pre-trained on 4 NVIDIA V100 for 2 days (500k iterations) with a batch size of 256. We use AdamW [KINGMA and BA, 2014;LOSHCHILOV and HUTTER, 2017] with 4000 warmups steps [VASWANI and collab., 2017]. During this stage, we do not perform any grid search.

Pre-training corpora

There is no large corpora freely available that contains a large number of transcripts of well segmented multilingual spoken conversation with code switching phenomenon.

Evaluation Framework

This section presents our evaluation protocol. It involves two different types of evaluation depending on the input context. The first group of experiences consists in multilingual evaluations with monolingual input context and follows classical downstream tasks [DZIRI and collab., 2019;[START_REF] Finch | Towards unified dialogue system evaluation: A comprehensive analysis of current evaluation protocols[END_REF] including sequence labeling [COLOMBO and collab., 2020], utterance retrieval [MEHRI and collab., 2019] or inconsistency detection. The second group focuses on multilingual evaluations with multilingual context.

Dialogue representations evaluation

Monolingual context

Sequence labeling tasks. The ability to efficiently detect and model discourse structure is an important step toward modeling spontaneous conversations. A useful first level of analysis involves the identification of dialogue act (DA) [STOLCKE and collab., 2000a] thus DA tagging is commonly used to evaluate dialogue representations. However, due to the difficulty to gather language-specific labelled datasets, multilingual sequence labeling such as DA labeling remains overlooked.

Next-utterance retrieval (NUR)

The utterance retrieval task [DUPLESSIS and collab., 2017;[START_REF] Saraclar | Lattice-based search for spoken utterance retrieval[END_REF] focuses on evaluating the ability of an encoder to model contextual dependencies. [LOWE and collab., 2016] suggests that NUR is a good indicator of how well context is modeled. Inconsistency Identification (II) Inconsistency identification is the task of finding inconsistent utterances within a dialogue context [SANKAR and collab., 2019b]. The perturbation is as follow: one utterance is randomly replaced, the model is trained to find the inconsistent utterance.5

Multilingual context

To the best of our knowledge, we are the first to probe representation for multilingual spoken dialogue with multilingual input context. As there is no labeled code-switching datasets for spoken dialogue (research focuses on on synthetic data [START_REF] Stymne | Evaluating word embeddings for indonesian-english code-mixed text based on synthetic data[END_REF], social media [PRATAPA and collab., 2018] or written text [KHANUJA and collab., 2020;[START_REF] Tan | Code-mixing on sesame street: Dawn of the adversarial polyglots[END_REF] rather than spoken dialogue). Thus we introduce two new downstream tasks with automatically built datasets: Multilingual Next Utterance Retrieval (mNUR) and Multilingual Inconsistency Identification . To best assess the quality of representations, for both mII and mNUR we choose to work with train/test/validation datasets of 5k conversations. The datasets, unseen during training, are built using the procedure described in subsection 6.2.4.

Multilingual next utterance retrieval. mNUR consists of finding the most probable next utterance based on an input conversation. The evaluation dataset is built as follow: for each conversation in language L composed of T utterances, a proportion p L 0 of utterances is replaced by utterances in language L 0 . D utterances that we call distractors 6 in language L or L 0 from the same movie. For testing, we frame the task as a ranking problem and report the recall at N (R@N) [SCHATZMANN and collab., 2005].

Multilingual inconsistency identification. The task of mII consists of identifying the index of the inconsistent sentences introduced in the conversation. Similarly to the previous task: for each conversation in language L composed of T utterances, a proportion p L 0 is replaced by utterances in language L 0 , a random index is sampled from [1,T] and the corresponding utterance is replaced by a negative utterance taken from the same movie.

Altering tasks difficulty

One of the interesting properties of II, mII, NUR, mNUR is the ability to alter the task difficulty in a controlled manner when sampling the negative utterances. For example, instead of randomly sampling the false utterances, the most similar to the true one as measured by a similarity metric [CELIKYILMAZ and collab., 2020;ZHANG and collab., 2019a] could be chosen. This flexibility could allow increasing the difficulty of the task as models get better.

Multilingual dialogue act benchmark

A plethora of freely available dialogue act dataset [GODFREY and collab., 1992;LI and collab., 2017;SHRIBERG and collab., 2004]) has been proposed to evaluate DA labeling systems in English. However, constituting a multilingual dialogue act benchmark is challenging RIBEIRO and collab. [2019b]. We introduce Multilingual dIalogue Act benchMark (in short MIAM). This benchmark gathers five free corpora that have been validated by the community, in five different European languages (i.e. English, German, Italian, French and Spanish), examples are provided in Table 6.4. We believe that this new benchmark is challenging as it requires the model to perform well along different evaluation axis and validates the cross-lingual generalization capacity of the representations across different annotation schemes and different sizes of corpora. In Figure 6.2 we illustrate the diversity of the gathered corpora through the lens of utterance length.

DA for English For English, we choose to work on the MapTask corpus. It consists of conversations where the goal of the first speaker is to reproduce a route drawn only on the second speaker's map, with only vocal indications. We choose this corpus for its small size that will favor transfer learning approaches (36k utterances).

DA for Spanish Spanish research on DA recognition mainly focuses on three different datasets Dihana, CallHome Spanish [POST and collab., 2013] and DIME [START_REF] Coria | Predicting obligation dialogue acts from prosodic and speaker infomation[END_REF][START_REF] Olguin | Predicting dialogue acts from prosodic information[END_REF]. Dihana is the only available corpora that contains free DA annotation [RIBEIRO and collab., 2019a]. It is a spontaneous speech corpora [BENEDI and collab., 2006] composed of 900 dialogues from 225 users. Its acquisition was carried out using a Wizard of Oz setting [START_REF] Fraser | Simulating speech systems[END_REF]. For this dataset, we focus on the first level of labels which is dedicated to the task-independent DA. DA for German For German, we rely on the VERBMOBIL (VM2) dataset [KAY and col- lab., 1992]. This dataset was collected in two phases: first, multiple dialogues were recorded in an appointment scheduling scenario, then each utterance was annotated with DA using 31 domain-dependent labels. The three most common labels (i.e. inform, suggest and feedback) are highly related to the planning nature of the data.

DA for French Freely available to academic and nonprofit research datasets are lim- ited in the French language as most available datasets are privately owned. We rely on the French dataset from the Loria Team [BARAHONA and collab., 2012](LORIA) where the collected data consists of approximately 1250 dialogues and 10454 utterances. The tagset is composed of 31 tags.

DA for Italian For Italian, we rely on the Ilisten corpora [START_REF] Basile | Overview of the evalita 2018 italian speech act labe ling (iliste n) task[END_REF]. The corpus was collected in a Wizard of Oz setting and contains a total of 60 dialogues transcripts, 1, 576 user dialogue turns and 1, 611 system turns. The tag set is composed of 15 tags.

Metrics:

There is no consensus on the evaluation metric for DA labelling (e.g., [GHOSAL and collab., 2019;PORIA and collab., 2018] use a weighted F-score while [ZHANG and collab., 2019c] report accuracy). As in Chapter 5 we report accuracy.

Baseline encoders for downstream tasks

The encoders that will serve as baselines can be divided into two different categories: hierarchical encoders based on GRU layers (HR) and pre-trained encoders based on Transformer cells [VASWANI and collab., 2017]. The first group achieve SOTA results on several sequence labelling tasks [LI and collab., 2018;LIN and collab., 2017]. The second group can be further divided in two groups: language specific (BERT) and multilingual BERT (mBERT) and pre-trained hierarchical transformers from ZHANG and collab. [2019b](HT) are used as a common architecture to test the various pre-training losses. Tokenizer We will work with both language specific and multlingual tokenizer. Model with multilingual tokenizer will be referred with a m (e.g mBERT as opposed to BERT).

In the following we provide details of language specific BERT and on baseline models.

Pre-trained encoder baselines

The first group of pre-trained encoders are based on BERT. A concatenation of utterances is fed to the model to obtain a conversation embedding. For our languagespecific models, we use the German BERT7 , the original BERT for English, BETO [CAÑETE and collab., 2020] for Spanish, Flaubert [LE and collab., 2019] for French and Italian BERT [START_REF] Schweter | Italian bert and electra models[END_REF] for Italian. We rely on the multilingual BERT (mBERT) [DEVLIN and collab., 2018] 8 provided by the transformers library [WOLF and collab., 2019] implemented using the pytorch [PASZKE and collab., 2017] framework. For pre-trained hierarchical transformers, we rely on the work presented in Chapter 5 and for each considered language, we pre-train a language-specific encoder.

Decoders

Given the different nature of the proposed downstream tasks, we use various type of decoders. DA classification: Methods to tackle sequence labelling on monolingual representations can be divided into two different classes. The first one perform classification on each utterance independently using Bayesian Networks [KEIZER and collab., 2002], SVMs [SURENDRAN and LEVOW, 2006] or HMMs [STOLCKE and collab., 2000b]. The second class, which achieves stronger results, leverages the adjacency utterances by using deep representations [BOTHE and collab., 2018;KHANPOUR and collab., 2016]. Sequence labelling can be improved when sufficiently many training points are available by modelling inter-tag dependencies using RNN-based decoders [CHUNG and collab., 2014;HOCHREITER and SCHMIDHUBER, 1997], and CRFs [START_REF] Chen | Emotionlines: An emotion corpus of multi-party conversations[END_REF]LAFFERTY and collab., 2001]. Thus, in this work, we choose to experiment with a MLP, a CRF and a RNN decoder based on GRU.

II and mII : For this task, the context embedding E C k is fed to a MLP. Both the encoder and the MLP are trained to predict the inconsistent utterance index by minimising a cross-entropy loss. Formally, this task is formulated as a classification problem with T classes.

NUR and mNUR: For this task, we first compute the context embedding E C k then the candidate utterance u L c c is embedded using the either f u µ or a chosen encoder to obtain E u L c c . Both representations are concatenated and given to a MLP. The architecture is trained to predict if the provided candidate utterance is a suitable next utterance by minimizing a binary cross-entropy. This experiment is similar to the one in [LOWE and collab., 2015].

Additional details on models

In this section, we describe models used as well as details on the pre-training parameters. In Table 6.5 we report the main hyper-parameters used for our model pre-training. We used GELU [HENDRYCKS and GIMPEL, 2016] activations and the dropout rate [SRIVASTAVA and collab., 2014] is set to 0.1. Although vanilla Transformers impose a fixed context size it can be relaxed [DAI and collab., 2019]. We follow [COLOMBO and collab., 2020;SANKAR and collab., 2019a] and set T = 5. We rely on the tokenizers provided by the HuggingFace library based on the SentencePiece [KUDO and RICHARDSON, 2018] and WordPiece [WU and collab., 2016] algorithms. In all experiments, for our models relying on the HT we use the same architecture as the SMALL model developped in Chapter 5 which contains 80 millions parameters.

Original BERT has 167 millions parameters and is pre-trained using 16 TPUs during several days with over 500K iterations.

For each task, model are fine-tuned and dropout [SRIVASTAVA and collab., 2014] is set to 0.1. The best learning rate is found in {0.01, 0.001, 0.0001} and chosen based on the validation loss. ) stands for the model pre-trained with the utterance level loss mL u on spoken data and mHT(µ written ) stands for a hierarchical encoder where sentence embeddings is computed using a pre-trained BERT encoder.

VM2 Map Task Dihana Loria Ilisten

Ablation study on pre-training data

We showcase the difference between pre-training with spoken and written corpora. We compare mHT(µ written ), a hierarchical encoder where each utterance is embedded using the representation of the [CLS] token given by the second layer of BERT, and mHT u (µ spoken ), a model pre-trained on OPS using L u only. The prediction is performed by feeding the utterance embeddings to a simple MLP. In Table 6.6, we report the results on MIAM. improves over the mBERT model. Interestingly, we observe that the monolingual pre-training works slightly better compared to the multilingual pre-training when training using the same loss. This result surprising results might be attributed to the limited size of our models [KARTHIKEYAN and collab., 2019]. We see that in both cases, introducing a loss with aligned multilingual conversations (MMUG or TMUG) induces a performance gain (+1.5%). This suggests that our pretraining with the new losses better captures the data distribution. By comparing the results of mMUG + TMUG with mMUG, we observe that the addition of crosslingual generation during pre-training helps. A marginal gain is induced when using MMUG over TMUG, thus we believe that the improvement of mMUG + MMUG over mMUG can mainly be attributed to the cross-lingual generation part. Interestingly, we observe that the combination of all losses out-performs the other models which suggests that different losses model different patterns present in the data.

Inconsistency Identification

In this section, we follow MEHRI and collab.

[2019] and evaluate our pre-trained representations on II with a monolingual context. A random guess identifies the inconsistency by randomly selecting an index in [1,T] which corresponds to an accuracy of 20% (as we have set T = 5). in three groups: hierarchical transformer encoders pre-trained using our custom losses, baselines (see subsection 6.3.4) using either multilingual or language specific tokenizer.

Toke. stands for the type of tokenizer: multi and lang denotes a pre-trained tokenizer on multilingual and language specific data respectively. When using lang tokenizer, MUG pre-training and finetuning are performed on the same language.

de-en de-es de-fr de-it en-es en-fr en-it es-fr es Table 6.9 -Results on the mII task with bilingual input context.

de en es fr it R@5 R@2 R@1 R@5 R@2 R@1 R@5 R@2 R@1 R@5 R@2 R@1 R@5 R@2 R@1 mBERT 65 6.10 -Results on the NUR task with monolingual input context. R@N stands for recall at N. de-en de-es de-fr de-it en-es R@5 R@2 R@1 R@5 R@2 R@1 R@5 R@2 R@1 R@5 R@2 R@1 R@5 R@2 R@1 mBERT 54.4 27.0 11.6 55.9 24.8 11.9 57.9 24.2 12.9 57.5 23.9 13.0 55.4 25.6 13.0 mBERT (4-layers) 54.1 26.5 11.9 55.7 24.8 12.4 57.2 24.1 12.4 57.0 23.5 13.1 55.6 23.1 12.9 mHR 52.1 25.5 12.1 54.9 14.6 10.7 56.1 22.9 11.3 56.9 24.9 13.0 53.9 23 en-fr en-it es-fr es-it fr-it R@5 R@2 R@1 R@5 R@2 R@1 R@5 R@2 R@1 R@5 R@2 R@1 R@5 R@2 R@1 mBERT 57.9 25. In this chapter, we explore spoken dialog within a mutlimodal scenario in the scope of emotion recognition. Since humans communicate through different modalities (i.e visual, acoustic, linguistic), efforts have been made towards developing complex fusion mechanisms based on neural networks. In these works, the linguistic modality appears to be the most effective. Furthermore, when the communication modality is the text, such as in chats or in theatre plays, the writers use punctuation marks to convey emotions. Linguists have showcased the prosodic aspect of punctuation marks acknowledging their importance in human communication. However, punctuation marks are often reduced to their semantic aspect and removed in most NLP tasks. In this chapter, we investigate the role played by punctuation marks in an emotion recognition task with current neural network architectures. Results showcase their abilities to transcribe prosodic cues. We explore their role further through ablation studies and a comparison with simple multimodal models.

Introduction

Emotion recognition in spoken dialogues is an increasingly popular area of Affective Computing research. Acknowledging humans communicate through a variety of channels (i.e visual, acoustic, linguistic), multimodal systems have integrated these different unimodal representations into one synthetic representation. So far, a consequent effort has been made to develop complex architectures allowing the fusion of these modalities. However, most of the accuracy is often carried by the linguistic modality [PORIA and collab., 2018;RAHMAN and collab., 2020]. A natural way to leverage emotion in textual conversation, employed by writers in plays or by people in written chat, is the use of punctuation1 . Linguistic studies have shown that the use of punctuation follows either a syntactic or prosodic approach [START_REF] Bruthiaux | Knowing when to stop: Investigating the nature of punctuation[END_REF][START_REF] Halliday | Spoken and Written Language, Language and Learning Series[END_REF]. [START_REF] Moore | What's the point? the role of punctuation in realising information structure in written english[END_REF] distinguishes between text to be read in silence and text to be read aloud. He shows that in the latter, punctuation

Data and Models

Data Description

For this experimental study, we use the MELD2 dataset [PORIA and collab., 2018], a multimodal dataset collected from the TV show Friends. The data consists in 13708 manually transcribed utterances (Table 7.1 provides statistics for each split), labelled in sentiment and emotion including : anger, sadness, disgust, surprise, fear, joy and neutral.I nTable 7.2 we present several examples from the MELD corpus and Table 7.1 provides statistics for each split. We choose this dataset for the following reasons: (1) due to its nature, it falls into the category of written-to-be-spoken text. This category of textual data gathers not only scripts for plays but also transcripts of spoken dialogues obtained either manually or to some extent, with Automatic Speech Recognition (ASR) models [BAGHER ZADEH and collab., 2018;BUSSO and collab., 2008;ZADEH and collab., 2016]. However, the latter suffers from the lack of punctuation marks. Since dialogues from MELD are manually transcribed, the transcribed punctuation marks are of good quality.

(2) When comparing to EK and collab. [2020], which considers all nonalphanumeric characters, we choose to consider a limited number of punctuation marks. Indeed, not all punctuation marks are prosodic: for example, it has been demonstrated that adults and children show some pitch declination at commas, regardless of their reading ability [KUHN and collab., 2010], while exclamation points and quotation marks appear to signal a rise in pitch [SCHWANENFLUGEL and collab., 2013]. Brackets, on the other hand, do not appear to have a clear relationship with expressive reading [START_REF] Bodenbender | The punctuation and intonation of parentheticals[END_REF]. Hence, we only consider the following punctuation marks : '<!> <?> <.> <,> <...>'.

(3) Table 7.4 shows the proportion of utterances containing specific punctuation marks for each emotion. As expected, some emotions are more closely related to the use of certain punctuation marks. For instance '<! ! !>' is present in 36% and 33% of sentences labelled with respectively joy and anger; where '<! ?>' is present in 75% of sentences labelled with surprise.

Alternatively the dot and comma appear in more than half the sentences labelled as neutral. Those observations are in line with the relationship we have previously exposed between the use of some punctuation marks and expressed emotions. (3)

Finally the MELD corpus provides acoustic features allowing us to compare the use of prosodic punctuation marks with fusion between the textual and audio modalities.

Baselines

We present here the baselines we selected for our experiments, as they are the most frequently used to extract context-independent utterance-level feature vectors in emotion recognition tasks.

Text-CNN In this framework introduced by KIM [2014], the embeddings of the words us to test the importance of punctuation as a discriminative feature in emotion recognition tasks. With the S p!¬p experiment, we want to show the impact of punctuation removal at testing time, when the model has been trained with it. S ¬p!p acts as a sanity check experiment: if a model is not trained to associate punctuation marks to particular emotion labels, we expect it to ignore these characters at testing time. Experiments are first conducted with only textual features, then with both textual and audio features.

Is punctuation a discriminative feature ?

We present our results in Table 7.6. When comparing S p!p with S ¬p!¬p , we observe that every model, trained on textual modality only, achieves a better performance when punctuation marks are kept in the dataset: up to 9 points for BERT and RoBERTa. Moreover, results with the S p!¬p setting show the extent to which models rely on these features to make predictions leading to a huge drop in performance when they are removed at inference time: from 10 points for transformer-based models to 15 for CNN R model. Table 7.5 gives performance details for each emotion class.

We observe an increase of the neutral class accuracy score for all models with the S p!¬p setting: by removing punctuation marks, classifiers lose precious prosodic information and consider such altered utterances as neutral. Keeping punctuation marks allows the models to efficiently leverage the ambiguity of utterances labelled as surprise, joy and anger.

The setting S ¬p!p does not show major differences for models based on convolutional network. However, pre-trained transformer models show an increase of 3 points of the F1 Weighted score, stemming from better results on the joy, anger and surprise labels. Table 7.5 -Baselines results on the test set of MELD using only textual modality. W-avg F1 denotes the weighted average of F1 score. Results have been averaged over 5 runs.

Ablation Study on Punctuation Marks

In this section, we investigate the role of each punctuation mark in emotion recognition tasks. Each model is trained with the original training set. At testing time, we remove one mark or a specific sequence of marks at a time. Results for BERT are presented in Table 7.7, we obtain similar results for other baselines. They show cues related to sadness. Such a behaviour can be misleading: for example, the sentence "I am so sad" is correctly labelled by BERT as sadness however "I am so sad !" would be classified as joy.

Punctuation as substitute for acoustic modality ?

Table 7.6 shows that for every setting, audio information improves the performance of the models. This is especially the case for models without punctuation: by comparing S T+A ¬p!¬p with S T ¬p!¬p , we notice an average increase of about 2 points. However, as previously mentioned the addition of punctuation marks alone, i.e S T p!p , improves the result up to 9 points. This seems to confirm the greater efficiency of punctuation marks to transmit prosodic cues. While our experiments were made with a simple fusion mechanism, similar observations have previously been made RAHMAN and collab. [2020] regarding the low performance improvement brought by other modalities. However in Table 7.8 -Baselines results on the test set of MELD using only both textual and acoustic modality. W-avg F1 denotes the weighted average of F1 score. Results have been averaged over 5 runs.

Chapter 7 Conclusion

This chapter is an preliminary study that aims to investigate ways to translate non-textual modality into textual modality. In particular, we focused on punctuations marks and we provided a quantitative analysis of their impact on emotion recognition. We showcase the ability of punctuation marks to encode some prosodic cues in textual modality, boosting classifiers performances especially with pre-trained models. Furthermore we showed how punctuation marks may be limited, as they seem to perform very well when encoding expressive tones related to anger, joy or surprise but fail at transcribing acoustic cues related to sadness or disgust. with English data, whereas, a dialogue system may be exposed to languages other than English or multilingual conversations. In Chapter 6, we proposed to extend the work presented in Chapter 5 to a multilingual setting. Based on the architectures previously introduced, we presented novel losses inspired by the CS phenomena that may occur within a speaker statement or in a conversation. We leverage the OpenSubtitles alignment files that are available in a large variety of languages to forge multilingual conversational contexts used to enrich the pre-training corpus with code-switched conversation. We evaluated our approach on a newly introduced multilingual benchmark called MIAM gathering DA annotated datasets in 5 European languages. Additionally, we test the learnt multilingual representations on two new tasks: contextual inconsistency detection and next utterance retrieval with both monolingual and multilingual input context. Finally, in Chapter 7, we explored the multimodal aspect of human communication with a focus on emotion recognition. In a spoken conversation, emotions are conveyed via a variety of channels such as visual, acoustic and linguistic. Current multimodal systems aim at integrating these different unimodal representations into one synthetic representation via complex fusion mechanisms. On the other hand, information in textual modality is well leveraged by neural architectures and has been shown as the most dominant modality regarding the system's performances. Hence, we aim at investigating the translation of non-textual modalities, i.e. visual and acoustic, into the textual modality. This last chapter is a preliminary study that focuses on punctuation marks as they are a simple and easily accessible textual representations of prosodical cues. We provided a quantitative study that showcases how effectively acoustic cues can be cram into punctuation marks boosting the performance of the predictor, especially with pre-trained models.

Limitations and future work

In this section, we put our contributions into perspective, discuss some of the limitations of our work as well as the numerous perspectives for the different parts of my work

Limitations and Future Directions Related to RQ1

In RQ1, we explored the problem of sequence labelling in the monolingual scenario, thus Part II opens the following research directions:

• Including the interaction dissymmetry and speaker information in our models. In Part II, we put a lot of effort into providing distributed representations of the dialogue context solely based on the textual content. We have neglected some aspects of the interaction such as the speaker information. Such information covers a wide range of phenomena from-turn taking [LIU and collab., 2020;SHANG and collab., 2020], intra-speaker and inter-speaker dependencies [SHEN and collab., 2020] to personality traits [LI and collab., 2016;ZHONG and collab., 2020]. We believe they are key information to better represent the dynamic of the conversation. In Chapter 5, we have provided a hierarchical transformer to encode the conversation. In particular, the dialogue level of this CHAPTER 8. CONCLUSIONS, LIMITATIONS AND FUTURE WORK encoder is trained to learn utterances dependencies. In future work, we aim at modifying this part of the network in order to include self-and inter-speakers dependencies.

• Explore the interplay between dialogue act and emotion In this thesis we have addressed DA and E/S classification as two separate tasks. However, as conversations are often driven by emotion there is a mutual dependency between DA and E/S [LI and collab., 2017]. BOTHE and collab. [2020] have highlighted this relationship, for instance, the Accept/Agree dialogue act is more likely to occurs when the utterance is labelled with a Joy emotion. Future work includes the study and modelling of this dependence via structured prediction. Furthermore, recently conversational corpus annotated in both DA and E/S have been realised [BOTHE and collab., 2020], following DailyDialog [LI and collab., 2017]. We plan to include these new datasets in our SILICONE benchmark.

• • Studying domain adaptation. In this thesis, we have provided neural models for spoken dialogue understanding. In Chapter 5 we have justified the choice of OpenSubtitles Corpus as a pre-training dataset with the discrepancy between the spoken nature of the downstream tasks and the usual pre-training corpus from written text. We have presented experiences that showcase this domain-shit and its impact on the system's performances. However, we did not provide a quantitative analysis of this discrepancy. For example, [START_REF] Elsahar | To annotate or not? predicting performance drop under domain shift[END_REF] studies the prediction of the performance drop when a model is evaluated on a new target domain. Three category of measures are considered to evaluate the distance between domains: H-divergence, reverse classification accuracy and confidence measures. These measures rely on parameterised classifiers that need to be learnt. Meanwhile, pre-trained language models are ubiquitous and provide an underused distributional representation of sentences. Hence, in future work, we aim at investigating untrained measures based on distribution distances.

Limitations and Future Directions Related to RQ2

In RQ2, we explored spoken dialogue understanding problems that involve working with multilingual and multimodal data. Therefore, Part III opens the following research directions:

• Enrich our models developed in Chapter 6 with new languages In the future, we plan to further work on OPS to obtain fine-grained alignments (e.g at the span and word levels) and enrich the definition of code-switching (currently limited at the utterance level). Moreover, when considering interactions with voice assistants and chatbots, users may not be able to express their intent in the language in which the voice assistant is programmed. Thus, we would like to strengthen our evaluation protocol by gathering a new DA benchmark with code-switched dialogues to improve the multilingual evaluation. Lastly, the cross-lingual few shot is a setting we also consider as future work, hence it would involve re-annotating datasets with a common annotation scheme following for example [BUNT and collab., 2019]. We would extend the MIAM corpus to new language and include E/S corpus thus forming a multilingual equivalent of SILICONE.

• Learning multimodal pre-trained representations In this thesis, we have addressed multimodal and conversational representations separately. In Chapter 5 and Chapter 6, we have provided textual pre-trained representations tailored for spoken dialogues. In Chapter 7 we have studied multimodal systems and showcased the importance of the textual modality due in particular to its pre-trained representations. Extending [BUGLIARELLO and collab., 2020] and [DAI and collab., 2021], it would be interesting to provide pre-trained multimodal representations tailored to spoken dialogues.

• Toward a textual representation of modality In Chapter 7 we showed that punctuation marks convey prosodical cues that allow us to leverage the ambiguity of utterances in emotion classification. Hence, punctuation marks are discriminative features in the scope of emotion classification, especially with pre-trained models. We believe that the performances of current textual encoders may offer a better opportunity to transmit particular (social) cues, when they are available than fusion with the acoustic or visual modality -as we have shown with punctuation. In future work, we plan to translate other modalities into a textual description. As an example, the MOSEI dataset [BAGHER ZADEH and collab., 2018] provides visual features that include facial action units. Each of these units has its own label, e.g. the 6th facial action unit is labelled as Cheek raiser. Instead of relying on the one-hot encoded representation of these labels, we propose to use their textual description. This text can now be directly concatenated to the corresponding utterance transcript or embedded through a masked language model and further used in common fusion mechanisms. This approach has several advantages: (1) modalities are placed into a common space, i.e. language, hence the system handles data of the same nature (2) non-textual modalities can benefit from powerful representations offered by pre-trained language models.

Abstract :

Conversational AI has received a growing interest in recent years from both the research community and the industry. Products have started to emerge (e.g. Amazon's Alexa, Google's Home, Apple's Siri) but performances of such systems are still far from humanlikeness communication. As an example, conversation with the aforementioned systems is often limited to basic question-response interactions. Among all the reasons why people communicate, the exchange of information and the strengthening of social bound appeared to be the main ones. In dialogue research, the two aforementioned problems are well known and addressed using dialogue act classification and emotion/sentiment recognition. Those problems are made even more challenging as they involve spoken dialogues in contrast to written text. A spoken conversation is a complex and collective activity that has a specific dynamic and structure. Thus, there is a need to adapt both natural language processing and natural language understanding techniques which have been tailored for written texts as it does not share the same characteristics. This thesis focuses on methods for spoken dialogue understanding and specifically tackles the problem of spoken dialogues classification with a particular focus on dialogue act and emotion/sentiment labels. Our contributions can be divided into two parts : in the first part, we address the problem of automatically labelling English spoken dialogues. In this part, we start by formulating this problem as a translation problem which leads us to propose a seq2seq model for dialogue act classification. Then, our second contribution focuses on a scenario relying on small annotated datasets and involves both pre-training a hierarchical transformer encoder and proposing a new benchmark for evaluation. This first part addresses the problem of spoken language classification in monolingual (i.e. English) and monomodal (i.e. text) settings. However, spoken dialogue involves phenomena such as codeswitching (when a speaker switch languages within a conversation) and relies on multiple channels to communicate (e.g. audio or visual). Hence, the second part is dedicated to two extensions of the previous contributions in two settings : multilingual and multimodal. We first address the problem of dialogue act classification when multiple languages are involved and thus, we extend the two previous contributions to a multilingual scenario. In our last contribution, we explore a multimodal scenario and focus on the representation and fusion of modalities in the scope of emotion prediction.

Institut Polytechnique de Paris 91120 Palaiseau, France

2. 1

 1 Components of traditional dialogue systems. ............... 3.1 Example of dialogue between two speakers from the MELD corpus. u i denotes the i -th utterances and y i the corresponding emotion. ..... 3.2 Transfer Learning ................................ 4.1 Seq2seq model architecture forDA classification. (a) The encoder is composed with three different levels representing a different hierarchical level in the dialogue. The utterances are encoded at: word level (purple), persona level (orange) and sentence level (green). (b) The decoder (blue) is responsible to generate for each utterance aDA exploiting the last state of the encoder as initial hidden state. ............ 4.2 Attention matrix visualisation on MRDA for the fixed context of 5 utterances. Green color for predicted label indicates a correct label, orange color indicates a mistake. (a) stands for the HGRU with attention, (b) stands for the HGRU with hard guided attention, (c) is HGRU with soft guided attention. ................................ 4.3 Confusion Matrix for our best performing seq2seq model on SwDA for 10 out of 42 tags. For label designation see Section 4.5.1. ......... 4.4 Confusion Matrix for our best performing seq2seq model on MRDA.For label designation see Section 4.5.1. ...................... 5.1 General structure of our proposed hierarchical dialogue encoder, with a decoder: f u µ , f d µ and the sequence label decoder (g dec µ ) are colored respectively in green, blue and red. ...................... 5.2 This figure shows an example of corrupted context. Here p C is randomly set to 2 meaning that two utterances will be corrupted. u 1 and u 4 are randomly picked in 5.2b, 5.2d and then masked in 5.2c, 5.2e. .. 5.3 Schema of the different models evaluated on SILICONE. In this figure f u µ , f d µ and the sequence label decoder (g dec µ ) are respectively colored in green, blue and red for the hierarchical encoder (see Figure 5.3a and Figure 5.3d). For BERT there is no hierarchy and embedding is performed through f u µ colored in grey (see Figure 5.3c, Figure 5.3d) .. 5.4 Histograms showing the utterance length for each dataset of SILICONE. 5.5 A comparison of pre-trained encoders being fine-tuned on different percentage the training set of SEM. Validation and test set are fixed over all experiments, reported scores are averaged over 10 different random split. ........................................ 5.6 Illustration of improvement of accuracy during pre-training stage on SEM for both a TINY and SMALL model. .................... 5.7 A comparison of different parameters initialisation on MELD s . Training is performed using a different percentage of complete training set. Validation and test set are fixed over all experimentation. Each score is the averaged accuracy over 10 random runs. ................ 96 6.1 6.1a and 6.1b illustrate pre-training losses using monolingual context. 6.1b and 6.1c show two scenarios for the MMUG loss using multilingual context. Double squares on the figure indicates the randomly selected utterance to predict. ..............................113 6.2 Histograms showing the utterance length for OPS (left) and MIAM (right). 117 7.1 Confusion matrix visualisation of BERT on different version of the MELD test set for the 5 most represented classes. From left to right, (a) no modification is applied, (b) <!> is removed (c) each utterance is appended with an exclamation mark <!> and (d) each utterance is appended with the punctuation mark <?>. .................137

4. 3

 3 Accuracy of a seq2seq on dev test and Baseline CRF on SwDA and MRDA. Bold results exhibit significant differences (p-value < 0.01) according to the Wilcoxon Mann Whitney test performed on 10 runs using different seeds. ....................................... 4.4 Example of predicted sequence of tags taken from SwDA. seq2seq is our best performing model, CRF stands for Baseline CRF , G. is the groundtruth label. ....................................... 4.5 Accuracy on the dev set of the different encoder/decoder combination MRDA andSwDA. ForSwDA, Wilcoxon test (10 runs with different seeds) has been performed for an HGRU encoder with a decoder with hard guided attention against an HGRU encoder with soft guided attention, soft guided attention, with attention, without attention pairwise tests exhibit p-value < 0.01. ............................. 4.6 Accuracy on the dev set of seq2seq model trained with sequence level loss. B train stands for the beam size during training, B inf for the one during inference 1 .For SwDA, Wilcoxon test (10 runs with different seeds) has been performed for B train = 2 and B inf = 2 against all other models.

  ...................................123 6.9 Results on the mII task with bilingual input context. ...........123 6.10 Results on the NUR task with monolingual input context. R@N stands for recall at N. ..................................123 6.11 Results on the mNUR task with bilingual input context. ..........123 7.1 MELD description ...............................135 7.2 Some examples of dialogues from MELD dataset. ..............135 7.3 Punctuation marks distribution over MELD splits. Bold number in parenthesis indicates the percentage of tokens. ..................136 7.4 Distribution of sentences containing punctuation marks among emotion classes. ...................................136 7.5 Baselines results on the test set of MELD using only textual modality. W-avg F1 denotes the weighted average of F1 score. Results have been averaged over 5 runs. ..............................138 LIST OF TABLES 7.6 Baseline results (weighted average of F1 score) on the test set of MELD using both textual (T) and audio (A) modalities. Results have been averaged over 5 runs. .............................139 7.7 Ablation study results for BERT on MELD test set. Each punctuation mark is removed one at a time during inference. Presented results are the difference of accuracy after removal and without modification. Results are obtained considering only utterance containing the punctuation marks to be removed. Results have been averaged over 5 runs. 139 7.8 Baselines results on the test set of MELD using only both textual and acoustic modality. W-avg F1 denotes the weighted average of F1 score. Results have been averaged over 5 runs. ...................140

  Part I, we start by introducing characteristics of spoken dialogue and the relevant related work for DA and E/S classification. We also present deep learning methods and models used in order to understand the contributions of this thesis. In Part II,we tackle the problem of monolingual spoken dialogue utterances classification in two scenarios: (1) when large annotated corpora are available (RQ1.1) (2) in low-resource tasks (RQ1.2). Finally in Part III, we extend the work presented in Part II to two settings: multilingual (RQ2.1) and multimodal (RQ2.2). More precisely, this dissertation is organized as follows: [Part I]. In this first part, we introduce the background of spoken dialogue understanding and recall deep learning methods and architectures related to NLP which will be used in Part II and Part III.

CHAPTER 2 .

 2 Figure 2.1 -Components of traditional dialogue systems.

CHAPTER 3 .

 3 Figure 3.1 -Example of dialogue between two speakers from the MELD corpus. u i denotes the i -th utterances and y i the corresponding emotion.

Figure

  Figure 3.2 -Transfer Learning

  5+1) AE is the length normalisation coefficient[WU and collab., 2016]. At each time step the B most likely sequences are kept (B corresponding to the beam size).

  -the-art models[KUMAR and collab., 2018; LI and collab., 2018a] use conditional random fields which model dependencies between tags on top of an GRU or a HGRU encoder which computed an embedding of the a variable number of utterances sentences. We have implemented our own CRF (Baseline CRF ) following the work of LI and collab. [2018a]:

Figure 4 .

 4 Figure 4.2 -Attention matrix visualisation on MRDA for the fixed context of 5 utterances. Green color for predicted label indicates a correct label, orange color indicates a mistake. (a) stands for the HGRU with attention, (b) stands for the HGRU with hard guided attention, (c) is HGRU with soft guided attention.

Figure 4

 4 Figure 4.3 -Confusion Matrix for our best performing seq2seq model on SwDA for 10 out of 42 tags. For label designation see Section 4.5.1.

Figure 4 . 4 -

 44 Figure 4.4 -Confusion Matrix for our best performing seq2seq model on MRDA. For label designation see Section 4.5.1.

Figure 5

 5 Figure 5.1 -General structure of our proposed hierarchical dialogue encoder, with a decoder:f u µ , f dµ and the sequence label decoder (g dec µ ) are colored respectively in green, blue and red.

  the set of positions of masked utterances in the context C k , Ck is the corrupted context, and p C is the proportion of masked utterances. We propose a visual illustration of the corrupted context Figure 5.2 by the MLM Loss.

  (a) Initial context composed by 5 utterances. (b) u 1 is chosen to be masked. (c) Corrupted context with utterance u 1 masked. (d) u 4 is chosen to be masked. (e) Corrupted context with utterance u 4 masked.

Figure 5

 5 Figure 5.2 -This figure shows an example of corrupted context. Here p C is randomly set to 2 meaning that two utterances will be corrupted. u 1 and u 4 are randomly picked in 5.2b, 5.2d and then masked in 5.2c, 5.2e.

  Figure 5.3 -Schema of the different models evaluated on SILICONE. In this figure f u µ , f d µ and the sequence label decoder (g dec µ ) are respectively colored in green, blue and red for the hierarchical encoder (see Figure 5.3a and Figure 5.3d). For BERT there is no hierarchy and embedding is performed through f u µ colored in grey (see Figure 5.3c, Figure 5.3d)

  To validate the importance of pre-training in a regime of low data, we train different HT (with random initialisation) on different portions of SEM and MELD s . Results shown in Figure 5.5 illustrate the importance of pre-trained representations.

CHAPTER

  Figure 5.7 -A comparison of different parameters initialisation on MELD s . Training is performed using a different percentage of complete training set. Validation and test set are fixed over all experimentation. Each score is the averaged accuracy over 10 random runs.

Figure 6

 6 Figure 6.1 -6.1a and 6.1b illustrate pre-training losses using monolingual context. 6.1b and 6.1c show two scenarios for the MMUG loss using multilingual context. Double squares on the figure indicates the randomly selected utterance to predict.

Figure 6 .

 6 Figure 6.2 -Histograms showing the utterance length for OPS (left) and MIAM (right).

4 -

 4 abbastanza

CHAPTER 8 .

 8 CONCLUSIONS, LIMITATIONS AND FUTURE WORK scenarios: multilingual and multimodal. Research on NLU and in particular on DA or E/S classification is very English-centric, as an example Part II exclusively deals

  Text generation tailored for dialogue Great advances have been made in natural language generation with large scale pre-training transformers based architectures such as [RADFORD and collab., 2019; ?]. They have shown the capacity to produce rich, lexically diverse and fluent content similar to text written by humans. As we showed throughout this manuscript a spoken conversation has features that distinguish it from a written text. Recently, in the scope of open-dialogue systems, ZHANG and collab. [2019] have taken a step in that direction. It would be interesting to pursue the work established in Chapter 5 toward the generation of dialogues. Future work could also include style transfer following COLOMBO and collab. [2019]; YANG and collab. [2020].

Titre:

  M éthodes neuronales pour la compr éhension des dialogues parl és Mots cl és : Apprentissage Profond, Traitement du Langage Naturel, Dialogues Parl és. R ésum é:L'intelligence artificielle conversationnelle a suscit é un int ér êt croissant ces derni ères ann ées, tant dans la communaut é des chercheurs que dans l'industrie. Des applications grand publique ont commenc é à voir le jour (par exemple, Alexa d'Amazon, Home de Google, Siri d'Apple), mais les performances de ces syst èmes sont encore loin d'une communication semblable à celle des humains. Par exemple, la conversation avec les syst èmes susmentionn és se limite souvent à des interactions de base de type question-r éponse. Parmi toutes les raisons pour lesquelles les gens communiquent, l' échange d'informations et le renforcement des liens sociaux semblent être les principales. Dans la recherche sur le dialogue, ces deux probl èmes sont bien connus et abord és à l'aide de la classification des actes de dialogue et de la reconnaissance des émotions/sentiments. Ces probl èmes sont d'autant plus difficiles àr ésoudre qu'ils concernent des dialogues parl és, contrairement aux textes écrits. Une conversation parl ée est une activit é complexe et collective qui poss ède une dynamique et une structure sp écifiques. Il est donc n écessaire d'adapter les techniques de traitement et de compr éhension du langage naturel qui ont ét é conc ¸ues pour les textes écrits car elles ne partagent pas les m êmes caract éristiques. Cette th èse se concentre sur les m éthodes de compr éhension des dialogues parl és et aborde sp écifiquement le probl ème de la classification des dialogues parl és avec un accent particulier sur les étiquettes des actes de dialogue et des émotions/sentiments. Nos contributions peuvent être divis ées en deux parties : dans la premi ère partie, nous abordons le probl ème de l' étiquetage automatique des dialogues parl és en anglais. Dans cette partie, nous commenc ¸ons par formuler ce probl ème comme un probl ème de traduction, ce qui nous am ène à proposer un mod èle seq2seq pour la classification des actes de dialogue. Ensuite, notre deuxi ème contribution se concentre sur un sc énario reposant sur de petits ensembles de donn ées annot ées et implique à la fois le pr é-entraînement d'un encodeur transformateur hi érarchique et la proposition d'un nouveau benchmark pour l' évaluation. Cette premi ère partie aborde le probl ème de la classification du langage parl é dans des contextes monolingues (i.e. anglais) et monomodaux (i.e. texte). Cependant, les dialogues parl és impliquent des ph énom ènes tels que le code-switching (lorsqu'un locuteur change de langue au cours d'une conversation) et s'appuient sur plusieurs canaux pour communiquer (par exemple, audio ou visuel). La deuxi ème partie est donc consacr ée à deux extensions des contributions pr éc édentes dans deux contextes : multilingue et multimodal. Nous abordons d'abord le probl ème de la classification des actes de dialogue lorsque plusieurs langues sont impliqu ées et nous étendons donc les deux contributions pr éc édentes à un sc énario multilingue. Dans notre derni ère contribution, nous explorons un sc énario multimodal et nous nous concentrons sur la repr ésentation et la fusion des modalit és dans le cadre de la pr édiction des émotions. Title : Neural Methods For Spoken Dialogue Understanding Keywords : Deep Learning, Natural Language Processing, Spoken Dialogue

  1.1 Example of conversation from DailyDialog corpus between two speakers A and B. The underlined words in purple explicitly indicate the emotions. The words in italic are ideas expressed by speaker B that are new for the other speaker A. .......................... 2.1 Extract of dialogue from Waiting for Godot ................. 2.2 Example of dialogue taken from the Switchboard Dialog Act Corpus. .

4.1 Example of conversation from Switchboard Dialogue Act Corpus. A is speaking with B. ................................. 4.2 Statistics for MRDA and SwDA. |C| is the number of Dialogue Act classes, |V| is the vocabulary size. Training, Validation and Testing indicate the number of conversations (number of utterances) in the respective splits.

  4.7 Accuracy of our best models (seq2seq) and Baseline CRF on SwDA The datasets are grouped by label type (DA vs E/S) and ordered by decreasing size. MT stands for Map Task, IEM for IEMOCAP and Sem for Semaine. ................................ 92 5.5 Results of ablation studies on SILICONE ................... 94 5.6 Comparison of GAP and MLM with a comparable number of parameters. For all models a MLP decoder is used on top of a TINY pre-trained encoder. ..................................... 95 5.7 Number of parameters for the encoders. Sizes are given in million of parameters. ................................... 96 6.1 Example of automatically built input context from OPS. .........112 6.2 Statistics of the processed version of OPS. ..................114 6.3 Statistics of the processed version of the alignment files from OPS. ...115 6.4 Examples of dialogues labelled with DA taken from MapTask, Dihana, VM2, Loria and Ilisten. AFF. stands for affirmation, FEED. for feedback and ACK.

and MRDA test sets. .................................. 5.1 Statistics of datasets composing SILICONE. E stands for emotion label and S for sentiment label; ? stands for datasets with available official split. Sizes of Train, Val and Test are given in number of conversations. 5.2 Architecture hyperparameters used for the hierarchical pre-training. . SILICONE. for acknowledgement. ....................118 6.5 Architecture hyperparameters used for the hierarchical pre-training. . 120 6.6 Ablation studies on pre-training data. We report the accuracy on MIAM

Table 1

 1

.1 -Example of conversation from DailyDialog corpus between two speakers A and B. The underlined words in purple explicitly indicate the emotions. The words in italic are ideas expressed by speaker B that are new for the other speaker A.

  ELMo [PETERS and collab., 2018] language model that leverages CS phenomena,

	CHAPTER 2. SPOKEN DIALOGUE UNDERSTANDING
	of the
	NLP community [ÇETINOGLU
	and collab., 2016]. For instance [AGUILAR and SOLORIO, 2020] provides an extension

Emotion Recogniton in Conversation

	study of emotion and sentiment in human communication by intelligent sys-
	tems is a whole cross-disciplinary research field called affective computing. It aims
	machines to recognize, predict, and interpret human emotions in order to make
	emotional responses. It covers a wide range of areas such as computer science, AI,
	cognitive science, neuroscience, neuropsychology and social science. In this chapter,
	we recall deep learning approaches that have been developed to tackle E/S predic-
	tion in spoken dialogue. Currently, these approaches can be divided into two main
	lines of research:

ERC: with recent public available datasets (e.g.

  Then, MIKOLOV and collab. [2013] proposed a shallow architectures based on RNN and two novel objectives i.e, continuous bag-of-words CBOW and Skip-gram to learnt word embeddings. Although effective in many NLP tasks, word embeddings suffer from two main shortcomings: (1) they rely on shallow neural architectures (2) they are context-independent. Indeed, since NLP tasks are beyond word-level, the

rest of the neural network needs to be trained on the downstream tasks. Hence, it appears natural to pre-train the encoder from top to bottom on sentence-level as proposed in

PETERS and collab. [2018]

. With the use of such deeper neural networks, in particular with the introduction of the aforementioned Transformers architecture, the number of parameters has quickly skyrocketed. Meanwhile, as previously said, a variety of supervised tasks do not gather enough data to completely exploit the power of such neural networks preventing a directly supervised training. Still, it is worth noticing that pre-training is not necessarily unsupervised: in Computer Vision (CV) models have been trained on ImageNet

[DENG and collab., 2009]

a dataset gathering 14 millions of pictures hand annotated over 20.000 categories. As previously mentioned there is no such dataset in NLP, for instance

[CONNEAU and collab., 2017]

has trained a sentence encoder on the SNLI

[BOWMAN and collab., 2015]

corpus, a

  The proposed seq2seq model in Chapter 4 is data-hungry and requires large corpora of labelled utterance to be trained. As an example Switchboard has over 100k labelled utterances, whereas the SEMAINE corpus has about 5k labelled utterances. Additionally, those models are very specific to the labelling scheme employed. Adapting them to different sets of emotions or dialogue acts would require more annotated data.

	Part II Abstract
	This part gathers our contributions related to RQ1 "How to leverage recent
	advances in neural networks to build a system that can automatically label
	English spoken dialogue utterances ?". This part is further divided into two different chapters: Chapter 4
	• Our formalization of sequence labelling shares common points with
	Neural Machine Translation (NMT) problems. Inspired by the recent suc-cesses of seq2seq approaches widely adopted NMT, Chapter 4 proposes A seq2seq Model for Sequence a seq2seq tailored towards DA classification paving the way for further innovations inspired by advances in NMT research. This seq2seq aims at improving the modelling of tags sequentially. Labelling
	Part II
	Monolingual Spoken Dialogue
	Understanding
	ABDEL-HAMID,O., L. DENG and D. YU. 2013, "Exploring convolutional neural net-
	work structures and optimization techniques for speech recognition.", in Inter-
	speech, vol. 11, Citeseer, p. 73-5. 45
	AGARAP,A .F .2018, "Deep learning using rectified linear units (relu)", CoRR, vol.
	abs/1803.08375. URL http://arxiv.org/abs/1803.08375. 44
	BAHDANAU,D., K. CHO and Y. BENGIO. 2014, "Neural machine translation by jointly

learning to align and translate", URL http://arxiv.org/abs/1409.0473,cite arxiv:1409.0473Comment: Accepted at ICLR 2015 as oral presentation. 46 BELINKOV,Y., N. DURRANI, F. D ALVI, H. SAJJAD and J. R. GLASS. 2017, "What do neural machine translation models learn about morphology?", CoRR, vol. abs/1704.03471. URL http://arxiv.org/abs/1704.03471. 53 • In Chapter 5, we address the problem of E/S and DA labelling in a data scarcity setting by proposing new pre-trained objectives. In this chapter, we learn generic representations that can quickly adapt to small labelled datasets. Chapter 4 abstract This chapter present our first contribution where we propose to tackle the problem of sequence labelling in spoken dialogue with a focus on DA classification. The task of predicting DA based on conversational dialogue is a key component in the development of conversational agents. Accurately predicting DAs requires a precise modeling of both the conversation and the global tag dependencies. We leverage seq2seq approaches widely adopted in Neural Machine Translation (NMT) to improve the modelling of tag sequences. Seq2seq models are known to learn complex global dependencies while currently proposed approaches using linear conditional random fields (CRF) only model local tag dependencies. In this work, we introduce a seq2seq model tailored for DA classification using: a hierarchical encoder, a novel guided attention mechanism and beam search applied to both training and inference. Compared to the state of the art, our model does not require handcrafted features and is trained end-to-end. Furthermore, the proposed approach achieves an unmatched accuracy score of 85% on SwDA, and state-of-the-art accuracy score of 91.6% on MRDA. The content of this chapter has been publicly released and can be found on an open preprint server: P. C OLOMBO &E.C HAPUIS and collab. [2020].

Table 4

 4

	.1, DA classifica-

Table 4 .

 4

1 -Example of conversation from Switchboard Dialogue Act Corpus. A is speaking with B.

  The closest setting where seq2seq model have been successfully used is dependency parsing[LI and collab., 2018b], where output dependencies are crucial to achieve state-ofthe-art performance. In our work we adjust NMT techniques to the specifics of DA

	Thus, it can be
	formulated as sequence to sequence problem very similar to NMT.
	The general architecture of our seq2seq models [SUTSKEVER and collab., 2014] follows

a classical encoder-decoder approach with attention

[LUONG and collab., 2015]

.

We use GRU cells

[CHO and collab., 2014]

, since they are faster to train than LSTM ones

[JOZEFOWICZ and collab., 2015]

. Recent advances have improved both the learning and the inference process, producing sequences that are more coherent by means of sequence level losses

[START_REF] Wiseman | Sequence-to-sequence learning as beam-search optimization[END_REF]

and various beam search settings

[VIJAYAKUMAR and collab., 2016; WU and collab., 2016]

. classification.

Table 4 .

 4 2 -Statistics for MRDA and SwDA. |C| is the number of Dialogue Act classes, |V| is the vocabulary size. Training, Validation and Testing indicate the number of conversations (number of utterances) in the respective splits.

		|C| |V|	Train	Val	Test
	MRDA	5 10K	51(76K)	11(15K) 11(15K)
	SwDA	42 19K 1003(173K) 112(22K) 19(9K)
	Tags in SwDA: SwDA extends the Switchboard-1 corpus with tags from the SWBD-

DAMSL tagset. The 220 tags were reduced to 42 tags. The resulting tags include dialogue acts like statement-non-opinion, acknowledge, statement-opinion, agree/accept, etc. The average speaker turns per conversation, tokens per conversation, and tokens per utterance are 195.2, 1,237.8, and 7.0, respectively.

Table 4 .

 4 4 -Example of predicted sequence of tags taken from SwDA. seq2seq is our best performing model, CRF stands for Baseline CRF , G. is the groundtruth label.

	4.6.2 Experiment 2: What is the best encoder?
	In Table 4.5, we present the results of the three encoders presented in Section 4
	on both datasets. ForSwDA and MRDA, we observe that a seq2seq equipped with a
	hierarchical encoder outperforms models with Vanilla RNN encoder, while reducing
	the number of learned parameters.
	The VGRU D does not play well with the PersoHGRU encoder. When combined
	with a guided attention mechanism, the PersoHGRU exhibits competitive accuracy

Table 4 .

 4 5 -Accuracy on the dev set of the different encoder/decoder combination MRDA

andSwDA. ForSwDA, Wilcoxon test (10 runs with different seeds) has been performed for an HGRU encoder with a decoder with hard guided attention against an HGRU encoder with soft guided attention, soft guided attention, with attention, without attention pairwise tests exhibit p-value < 0.01.

Table 4 .

 4 [START_REF] Dialog | [END_REF] shows that this fine tuning steps improves the performances of 1% on SwDA (84% vs 85%) and 1.2% on MRDA (90.4% vs 91.6%).

			SwDA	RMDA
	B inf	B train	2525
	1		84.8 84.7 91.3 91.6
	2		84.9 84.8 91.3 91.6
	5		85.0 84.9 91.5 91.6

Table 4 .

 4 6 -Accuracy on the dev set of seq2seq model trained with sequence level loss. B train stands for the beam size during training, B inf for the one during inference 7 .For SwDA, Wilcoxon test (10 runs with different seeds) has been performed for B train = 2 and B inf = 2 against all other models. For MRDA, Wilcoxon test has been performed (10 runs with different seeds) for B train = 5 and B inf = 1 against all model with B train = 2.

seq2seq BEST : Our seq2seq BEST model is composed of a HGRU encoder and a decoder with hard guided attention finetuned with B train = 2 and B inf = 5 for SwDA and B train = 5 and B inf = 1 for SwDA.

Table 4

 4

	, 2018]

.7 shows the performances of best performing model seq2seq BEST on the test set. seq2seq BEST achieves an accuracy of 85% on the SwDA corpora. This model outperforms

[START_REF] Chen | Emotionlines: An emotion corpus of multi-party conversations[END_REF]

and

[RAHEJA and TETREAULT, 2019]

which achieve an accuracy of 82.9%. On MRDA, our best performing model reaches an accuracy of 91.6% where current state-of-the-art systems,

[START_REF] Chen | Emotionlines: An emotion corpus of multi-party conversations[END_REF] KUMAR and collab.

Table 4 .

 4 7 -Accuracy of our best models (seq2seq) and Baseline CRF on SwDA and MRDA test sets.

  are often medium-sized (e.g. Cornell Movie Corpus[START_REF] Lee | Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs[END_REF],Ubuntu [LOWE and collab., 2015], MultiWOz[BUDZIANOWSKI and collab., 2018a]). In our work, we focus on OpenSubtitles [LISON and TIEDE-MANN, 2016] 4 because (1) it contains spoken language, contrarily to the Ubuntu corpus[LOWE and collab., 2015] based on logs; (2) as Wizard of Oz [BUDZIANOWSKI and collab., 2018a] and Cornell Movie Dialog Corpus [DANESCU-NICULESCU-MIZIL

Table 5 .

 5 -RESOURCE DA AND E/S Avg SwDA MRDA DyDA DA MT Oasis DyDA e MELD s MELD e IEMO SEM

	CLASSIFICATION TASKS

1 -Statistics of datasets composing SILICONE. E stands for emotion label and S for sentiment label; ? stands for datasets with available official split. Sizes of Train, Val and Test are given in number of conversations.

(a) SILICONE DA (b) SILICONE S/E Figure 5.4 -Histograms showing the utterance length for each dataset of SILICONE. CHAPTER 5. A PRE-TRAINED MODEL FOR LOW-RESOURCE DA AND E/S

Table 5 .

 5

4 -Performances of different encoders when decoding using a MLP on SILICONE. The datasets are grouped by label type (DA vs E/S) and ordered by decreasing size. MT stands for Map Task, IEM for IEMOCAP and Sem for Semaine.

Table 5 .

 5 4 provides an exhaustive comparison of the different encoders over the SILICONE benchmark. As previously discussed, we adopt a plain MLP as a decoder to compare the different encoders. We show that SILICONE covers a set of challenging tasks as the best performing model achieves an average accuracy of 74.3. Moreover, we observe that despite having half the parameters of a BERT model, our proposed model achieves an average result that is 2% higher on the benchmark. SILICONE covers two different sequence labelling tasks: DA and E/S.InTable 5.4 and Table 5.3,

we can see that all models exhibit a consistently higher average accuracy (up to 14%) on DA tagging compared to E/S prediction. This performance drop could be explained by the different sizes of the corpora (see Table

5

.1). Despite having a larger number of utterances per label (u/l ), E/S tasks seem generally harder to tackle for the models. For example, on Oasis, where the u/l is inferior than those of most E/S datasets (MELD s , MELD e , IEMO and SEM), models consistently achieve better results. CHAPTER 5. A PRE-TRAINED MODEL FOR LOW-RESOURCE DA AND E/S CLASSIFICATION TASKS Figure 5.5 -A comparison of pre-trained encoders being fine-tuned on different percentage the training set of SEM. Validation and test set are fixed over all experiments, reported scores are averaged over 10 different random split.

  5. A PRE-TRAINED MODEL FOR LOW-RESOURCE DA AND E/S

			CLASSIFICATION TASKS
		Avg DA Avg E/S
	BERT (4 layers)	80.5	60.2
	HT(µ BERT°2layers )	80.5	61.1
	HT (µ u MLM )	80.8	64.0
	Table 5.5 -Results of ablation studies on SILICONE

  . Let u

	Index Speaker Monolingual Input	Multilingual Input
	0	A	Good afternoon.	Good afternoon.
	1	A	I'm here to see Assistant Director Harold Cooper. Je suis ici pour voir l'assistant directeurr Harold Cooper.
	2	B	Do you have an appointment?	Do you have an appointment?
	3	A	I do not.	Non.
	4	A	Tell him it's Raymond Reddington.	Dites lui que c'est Raymond Reddington.
				L i i be an input utterance and	ũL i i its cor-

Table 6 .

 6 1 -Example of automatically built input context from OPS.

Table 6 .

 6 2 -Statistics of the processed version of OPS.

		de	en	es	fr	it
	# movies	46.5K 446.5K 234.4K 127.2K 134.7K
	# conversations	1.8M	18.2M 10.0M	5.2M	4.2M
	# tokens	363.6M	3.7G	1.9G	1.0G	994.7M

  English, Spanish, German, French and Italian). Table6.2 gathers statistics on the considered multilingual version of OPS. To obtain conversations from OPS, we consider that two consecutive utterances are part of the same conversation if the inter-pausal unit[KOISO and collab., 1998](i.e silence between them) is shorter than ± T = 6s. If a conversation is shorter than the context size T, they are dropped and utterance are trimmed to 50 (for justification see Figure6.2). Obtaining aligned conversations We take advantage of the alignment files provided in OPS. They provide an alignment between utterances written in two different languages. It allows us to build aligned conversations with limited noise (solely high confidence alignments are kept). Statistics concerning the aligned conversations can be found in Table6.3 and an example of automatically aligned context can be found in Table6.1. The use of more advanced methods to obtain more fine-grained alignment (e.g word level alignment, span alignment inside an utterance) is left as future work.

	Collecting our pre-training corpus involves two steps: the first step consists of seg-menting the corpus into conversations, in the second step, we obtain aligned conver-sations. Conversation segmentation Ideal pre-training corpora should contain multilingual spoken language with dialogue structure. In our work, we focus on OPS [LISON and TIEDEMANN, 2016] 4 because it is the only free multilingual dialogue corpus (62 dif-ferent languages). After preprocessing (see subsection 5.2.5), OPS contains around 50M of conversations and approximately 8 billion of words from the five different de-es de-fr de-it en-es # utt. 23.4M 19.9M 17.1M 14.1M 63.5M # tokens. 217.3M 194.1M 167.0M 139.5M 590.9M en-fr en-it es-fr es-it fr-it # utt. 44.2M 36.7M 37.9M 31.4M 23.8M languages (i.e de-en # tokens. 413.7M 347.1M 362.1M 304.6M 248.5M

Table 6 .

 6 3 -Statistics of the processed version of the alignment files from OPS.

Table 6 .

 6 [START_REF] Dialog | [END_REF] -Ablation studies on pre-training data. We report the accuracy on MIAM for the mHT. mHT u (µ spoken

	Total

  Results demonstrate an overall higher accuracy when the pre-training is performed on spoken data. This supports the choice of OPS as pre-training corpora and demonstrates that the origin of the pre-training data matters. In Table6.8 we dissect the relative improvement brought by the different parts of the code-switched inspired losses and the architecture to better understand the relative importance of each component. Similarly to Chapter 5, we see that the hierarchical pre-training on spoken data (see mMUG)

Overall, pre-trained models achieve better results. Contrarily to what can be observed in some syntactic tagging tasks

[START_REF] Zhang | Language modeling teaches you more than translation does: Lessons learned through auxiliary syntactic task analysis[END_REF]

, for DA tagging pre-trained models achieve consistently better results on the full benchmark. This result of multilingual models confirms what is observed with monolingual data (see

[MEHRI and collab., 2019]

): pre-training is an efficient method to build accurate dialogue sequence labellers.

Comparison of pre-training losses

Table 6

 6 Table 6.8 -Accuracy of pre-trained and baseline encoders on MIAM. Models are divided

	CHAPTER 6. CROSS-LINGUAL PRE-TRAINING METHODS FOR SPOKEN DIALOG
		Toke. VM2 Map Task Dihana Loria Ilisten Total
	BERT	lang 54.7	66.4	86.0	50.2	74.9	66.4
	BERT -4layers	lang 52.8	66.2	85.8	55.2	76.2	67.2
	HR+ CRF	lang 49.7	63.1	85.8	73.4	75.2	69.4
	HR+ MLP	lang 51.3	63.0	85.6	58.9	75.0	66.8
	MUG Chapter 5	lang 54.0	66.4	99.0	79.0	74.8	74.6
	mBERT	multi 53.2	66.4	98.7	76.2	74.9	73.8
	mBERT -4layers	multi 52.7	66.2	98.0	75.1	75.0	73.4
	mHR+ CRF	multi 49.8	65.2	97.6	75.2	76.0	72.8
	mHR+ MLP	multi 51.0	65.7	97.8	75.2	76.0	73.1
	mMUG	multi 53.0	67.3	98.3	78.5	74.0	74.2
	mMUG + TMUG	multi 54.8	67.4	99.1	80.8	74.9	75.4
	mMUG + MMUG	multi 56.2	67.4	99.0	78.9	77.6	75.8
	mMUG + TMUG + MMUG multi 56.2	66.7	99.3	80.7	77.0	76.0

.7 gathers the results. Similarly conclusion than in section 6.4.1 can be drawn: pre-trained models achieve better results and the best performing model is obtained with mMUG+MMUG+TMUG.

.4 30.1 33.6 33.9 36.2 34.0 32.1 33.4

						-it fr-it Avg
	mBERT	31.2	28.0	28.0 27.6	28.4	33.0 32.1 35.1 31.0 28.7 30.3
	mBERT (4-layers)	30.7	28.7	28.2 27.1	28.7	33.1 30.9 35.1 30.1 28.1 30.1
	mHR	28.7	27.9	26.9 27.3	25.5	25.1 30.6 34.3 30.0 26.8 28.3
	mMUG	34.5	30.1	30.1 27.7	28.2	33.1 32.1 35.4 32.0 29.5 31.2
	mMUG + TMUG	34.0	32.0	32.2 29.1	28.3	32.9 32.4 35.1 33.0 29.3 31.8
	mMUG + MMUG	35.1	33.8	34.0 30.1	29.4	32.8 32.6 36.1 33.9 31.6 32.9
	mMUG + TMUG + MMUG 35.7	34.0	32.5 31

MMUG 68.3 27.4 21.2 68.9 27.8 18.3 69.3 27.1 17.9 67.4 25.3 17.4 70.2 30.0 18.7 Table

		.1 27.1 20.1 62.1 26.1 16.8 62.4 24.8 15.3 63.9 22.9 13.4 66.1 27.8 16.9
	mBERT (4-layers)	65.1 27.5 20.2 61.4 25.6 15.1 62.3 24.6 15.9 63.4 22.8 12.9 65.6 27.4 15.8
	mHR	65.0 27.1 20.0 60.3 25.0 15.2 61.0 23.9 14.7 63.0 22.9 13.0 65.4 27.3 15.8
	mMUG	66.9 28.0 20.0 65.9 26.4 16.3 66.7 26.4 16.4 66.2 25.2 17.2 68.9 28.9 17.2
	mMUG + TMUG	67.2 28.2 20.1 68.3 29.8 17.5 69.0 26.9 17.3 67.1 25.4 17.3 69.9 29.4 18.6
	mMUG + MMUG	66.9 28.1 20.7 68.1 26.7 18.0 68.7 26.9 17.5 67.2 25.2 17.4 69.7 29.4 18.6
	mMUG + TMUG +

.5 28.7 13.7 66.1 31.4 14.5 60.1 25.5 13.6 63.1 25.9 14.2Table 6 .

 6 11 -Results on the mNUR task with bilingual input context.

		4 12.3 57.1 23.5 12.1 57.8 27.9 12.2 54.2 22.1 11.2 58.1 22.9 12.5
	mBERT (4-layers)	57.8 23.2 12.1 57.1 23.4 11.9 57.1 27.6 12.1 55.1 22.0 11.1 58.9 22.6 12.7
	mHR	55.9 20.9 11.6 56.8 22.9 11.8 54.9 27.0 12.0 53.9 21.0 11.6 56.1 21.9 11.4
	mMUG	61.9 24.9 12.9 61.4 27.6 11.9 64.6 29.7 13.9 59.0 24.2 13.4 59.7 23.6 12.2
	mMUG + TMUG	62.9 25.2 14.3 62.7 27.8 12.9 64.9 29.9 13.8 60.1 25.1 13.5 61.5 25.8 13.1
	mMUG + MMUG	63.9 26.3 14.7 61.5 27.6 13.1 65.0 30.2 13.1 60.1 25.3 12.9 63.1 25.9 13.6
	mMUG + TMUG + MMUG 64.0 26.7 14.1 63

  Table 7.3 shows the punctuation marks distribution in MELD.

	Datasets	# Dialogues Train Val Test Train Val # Utterances Test
	MELD	1039 114 280 9989 1109 2610

Table 7

 7

		.1 -MELD description
	Speakers	Utterances	Emotions
	Joey	What is it ? Hey !	sadness
	Rachel	Really it 's nothing . I 'm just	sadness
	Joey	Rach come on , what ?	neutral
	Rachel	I 've just been thinking about how my baby	sadness
		and I are gon na be all alone .
	Joey	What are you talking about alone ? What	surprise
		about Ross ?
	Monica	What is it ? !	surprise
	Rachel	I do n't know ! But maybe if we keep that	disgust
		drawer shut , it 'll die .
	Monica	I ca n't believe we 're living here !	sadness
	The Teacher	Excellent ! What Rachel has shrewdly ob-	joy
		served here
	Phoebe	You completely stole my answer !	anger
	Rachel	Well , honey that was pretty obvious.	sadness
	Phoebe	Well how would you know ? ! You did n't	anger
		even read it !
	Table 7.2 -Some examples of dialogues from MELD dataset.

Table 7 .

 7 3 -Punctuation marks distribution over MELD splits. Bold number in parenthesis indicates the percentage of tokens.

  Table 7.8 we notice that S T+A p!p models achieve better performances on disgust and fear than models with S T p!p setting.

	Models	Setting	Emotions sadness surprise neutral joy anger disgust fear W-Avg F1
		S p!p	16.00	56.27	82.93	50.83 41.92	0.00	0.00	57.75
	CNN R	S ¬p!¬p	20.76	51.35	70.70	36.96 37.57	0.00	0.00	49.82
		S p!¬p	19.46	15.56	90.95	18.97	8.31	3.00	0.44	42.93
		S p!p	25.73	56.98	76.12	52.78 44.35	8.33	0.00	57.88
	CNN G	S ¬p!¬p	23.14	48.57	68.89	43.04 40.51	1.33	0.00	50.74
		S p!¬p	28.32	16.43	85.35	25.44 19.11	19.67	1.33	46.93
		S p!p	35.35	65.71	74.33	58.74 52.97	9.33	12.00	62.25
	BERT	S ¬p!¬p	32.65	58.17	67.43	52.72 48.69	5.00	10.67	55.84
		S p!¬p	34.92	27.78	80.87	31.40 25.69	11.00	20.00	51.20
		S p!p	32.22	62.22	77.44	62.52 49.27	15.33	9.78	62.86
	RoBERTa	S ¬p!¬p	38.59	44.68	78.20	37.77 32.52	16.00	22.67	55.04
		S p!¬p	36.86	30.24	85.60	30.26 24.79	19.00	15.11	52.81

In SwDA arround two third of the sentence have at least a AA or BB

T is an hyperparameter, experiments have shown that 5 leads to the best results.

In our work we rely on same pre-trained embedding word2vect[MIKOLOV and collab., 2013] instead of GloVe [PENNINGTON and collab., 2014].

This analysis can be supported by

5.1.1 in SwDA coder manual https://web.stanford.edu/ ~jurafsky/ws97/manual.august1.html

Further investigations with several persona based model inspired from the work of LI and collab.[2016b] shows the same poor improvement (in terms of accuracy)

The considered beam size are small compared to other applications[LI and collab., 2016a]. While increasing the beam size, we see that the beam search become very conservative[GIMPEL and collab., 2013] and tends to output labels highly represented in the training set (e.g., sd forSwDA).

Upon publication, we will release the code, models and especially the preprocessing scripts to replicate our results.

We split according to the code in https://github.com/NathanDuran/Maptask-Corpus.

We use a random split from https://github.com/NathanDuran/BT-Oasis-Corpus.

Comparison with concurrent work is more difficult as system performance heavily depends on the number of classes and label processing varies across studies[START_REF] Clavel | Sentiment analysis: from opinion mining to human-agent interaction[END_REF].

We consider the two first layer for a fair comparison based on the number of model parameters.

We investigate a similar setting for GAP which lead to poor results, the loss hit a plateau suggesting that objectives are competing against each other. More advanced optimisations techniques[SENER and KOLTUN, 2018] are left for future work.

We refer to code-switching at the utterance level, although it is more commonly studied at the word or span level[BANERJEE and collab.,

[START_REF] Agarap | Deep learning using rectified linear units (relu)[END_REF] BAWA and collab., 2020;[START_REF] Fairchild | Determiner-noun code-switching in spanish heritage speakers[END_REF] POPLACK, 1980] 2 Although our pre-training can be easily generalised to 62 languages, we use a limited number of languages to avoid exposure to the so-called "curse of multilinguality"[CONNEAU and collab., 2019]

In practice, we follow SANKAR and collab. [2019a] and set the context length to 5 consecutive utterances.

http://opus.nlpl.eu/OpenSubtitles-alt-v2018.php

To ensure fair comparison, contrarily to [MEHRI and collab., 2019] the pre-training is different from the evaluation tasks.

D is set to 9 according to[LOWE and collab., 2015]

https://deepset.ai/

https://github.com/google-research/bert/blob/master/multilingual.md

In this study, we focus on spoken dialogues and therefore do not consider the use of emoji

https://affective-meld.github.io/

However this observation should not be over interpreted as the MELD dataset does not gather enough examples for disgust and fear

Acknowledgment

LIST OF TABLES CHAPTER 4. A SEQ2SEQ MODEL FOR SEQUENCE LABELLING

The decoder (blue) is responsible to generate for each utterance aDA exploiting the last state of the encoder as initial hidden state.

Chapter 4 Conclusion

In this work, we have addressed the RQ1.1 and proposed a novel approach to the sequence labelling task in spoken dialogue. We have shown that our seq2seq model, using a newly devised guided attention mechanisms, achieves state-of-the-art results thanking its ability to better model global dependencies. However the proposed architecture need to be trained in a fully supervised manner and required large annotated corpora only available for DA.

Theses shortcomings have led to RQ1.2 and are addressed in the next chapter.

Results on SILICONE

This section gathers experiments performed on the SILICONE benchmark. We first analyse an appropriate choice for the decoder, which is selected over a set of experiments on our baseline encoders: a pre-trained BERT model and a hierarchical RNN-based encoder (HR). Since we focus on small-sized pre-trained representations, we limit the sizes of our pre-trained models to TINY and SMALL (see Table 5.7).

We then study the results of the baselines and our hierarchical transformer encoders (HT) on SILICONE along three axes: the accuracy of the models, the difference in performance between the E/S and the DA corpora, and the importance of pre-training.

As we aim to obtain robust representations, we do not perform an exhaustive grid search on the downstream tasks.

Experimental Hyper-parameters for SILICONE

For all models, we use a batch size of 64 and automatically select the best model on the validation set according to its loss. We do not perform exhaustive grid search either on the learning rate (that is set to 10 °4), nor on other hyper-parameters to perform a fair comparison between all the models. We use ADAMW [KINGMA and BA, 2014;LOSHCHILOV and HUTTER, 2017] with a linear scheduler on the learning rate and the number of warm-up steps is set to 100. For all model we used a tokenizer based on WordPiece [WU and collab., 2016]. We used GELU [HENDRYCKS and GIMPEL, 2016] activations and the dropout rate [SRIVASTAVA and collab., 2014] is set to 0.1. We report in Table 6.5 the main hyper-parameters used for our model pre-training.

Decoder Choice

Current research efforts focus on single label prediction, as it seems to be a natural choice for sequence labelling problems (subsection 5.2.1). Sequence labelling is usually performed with CRFs [CHEN and collab., 2018b;KUMAR and collab., 2018] and GRU decoding [COLOMBO and collab., 2020], however, it is not clear to what extent inter-label dependencies are already captured by the contextualised encoders, and whether a plain MLP decoder could achieve competitive results. As can be seen the GAP loss could require a finer choice of hyper-parameters). Finally, we see that supervising at both dialogue and utterance level helps for MLM 11 .

Multi level Supervision for pre-training

In this section, we illustrate the advantages of learning using several levels of supervision on small datasets. We fine-tune different model on SEM using different size of the training set. Results are shown in Figure 5.5. Overall we see that introducing sequence level supervision induces a consistent improvement on SEM.

Improvement over pre-training

In this experiment we illustrate how pre-training improves performance on SEM

Numerical Results

In this section, we empirically demonstrate the effectiveness of our code-switched inspired pre-training on downstream tasks involving both monolingual and multilingual input context.

Monolingual input context

DA labeling Global analysis. Table 6.8 reports the results of the different models on MIAM. Table 6.8 is composed of two distinct groups of models: language specifics models (with language-specific tokenizers) and multilingual models (with a multilingual tokenizer denoted with a m before the model name). Overall, we observe that mMUG augmented with both TMUG and MMUG gets a boost in performance (1.8% compared to mMUG and 2.6% compared to a mBERT model with a similar number of parameters). This result shows that the model benefits from being exposed to aligned bilingual conversations and that our proposed losses (i.e. TMUG and MMUG) are useful to help the model to better catch contextual information for DA labeling.

Language-specific v.s. multilingual models. By comparing the performances of HR (with either a CRF or MLP decoder), we can notice that for these models on DA labelling it is better to use a multilingual tokenizer. As multilingual tokenizers are not tailored for a specific language and have roughly twice as many tokens than their language-specific counterparts, one would expect that models trained from scratch using language-specific tokenizers would achieve better results. We believe this result is related to the spoken nature of MIAM and further investigations are left as future work. Recent work RUST and collab. [2020] has demonstrated that pre-trained language models with language-specific tokenizers achieve better results than those using multilingual tokenizers. This result could explain the higher accuracy achieved by the language-specific versions of MUG compared to mMUG. We additionally observe that some language-specific versions of BERT achieve lower results (e.g Dihana, Loria) than the multilingual version which could suggest that these pre-trained BERT might be less carefully trained than the multilingual one; in the next part of the analysis we will only use multilingual tokenizers.

Next utterance retrieval

In this section, we evaluate our representations on NUR using a monolingual input context. As we use 9 distractors, a random classifier would achieve 0.10 for R@1, 0.20 for R@2 and 0.50 for R@5. The results are presented in Table 6.10. When comparing the accuracy obtained by the baselines models (e.g mBERT, mBERT (4-layers) and HR) and our model using the contextual losses at the context level for pre-training (i.e MUG, TMUG and MMUG) we observe a consistent improvement.

Takeaways Across all the three considered tasks, we observe that the models pre-trained with our losses achieve better performances. We believe it is indicative of the validity of our pre-training.

Multilingual input context

In this section, we present the results on the downstream tasks with multilingual input context.

Multilingual inconsistency identification

Table 6.9 gathers the results for the mII with bilingual input context. As previously a random baseline would achieve an accuracy of 20%. As expected predicting inconsistency with bilingual context is more challenging than with a monolingual context: we observe a drop in performance of around 15% for all methods including the multilingual BERT. Our results confirm the observation of WINATA and collab. [2021]: multilingual pre-training does not guarantee good performance in code switched data. However, we observe that the losses, by exposing the model with bilingual context, obtain a large boost (absolute improvement of 6% which correspond to a relative boost of more than 20%). We also observe that MUG+MMUG+TMUG outperforms mBERT on all pairs, with fewer parameters.

Multilingual next utterance retrieval

The results on bilingual context for mNUR are presented in Table 6.11. mNUR is more challenging than NUR. Overall, we observe a strong gain in performance when exposing the model to bilingual context (gain over 9% absolute point in R@5). Takeaways: These results show that our code-switched inspired losses help to learn better representations in a particularly effective way in the case of multilingual input context.

Chapter 6 Conclusion

In this work, we showed the strong impact of the studied set of pre-training losses, tailored for multilingual spoken dialogue data. These losses induce a significant improvement by leveraging the parallel conversations extracted from OpenSubtitles. They achieved solid results on the new presented benchmark for dialogue act tagging (MIAM), available on 5 European languages. We also observe a significant improvement on the two new introduced tasks: multilingual inconsistency identification and multilingual next utterance retrieval.

CHAPTER 7. MULTIMODAL NATURE OF PUNCTUATION: APPLICATION TO EMOTION RECOGNITION

and prosody have the same function. From this perspective, when dealing with written-to-be-spoken texts or speech transcripts, punctuation can be viewed as a metalanguage that transcribes prosodic cues such as intonation and pausing into a written language.

The link between prosody and the emotion conveyed in spoken language has been well established [JUSLIN and LAUKKA, 2003;[START_REF] Mozziconacci | Prosody and emotions[END_REF]. Therefore, prosody has become a key component of speech emotion recognition or multimodal emotion recognition systems [BAGHER ZADEH and collab., 2018;BALTRUŠAITIS and collab., 2019;PORIA and collab., 2018;ZADEH and collab., 2016]. When considering only the textual modality, changing the punctuation sequence of an utterance can drastically alter the prosody related to it and in turn the perceived emotion. For example, the sentence "Hello, how are you today ?" can be read as neutral while adding exclamation marks as in "Hello ! how are you today ? !" is more dynamic and enthusiastic, thus conveying the utterance of a joyful tone. If humans are sensitive to such phenomena, studies on how they may affect deep learning models are still lacking. Whereas punctuation marks have been used as hand crafted features in sentiment analysis or sarcasm recognition [START_REF] Agrawal | Kea: Sentiment analysis of phrases within short texts[END_REF]RUBIN and collab., 2016], in most NLP tasks they are removed during pre-processing, as NLP models achieve higher accuracy on syntactically correct utterances [BAGHER ZADEH and collab., 2018;DELBROUCK and collab., 2020;GU and collab., 2018;KRATZWALD and collab., 2018;RAHMAN and collab., 2020]. Moreover a recent work [EK and collab., 2020] shows that in Natural Language Inference (NLI) neither BERT [DEVLIN and collab., 2019] nor RNN-based models take into account relevant changes in punctuation marks, but BERT seems robust to irrelevant changes. However, this study mixes data from different genres, i.e written and spoken English and focuses on the semantic aspect of punctuation. To the best of our knowledge, there is no quantitative study taking into account the influence of punctuation marks on emotion recognition task for spoken data. In Automatic Speech Recognition (ASR), efforts have been made toward punctuation prediction [CHO and collab., 2017;COURTLAND and collab., 2020;[START_REF] Lu | Better punctuation prediction with dynamic conditional random fields[END_REF]. However, these works are mostly motivated by text readability, or downstream tasks such as part-of-speech (POS) tagging or Natural Language Translation -which focus on the semantic side of punctuation. Hence, the impact of punctuation as a prosodic marker on modern deep learning architectures in emotion recognition tasks remains overlooked.

In this work, we investigate the ability of different state-of-the art neural architectures to leverage punctuation for emotion recognition tasks. We address the following research questions in two stages: (1) Are punctuation marks discriminating features for emotion recognition systems based on speech transcripts ? (2) Can punctuation marks be a substitute for acoustic modality ? In order to answer these two questions, we first present the data and the distribution of punctuation marks among emotion categories, as well as the investigated neural models section 7.2. Then, we compare models trained and tested w/o punctuation marks and w/o acoustic information for emotion recognition tasks and provide an in-depth analysis of the obtained results section 7.3.

CHAPTER 7. MULTIMODAL NATURE OF PUNCTUATION: APPLICATION TO EMOTION RECOGNITION

composing the utterances are fed into a convolutional neural network (CNN). We denote by CNN R and CNN G the models corresponding to words embeddings being set randomly and with Glove embedding [PENNINGTON and collab., 2014] respectively.

Transformer-based architectures They outperform classical architectures relying on word embeddings such as glove. We use BERT and RoBERTa [LIU and collab., 2019] as they achieve competitive results and are used as sentence embedding extractors for state-of-the-art methods [GHOSAL and collab., 2020;LI and collab., 2020].

Sentence embeddings are obtained through the representation of the [CLS] token provided by the last layer. For all our baseline classification models, the sentence embedding is fed to a multilayer perceptron, trained with dropout [SRIVASTAVA and collab., 2014]. We do not use the contextual information nor any additional information from the other modalities in order to identify the emotion or sentiment of an utterance. To obtain multimodal representations, we concatenate the sentence representations given by the aforementioned models with audio features provided with the MELD dataset.

Training Details We use dropout [SRIVASTAVA and collab., 2014] MELD corpus is lower cased at training and testing time. We use tokenizer introduced by DEVLIN and collab.

[2019] and used the SentencePiece algorithm [KUDO and RICHARDSON, 2018]. We introduce a variant of MELD where we remove all punctuation marks, and that we note ¬p. Similarly, we denote by p the original dataset (keeping all punctuation marks). We then define four settings for our experiments: that <!> and <?> are the most discriminating features, with <!> being more related to anger or joy and <?> to surprise. On the other hand, dot and comma seem to have no effect on the prediction, as their use follows both a syntactic and prosodic approach leading to poor improvement. This section presents the ablation study of punctuation marks for BERT. We obtain similar results for other baselines. In order to showcase the influence of each punctuation mark on the prediction we train BERT To highlight the impact that some punctuation marks may have on predictions, we systematically append utterances of the original corpus with the most discriminating punctuation marks. Results for BERT are presented in Figure 7.1. Overall, this completely polarizes neutral utterances between surprise/joy/anger or surprise/anger, when <!> or <?> are used respectively. This behavior is expected, as adding punctuation marks will obviously polarize neutral sentences for human readers too: 'I don't know' is perceived as neutral but 'I don't know !' would be perceived as anger. However, we notice that punctuation marks fail at transcribing prosodic Chapter 8

Experiments and Analysis

Conclusions, Limitations and Future Work

Conclusions

Throughout this thesis, we have presented different neural methods for spoken dialogue understanding. A particular emphasis has been placed on DA and E/S classification. We explored and developed new methods in various scenarios i.e. monolingual, lack of annotated data, multi-lingual and multimodal.

Part II is related to RQ1. In this part, we focused on monolingual conversational data (in English) and addressed the problem of sequence labelling. In its formulation (see Chapter 3), this problem is very similar to NMT problems. Thus, our first contribution presented in Chapter 4 leveraged these similarities, as we described a dialogue as a sequence of utterances to be mapped into a sequence of tags. This allowed us to apply NMT methods and architectures such as Seq2seq models widely used in this field. Compared to previous works, this approach gets rid of the currently used CRF and handcrafted features and allows for an end-to-end approach relying solely on neural networks. We proposed an encoder tailored to the hierarchical nature of dialogue and a novel attention guided mechanism. Moreover, the small size of the output space allowed us to apply a beam search during both training and inference.

We showcase new SOTA results on SwDA and competitive results on MRDA.

However, in this approach, most of the network is trained from scratch which prevents us from using small annotated corpora as it is often the case with datasets labelled in E/S. Meanwhile, architectures pre-trained on large corpora (e.g. BERT, RoBERTa, GPT) have proved to be very effective for transferring generic representations of text to low-resource tasks. Nevertheless, these representations are pretained on written datasets and have been limited to the sentence level. In Chapter 5,we addressed these limitations and proposed a pre-trained model tailored to spoken dialogues. Our solution involves a hierarchical transformer which is pre-trained on OpenSubtitles, a large collection of spoken dialogue transcripts from movies and series. We adapted two well-known pre-training objectives to the encoder hierarchy. As the evaluation of such representations is challenging, we collected a new benchmark called SILICONE which gathers 10 datasets of different sizes that are annotated in DA and E/S. We demonstrated that our approach is suited for small and medium-size datasets and allows us for a reduction of the number of parameters while achieving better performances.

Part III is related to RQ2 and we extend the work presented in Part II in two different