Keywords: machine translation, user-generated content, non-canonical language, natural language processing

The rapid advancements in telecommunications over the past few decades have revolutionized the way people exchange information. Thanks to these advancements, the average user can now communicate with others across the globe in real-time and with minimal delay. With approximately 60% of the global population having Internet access, billions of individuals interact by sharing user-generated content (UGC) in various forms. This kind of content, which often includes reviews and opinions, provides a valuable source of information, offering a comprehensive view of global trends. Machine Translation (MT) plays a vital role in enabling smooth communication and facilitating the automatic processing of UGC for data mining purposes. However, translating UGC presents unique challenges compared to translating traditional text. UGC is highly productive and exhibits various phenomena such as repeated characters, typographical errors, contractions, jargon, and unconventional sentence structures. These specificities lead to a significant number of Out-of-Vocabulary tokens (OOVs) and rare sequences, which pose problems since they are not adequately represented in the standard parallel corpora used to train MT models. Additionally, conventional domain adaptation techniques like fine-tuning have limited success in addressing these challenges. They suffer from per-

Résumé: Les avancées rapides des télécommunications au cours des dernières décennies ont révolutionné la manière dont les gens échangent des informations. Grâce à ces progrès, l'utilisateur moyen peut désormais communiquer avec d'autres personnes à travers le monde en temps réel et avec un délai minimal. Avec environ 60% de la population mondiale ayant accès à Internet, des milliards d'individus interagissent en partageant du contenu généré par les utilisateurs (UGC) sous diverses formes. Ce type de contenu, qui comprend souvent des critiques et des opinions, constitue une source précieuse d'informations, offrant une vue d'ensemble des tendances mondiales. La traduction automatique joue un rôle vital en permettant une communication fluide et en facilitant le traitement automatique de l'UGC à des fins d'exploration de données. Cependant, la traduction des UGC présente des défis uniques par rapport à la traduction d'un texte traditionnel. L'UGC est très productif et présente divers phénomènes tels que des caractères répétés, des erreurs typographiques, des contractions, du jargon et des structures de phrases non conventionnelles. Ces spécificités entraînent un nombre important de mots hors vocabulaire (OOV) et de séquences rares, qui posent des problèmes car ils ne sont pas représentés de manière adéquate dans les corpus parallèles standard utilisés pour entraîner les modèles de traduction automatique. En outre, les techniques conventionnelles d'adaptation au domaine, telles que le "fine-tuning", n'ont qu'un succès limité dans la résolution de ces problèmes. Elles souffrent d'une dégradation des performances lorsqu'elles sont appliquées aux données du domaine et ne sont pas en mesure de suivre l'évolution constante de la nature de l'UGC. Dans cette étude, nous nous concentrons sur la tâche de traduction automatique des UGC dans le scénario "zero-shot", où nous nous abstenons d'utiliser des données d'apprentissage spécifiques aux UGC. Notre objectif est de développer des architectures de traduction automatique plus généralisées, capables de gérer le "distributional shift", inhérente à l'évaluation de la traduction des UGC. Dans la phase initiale de notre recherche, nous avons consacré nos efforts à l'identification et à la quantification des spécificités de l'UGC qui entravent la performance de la traduction. Nous avons également créé des cadres d'évaluation et des collections de données pour nous aider dans cette tâche. À l'aide de modèles "off-theshelf", nous étudions les difficultés rencontrées par les systèmes de traduction automatique lorsqu'ils traduisent des UGC et nous établissons un lien entre les erreurs et les mécanismes sous-jacents. Ensuite, nous nous penchons sur l'étude et la proposition de différentes méthodes pour relever les défis posés par l'UGC. Ces méthodes comprennent l'exploration des pipelines de normalisation, l'emploi de techniques de tokenisation plus granulaires et l'utilisation de modèles de variables latentes pour améliorer la robustesse des systèmes de traduction automatique. Pour chacune de ces approches, nous évaluons systématiquement les performances et la robustesse des systèmes, nous effectuons une analyse détaillée des erreurs et nous proposons des pistes prometteuses pour aborder la traduction automatique des UGC dans une évaluation "zero-shot".

formance degradation when applied to indomain data and are unable to keep up with the ever-evolving nature of UGC. In this study, we focus on the task of automatically translating UGC in the zero-shot scenario, where we restrain from using any UGC-specific training data. Our aim is to develop more generalized MT architectures that can handle the distributional drift inherent in UGC. In the initial phase of our research, we dedicated our efforts to identifying and quantifying the specificities of UGC that hinder translation performance. We have also created evaluation frameworks and data collections to aid in this endeavor. Using off-theshelf models, we investigate the challenges faced by MT systems when translating UGC and link the errors to their underlying mechanisms. Subsequently, we delve into the study and proposal of different methods to address the challenges posed by UGC. These methods include exploring normalization pipelines, employing more granular tokenization techniques, and utilizing latent variable models to enhance the robustness of MT systems. For each of these approaches, we systematically evaluate the performance and robustness of the systems, conduct a detailed error analysis, and offer insights into promising avenues for tackling the automatic translation of UGC in the zero-shot setting.

I am deeply grateful to the LISN and Inria labs for making my time as a Ph.D. candidate such an enjoyable and enriching experience. I am extremely fortunate to have crossed paths with amazing and talented colleagues and researchers. I will always cherish the time I spent with Aina and Leonardo and I thank them for their companionship, valuable perspectives and help during this journey. I would also like to thank Christian for his kindness, friendship and unconditional support. I was also very lucky to met inspiring and experienced fellows it the beginning of my Ph.D., in particular, Matthieu, Rachel, Lauriane, Gaël, Pooyan and Yoann; I am thankful for their advice and for the enlightening discussions we had, which greatly contributed to this work. My profound gratitude also goes to Arij and Wissam, for their help, creative ideas and rich insights.

I must acknowledge the financial support provided by the ANR's ParSiTi project, without which this research would not have been possible. Additionally, I am grateful for the computational resources from the IDRIS (LabIA and Jean-Zay platforms) and the Inria's clusters (RIOC and CLEPS), which were instrumental in this research work.

The support from my friends was a determinant force during my Ph.D. experience. A i special mention goes to Natalio for his technical advice on the aesthetics of the figures in this dissertation.

Finally, I want to thank my family for their strong support and loving care, which have been the primary motivations behind my academic journey. I owe this work to them.

Summary

In this work, we address the automatic translation of user-generated content (UGC), that is to say, the content published by users on online platforms, such as social media discussion channels or public user reviews. Translating UGC is both a necessary step to make this data amenable to automatic processing and a way to test the robustness of NLP pipelines to this kind of content: the way we now communicate through social networks has resulted in a fundamental change in the quantity and fluidity of exchanges, which makes UGC a particularly rich source of information: by automatically processing these massive, unstructured, and diverse texts, we can provide insights of large-scale trends in -virtually -real-time if (and only if) we are able to cope with the extreme, ever-evolving lexical and morphological variability that characterize this kind of content.

At a time when neural networks and, more specifically, transformers, seem to provide a solution to all NLP problems, translating UGC also constitutes an important contribution to the question of identifying the limits of this architecture, a problematic currently at the heart of many works in the NLP community. Indeed, due its high productivity, its multi-lingual, language-specific and topic-specific nature, UGC is the epitome of expecting endless possibilities. Deploying MT systems that are capable of handling UGC is highly challenging as this kind of content is not present in the corpora usually considered to train MT systems and the ever-evolving nature of UGC, as well as the costs of cleaning and annotating UGC data do not allow us to build the parallel corpora necessary to learn or even to fine-tune a neural translation model. This is why we put ourselves in a particularly difficult zero-shot scenario likely to highlight the limits of NMT.

We start this dissertation by studying UGC specificities that make it different from standard and widely-available parallel corpora. In this respect, we investigate distinct UGC linguistic features that we later link to translation quality. We analyze and compare the prediction errors that occur when translating UGC using different off-the-shelf MT baselines.

As a first approach to the task at hand, we explore UGC normalization using automatically extracted phonetization to take into account phonemic information not contained in the corpus.

Moreover, phonemic similarities often lead to typographical errors and compressed/abbreviated phonemic forms, which we study and aim to cope with. Subsequently, we explore the robustness properties of MT to UGC when using different translation granularities (e.g. character-based or subword-based tokenization) and end-to-end data-driven tokenization rules. To assess the robustness of such models, we built an UGC evaluation framework that can isolate each of UGC characteristic to assess their individual impact on automatic translation. Finally, in the last part of this research, we explore, compare and propose MT architectures with more robust learning representation by means of variational inference, and study how such models can enforce generalization over UGC by alleviating overfitting.

Résumé

Dans ce travail, nous abordons la traduction automatique du contenu généré par l'utilisateur (UGC), c'est-à-dire le contenu publié par les utilisateurs sur des plateformes en ligne, comme les canaux de discussion des médias sociaux ou commentaires publiques d'utilisateurs. La traduction de l'UGC est à la fois une étape nécessaire pour rendre ces données exploitables pour un traitement automatique et un moyen de tester la robustesse des pipelines NLP pour ce type de contenu. En outre, en s'attelant à cette tâche, la technologie s'attaque à la barrière linguistique, permettant une conversation plus fluide entre les locuteurs du monde entier, tout en étant capable d'utiliser la langue avec laquelle les utilisateurs sont plus à l'aise. En effet, les réseaux sociaux ont changé notre façon de communiquer, permettant une portée et une fluidité d'échange sans précédent. Cela fait que l'UGC constitue une source d'information particulièrement riche : en traitant automatiquement ces textes massifs, non structurés et diversifiés, nous pouvons fournir des tendances à grande échelle en -virtuellement -temps réel si (et seulement si) nous sommes capables de faire face à l'extrême variabilité lexicale et morphologique, en constante évolution, qui caractérise ces textes.

En raison de sa productivité élevée, souvent multilingue, spécifique à une langue ou un ou spécifique à un sujet, et dont les acteurs sont des millions de pairs décentralisés à travers le monde, la traduction des UGC est l'exemple même de l'attente de possibilités infinies. Ceci constitue un scénario particulièrement difficile pour déployer des systèmes de MT capables de traiter le UGC car ses propriétés ne sont pas présentes dans les corpus standard, largement utilisés pour entraîner les systèmes de traduction automatique. De plus, la grande diversité des UGC et la présence de formes émergentes de communication, ainsi que les coûts de nettoyage et d'annotation des données de l'UGC, imposent de fortes contraintes à la production de grands corpus parallèles de ces textes pour l'entraînement ou l'adaptation au domaine. v Nous rapportons et analysons les principaux problèmes des architectures de MT actuelles lors du traitement de l'UGC et nous orientons notre étude pour aborder cette tâche sous une évaluation détaillée, en rapportant systématiquement les résultats sur des tests UGC qui ont une nature et source diverses, ainsi qu'en contraignant nos ressources d'entraînement à des corpus standards et bien formatés. Dans cette ligne de pensée, nous nous concentrons principalement sur les propriétés de robustesse des modèles de MT et visons à proposer des systèmes de traduction plus intrinsèquement robustes et généraux.

Nous commençons cette thèse en étudiant les spécificités des UGC qui le rendent différents des corpus parallèles standard et largement disponibles. A cet égard, nous étudions les traits linguistiques distincts de l'UGC que nous lions ensuite à l'impact sur la performance lors de la traduction de ce type de texte. Nous analysons et comparons les erreurs de prédiction qui se produisent lors de la traduction d'UGC à l'aide de différents systèmes baselines de MT standard.

Comme première approche de la tâche à accomplir, nous explorons la normalisation de l'UGC à l'aide de la phonétisation extraite automatiquement afin de prendre en compte les informations phonétiques non contenues dans le corpus. De plus, les similarités phonémiques conduisent souvent à des erreurs typographiques et à des formes phonétiques compressées/abrégées, que nous étudions et cherchons à résoudre. Par la suite, nous explorons les propriétés de robustesse de la MT à l'UGC lors de l'utilisation de différentes granularités de traduction et des règles de tokenisation apprises de bout en bout; et nous élaborons sur les capacités de vocabulaire ouvert des systèmes MT au niveau des caractères. Pour évaluer la robustesse de ces modèles, nous avons conçu un cadre d'évaluation de l'UGC qui peut isoler chacun des phénomènes de l'UGC annotés afin d'évaluer leur impact individuel sur la traduction automatique. Enfin, dans la dernière partie de cette recherche, nous explorons, comparons et proposons des architectures de traduction automatique qui ont pour but d'apprendre représentations robustes en utilisant des méthodes d'inférence variationnelle qui visent la généralisation sur l'UGC en réduisant l'overfitting.

List of Figures

1.1 Typical social media thread initiated by a seed photo and its automatic translation.

Inspired from a real conversation about a series of demonstrations that took place on Greece. Bing was used (on the 14/05/2015) as it was then the official MT engine for Twitter and Facebook. . . . . . . . . . . . . . . . . . . . . . . . . . . .

List of Tables

3.1

Examples showing common noisy phenomena in UGC [START_REF] Seddah | The french social media bank: a treebank of noisy user generated content[END_REF]. . .

Chapter 1

Introduction

In this chapter, we introduce the scope, questions, and contributions of this dissertation, before presenting the main scientific context in which this work takes place. We will start by describing the interest of correctly processing automatically User-Generated Content (UGC), that is to say the content published by users of online platforms, such as social media discussion channels.

We then establish the set of questions that will be addressed throughout this dissertation, the outline of its contents, and the contributions of this study.

Context

In recent decades, forms and channels of communication have undergone a steep change following the revolutionary onset of the Internet and its associated technologies. This has given regular users access to a vast and increasing number of platforms and media for content exchange, and a reach and organized information sharing that is unprecedented in history. The large amount of data on the Internet needs to be automatically processed to take full advantage of it.

In this work, we specifically focus on written forms of User-Generated Content (UGC) and study how Natural Language Processing (NLP) methods perform when treating such a kind of text. More precisely, we focus on Machine Translation (MT), a fundamental task to tackle in order to facilitate and ensure information sharing between speakers of different languages.

Concretely, working on UGC MT is putting efforts towards making communication and information exchange more fluid between individuals, as millions of users interact, employing hundreds of different natural languages and dialects every day. In addition, the immense and diverse volume of UGC can be regarded as a rich and ubiquitous source of data. As such, automatically processing these texts is of great interest both from an economic and societal point of view, as we will see in Chapter 3. Moreover, online communication catalyzed the rapid and massive emergence of new forms of texts accompanied by increased volume in multilingual user interaction. In this context, the "Parsing the Impossible: Translating the Improbable" (ParSiTi) project (ANR-16-CE33-0021), which funded this doctoral work, seeks to adapt current language technologies to cope with the challenges raised by UGC. Indeed, most of the NLP research mainly focuses on clean edited high-resource texts, overseeing the non-canonical, multilingual and contextual nature of such information streams and the ParSiTi project, identifies two main scientific challenges related to the automatic processing of UGC. The first one is the lack of generalization of current NLP models, which emerges from the shallow or void learning of linguistic knowledge in state-of-the-art NLP models.

As these models consider tokens (i.e. phrases, words our sub-word units) as unstructured discrete units, they will consider the UGC French sentence "ki ca ?" and its canonical version "qui ça ?" as two unrelated sentences, whereas the average French-speaking user could easily identify the similarities (phonemic resemblance for the first token, "ki", and lack of cedilla for "ca"), and thus correctly interpret the phrase. There are also multilingual and strongly contextual utterances in UGC, which can, for instance, be seen in Table 1.1, where understanding the user's discussion is challenging without the context given by the photo. Specifically, the main focus of this work, UGC automatic translation, is an important problem for open and real-world MT systems, which must be able to cope with such arbitrarily diverse text constructs that ultimately trigger the apparition of out-of-vocabulary (OOV) tokens such as non-existent words or foreign alphabets. The lack of MT robustness has led to unfortunate mistranslations (sometimes at large scale) in the past, as we will discuss further in Chapter 3.

Our approach

In this research work, we intend to design MT architectures that are robust to out-of-distribution (OOD) texts. More specifically, we will consider two kinds of OOD texts: out-of-domain canonical texts and noisy UGC. As we will discuss in Chapter 3, UGC is a multi-domain type of text with (@rigolboche)

Original source

Bing © translation ! T'as vu il l'a bien cherché wsh #AperoChezRicard ! Did you see he looked for it wsh #AperoChezRicard ! +10000, shah! ! +10000, shah! ! tabuz, lavé rien fé ! tabuz, washed nothing fe ! ki ca ? lemecousonchien ?

! ki ca ? theguyorhisdog? ! Wtf is wrong with him ? #PETA4EVER

! Wtf is wrong with him ? #PETA4EVER. ! ki ca ? le chien ? looool ! ki ca ? the dog? looool Figure 1.1: Typical social media thread initiated by a seed photo and its automatic translation. Inspired from a real conversation about a series of demonstrations that took place on Greece. Bing was used (on the 14/05/2015) as it was then the official MT engine for Twitter and Facebook.

many specificities.

To achieve this goal, we will use, throughout this work, a consistent experimental protocol in which we systematically evaluate our models on both canonical OOD (i.e. different domain from the train data), and OOD UGC test sets. Such an evaluation protocol, in addition to reporting results of all our systems on in-domain test sets, is intended to assess whether the models developed for translating UGC are still able to translate canonical corpora.

Furthermore, since we aim to improve robustness to a wide set of OOD conditions, we decided not to use any target-specific information, and we have restricted ourselves to develop only zero-shot methods. Specifically, we assume our systems to have no access to UGC during training. Indeed, because of the vast diversity of UGC specificities, it is impossible to collect all possible variations in a single corpus and UGC always contains new emergent forms of written expression: for instance, Martínez Alonso et al. (2016) shows that normalization schemes designed for UGC data collected in 2011 were not suitable for data collected in 2014. It is therefore necessary to develop models that perform well, not only on a given UGC corpus, but that will continue to do so in the future or under different circumstances (platform, language, domain, etc.) without constantly annotating data, which is a particularly challenging task. That is why, our study focuses on characterizing and addressing UGC performance from a robustness perspective rather than using target-distribution information, for instance, by fine-tuning pretrained models on UGC data. Modeling and explaining MT performance on noisy UGC remains today a challenge. To address this aspect, we introduce an extensive UGC evaluation framework to assess the detrimental impact of UGC's distinct specificities. It is noteworthy that, when this doctoral research work started, a lack of UGC evaluation resources in the literature prevails, making it difficult to study their impact on NLP tasks. Since, a rather limited number of parallel corpora and evaluation frameworks (including ours) have been proposed. The parallel test set annotated with UGC specificities we developed during this thesis in Chapter 6 is one of the main contributions of this work.

Challenges raised by UGC: questions to address

Now that the scope of our work has been introduced and that we have stated the premises and interests of this study, we will detail the plan of this report and our main contributions. In a first step, we identify and quantify the impact of UGC specificities on state-of-the-art MT systems.

Then, we propose and discuss different approaches to address these problems, ranging from pre-processing and normalization via a phonetization process, to considering different NMT architectures that could deal with the noise present in our evaluation. In all of our experiments, we systematically show an error analysis to highlight and explain the caveats and the observed improvements of our methods. Now we describe the research questions raised by the translation of UGC that we address in this work.

How does UGC impact translation quality?

At first, we want to evaluate the impact of UGC specificities on MT translation quality. To do so, we review, in Chapter 2, popular state-of-the-art MT architectures. We have considered both classical phrase-based models and models based on neural networks. Our first results raise several questions:

• What is the difference between phrase-based and neural machine translation systems, and how comparably robust are they to UGC?

• How do NMT systems behave when processing such kinds of texts and can translation errors be linked to their architecture?

In Chapter 4, we compare the MT predictions of UGC and quantitatively assess the performance of different MT systems and evaluate their robustness. In the second part of this chapter, we study the internal working of NMT systems to explain some of the observed errors as well as models behavior.

Which UGC specificities result in translation errors? The work described in Chapters 4 and 6 aims to characterize which of the specificities, described in Chapter 3, make translation more complicated, thus leading to a lower translation quality. More precisely, we address the following research questions:

• Do UGC's specificities distribution conditions performance for a given NMT architecture?

• How much do such UGC specificities impact translation quality individually?

In this work, the potential interactions between different types of UGC are addressed carefully by isolating and documenting translation quality detriment caused by them.

What kind of methods can be used to cope with UGC specificities for zero-shot transla-

tion? To answer this question, we propose and explore normalization methods and promising NMT architectures to overcome the variability of UGC due to its inherent links to users' creativity.

In the scope of this work, we keep a systematic effort toward studying zero-shot methods, i.e. without using neither parallel nor monolingual UGC. Some questions addressed in this part of the thesis are:

• Can supplementary natural language information, such as phonetization, provide a solution to the challenges raised by the translation of UGC?

• Can subword-level NMT models' properties take advantage of morphological information to resolve OOV occurrences imposed by UGC?

• Does the use of latent variables in translation models reduce the impact of the UGC noise on the translation quality?

Under the zero-shot translation condition, we intend to review methods that augment the capacity of MT models by either providing information, as the model described in Chapter 5 that makes use of automatically-extracted phonetization or leveraging on subword morphological features, as we explore in Chapter 6. Finally, in Chapter 7, we explored latent-vector models, which, notably, take into consideration stochastic perturbations to the model, and we give insights on their robustness to UGC.

Contributions of this work

Addressing these questions and analyzing the experimental evidence that they resulted in, resulted in several contributions:

• The caveats and considerations for the range of different MT systems when processing UGC were noticed and documented, including the effects that this noise has on NMT, specifically, the attention mechanisms. Additionally, we investigate and report a robustness perspective of such NMT models that we further link to specific morphosyntactic specificities in this kind of texts.

• A new automatic normalization pipeline was developed to account for phonetic writing and letter omission by leveraging pronunciation similarities in French, which proved to be especially effective for aiding in the translation of UGC short sentences.

• The effect of translation level for translating noisy texts was studied, and we describe new

considerations regarding the open-vocabulary capabilities of character-based MT involving the choice of vocabulary and show that it is possible to reduce the effect of rare or OOV characters in MT.

• A series of corpora and evaluation protocols were designed specifically to unveil the impact of the different UGC specificities introduced in Chapter 3 on translation quality. In order to do so, we released an evaluation framework to control the UGC specificity present during evaluation, notably preserving only one specificity at the time to disentangle their individual effects on performance. This resource also enabled us to account for the performance impact of a significantly wider range of such specificities than that was reported in the state-of-the-art literature, and provided a new and extensible framework for evaluating the robustness capabilities of any MT system.

• We explore models that use latent representations by including variational Bayesian methods in state-of-the-art MT models (namely Transformer) in order to assess whether such approaches can produce inherently more robust translation systems from noisy inputs.

We also propose a novel variational MT architecture that achieves better results compared to a strong baseline model, especially, when processing UGC. We show how these methods act as noise regularizer, enforcing the model to learn more robust representations.

We also study and showcase the semi-supervised training capabilities of these models by introducing a source-side sentence reconstruction loss term. Finally, we conduct translation experiments with our proposed model in a different language pair (Japanese-English) and

show that our findings also hold in this experimental setup.

Dissertation outline

In this work, we study the impact of UGC on MT and characterize this type of text in order to link translation quality to specific morphosyntactic phenomena presented in UGC. Then, we propose and discuss methods to cope with the noise present in UGC to bridge the performance gap between such texts and canonical texts.

We start by describing the Phrase-Based Statistical Machine Translation (PB-SMT) and NMT models we consider in this work. Furthermore, we discuss and explain MT notions and methods traditionally used to cope with noise and domain-drift scenarios by leveraging segmentation and translation granularity. We also explain the automatic metrics used to assess the performance and robustness capabilities of the MT systems used in this work.

We then discuss, in Chapter 3, the interest and challenges raised by UGC for NLP models, specifically translation models. We also explain, in this chapter, the importance from a user perspective of correctly processing UGC to extract useful information about a wide range of topics. We additionally highlight the importance of devising robust MT models for real-world applications, as they can hardly avoid UGC when deployed in public and real-world scenarios and applications.

Once the context surrounding this doctoral thesis, the difficulties to overcome when translating UGC, and the main MT and evaluation methods are introduced, we develop an in-depth study of our first MT baselines in Chapter 4. In this part, we describe the caveats of different MT systems when they are processing UGC, including the effects that this type of texts has on NMT architectures, specifically, the attention mechanisms. Additionally, we investigate and report a robustness perspective of the studied MT systems and discuss how different MT paradigms are sensitive to noise in the input.

In order to set a first baseline for improving UGC MT, we explain, in Chapter 5, our approach to address the phonetic writing specificities of French UGC such that we can automatically generate token-level normalization to reduce the presence of noise in the system input. We report improvement over several UGC test sets while hurting performance on canonical tests because of the artificial ambiguities induced by the method. We also report how sentence length impact the performance of our methods.

We continue our search for improving modeling by studying the impact of translation granularity on MT robustness in Chapter 6. The motivation of the method developed in this chapter is to improve the capacity of NMT to learn token representation to representations to ensure that noisy tokens and their normalized form have "close" or "similar" representation, rather than pinpointing a UGC specificity such as we did in Chapter 5. By doing so, we aim to attack a broader range of UGC phenomena, thus leading our research to rigorously assess performance and robustness in order to know what our models are doing better.

In the same track of thought, although the robustness of different types of MT systems was assessed, the reasons why UGC translation performs typically worse than canonical evaluation sets, remains, at best, obscure and corpus-level metrics have proven to be too coarse of a metric to explain the impact on the performance of the variety of UGC features described in Chapter 3. This is why, in Chapter 6, we have designed an evaluation framework that, to the best of our knowledge, contains the widest range of UGC specificity typology in the literature with 13 different UGC specificity annotations. Also, to address the potential interaction between such UGC phenomena, we designed a code base that can isolate them and generate a series of subcorpus with any given distribution of such types. Such an evaluation framework proved useful to compare different MT systems, concretely, NMT models with finer translation granularities (subword and character segmentation), which could benefit from a larger tokenlevel coverage with a reduced number of vocabulary elements. Indeed, by using our proposed evaluation resource, we were able to assess robustness properties of the character-based NMT models in comparison to our baseline vanilla NMT systems, where the former showed to be considerably more robust to misspellings (character changes, missing diacritics), tokenization errors, graphemic and punctuation stretching, and inconsistent case changes.

Moreover, as character-level NMT showed the worst robustness capabilities when confronted to special and rare characters, we performed a study to revisit the open-vocabulary properties of such char-level MT systems and these results showed that we can get improved translation performance by correctly managing unknown character by the means of setting the character vocabulary size correctly, often overlooked in today's character-level MT literature.

As a final stage of this work, in Chapter 7, we investigate another approach to zeroshot noisy UGC, for which we have introduced and studied new Variational Neural Machine Translation (VNMT) architectures that outperformed strong baselines when translating UGC. In this phase chapter of the dissertation, we provide further insight on how VNMT latent neural representations are more robust to UGC, by proposing novel approaches of visualization and explanation surrounding such models. In addition, we conducted a series of detailed ablation experiments to justify and understand the impact of our design choices.

Finally, in Chapter 8, we conclude and give some perspectives and interesting future avenues to cope with UGC MT that emerged from this work. Chapter 2

Publications related to this thesis

Machine translation: a review of methods and techniques

Now that we have introduced the problems and questions we will be addressing in this dissertation, we present the different Machine Translation frameworks and methods used in our work. We will start by describing the two most popular families of MT approaches: phrase-based statistical systems (Section 2.1) and neural architectures (Section 2.2), which are the systems used in the experiments throughout this work. Then, in Section 2.3, we will describe and discuss MT tokenization methods as well as other approaches and considerations introduced to address the Out-of-Vocabulary (OOV) problem, notably present in out-of-distribution evaluation. In the final part of this chapter (Section 2.4) we discuss different methods to evaluate the performance of the models and explain their behavior.

Statistical MT

Statistical Machine Translation (SMT) [START_REF] Weaver | Translation. Machine translation of languages: fourteen essays[END_REF][START_REF] Brown | A statistical approach to language translation[END_REF]) is a set of methods that rely on statistics to translate sentences by modeling the relationship between two parallel bilingual corpora without the need to use explicit linguistic rules.

Noisy-channel paradigm

In SMT, the problem of choosing the best translation ŷ given the input sentence x can be formally formulated as the following optimization problem:

ŷ = arg max y p(y|x) = arg max y p(x|y) ⇥ p(y) (2.1)
in which p(x|y) is estimated by the translation model, which will be introduced next, and p(y), computed by the target language model, a well-known probabilistic model that captures dependencies between words of a given sentence. This framework combining translation and target language models is known as the noisy-channel model [START_REF] Shannon | A mathematical theory of communication[END_REF]).

It should be noted that the probability search space of translations is considerably large as, even if we have the n correct target words, there will be n! permutations, and finding the best-scoring permutation (i.e. solving the arg max) is NP-complete [START_REF] Udupa | Computational complexity of statistical machine translation[END_REF]. That is why SMT relies on approximate search methods, such as beam search, to find the best translation.

Phrase-based approach Phrase-based statistical machine translation (PB-SMT) is the approach in this framework that achieves the best performance and had long been the state-of-theart method in SMT [START_REF] Koehn | Phrase-Based Models[END_REF]. Phrase-based models process phrases made of sequence of words (i.e. n-grams) that are translated using a translation table, containing, for each sequence of source words of the training data, the possible corresponding target word sequences and their probabilities. For instance, the translation table associates the German word "natürlich" to its possible translations: "of course", "of course ," and "naturally ". The phrase table is built automatically from the word alignments.

In a phrase-based model, the source sentence is broken down in K phrases x k and the probability p(x|y) is estimated as:

p(x|y) = K Y k=1 (x k |y k ) (2.2)
This equation uses the probability (x k |y k ) of translating y k to x k that is stored in the translation table, automatically extracted from the training data [START_REF] Koehn | Phrase-Based Models[END_REF]. This first (simplified) model has been devised to allow taking into consideration other features (e.g. to describe words reordering) in the estimation of the translation probability [START_REF] Bertoldi | Improved minimum error rate training in moses[END_REF]). Equation 2.2 highlights the main characteristics of phrase-based models: computing the score of a translation hypothesis relies on finding an exact match of a source phrase (i.e. a subsequence of the source sentence) in the phrase table, which is particularly problematic for the translation of UGCs since the high variability present in the source sentences strongly reduces the probability that a given sequence of words has been seen during training and ultimately extracted in the phrase table.

Neural machine translation

Neural Machine Translation (NMT) is a set of methods that employs artificial neural network architectures to translate sentences. These models are capable of automatically learning relevant representations of the source and target sentences from the training set and do not rely any longer on exact matches of source phrases as PB-SMT systems. The MT task is part of a wide family of architectures called sequence-to-sequence models that can be used to learn the mapping of an observations' sequence to a label sequence, and it is behind multiple other NLP tasks, such as language modeling and summarization. NMT has proven to outperform statistical MT models, setting a new state-of-the-art for MT when compared to SMT approaches [START_REF] Bentivogli | Neural versus phrasebased machine translation quality: a case study[END_REF][START_REF] Mutal | Differences between SMT and NMT output -a translators' point of view[END_REF].

In this section, we review the most popular NMT architectures that we used in our experiments.

First, we describe the encoder-decoder model, which is at the heart of NMT and other sequence modeling tasks. We then focus on two neural frameworks, namely, RNN-based models and

Sequential RNN-based models

The first type of NMT models that were proposed are based on Recurrent Neural Networks (RNN). This kind of architecture processes the input sentence token by token and progressively builds a "hidden vector", h, as introduced previously. For a given encoding step (h d t , where t 2 [1...T S ] represents each one of the source sentence tokens), depends on the previous tokens 1, ..., t 1, allowing the RNN to propagate the information throughout the sentence and these representations to be contextualized. The last hidden vector contains the information that summarizes the whole source sentence and is used as the input of the decoder. In addition, bidirectional RNN encoders [START_REF] Schuster | Bidirectional recurrent neural networks[END_REF] have been shown to consistently outperform uni-directional encoders [START_REF] Graves | Bidirectional LSTM networks for improved phoneme classification and recognition[END_REF][START_REF] Sundermeyer | Translation modeling with bidirectional recurrent neural networks[END_REF]) by building two h vectors using the left-to-right and right-to-left streams (i.e. processing the source sentence from first to last token, as well as from end to start). It is worth mentioning that the decoder is always uni-directional since we do not have access to future target tokens, as they are generated one by one conditioning each decoding step to the past produced tokens.

In Figure 2.2, we show a typical encoder-decoder RNN MT architecture featuring a bidirectional encoder. Left-to-right and right-to-left encoding streams are marked as forward and backward, respectively. In a first stage, each token of the input sequence, x, is passed to each of the encoder's RNN cells, whose hidden states (h t for t in [1, T S ], corresponding to the T S -long source), are sequentially computed and propagated through these cells until the last state, h T S , is generated. In addition, as discussed previously, the token-by-token encoding is usually done in a bidirectional fashion, and the final hidden vector is defined as the concatenation of the resulting representations for both directions ([c forw T S ; c back T S ] in the figure). Figure 2.2 portrays this approach, showing left-to-right flow (forward) in dark green, and right-to-left (backward) in light green.

In a second stage, the decoder generates each of the target tokens in an autoregressive fashion, i.e. by outputting one token at a time, conditioned on both the context vector and the last token produced by the decoder, while propagating the inner hidden state of each of the decoder's RNN units. Finally, a softmax layer maps the RNN outputs to a probability distribution over each possible token of the target language vocabulary for each decoding step.

The optimization problem for NMT, as well as it sequential autoregressive decoding can be expressed as:

arg max p(y|x) = arg max J Y i=1 p(y i |y j<1 , x) (2.3)
where is the architecture's trainable parameters and J is a maximal output length, which is used to stop the generation in case the <eos> symbol is not generated for the (x, y) pair.

In Figure 2.2, red and green blocks represent the encoder and decoder RNN neurons, respectively. These are most commonly chosen [START_REF] Yang | A survey of deep learning techniques for neural machine translation[END_REF]) to be Long-short Term Memory (LSTM) networks [START_REF] Hochreiter | Long short-term memory[END_REF], which have been shown to outperform vanilla RNNs [START_REF] Chung | Empirical evaluation of gated recurrent neural networks on sequence modeling[END_REF][START_REF] Chase | A critical review of recurrent neural networks for sequence learning[END_REF]. In Figure 2.3 we can see the three gated mechanisms introduced within an LSTM cell, which, in contrast to vanilla RNNs, do not store all the information in its state for a training sequence, but selectively learns to remember or forget [START_REF] Skrlj | Towards robust text classification with semantics-aware recurrent neural architecture[END_REF]). For all RNN-based NMT experiments presented in this work, we use the LSTM architecture, according to the current trends in the field [START_REF] Yang | A survey of deep learning techniques for neural machine translation[END_REF]. Nonetheless, to cope with the computational overhead due to longer sequences for the character-level RNN models in Chapter 6, we use a simplification of LSTM, consisting of a single gated output. It is known as Gated Recurrent Unit (GRU) [START_REF] Cho | Learning phrase representations using RNN encoderdecoder for statistical machine translation[END_REF], and it achieves similar performances to LSTM while being more computationally efficient [START_REF] Chung | Empirical evaluation of gated recurrent neural networks on sequence modeling[END_REF].

Attention mechanisms

As discussed in Section 2.2.1, the decoder network, is conditioned on the encoder's hidden vector h t , for each target token produced in an instant t according to the following equations:

h t = sigm(W hx x t + W hh h t 1 ) y t = W yh h t (2.4)
where W hx and W hh are the encoder's parameters and W yh the decoder's parameters.

Since the last hidden vector (h T S ) contains the representation for the whole input sequence, learning the alignments, i.e. the most important tokens in the input when producing a given target token; becomes a sub-task delegated primarily to the decoder network. In order to accomplish this, a method introduced by [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF], called attention, aims to unburden the decoder by explicitly attributing the alignment modeling to the encoder. Since then, attention methods have greatly driven the research in NMT and proved to improve sequence-to-sequence architectures.

Concretely, at each decoding step t, a context vector c t is computed as a weighted vector along the h 1:T S components of the hidden vector output by the encoder. This context vector is intended to help the decoder focusing more on a hidden vector component (that is, on a corresponding source token) than others, computed at each decoding step by the alignment model as:

c i = J X j=1 a ij h i (2.5)
where a t is the attention vector that assigns a certain weight to each token component of the hidden vector. The vectors a and c are shown accordingly in Figure 2.2. In order to calculate a t , a scoring function is combined with a softmax function to output a distribution on the last decoder's state s t 1 and the encoder's hidden vectors. This equation is formulated as follows:

a t = sof tmax(scoring(s t 1 , h)) (2.6)
In this work, we study the attention behavior to perform error analysis and investigate whether such errors can be explained in terms of faulty attention weight distribution for a given decoding time step.

The concept of attention rapidly evolved and eventually led to a series of models in which the sequences and order of tokens can be fully captured by positional embeddings (Vaswani et al. 2017). This evolution resulted in a breakthrough in NMT research and in a new kind of architecture family: transformers.

Transformers

The Transformer model (Vaswani et al. 2017) demonstrates how self-attention networks can perform sequence-to-sequence tasks without explicitly modeling the sequential nature of a sentence, i.e. sequential decoding and training of the network's parameters in equation 2.4.

Instead, the whole input token sequence is processed simultaneously in parallel and the position of each token is encoded by a positional embedding, where the several parallel and independent self-attention mechanisms provide information for each token of the input. Thus, Transformer self-attention layers connect the sequence tokens with a constant number of operations, whereas RNN networks require O(n) operations to for n-length sequences, resulting in improved performance of the former for long-range dependencies (Vaswani et al. 2017). As a consequence, the Transformer has access to more information simultaneously during the output generation than RNN approaches. Hence, this model has been widely adopted as the state-of-the-art architecture [START_REF] Xia | Tied transformers: Neural machine translation with shared encoder and decoder[END_REF], as it has also been consistently shown to perform better than RNN. In addition, this model is highly scalable through parallelization, contrary to sequential architectures.

Transformer is another instance of the encoder-decoder paradigm, as shown in Figure 2.4.

The principal component of this architecture is a set of multi-head self-attention networks, where several independent attention mechanisms model the inner relations between tokens

After reviewing the main MT architectures used in this work, we discuss other complementary methods that influence translation quality. Specifically, we will describe tokenization methods and granularity of translation, i.e. using words, subword or characters in the input and output vectors, which have a noticeable impact on translation quality [START_REF] Sennrich | Neural machine translation of rare words with subword units[END_REF] and play an important role in the vocabulary coverage learned by the MT model [START_REF] Duygu Ataman | On the importance of word boundaries in character-level neural machine translation[END_REF].

Tokenization methods

In this section, we introduce the tokenization methods used during this work and discuss the motivation for using different token granularities in MT. Tokenization is especially important when there are many out-of-vocabulary (OOV) tokens, that is to say, tokens that have not been seen

during training. This happens when translating morphologically rich or low-resource language or in the context of distributional shifts such as UGC translation. Coping with these OOV tokens is an important problem, as they often degrade translation performance [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF] and tokenization is the simplest way to reduce their number.

MT systems were initially designed using words as the basic input unit: A rule-based tokenization algorithm appends spaces between independent sequence of characters (such as text and punctuation) in order to reduce vocabulary and make training easier [START_REF] Ataman | An evaluation of two vocabulary reduction methods for neural machine translation[END_REF]. However, this approach results in many OOV because of the considerable lexical diversity of words in many languages and the poor compositional properties of many models. Translating these OOV raises many challenges as these words cannot be properly modelled, affecting translation quality. Indeed, OOVs have to be mapped to a special <UNK> symbol that is generally kept unchanged during translation and is therefore still present in the MT output. This token can be directly copied into the output by incorporating a copy mechanism [START_REF] Gu | Incorporating copying mechanism in sequence-to-sequence learning[END_REF], or subsequently translated by replacing it in a post-processing step [START_REF] Jean | On using very large target vocabulary for neural machine translation[END_REF][START_REF] Mi | Coverage embedding models for neural machine translation[END_REF], or via embedding substitution [START_REF] Haddad | Handling oov words in nmt using unsupervised bilingual embedding[END_REF][START_REF] Yang | Using word embeddings in twitter election classification[END_REF].

The "post-processing" approach is straightforward and consequently, by far, the most widely used in today MT systems (even if it often seems to be ignored in the literature with the development of NMT). Its main advantage is that this approach is completely independent of the MT architecture used and, thus, can be easily applied to any MT system. The alignment between the source sentence and the translation hypothesis can either be predicted directly using a word-alignment method such as IBM models [START_REF] Mi | Coverage embedding models for neural machine translation[END_REF] or can be deduced from the attention matrix as proposed by [START_REF] Jean | On using very large target vocabulary for neural machine translation[END_REF]. Throughout this work, we will use the former approach and replace <UNK> in the evaluation translation hypothesis using source-target alignments predicted with fastalign [START_REF] Dyer | A simple, fast, and effective reparameterization of IBM model 2[END_REF]).

This strategy is, however, very limited and several alternatives have been proposed to actually translate the OOVs rather than copying them unchanged from the source sentence. The most straightforward way to deal with OOV is to consider smaller, sub-word tokens and to leverage on composition to represent words [START_REF] Sennrich | Revisiting low-resource neural machine translation: A case study[END_REF]).

The finest grain segmentation possible consists in cutting out the input sentence in a stream of characters and modeling it directly (resulting in a character-based model). This approach raises two main problems, the first being the computational cost of the resulting methods: the number of characters in a sentence is substantially larger than the number of words, and thus the computation times are mechanically increased. For instance, [START_REF] Luong | Achieving open vocabulary neural machine translation with hybrid word-character models[END_REF] report spending about 3 months training a purely character-level model. Another important problem, and a fundamental drawback of character-based models, is the difficulty of modeling long-range dependencies [START_REF] Wood-Doughty | Convolutions are all you need (for classifying character sequences)[END_REF][START_REF] Al-Rfou | Character-level language modeling with deeper self-attention[END_REF].

To avoid the pitfalls of character models, subword-level segmentation models have been proposed: Their goal is to uncover a word segmentation that limits the number of units created (to avoid complexity problems) while limiting the size of the vocabulary (to ensure that the frequency of units is sufficiently high). Notably, [START_REF] Sennrich | Neural machine translation of rare words with subword units[END_REF] propose to use the Byte-Pair Encoding (BPE) algorithm to produce a subword vocabulary and [START_REF] Schuster | Japanese and korean voice search[END_REF] introduce Wordpiece.

The use of subword-level segmentation has proved to consistently outperform word-based MT and is currently systematically used in all NMT systems. We adopt BPE tokenization throughout this work unless stated otherwise (our character-based models in Chapter 6). When using BPE, the vocabulary is built by merging the most frequent sequences of characters to reach a number of operations set as a hyperparameter. The intuition is that frequent n-gram structures constitute subword particles (e.g. prefixes or suffixes) and, by leveraging on their compositionality, we can decompose potential OOV during evaluation in a sequence of in-vocabulary n-grams.

BPE ensures that there are almost no OOV tokens in both the train and test sets, with the exception, for instance, of words in the test set that contain characters that do not appear in the train set. However, translation quality still suffers from the actual OOV words that existed before BPE segmentation [START_REF] Araabi | How effective is byte pair encoding for outof-vocabulary words in neural machine translation?[END_REF], as these words are often divided into many BPE tokens.

Analyzing translation hypothesis

In this work, we will use two methods to analyze the predictions of the different MT systems we consider: the first method is a qualitative method that uses the attention matrices at the heart of NMT models to explain the decisions made by the translation system; the second method relies on the automatic metrics usually used to evaluate the translation systems. Finally, we will also introduce several metrics to assess translation robustness to noise in the input sentence that we use throughout this work.

Visualization: understanding NMT decisions

Although NMT systems are commonly regarded as black-box systems because their decision process is difficult to interpret, a popular way to gain insight into the inner workings of these systems is through self-and cross-attention matrices. Using attention to explain neural network decisions is subject to a long-lived debate [START_REF] Serrano | Is attention interpretable[END_REF], but has been proven to be informative in many works [START_REF] Ding | Visualizing and understanding neural machine translation[END_REF]. The attention matrix quantifies the "importance" of a (source or target) token to generate each one of the target tokens. We will use this method in Chapter 4 to highlight some of the pitfalls of NMT systems when translating UGC.

Source-Target Attention Figure 2.5 gives an example of a cross-attention matrix (as the one shown in Figure 2.2) between an English source sentence (horizontal axis) being translated into French (vertical axis shows the predicted target tokens). It shows how the decoder focuses on some specific source words to different degrees when producing a given target word. Attention weights from Equation 2.6 are depicted in grey-scale (the whiter, the closer to 1). These are computed by the model as the attention scoring function is learned during training. For instance, it can be noted that to generate the French tokens "ne peut plus" the decoder mainly relies on the English tokens "can longer produce". This shows how attention can give special importance to the most pertinent encoder hidden vectors (each corresponding to a source-side token).

Figure 2.5: Attention matrix example showing the alignments between the French sentence predicted by an NMT system (y-axis) given the English sentence (x-axis). Reproduced from [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF]. The intuition behind this is that each one of the heads will be able to capture different relations between the sentence tokens.

Self-Attention

Evaluation metrics

(Sacre)BLEU As a corpus-level automatic translation quality metric, we systematically report results using case-sensitive BLEU [START_REF] Papineni | Bleu: a method for automatic evaluation of machine translation[END_REF], as it is the standard way to quantify translation performance of the MT systems. More specifically, we consistently report results using its tokenization-agnostic versions, that is, SacreBLEU (Post 2018) and multi-bleu-detok.perl 1 , both taking the detokenized translation hypothesis as input and performing their own internal tokenization, which can be easily reproduced. This follows the recent trend of evaluating translations without any specific tokenization (other than space delimiters between words) to ensure the reproducibility of the evaluation regardless of the different tokenization rules that different researchers may use. Indeed, the necessity to report results using detokenized evaluation frameworks has been widely acknowledged [START_REF] Marie | Scientific credibility of machine translation research: A meta-evaluation of 769 papers[END_REF]) as this BLEU metric flavor is increasingly popular over traditional BLEU score. Edit distance Metrics based on BLEU are evaluating translation quality at the corpus-level: they are not accurate when ranking single sentences or small corpora due to the high sparseness of n-grams in these conditions [START_REF] Ye | Sentence level machine translation evaluation as a ranking[END_REF]. In consequence, correlation between human evaluation and sentence-level BLEU can be astonishingly low [START_REF] Liu | Syntactic features for evaluation of machine translation[END_REF][START_REF] Li | Modeling monolingual character alignment for automatic evaluation of chinese translation[END_REF]. In order to have a sentence-level metric, which enables single translation outputs to be ranked and compared, we have chosen to use token-level edit distance between predictions and references [START_REF] Przybocki | Edit distance: A metric for machine translation evaluation[END_REF]. Having a sentence-level metric proves useful in Chapter 6 for a fine-grained characterization of UGC and the challenges it raises.

chrF The BLEU metric has the limitation of needing an exact token-level match between prediction and reference, thus, a change in the lexicon or relatively small and fine-grained errors (e.g. a predicted token differing in a single character to that of the reference translation) are not quantifiable. Hence, we also use the MT character-level chrF (Popović 2015) evaluation metric, which we use to compare to BLEU metrics and explain UGC translation quality taking into account character-level editions. This metric proved to be the most reliable when characterizing the robustness of MT to UGC in Chapter 6 and Chapter 7.

chrF is defined as the F-score on character n-grams that overlap between the hypothesis and reference:

chrF = (1 + 2 ) chrP • chrR 2 • chrP + chrR (2.7)
where chrP , the character n-gram precision, is the percentage of n-grams in the hypothesis contained in the reference; and chrR, the character n-gram recall, is the percentage of n-grams in the reference also present in the hypothesis. is a parameter that gives proportionally more importance to recall than to precision. In this work, we have chosen = 1, which gives the same importance to the recall and precision.

Assessing the performance impact of noise in the source sentence

We are particularly interested in assessing the translation quality impact caused by noise by the means of comparing the performance achieved by a given MT system when translating "raw" noisy sentences to the translation output by the very same system when translating the "normalized" version of the same sentences (i.e. sentences with the same meaning in which the noise has been "removed"). As notation to further discussing this kind of metrics, we will refer to

x as the original noisy source, x as its normalized version, and y, ỹ their respective translation hypothesis.

It should be noted that these metrics require "normalized" corpora, in which noisy source sentences have been "corrected" to a form that complies with the language's rules. Only a few normalized corpora are freely available, as this kind of annotation is difficult to accomplish because of potential ambiguity in the source text. This problem is all the more important because it is often necessary to produce new references corresponding to the expected translations when the sources do not contain noise, since removing it from the sentences can also remove information and thus modify the corresponding reference translation (e.g. if emojis or emoticons are removed from the input, they should also be suppressed in the reference). These considerations raise a major resource-intensive limitation that makes such metrics costly and renders annotation consensus notably hard. In addition, this kind of annotated UGC corpora is highly dependent on the specificities and scheme used. For instance, [START_REF] Fujii | Phemt: A phenomenon-wise dataset for machine translation robustness on user-generated contents[END_REF] provides a UGC corpus annotated in terms of four static UGC specificities, but it remains difficult to extend it to other noise categories.

Throughout this work, we use two metrics to assess the robustness of each NMT model by quantifying to what extent the noise in the source impacts translation quality. They are based on a monotonically increasing metric with respect to the translations' quality, and which we proceed to elaborate on using the BLEU score; although they can be adapted to be used for the-lower-the-better metrics trivially.

Noise impact ratio (NR)

The noise impact ratio [START_REF] Fujii | Phemt: A phenomenon-wise dataset for machine translation robustness on user-generated contents[END_REF] is the quotient between the BLEU score obtained by translating the original, noisy sentences evaluated using the original references, and the BLEU score achieved by inputting the normalized version of the same source sentences instead and evaluating using the accordingly normalized references. It is calculated as:

NR(x, x, y, ỹ) = BLEU(x, y) BLEU(x, ỹ) (2.8)
In other words, this metric compares the translation quality achieved by an MT system when translating noisy sentences to the one that would have been obtained if no noise had been present in the source. Hence, an NR score of 1.0 means that the noise in the source sentence does not impact the MT quality at all (i.e. the MT system was able to automatically remove the noise present in the source sentence or least ignore it); conversely, the smaller the score, the greater the impact that noise has on translation quality.

Target-source noise ratio (TSNR) Proposed by [START_REF] Anastasopoulos | An analysis of source-side grammatical errors in NMT[END_REF], this metric aims to assess how much perturbation is artificially added during the MT system due to the noise initially present in the input sequence. It is defined as:

T SNR(x, x, y, ỹ) = 100 BLEU(y, ỹ) 100 BLEU(x, x) (2.9)
This is to say, by using this metric, we quantify to what extent the noise present in the input sentence (estimated by BLEU (x, x)) is compared to the noise output by the system when producing the target translation (estimated by BLEU (y, ỹ). Concisely, a TSNR value close to 1

indicates that the noisy textual perturbations (assessed using BLEU) in the input are comparable to those of the resulting translation. This is to say that the drift UGC specificities present in the source On the other hand, a low TSNR value suggests that the MT system is less prone to propagate the source-side noise and, consequently, generating a relatively more similar translation to the translation of the normalized version.

Conclusions

In this chapter, we have given a broad vision of the methods used in this thesis and how they compare, which commonly translates into advantages and disadvantages regarding memory size, computation time, and models' complexity.

We have discussed the main MT model families, focusing on the different NMT architectures we address in the next chapters. In addition, we have presented how we exploit the information provided by the attention matrices for the error analysis when exploring how different MT technologies behave when translating UGC in Chapter 4.

In the final part of this chapter, we presented and discussed the metrics we use in the next chapters to evaluate the translation quality of the explored MT systems. Especially, we reviewed TSNR and NR, that we use to assess MT robustness to UGC. We also discussed challenges regarding the use of such metrics, which require additional annotation efforts to normalize UGC, ultimately leading to complex normalization guidelines and difficult inter-annotator consensus.

Now that we have established our scientific framework and explained the set of MT methods that will be used throughout this work, we proceed to describe and discuss UGC and elaborate on the challenges and the interests it poses for NLP, specifically MT.

Chapter 3

User-generated content: an NLP nightmare

In this chapter of the dissertation, we introduce in further detail the main task to be addressed in this work and why it is as problematic as it can be useful to conceive robust NLP models that can cope with a wide range of linguistic phenomena. As we will discuss below, UGC is widely and increasingly present in today's discussion channels and can provide an interesting source of data, although it is characterized by extremely high lexical and semantic variability.

What is UGC and what is at stake?

UGC consists of different types of content, such as pictures, videos, audio, and texts created and published on the Internet by its users [START_REF] George | Jackie, web 2.0 and user-generated content: Legal challenges in the new frontier[END_REF]. Concretely, under the scope of this work, we address text-based forms of UGC, specifically, noisy social-media UGC, or non-canonical text, which is a kind of written content that is produced by users in online discussion channels such as Twitter, Facebook or Doctissimo. This type of texts has been extensively studied over the years, e.g. [START_REF] Foster | cba to check the spelling": Investigating parser performance on discussion forum posts[END_REF][START_REF] Seddah | The french social media bank: a treebank of noisy user generated content[END_REF][START_REF] Eisenstein | What to do about bad language on the internet[END_REF][START_REF] Sanguinetti | Treebanking user-generated content: a UD based overview of guidelines, corpora and unified recommendations[END_REF]. In this context, users are often able to add elements that are typically not presented in canonical corpora in order to denote emotions, shortening typing time or efforts and, thus, promoting spelling, grammatical, and semantic diversity, whether purposely or not. Despite the challenges posed by UGC, which we will discuss in this chapter, such content can provide a direct way to estimate customer needs and trends (Bahtar and Muda 2016; Timoshenko and Hauser 2019) and thus can represent a powerful asset for data mining due to their ubiquitous presence on the Internet [START_REF] Krumm | User-generated content[END_REF][START_REF] Matthew | The empowered customer: User-generated content and the future of marketing[END_REF].

As an easily available source of data, UGC has also provided a interesting, if not valuable, source of data for different NLP tasks, such as hate speech detection [START_REF] Mossie | Social media dark side content detection using transfer learning emphasis on hate and conflict[END_REF][START_REF] Gomez | Exploring hate speech detection in multimodal publications[END_REF], prevention against human trafficking [START_REF] Chambers | Character-based models for adversarial phone extraction: Preventing human sex trafficking[END_REF], sentiment analysis [START_REF] Schmunk | Sentiment analysis: Extracting decision-relevant knowledge from ugc[END_REF], fact-checking [START_REF] Bondielli | A survey on fake news and rumour detection techniques[END_REF] and mental health assessment [START_REF] Calvo | Natural language processing in mental health applications using non-clinical texts[END_REF].

Almost by definition, UGC covers multiple different domains; is multi-modal in nature [START_REF] Khoshamooz | Multi-domain user-generated content based model to enrich road network data for multi-criteria route planning[END_REF], addresses diverse topics and because of the diversity of the users, entails a large degree of language variation [START_REF] Sanguinetti | Treebanking user-generated content: a UD based overview of guidelines, corpora and unified recommendations[END_REF]). Interestingly, these variations are perceptible also through many orthographic variants that span over namedentities [START_REF] Jijkoun | Named entity normalization in user generated content[END_REF]. Specific idioms and domain-related jargon make clear the fact that UGC does not constitute an homogeneous domain and instead could be seen as a multidimensional space where each axis would relate to one specific aspect (domain, divergence to the orthographic/grammatical norm, sentiment expression, socio-demographic characterization of the user, etc.) [START_REF] Foster | cba to check the spelling": Investigating parser performance on discussion forum posts[END_REF][START_REF] Seddah | The french social media bank: a treebank of noisy user generated content[END_REF][START_REF] Eisenstein | What to do about bad language on the internet[END_REF][START_REF] Héctor | From noisy questions to Minecraft texts: Annotation challenges in extreme syntax scenario[END_REF].

Exploiting this type of text and, therefore, designing translation systems capable of treating them has been an crucial ongoing research question in the literature [START_REF] Chen | Twitter-a new pathway to access product innovation ideas -can machine learning help pepsico identify innovative ideas in user-generated content platforms?[END_REF]). As coping with the noise that rifles UGC still poses many challenges1 , the questions about the extent to which an UGC phrase effectively conveys information remains, as opposed to being gibberish or the cause of an unfortunate mistranslation.

For example, the automatic translation of a noisy Arabic version of "good morning" triggered a terrorist alert when Facebook mistranslated it, resulting in an unjustified arrest in 2017 [START_REF] Guardian | Facebook translates 'good morning' into 'attack them', leading to arrest[END_REF]. A similar unfortunate situation took place massively when the Facebook translation engine translated thousands of Burmese posts that contained the name of the Chinese president, Xi Jinping, into a slur word in English, forcing the company to issue a public apology (Times 2020). Beyond these scandals, modern NLP pipelines continue to make mistakes when faced with noisy input, and proposing architectures that overcome these issues is still an open question.

Encoding simplification: covers a set of phenomena aiming to reduce writing efforts (English trans.) Word omission "je n'aime pas" ! "j'aime pas" I don't like Irregular tokenization "I was" ! "Iwuz" (contraction) "c'était" ! "c t" (over-splitting) It was Spoken language "je ne sais pas" ! "chais pa" I don't know Spelling and typographical errors "Je fsit" ! "Je fais" I do Grammatical errors "There are a lots", "Elle es grand" Jargon "mec", "wesh" Dude/man Profanities (occasionally masked) Dialects Code-switching "C'est trop good" It's too good Phonetic writing errors "J'ai regarder" ! "J'ai regardé" I watched Internet slang "to be honest" ! "tbh" Inconsistent casing "Allez !" ! "ALLEZ !" Go Table 3.1: Examples showing common noisy phenomena in UGC [START_REF] Seddah | The french social media bank: a treebank of noisy user generated content[END_REF].

Marks

Comparing UGC to canonical texts

UGC has many specific characteristics that explain why it differs from canonical texts [START_REF] Sanguinetti | Treebanking user-generated content: a UD based overview of guidelines, corpora and unified recommendations[END_REF][START_REF] Michel | MTNT: A testbed for machine translation of noisy text[END_REF], which are, in contrast, clean and well-formed corpora that comply with the set of linguistic rules of a given language.

Regarding the different noise specificities that are prevalent in UGC, we display, in Table 3.1, the ones we consider and discuss in further detail in Section 3.3.

Because of these specificities, UGC text differs from widely distributed canonical corpora, used to train reproducible (and usually license-friendly) NLP models. These differences, which, among others, result in a high number of Out-of-Vocabulary (OOV) tokens, raise many challenges to the automatic processing of UGC. First, large UGC corpora are expensive to annotate and can intrinsically impose major consensus issues due to ambiguities and, occasionally, incomprehensible phrases. Second, due to the extremely high productivity of UGC, which permeates this type of text at all linguistic levels (lexical, grammatical, and syntactic); there are too many morphological variations of every possible linguistic construct to account for all of them. Specifically, MT, which constitutes the main scope of this work, illustrates the difficulties of processing UGC, as performances are affected when translating due to its specificities.

In this regard, we now present different lexical, grammatical, and syntactical UGC speci-ficities that make it different from canonical corpora in terms of a well-defined linguistic and morphological classification. Thus, in order to illustrate the difficulties imposed by translating UGC, we show some examples and their translations by public and popular translation systems in Table 3.2. Indeed, a decrease in translation quality has been documented when MT systems have to translate this kind of text [START_REF] Khayrallah | On the impact of various types of noise on neural machine translation[END_REF][START_REF] Fujii | Phemt: A phenomenon-wise dataset for machine translation robustness on user-generated contents[END_REF]. In this respect, we use the phenomena discussed in the following to explain and identify the caveats of MT when facing UGC in terms of its distinctive linguistic traits.

UGC specificities

In this section, we proceed to discuss in detail the UGC's idiosyncrasies previously introduced. In order to characterize UGC, we consider a list of 13 syntactical, grammatical, and morphological UGC specificities presented in classified and which we investigate in terms of their impact on translation quality. Our defined types differ from the original typology only by not considering code-switching and profanities but, on the other hand, adding a category for interjections.

To introduce the considered typology, we will start by explaining the main groups of UGC's specificities in Table 3.1, which in turn we divide in the fine-grained typology presented for the experiments in this thesis. Essentially, these groups are classified in a purpose-oriented manner, i.e. they encompass the lexical, grammatical, and syntactical peculiarities, characterized in terms of concrete morphosyntactic specificities of user expression while associating a reason for employing such linguistic resources. For instance, in UGC, users often simplify the message to reduce writing efforts or express emotions more vividly, which often leads to a very high lexical variability, such as those seen for the "Word omission" and "Punctuation reduplication" categories in Table 3.1.

We also base some of the UGC specificities on a context and domain-related axis (Martínez Alonso et al. 2016), for example, Named Entities (NE) evoking a certain television series.

Encoding simplification

This kind of UGC's phenomenon involves a large set of noisy occurrences, commonly used by users for shortening the messages' length, while enjoying the freedom to introduce new morphological variations for otherwise known and comprehensible words. Arguably, they are responsible for a large number of spelling errors, consequently introducing OOVs and rare token sequences. In Table 3.4, we present artificial examples of how these UGC specificities could affect the quality of the translation individually, which we will comment on in more detail.

Letter change, deletion or addition Words that present some form of letter alternation or modification from the correct form, introduced by the user willingly or not. They often lead to OOV tokens and some can compromise the full meaning of the translation, e.g. "Je suis alé à la bibliothèque" is translated from French to English by Google Translate as "I am random at the library " in example ¨. The error appears to be that the MT system predicted alé as aléatoire, which is translated as random in English, whereas the correct spelling should have been allé

(went in English). This encoding simplification phenomenon can be seen as the omission of the missing letter (l) to match the correct French word.

Diacritics omission or error

As in character-level modifications for an existing word, diacritic errors often cause OOVs, but the size of the search space of possible combinations is substantially reduced by the fact that each letter can only be changed by the same letter with different diacritics (i.e. accents). State-of-the-art MT models later proved remarkable robustness to these UGC specificities further in this work, and we found no example containing only this type that produces an incorrect translation in GOOGLE TRANSLATE when consulted. Phonetization It takes place when a word is written using an alternative orthography (existing or not) that shares a similar pronunciation. This phenomenon involves knowing the pronunciation to match it with its corresponding canonical form. For instance, in example AE in Figure 3.4, "nez" is translated using its literal meaning to "nose" in English, whereas the correct source French token is "né", sharing the same pronunciation /ne/. The same happens for the tokens "sa" et "ça", leading to substantial changes in translation, as can be seen.

Wrong verb tense

We study grammar conjugation errors that can lead to ambiguities and could cause errors in translations.

Wrong gender or grammatical number

We also take into account grammar inconsistencies to explore whether they represent an inconvenience to MT performance and a source of ambiguity during translation. These are annotated with respect to grammatical differences in gender and number.

Abbreviation These constitute a rather high compression modification of the message, including the omission of letters for any given word and compacting multi-word expressions (MWE) in a reduced sequence of characters. They most often lead to OOV tokens that could be mapped to a sequence of known target tokens, although it often proves challenging since it requires, to a large extent, a lexicon. In example ±, the abbreviation "bien" ! "bn" for a fairly standard French expression fails to produce the correct translation ("very much").

Marks of expressiveness

These account for any modification or addition of words, letters and punctuation symbols in the message in order to convey emotions. In this work, we considered three specific UGC morphological features that are related to such a group. Examples of the impact on translation quality are shown in Table 3.5, referenced in this subsection.

Emoticons and emojis

The former corresponds to sequences of characters to state the intended tone and denote emotions (often by representing facial expressions using punctuation symbols) (Dictionary 2017), and consequently introduces rare sequences of often known source tokens. On the other hand, emojis, are specially encoded characters that represent miniaturized figures, serving, in a general way, to the same purposes as emoticons, but introducing an ever-increasing list of valid emojis (3,251 emojis in Unicode Standard, as of September 2020).

Although emojis are progressively present in today's linguistic landscape (the 2015 Oxford Language word was, indeed, an emoji (Verge 2015)), their high number and variability often results in character-level OOV during evaluation. In example ¨, the Unicode-encoded emoji ("U-1F972"), a character-level OOV, is ignored during translation. Conversely, the emoticon ":O)", representing a surprised expression, being a rare sequence not present in canonical texts, is treated by the MT engine as "où", producing its corresponding English translation "where".

Grapheme stretching and Punctuation reduplication They consist of repetitive structures

to emphasize emotion by expanding certain words or punctuation. These are most often manifested in rare repetition sequences of characters and punctuation marks. Example ≠

shows how these specificities can affect translation by adding repeated characters. Punctuation repetitions ("!!!!") are arguably easier to treat, as they can be tokenized and recopied in the output, while words undergoing stretching have to be mapped to a known word and translated accordingly ("supeeeerrrr " in the example).

Inconsistent casing This UGC specificity involves any kind of casing occurrence that does not comply with the language rules. They introduce different (possibly alternated) upper-or lowercase versions of characters or full words, which, in turn, are represented by completely different elements in the vocabulary. However, current MT systems employ methods such as translating the full lowercase version of the text and employing placeholders to recover the actual source-side casing, known as in-line casing (Berard et al. 2019b). The public translation engine appears to be robust to this UGC specificity, as can be seen in example AE. However, it is an additional confounding factor for translation; for instance, in the presence of a letter deletion, Google's translation system outputs: "C'est très bin" ! "It's very good", whereas, when an inconsistent case is introduced, the result is "C'est très BIN" ! "It's very BIN".

Interjections These are words often related to MWE that also set the intended tone of the message. They are specifically difficult to process correctly without any lexicon or example dataset, since they represent rare sequences that can be extended into spans of several words.

In example Ø, the interjection "ptdr ", which stands for "peté de rire", which is analogous to lol (laughing out loud) in English, is not recognized by the translator and is copied into the output.

Some interjections are correctly translated by Google's engine, e.g. it is the case if we substitute "ptdr " by "mdr ", another known French to evoke laughter, which is translated as "lol".

Boundary shifting

This category groups the phenomenon of changing the segmentation between words, either to reduce writing efforts or due to typographical errors.

UGC context-dependent specificities

These make reference to any peculiarities of the message related to the specific nature of the channel or the exchange platform used. Special character sequences These are occurrences of characters that, due to the UGC's extreme diversity could be, in principle, any type of character, possibly in a different encoding.

They can also depend on the UGC platform having distinguishable uses to access functionality, e.g. the characters # and for hashtags and user mentions, respectively, in TWITTER or URL strings (http(s):// ). In example ¨, the character # prevented the correct translation of "tenez bon"

! "hold on".

Named entities They are NE that are in line with the informational context surrounding the UGC, e.g. names of films or games (possibly written in a foreign language). Some other NE depend on the publication platform, as is the case of RT and TT, denoting Retweet and Trending Topics in TWITTER discussions. In example ≠, the name of a hypothetical game (fish fins) is incorrectly translated when it should be kept unmodified in the output.

UGC as an ever-changing way of expression

In addition, motivated by the observation that the potential variations in UGC are too wide and variable to be accounted for, and, lacking sufficiently large translated UGC training datasets in open access, we set as our main approach not to use any domain-adaptation technique nor target-specific (UGC) data. Furthermore, a constant evolution of ways of expression, such as the increasing use of an ever-growing set of available emojis and endless possible discussion topics, gives UGC a considerable dynamic through time and location of the users [START_REF] Broni | Emoji use at all-time high[END_REF], and that has recently been shown to strongly contribute to emergent linguistic constructs [START_REF] Raviv | The role of social network structure in the emergence of linguistic structure[END_REF].

Furthermore, [START_REF] Oren | Distributionally robust language modeling[END_REF] In this zero-shot scenario, we intend to resolve UGC specificities without using any UGC data, for which we explore and propose methods that leverage some heuristics or fine-grained morphological properties. Under these premises, we proceed to state the challenges and main research questions for this thesis work.

Datasets

In this section, we describe the datasets for training, developing, and testing all of the MT models investigated throughout this work. We elaborate both on the canonical and commonly used corpora for MT, as well as the UGC ones, which we compare in terms of automatic metrics for noise and domain-drift assessment (discussed in Section 2.4.2).

Canonical data

We train our models on two different canonical parallel corpora. We first consider the traditional corpus for training MT systems, namely the WMT data consisting of the europarl (v7) corpus2 and the newscommentaries (v10) corpus3 . We use the newsdiscussdev2015 corpus as a development set. This is exactly the data confgiuration used to train the system described in [START_REF] Michel | MTNT: A testbed for machine translation of noisy text[END_REF], which will be used as a reference throughout this work.

We also consider the French-English parallel portion of OpenSubtitles'18 [START_REF] Lison | Opensubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora[END_REF]) as a second training set. This corpus is a collection of crowd-sourced and peer-reviewed edited subtitles for movies. We assume that because it is made up of informal dialogues, such as those found in popular sitcoms, sentences from OpenSubtitles will be much more similar to the UGC data than the WMT training dataset. However, it must be noted that UGC differs significantly from subtitles in many aspects: emotion denoted with repetitions, typographical and spelling errors, emojis, etc.

To allow for a fair comparison between systems trained on WMT and on OpenSubtitles, we consider a small version of the OpenSubtitles corpus, that has nearly the same number of tokens as the WMT training set and a large version that contains all OpenSubtitles parallel data.

To evaluate our system on in-domain data, we use two test sets, namely, newstest'14, as well as 11,000 sentences extracted from OpenSubtitles, which we refer to as OpenSubTest throughout this dissertation. Statistics for our different datasets are presented in Table 3.9.

Non-canonical UGC data

To evaluate our models, we consider two data sets of manually translated UGC.

The first is a collection of French-English parallel sentences manually translated from an extension of the Parallel French Social Media Bank [START_REF] Seddah | The french social media bank: a treebank of noisy user generated content[END_REF], which contains texts collected on Facebook, Twitter, as well as from the forums of JeuxVideos.com and Doctissimo.fr. 4We refer to this corpus as PFSMB for Parallel French Social Media Bank, and it consists of 1,554 comments in French annotated with different kinds of linguistic information: Part-of-Speech tags and surface syntactic representations. Phrases have been translated from French to English by a native French speaker and an extremely fluent, near-native English speaker. Typographic and grammatical errors were corrected in the gold translations, but the linguistic register was kept. For instance, idiomatic expressions were mapped directly to the corresponding ones in English (e.g. "mdr" has been translated to "lol" and letter repetitions were also kept (e.g.

"ouiii" has been translated to "yesss"). For our experiments, we divided the PFSMB into a test set and a blind test set containing 777 comments each.

We also consider, for these experiments, the MTNT corpus (Michel and Neubig 2018), a dataset made up of French sentences that were collected from Reddit and translated into English by professional translators. We use their designated test set and add a blind test set of 599 sentences we sampled from the MTNT validation set. The PFSMB and MTNT corpora both differ in the domains they consider, their collection date, and the way sentences were collected to ensure that they are sufficiently noisy. Several statistics of these two corpora are reported in Table 3.9. As expected, our two UGC test sets have a substantially higher token-to-type ratio (TTR) than the canonical test corpora, indicating greater lexical diversity.

As discussed previously, we have divided the UGC test sets into two subsets: common test sets and blind test sets. This is motivated by the necessity of unforeseen test corpora in order to fairly report results of our methods after choosing the best configurations and MT systems over the common UGC tests. To gain more insights into the sizes and basic statistics of our entire dataset collection, please refer to Table 3.9, where the statistics of the mentioned corpora are shown.

Some examples of samples from the PFSMB and MTNT UGC corpora, and their corresponding reference translation, can be found in Table 3.9: Statistics on the French side of the corpora used in our experiments. TTR stands for Type-to-Token Ratio, ASL for average sentence length.

Quantifying the difference between canonical texts and UGC

As a first exploratory approach for the corpora being considered, we can note their token-level statistics in Table 3.9. It is worth noting that the TTR of our four UGC test sets are roughly 42% higher than the train and canonical test corpora, which indicates a proportionally higher lexical variability for UGC.

Several metrics have been proposed to quantify domain drift and extraneous n-grams apparitions overall between two corpora. In particular, the perplexity of a language model and the Kullback-Leibler (KL)-divergence between the character-level 3-gram distribution of the train and test sets were two useful measurements capable of estimating the noise-level of UGC corpora as shown respectively by [START_REF] Seddah | The french social media bank: a treebank of noisy user generated content[END_REF][START_REF] Héctor | From noisy questions to Minecraft texts: Annotation challenges in extreme syntax scenario[END_REF]. We also employed the perplexity (PPL) of a 5-gram Knesser-Nay language model [START_REF] Ney | On structuring probabilistic dependences in stochastic language modelling[END_REF] trained on Large OpenSubtitles as a measure to assess the out-of-domain (OOD) [START_REF] Haddow | Analysing the effect of out-of-domain data on SMT systems[END_REF]) of a given corpus. the frequency of such a BPE token j in the test corpus, we computed it as follows:

1 N • 16K X j=1 freq j • #unique_neighbors j #neighbors j
This corresponds to the weighted average of the diversity of BPE neighbors for the whole fixed-size vocabulary.

Scarcity of UGC resources for NLP

As previously discussed, UGC has inherent characteristics, such as addressing any possible kind of domain and raising considerable challenges to overcome for consensus and annotation (Martínez Alonso et al. 2016) due to its extreme variability. This makes their evaluation intrinsically a low-resource scenario in NLP [START_REF] Meftah | Neural Transfer Learning for Domain Adaptation in Natural Language Processing[END_REF], as parallel UGC corpora is scarce in publicly available datasets [START_REF] Lohar | Machine translation of user-generated content[END_REF].

In contrast, availability of good quality data in sufficiently large quantities strongly conditions the development and training of NLP models for new domains [START_REF] Bamman | Natural language processing for the long tail[END_REF]. Recently, some valuable and peer-reviewed UGC collections have appeared, specifically for MT, for example, the MTNT and 4Square corpora. However, they have primary adaptation and evaluation purposes, containing dozens of thousands of training sentences, substantially less than the usual number of samples needed to train today's NMT models. Additionally, noise present in large and automatically scraped UGC corpora for training purposes, such as the ParaCrawl5 has shown to negatively impact the quality of NMT [START_REF] Rikters | Impact of corpora quality on neural machine translation[END_REF], which in turn leads to a consequent loss of performance over in-domain corpora, even when UGC is available for training.

Related works in UGC translation

In this section, we review the works related to the series of MT methods and protocols developed to cope with UGC throughout this dissertation. We also link each of the following sets of MT literature to our proposed approaches in each chapter to justify their relevance at each stage of this work.

MT of UGC: general overview

We start by discussing the literature on the impact on translation performance caused by UGC translation. In this sense, we first review the work that characterizes UGC translation and its performance related to canonical text MT.

Automatic translation of UGC has proven to be a difficult task compared to canonical text MT (Berard et al. 2019a;[START_REF] Fujii | Phemt: A phenomenon-wise dataset for machine translation robustness on user-generated contents[END_REF], also involving challenges in terms of robustness and optimal vocabulary choice, due to an arbitrarily high variation of terms and orthography, informal language, and spelling errors (Berard et al. 2019b). Its differences, compared to other domain adaptation MT task, lie in the fact that UGC can encompass multiple topics while maintaining increased lexical variation and grammar inconsistencies due to its rich productivity [START_REF] Michel | MTNT: A testbed for machine translation of noisy text[END_REF], as previously discussed.

We then review the literature comparing NMT and PB-SMT, highlighting their application for conducted experiments to determine how source-side error estimation can be related to different grammatical errors in translations using NMT models. However, none of the aforementioned works focus on processing noisy UGC corpora, and they do not consider the use of phonetizers and pronunciation similarity to recover correct tokens.

Normalization and handling ambiguities via DAGs

They aim to correct known tokens, such that a neural language model chooses the best output when an uncertain input is present (typically words with similar pronunciation from an ASR output). Instead, our approach calculates the phonetization of the source token and candidates are proposed based on their phonemic similarity to it, whether this original word is an OOV or not, i.e. an alternative phonemic orthography, or incorrectly using another existing word due to their pronunciation resemblance.

On the same trend, [START_REF] Qin | System combination for out-of-vocabulary word detection[END_REF]

Character-level MT for UGC

The impact of noise on the charCNN and char2char models has been evaluated by [START_REF] Belinkov | Synthetic and natural noise both break neural machine translation[END_REF] and [START_REF] Ebrahimi | On adversarial examples for character-level neural machine translation[END_REF] by artificially adding noise to canonical texts (the TEDTalk dataset). Their results show that the different character-level models fail to translate even moderately noisy texts when trained on 'clean' data, and that it is necessary to train a model on noisy data (either natural or synthetic) to make it robust. Note that, as discussed in Chapter 3, and as of the date of this dissertation, there is no UGC parallel corpus large enough to train an NMT model, and we must rely on the model's ability to learn, from canonical data only, noise-and error-resistant representations of their input that are robust to the specificities found in UGC. This is why, in this work, we are interested in studying the MT performance in a zero-shot scenario when translating noisy UGC, as discussed previously in Section 3.4.

To the best of our knowledge, no work has studied the performance of the characterbased NMT architectures on an actual UGC scenario with real-world gathered and fine-grained annotated noisy sentences. This sets the main motivation for the character-level MT study in Chapter 6.

Conclusion

In this chapter, we reviewed substantial reasons to be interested in designing robust MT systems that can correctly translate noisy UGC, especially for public and widespread translation engines.

As noticed, the ubiquity of UGC in the modern communication world, and the richness of being massively produced by a large number of users, make them also valuable and to be taken into consideration for different NLP tasks.

We discuss the specificities of UGC and provide a fine-grained typology that can be used

MT noisy translation with state-of-the-art systems

In this chapter, we start by reviewing and analyzing the behavior of off-the-shelf modern PB-SMT and NMT methods when processing noisy UGC, in order to identify difficulties related to UGC translation. In this respect, we intend to use both automatic metrics and detailed error analysis discussed in Section 2.4.2.

Even if NMT has become the predominant MT approach in recent years [START_REF] Yang | A survey of deep learning techniques for neural machine translation[END_REF], we decided to include a PB-SMT system in our comparison because, to the best of our knowledge, the robustness of this family of methods has never been evaluated on UGC. Indeed, PB-SMT's translation tables may behave differently to UGC specificities; thus, these may be interesting.

Zero-shot MT and UGC

As we have discussed in Section 3.7, due to its productivity in terms of new forms, new structures, or new domains, UGC is extremely variable and constantly evolving. Robust UGC MT is expected to cope with an arbitrarily wide range of unknown inputs, as UGC has an evolving nature [START_REF] Lobato | Histories of user-generated content: Between formal and informal media economies[END_REF], ultimately resulting in new terms over time (and consequently potential OOV tokens).

Furthermore, building a parallel UGC dataset large enough to train MT systems is not possible due to the scarcity of good, consistent and license-friendly real-world UGC data, as discussed in Chapter 3. This is why, in this work, we focus on a zero-shot scenario, with training data constrained to canonical, standard, and publicly available corpora, whereas UGC data are used only to evaluate the robustness of our systems. This choice, motivated by the nature of the data and the task at hand, is particularly restrictive and makes the development of translation systems substantially more difficult. In particular, it strongly constraints the use of all the finetuning methods, today, at the heart of many NLP methods [START_REF] Ramponi | Neural unsupervised domain adaptation in NLP-A survey[END_REF]. This experimental configuration was recently used as a new track in the WMT 2020 Robustness Shared Task [START_REF] Specia | Findings of the WMT 2020 shared task on machine translation robustness[END_REF].

In short, our aim is to improve the inherent robustness of MT models. Thus, in order to unequivocally account for the improvements to our methods, we remove any impact on performance caused by the means of using UGC data during training.

First baselines: getting started

To set a first comparison point and assess whether our proposed methods and pipelines result in any performance improvement, we started by training three mainstream off-the-shelf MT architectures using PB-SMT and NMT approaches. For the former, we used the well-known

Moses framework (Koehn et al. 2007a), and for the latter, we chose the two most commonly This first study also aims to provide a comparison and error analysis of PB-SMT and NMT when translating noisy UGC. More precisely, our contributions in this chapter are threefold:

• We compare the performance of PB-SMT and NMT systems when translating either canonical or noisy UGC text;

• we analyze, both quantitatively and qualitatively, several cases in which PB-SMT outperforms NMT on highly noisy UGC, and we discuss the advantages, in terms of robustness, that PB-SMT offers over NMT approaches;

• we explain how these findings support previous observations on the limits of recurrent

Addressing tokenization: adopting the first mainstream robustness techniques

Following the usual practice used in the literature to address the OOV problem, and as stated in Section 2.3, we have adopted Byte-Pair Encoding (BPE) tokenization [START_REF] Sennrich | Neural machine translation of rare words with subword units[END_REF].

This method has enjoyed extensive literature support, and its principle is aimed at leveraging on subword n-gram unit composition in order to resolve possible OOV. That is, using this tokenization method, the MT systems will decompose words in a sentence into a sequence of the K most frequent n-grams occurring in the train set to output a translation, where K, the vocabulary size, is the only parameter to be set. It is worth mentioning that, as discussed in Section 2.3, any MT system will automatically replace OOV by a special <UNK> token during inference, and, although BPE ensures almost no OOV tokens [START_REF] Araabi | How effective is byte pair encoding for outof-vocabulary words in neural machine translation?[END_REF], OOV characters (e.g. emojis) and rare characters (e.g. under-represented alphabets in the train set) are the exceptions.

We have chosen to use a BPE tokenization with a vocabulary size of 16K token, as it is a common parameter value for BPE in literature. We only consider this kind of tokenization since it has been virtually used in all NMT systems and systematically outperforms word-level tokenization [START_REF] Ding | A call for prudent choice of subword merge operations in neural machine translation[END_REF][START_REF] Yang | A survey of deep learning techniques for neural machine translation[END_REF]. Each tokenization model is trained on the corresponding dataset used to train the MT system, as described in Section 3.5. Surprisingly enough, it appears that the PB-SMT model outperforms both NMT architectures when translating PFSMB, the noisiest test set (please refer to Table 3.10 in Chapter 3). We can observe mixed results for the less-noisy UGC MTNT corpus. On the other hand, a consistent trend emerges: NMT consistently outperforms PB-SMT when translating canonical out-and in-domain test sets (newstest'14 and OpenSubTest). Overall, these results support the observations that NMT produces, a priori, the best results under in-domain evaluation conditions.

Results

However, such performance hides poor robustness for UGC test sets. This manifests itself as a performance gap between PB-SMT and NMT that reaches the highest value when comparing in-domain translation performance (+4.4 BLEU score on average 2 ). In the same train of thought, a considerable performance decrease can be observed when evaluating another less related canonical corpus (+2.6 BLEU on average for newstest'14), and continues to decrease for the relatively clean MTNT UGC corpus (+2 BLEU) to finally be favorable to PB-SMT for PFSMB (-3.1 BLEU).

These results are consistent with observations concerning the comparison of PB-SMT and NMT under low-resource conditions [START_REF] Dowling | SMT versus NMT: preliminary comparisons for irish[END_REF], out-domain evaluation (Koehn and Knowles 2017) and noise robustness [START_REF] Khayrallah | On the impact of various types of noise on neural machine translation[END_REF]; which highlight the relatively low performance of vanilla NMT systems compared to PB-SMT under these conditions. In the following sections, we will analyze these results to further highlight the difficulty of NMT models to generalize to UGC sentences by assessing their robustness capabilities.

Impact of vocabulary size

We have also trained a version of each of our architectures, i.e.

PB-SMT, seq2seq and Transformer, using a 32K vocabulary size in the LARGE OP E NSU B T I T L E S training corpus to evaluate the impact of the BPE vocabulary size on translation quality. This was motivated by the observation that, as can be seen in Table 4.1, our NMT models performed considerably worse than PB-SMT with a 16K vocabulary on LARGE OP E NSU B T I T L E S, even suffering a performance drop compared to SMALL OP E NSU B T I T L E S, which shares the same domain and origin, while being substantially smaller than the former.

The results achieved by this system are reported in Table 4.2, for the larger vocabulary version (32K), PB-SMT seems to severely lose generalization capabilities: it achieves a similar performance for in-domain test sets, whereas performance on other test sets drops sharply.

In contrast, NMT witnesses a generalization improvement. A similar observation has already been reported by [START_REF] Al | The impact of arabic morphological segmentation on broadcoverage english-to-arabic statistical machine translation[END_REF] for PB-SMT, and is explained by an excessive token fragmentation, which limits the contextual information captured by the phrases.

In all experiments described in this work, we chose to keep the 16K BPE segmentation, as this setting results in the best performance for both of our training configurations.

Error analysis: are PB-SMT systems better than NMT architectures when processing UGC?

The goal of this section is to analyze both quantitatively and qualitatively the output of NMT systems to explain their poor performance when translating UGC. Several works have already identified two main limits of NMT systems: translation dropping [START_REF] Sato | Japanese-english machine translation of recipe texts[END_REF], manifested as the production of sensible shorter prediction than the ground-truth translation, and excessive token generation, also known as over-generation (Roturier and Bensadoun 2011; Kaljahi et al. This phenomena were also observed in more recent works [START_REF] Ramesh | An error-based investigation of statistical and neural machine translation performance on hindito-tamil and english-to-tamil[END_REF], where NMT was found to produce translated token repetitions, omissions and spurious words when considering other low-resource scenarios. Next, we will analyze in detail how these two problems impact the MT models.

Translation dropping By manually inspecting the system outputs, we found that NMT models tend to produce shorter outputs than the translation hypotheses of the PB-SMT system, often avoiding translating the noisiest parts of the source sentence. For instance, the French sentence "Bon je veux regard te _en w _olf [...]"3 (Well I want to watch Teen Wolf [...]) is translated to a considerably shorter output by seq2seq, "I want to look at you", which results in an obvious change in meaning by the NMT system, also known as hallucinations [START_REF] Vikas Raunak | The curious case of hallucinations in neural machine translation[END_REF].

Analyzing the attention matrix for this observation, in Figure 4.2, shows that this problem is caused by a rare BPE token ("te"), part of the Named Entity "teen wolf", which is confused :(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((( (((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((( PB-SMT stop talking about teen wolf please :(( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( ˇ" ( with the very common French pronoun "te" (you in English), where the attention weights are maximal across the sequence. As a consequence, the seq2seq model suddenly stops translating because the hypothesis "I want to look at you" is a very common English sentence with a much lower perplexity than the (correct) UGC translation. A similar pattern can be observed with the Transformer architecture in the case of rare token sequences on the source side, such as in examples ¬ and ƒ in Table 4.3, causing the translation to stop abruptly, whereas this is not the case for seq2seq and PB-SMT.

The phrase-based model, PB-SMT, does not suffer from this issue, as there is no entry in the phrase table that matches the sequence of BPE tokens of the source sentence. This illustrates how alignment tables can be more efficient than soft-alignment produced by attention mechanisms for highly noisy cases, in particular when the BPE tokenization generates ambiguous tokens, which confuses the NMT model.

In order to quantify the translation dropping phenomenon, we show, in Figure 4.1, the distribution of the ratio between the reference (ground-truth) translation sentence lengths and the ones produced by PB-SMT and NMT for PFSMB. This figure shows that both NMT systems, (i.e. RNN-based and Transformers models), have a consistent tendency to produce shorter sentences than expected, while PB-SMT does not. These results show that off-the-shelf NMT systems produce overall shorter translations, as other authors have noticed [START_REF] Zhang | Improving neural machine translation through phrase-based soft forced decoding[END_REF].

Moreover, there is a substantial percentage of NMT predictions that are 60% shorter than the references, demonstrating the presence of translations that are dropped or shortened.

Over-generation A second well-known issue with NMT is that the model sometimes repeatedly outputs tokens that lack any coherence, thus adding considerable artificial noise to the output [START_REF] Tu | Modeling coverage for neural machine translation[END_REF].

When manually inspecting the output, it can be noticed that this phenomenon occurred in UGC sentences that contain a rare, and often repetitive sequence of tokens, such as "ne spooooooooilez pas teen wolf non non non et non je dis non" (don't spoooooil Teen wolf no and no I say no), in which the speaker's emotion is expressed by repeated characters (graphemic and punctuation stretching) and word repetition. These correspond to the "Marks of Expresiveness" UGC specificities category, reviewed in Section 3.3.

The attention matrix obtained when translating such sentences with a seq2seq model often shows that the attention mechanism becomes stalled due to the repetition of some BPE token, as shown, for example, in the attention matrix in Figure 4.3. More generally, it is noticeable that there are many cases in which the attention weights start increasingly focusing on the end-of-sentence token until the translation is terminated, while ignoring the source sentence tokens thereafter.

The transformer model exhibits similar problems (for instance, it translates the previous example to "No no no no no no no no no no no no no no no no no no"). The PB-SMT system does not suffer from this problem and arguably produces the best translation: "don't spooooooozt Teen Wolf, no, no, no, no, I say no".

Source-side artificially amplified noise

As we just observed in the qualitative analysis and translation examples, noisy occurrences can affect the MT system's performance to a greater extent than the original source-side, e.g. single noisy tokens in the input can produce a larger-span negative impact on the system representations of other tokens in the sentence. Therefore, to assess and quantify such a phenomenon and compare our MT systems within this scope, we report in Table 4.4 the Target-Source Noise Ratio (TSNR), introduced by (Anastasopoulos 2019) and previously described in Chapter 2.

As a reminder, a TSNR value greater than 1 indicates that the MT system adds more noise on top of the source-side noise, i.e. the rare and noisy tokens present in the source create even more noise on the output. Consequently, the higher the score, the more amplified the input-side UGC noise is (noisy artefacts caused by the MT system). That is, most of the noise and errors present in the translation hypotheses are due to errors of the MT system when processing noisy inputs.

This metric assumes that a corrected version of each source sentence is available. Hence, in order to compute this metric, we first manually corrected 200 source sentences of the PFSMB test corpus. By doing so, we produced a normalized canonical version of the French UGC source sentences, which is meant to be compared to the original (noisy) version. In Table 4 report TSNR scores for our MT systems trained using our three training corpora, i.e. WMT, SMALL OP E NSU B T I T L E S and LARGE OP E NSU B T I T L E S. First, it can be noticed that for all systems, the value is considerably larger than 1 (between 4 and 5), which means that these MT models amplify the UGC noise occurrences in a proportion of roughly 4-fold, on average. Next, we can observe that PB-SMT has a better TSNR score, thus adding fewer artefacts (including dropped tokens and over-translations) to the output. It is also worth noting that the gap between the PB-SMT and NMT architectures (about 0.3 BLEU points) is much larger when training on WMT than when training in our OpenSubtitles (about 0.1 BLEU points). This could be explained by a more related linguistic register of the latter to social media discussion UGC, than that of the former. Indeed, the perplexity of the PFSMB corpus on a WMT language model is 826, compared to roughly 599 when using OpenSubtitles.

Conclusions and perspectives

In this chapter, we evaluate the capacity of different off-the-shelf MT architectures to translate UGC, and we perform a qualitative analysis of translations to understand when and why these systems lack robustness to UGC. An in-depth analysis of attention-based mechanisms depicts common causes of low translation performance and how UGC can "crash" the translation process.

Our experiments show that PB-SMT systems are more robust to noise than NMT models, and we provide several explanations about this relatively surprising fact. Concretely, we observed and investigated the discrepancy between BPE tokens as interpreted by the translation model at decoding time; and the addition of lexical noise factors that can get amplified during translation.

We have also shown, by producing a new dataset with more variability, that using more training data was not necessarily the solution to cope with the specificities of UGC. The aim of this work is, of course, not to discourage the NMT system deployment for UGC, but to better understand what aspect of PB-SMT methods contributes to noise robustness.

We have made our best effort to rule out the vastly documented scenario where NMT is in disadvantage compared to PB-SMT due to low-resource availability [START_REF] Koehn | Moses: Open source toolkit for statistical machine translation[END_REF] Our findings are in line with the NMT issues described in the works discussed in Section 3.8, where rare vocabulary or unusual token sequences often trigger bad or truncated translations by NMT. The conclusion of (Anastasopoulos 2019) that found a correlation between the performance detriment of NMT and the noise in the source sentence is confirmed. Nevertheless, we also show that PB-SMT seems much more robust, and its performance on highly noisy UGC corpora is much more stable performance-wise compared to both studied NMT architectures (RNN and Transformer).

Now that we have our MT baselines and first exploratory results of translating UGC, in the next chapter, we start searching for a simple and modular normalization baseline by conceiving a pre-processing pipeline that bridges the gap between UGC and canonical texts through correcting phonetic writing, to subsequently let the MT systems translate the normalized form.

As we will see, this normalization pipeline leverages on the phonemic similarity between original UGC tokens and possible normalization candidates, which allows us to make corrections without any UGC normalization database.

Chapter 5

Addressing a frequent feature of

French UGC: phonetic writing

Having our MT baselines and first results on translating UGC in the previous chapter, we propose a simple and modular normalization baseline. We design a pre-processing pipeline that bridges the gap between UGC and canonical texts through correcting noisy occurrences, and later having the MT systems translate the normalized form.

As discussed in Chapter 3, there is an extensive presence of phonetic writing in UGC [START_REF] Moens | Mining User Generated Content[END_REF], and the contribution of such a phenomenon has been studied specifically in French digital media writing styles (Wachs 2017). In this regard, we have developed an automatic normalization pipeline that leverages on the phonetic similarity between original UGC tokens and potential normalization candidates, which enables us to make corrections without any UGC normalization database. By doing so, we intend to recover from noise in the source by correcting tokens and, then, translate the normalized sentences using off-the-shelf MT systems.

In further detail, in this section, we present our first attempts to improve MT translation by proposing a re-ranking pipeline that aims to correct phonetic writing. As seen in Chapter 5, phonetic writing ultimately results in an increase of OOVs, e.g "j'ai fait" (I have done) ! "g fé", both phrases being homophones in French, although the latter is incorrect and does not exist in canonical texts. The phonetic writing UGC specificity can also cause incorrect syntactic use of known words, e.g. "j'ai regardé" (I watched) ! "j'ai regarder" (I have watch). In this sense, this method aims to transform these unusual and non-canonical constructs into their canonical form, which can be translated by any standard MT system in a subsequent stage.

Our contributions are threefold:

• We propose a pre-processing pipeline to normalize UGC and improve MT quality without using UGC training data in any form;

• by quantifying the corrections made by our normalizer in our UGC corpora, we quantify the presence of noise due to phonetic writing and demonstrate that using the phonetic information can be potentially exploited to produce corrections of UGC without the necessity of any kind of annotated data;

• we explore the performance improvement that can be achieved in machine translation by using a phonetic similarity heuristic to generate different normalization candidates.

The work presented in this chapter was presented at the W-NUT workshop (Núñez et al.

2019

), co-located with EMNLP 2019.

Phonetic correction model

To automatically process phonetic writing and map UGC to their correct spelling, we propose a simple model based on finding, for each token of the sentence, words with similar pronunciations.

Next, we select the best spelling alternative, using a language model. More precisely, our approach is made up of 4 steps:

1. Phonetizing: the pronunciation of each source token is automatically produced. All words in the input sentence as misspelled tokens are not necessarily OOVs (e.g. "j'ai manger"

-literally "I have eat" -which must be corrected to "j'ai mangé" -"I have eaten", the French words "manger" and "mangé" having both the same pronunciation /mÃ.ge/); 2. Retrieving in-vocabulary tokens with similar pronunciation: using these phonetic representations, the method looks, for each word w of the input sentence, for every word in the training vocabulary with a "similar" pronunciation to w according to an ad-hoc metric we discuss below;

3. Encoding in Directed Acyclic Graphs (lattices): we represent each input sentence by a lattice of n + 1 nodes, where n is the number of words in the sentence, in which the edge between the i-th and (i + 1)-th nodes is labelled with the i-th word of the sentence. Alternative spellings can then be encoded by adding an edge between the i-th and (i+1)-th nodes, labelled by a possible correction of the i-th word. 4. Normalization: using a language model, the probability to observe each alternative spelling of every normalized token of the sentence is computed (note that, by construction, the original UGC sentence is also contained in the lattice) and find the most probable path (and therefore potential normalization) of the input sentence. Note that finding the most probable path in a lattice can be done with complexity proportional to the size of the sentence, even if the lattice encodes a number of paths that grows exponentially with the sentence size [START_REF] Mohri | Semiring frameworks and algorithms for shortest-distance problems[END_REF]. For these experiments, we used the OpenGRM [START_REF] Roark | The OpenGrm open-source finite-state grammar software libraries[END_REF] and OpenFST [START_REF] Allauzen | Openfst: A general and efficient weighted finite-state transducer library[END_REF]) frameworks that provide a very efficient implementation to score a lattice with a language model. This process can be seen as a naive spell-checker, in which we only consider a reduced set of variations, tailored to the specificities of UGC texts. We will now detail the first two steps discussed above.

Generating the pronunciation of input words

To predict the pronunciation of an input word, i.e. its representation in the International Phonetic Alphabet (IPA), we use the gtp-seq2seq

python library 1 to implement a grapheme-to-phoneme conversion tool based on a Transformer model (Vaswani et al. 2017). We use a 3 layers model with 256 hidden units that is trained on a pronunciation dictionary automatically extracted from Wiktionary (our dataset is further described below). This vanilla model achieves a word-level accuracy of 94.6%, that is, it is able to find the exact correct phonetization of almost 95% of the words of our held-out data.

1 https://github.com/cmusphinx/g2p-seq2seq

We also consider, as a baseline, the pronunciation generated by the Espeak program. 2 that uses a synthesis method to produce phonetizations based on acoustic parameters.

Finding words with similar pronunciation In order to generate alternative spellings for each input word, we look in our pronunciation dictionary for alternative candidates based on phonetic similarity. We define the phonetic similarity of two words as the edit distance between their IPA representations, all edit operations being weighted depending on the articulatory features of the sounds involved. Thus, to compute the phonetic similarity, we used the implementation (and weights) provided by the PanPhon library [START_REF] Mortensen | Panphon: A resource for mapping IPA segments to articulatory feature vectors[END_REF]).

To account for peculiarities of French orthography, we also systematically consider alternative spellings in which diacritics (acute, grave, and circumflex accents) for the letter "e" (which is the only one that changes the pronunciation for different accentuation in French) were added whenever possible. Indeed, users often tend to 'forget' diacritics when writing online, and this kind of spelling error results in phonetic distances that can be large (e.g. the pronunciation of bebe and bébé is very different). We do not add diacritics on other French vowels, as accents do not change the French pronunciation for these, and, doing so increases the phonetic search space substantially.

We ultimately only keep as candidates words those that are present in the training corpus presented in Chapter 4 to filter out OOV and non-existent words.

Pronunciation dictionary To train our Grapheme-to-Phoneme model, we use a dictionary mapping words to their pronunciation (given by their IPA representation). To the best of our knowledge, there are no easily accessible pronunciation dictionaries for French. In our experiments, we have considered a pronunciation dictionary automatically extracted from Wiktionary dumps based on the fact that, at least for French, pronunciation information is identified using special templates, making its extraction straightforward [START_REF] Pécheux | Reassessing the value of resources for cross-lingual transfer of pos tagging models[END_REF]).

The dictionary extracted from French Wiktionary contains 1,571,090 words. We trained our G2P phonetizer on 1,200,000 examples, leaving the rest to evaluate its performance. When looking for words with similar pronunciation, as discussed in the previous section, we consider only the words that appear in our parallel MT training data (described in Chapter 4) to speed 2 espeak.sourceforge.net up the search and rule out any possible OOV during evaluation. After filtering, our dictionary contained pronunciation information for roughly 82K French words. On the other hand, in Figure 5.2, we show the effect of changing the maximum phonetic distance threshold in order to consider that any given word in the training word-level vocabulary should be retained as a normalization candidate in the lattice. It is straightforward to see that a higher value of this parameter leads to a linear increasing quantity of word-level replacements.

Statistics of the normalization process

However, due to the combinatorial nature of the lattices, a value that is too high has, as a direct outcome, an exploding quantity of candidate normalization phrases, and consequently, a higher computational cost.

To avoid an explosion of the number of alternatives that we consider, and upon experimentation, we determined that the best phonetic distance threshold value was 0.1 for these experimental conditions, which, as can be seen in Figure 5.2, corresponds to keeping only homophone normalization candidates, in which we could observe the most conservative normalization without introducing excessive noise to possible DAG paths, as discussed previously. This results, as can be seen in the figure, in a total of 600 replacements that take place in the PFSMB test set, which corresponds to changing 5% of the original tokens.

Machine translation results

Table 5.2 show the MT BLEU performances, ultimately produced by the proposed method along with the baselines, PB-SMT and Transformer MT systems, described in Chapter 4, and whose results are reported again to make comparison easier. We noticed a significant improvement in the results for the UGC test corpora when using the Transformer architecture trained with the SMALL OP E NSU B T I T L E S training set. Specifically, a BLEU score improvement for the PFSMB and MTNT test corpora when using G2P and Espeak phonetic normalization in Table 5.2, compared to the baseline MT systems. Interestingly, these improvements only hold for the Transformer model, whereas we consistently obtain a slight decrease of BLEU scores when the normalized text is translated using the PB-SMT model.

Concisely, we noticed an improvement of +1.5 BLEU for PFSMB using the G2P phonetizer, and +0.5 for MTNT when using the Espeak phonetizer instead. Regarding the performance decrease in the canonical test corpora, newstest'14 and

PB-SMT

OpenSubtitles, we can observe that there is usually a considerable under-performance on the latter (-0.65 BLEU averaging over the 6 models and training set configurations), which is not as noticeable in the former (-0.1 BLEU in the worst case). This could be explained by the substantially longer sentences in newstest'14 compared to OpenSubtitles, which have roughly 6 times more words on average, according to Table 3.9 in Chapter 3. That is, when sentences are longer, the number of possible paths in the lattice grows exponentially, thus increasingly adding confusion to the language model's decisions, which will ultimately produce the most probable normalization. This observation strongly suggests that the performance of the normalizing method depends on the length of the target sentence that is to be normalized. Motivated by this observation, we have also calculated the BLEU score of the PFSMB corpus by groups of sentences' length in order to further investigate why this method improves the translation quality of Transformer but not for PB-SMT models. The results reported in Figure 5.3 show that the highest improvement resulting from phonetic normalization is present in short sentences (between 1 and 10 words). It is worth noting that this is the only situation in which Transformer outperforms PB-SMT. Hence, the higher overall Transformer BLEU score over PB-SMT is certainly due to a relatively high successful normalization over the shortest sentences of the PFSMB test set. This is in line with the documented fact that NMT is consistently better than PB-SMT on short sentences [START_REF] Bentivogli | Neural versus phrasebased machine translation quality: a case study[END_REF]) and, in this concrete example, it seems that Transformer can take advantage of this when this normalization pipeline is applied.

Furthermore, these results could be considered as evidence supporting the conclusion that the proposed method generally performs better for short sentences, as observed in Table 5.2.

Furthermore, we have applied our method to a set of blind tests of UGC corpora MTNT and PFSMB. These results are displayed in Table 5.3; we also show the performance of the (Michel and Neubig 2018)'s baseline system on such test sets. The translation system is selected as the best for each of the UGC sets in Table 5.2. For this test corpus, we noticed a 0.5 and a 3 BLEU point decrease for the Transformer and PB-SMT systems, respectively, when our normalizer is used on the MTNT blind test. On the other hand, we obtained a 0.7 BLEU point loss for Transformer and a 3.6 point drop for PB-SMT, both evaluated on the PFSMB blind test. These results suggest that, when we do not tune our method (using an adequate UGC development set) looking for the best translation system, and for certain UGC sets, our approach introduces too much artificial noise, and MT performance can therefore be negatively impacted.

Qualitative analysis

Table 5.4 reports some examples of the output of this method, along with their translation before and after correction by our phonetic normalization process.

For Example ¿ in Table 5.4, it can be observed that the normalizer enables the MT system to produce the first part of the translation ("When I get to the taff ") correctly. This is the result of correcting the French homophones "arriver " into "arrivé", i.e. from the infinitive to the past form. It is very interesting to note that the robustness of the Transformer using subword units, seems to be good enough to correctly translate the typographical error "ce met a battre", thus, the correct proposed normalization ("se met à battre") does not impact the MT result but 5.4: Examples from our noisy UGC corpus normalizations. We show the original UGC source (src), reference translation (ref), the normalized source produced by our approach (norm src), the translation produced by Transformer from the original source (raw MT) and the one using the normalized source (norm MT).

certainly has an effect over the correctness of the French phrase.

Regarding Example ¡ in Table 5.4, it should be noted that the normalized proposition significantly improves MT translation, producing an output closer to the reference translation, compared to the raw MT output. The key normalization change is the misused French token "faite" (pronounced /fEt/) -"does" in English -by its correct homophone "fête" -"celebrates" in English. It can also be noticed that the robustness of the MT system is once again capable of correctly translating a phonetic writing contraction "c" as the two correct tokens "c'est".

Example ¬ in Table 5.4 shows how semantically different can be a misused French word due to confusing homophones. We can observe that the normalization replacement "nez" ("nose" in English) ! "né" ("born" in English), which are French homophones, drastically changes the meaning of the output translation. Additionally, the correction "marqué" ! "marquer"3 (changing to correct verb tense) also causes the translation to be closer to the reference.

Finally, in Example √ in Table 5.4 we show some caveats and limitations for our proposed method, where the correct original plural "Cartes bancaires ... retrouvés" was changed to the singular form "Carte bancaire ... retrouvé". This is due to the homophonic pronunciation of most French singular and plural pronunciations. Whenever there is no discriminant token with different pronunciation, such as an irregular verb, the language model has trouble choosing the correct final normalized phrase since both plural and singular propositions are proposed as candidates.

Thus, either can be indistinctly kept as final normalization, since both forms are correct and theoretically very similar in their perplexity measure.

Conclusions

In this chapter, we have proposed a pre-processing method that relies on phonetic similarity to normalize UGC. Our method can improve the translation quality of UGC of modern off-the-shelf NMT systems. Conversely, we have performed an error analysis showing that the MT system successfully translates phonetic-related errors with its increased robustness. However, it must be noted that we obtained negative results on a blind test evaluation, suggesting that the phonetic normalization approach introduced more noise than useful corrections on totally unseen data. This highlights the importance of holding out data in the form of blind test sets so that the real efficiency of an MT system can be verified. In addition, we have applied our normalizer to clean canonical test data and have shown that it slightly hurts MT performance. More in-depth studies are needed to assess whether our proposed normalization pipeline can correct phonetic-related errors in UGC for other languages and other difficult UGC scenarios, such as video game chat logs [START_REF] Héctor | From noisy questions to Minecraft texts: Annotation challenges in extreme syntax scenario[END_REF], while maintaining good performance on canonical texts.

Although our normalization pipeline showed some interesting corrections and helped NMT recover from errors that otherwise it could not, it also turned out to add artificial noise to some extent and displayed poor generalization over our blind test, showing a negative impact for the PB-SMT translation system. In this sense, our proof-of-work could benefit from other heuristics and constraints for the re-ranking, as well as further investigating and refining the phonetic distance measuring. Subsequently, we decided to focus our efforts on exploring other alternatives to cope with UGC by using end-to-end learning representations of NMT models to account for a wide spectrum of different specificities (discussed and listed in Section 3.3) that can be resolved using morphological cues, instead of pinpointing a single type of UGC idiosyncrasy. We thus investigate finer granularity-level translation (i.e. char-based) to assess how well char-based performs when translating UGC compared to the coarser translation granularities used so far in the dissertation. Furthermore, we also realized that a single corpus-level BLEU score was not helpful to assess whether the proposed MT models perform better or worse than the baselines with respect to any given UGC specificity. This led us to propose, in the next chapter, a UGC evaluation framework that links performance change with concrete UGC specificities, uncovering many of the performance and robustness caveats and advantages of a given MT system in terms of well-defined UGC specificities.

on this issue, we study the open-vocabulary capacities of character-based models and proposed a simple way to deal with charOOVs, by strongly shortening the character vocabulary size, which we also found had low impact on canonical and in-domain translation. Specifically, our approach uses substantially smaller character vocabulary size than standard characterlevel model configurations constitutes a call for action to correctly tune the vocabulary size, a parameter that has been overlooked in the literature.

During our experiments, we noticed that, in order to correctly and better assess the advantages of our proposed methods for translating UGC, a single metric that evaluates the translation for a corpus with several confounded phenomena was not sufficient to identify how distinct UGC specificities are handled differently by a given model. This led us to develop a UGC evaluation framework that aims to rigorously characterize the capabilities of an MT system to cope with UGC evaluation. This framework is described in Section 6.5.1.

In the final part of this chapter (Section 6.7), we combine character-level NMT with a phoneticized version of the source to produce the target translation characters, which we refer to as phon2char, and whose robustness capabilities are also assessed using our evaluation framework.

Why character-based MT?

We intend to use character-level NMT approaches to investigate to what extent they are robust to UGC specificities and noise at test time. Character-based MT seems to be a promising method to translate UGC because of its open-vocabulary property: character-based systems can model the inner semantics of potential word-level OOVs and will translate them, instead of simply generating a <UNK> token as in usual word or sub-word tokenizations. This feature is appealing for translation when spelling errors and, more generally, OOVs, can be mapped to canonical (sequences of) tokens that are present during training. Indeed, in this case, the usual methods for treating OOV (copying them from the source using alignment-based methods and the <UNK> token) will not attempt to translate them. It should however be noted that implementing this intuition has to avoid a potential pitfall: in UGC many OOV are named entities (such as hashtag, people names, etc.) that are often the same in the source and target languages.

Character-based models

As discussed in Section 2.3, character-level models pose inherent challenges, namely, increased computational costs and long-range dependency modeling. In order to overcome these difficulties, convolutional neural network encoding has been explored to get the best of both worlds, i.e. maximizing the capacity-to-compression ratio [START_REF] Cherry | Revisiting character-based neural machine translation with capacity and compression[END_REF]) of a given NMT base architecture. We investigate two of these character-based architectures. The first model we consider, charCNN [START_REF] Kim | Character-aware neural language models[END_REF], is a classic encoder-decoder in which the encoder uses character-based embeddings in combination with convolutional and highway layers to replace the standard lookup-based word representations. The model considers, as input, a stream of words (i.e. it assumes the input has been tokenized beforehand) and tries to learn a word representation that is more robust to noise by unveiling regularities at the character level. Thus, it should be noted that, despite learning character-based representation for the source words, charCNN processes the input and decoding at the word level. This architecture was initially proposed for language modeling Costa-jussà and Fonollosa ( 2016); shows how it can be used in an NMT system and reports improvements up to 3 BLEU points when translating from a morphologically-rich language, German, to English.

The second model we consider does not rely on an explicit segmentation into words: [START_REF] Lee | Fully character-level neural machine translation without explicit segmentation[END_REF] introduce the char2char model that directly maps a source character sequence to a target character sequence without any segmentation (spaces being considered as a vocabulary element) thanks to a character-level convolutional network with max-pooling at the encoder. It can be considered as an open-vocabulary model: it can generate any word made of any of the N most frequent characters of the train set (where N is a model hyperparameter) and only outputs a <UNK> token in the presence of a character that is not in this (char-) vocabulary. [START_REF] Lee | Fully character-level neural machine translation without explicit segmentation[END_REF] show that this model outperforms subword-level (i.e. BPE-based) translation models on two WMT'15 tasks (De-En and Cs-En) and gives comparable performance on two tasks (Fi-En and Ru-En). [START_REF] Lee | Fully character-level neural machine translation without explicit segmentation[END_REF] additionally report that in a multilingual setting, the character-level encoder significantly outperforms the subword-level encoder on all the language pairs. As expected, all models perform better when trained on the OpenSubtitles corpus than when trained on WMT, as the former is intuitively more similar to UGC data than the latter.

Results

Moreover, it appears that character-based models are largely outperformed by BPE-based models for most train-test combinations and, therefore, that their relative capacity to learn word representations that are robust to noise can be questioned. We explore some ways to reduce the performance gap between these two NMT approaches in the following.

Another interesting observation is that, while Transformer achieves the best results in all test sets when trained using OpenSubtitles, it is outperformed by seq2seq on the WMT training configuration. The situation is similar for character-based models. This observation suggests that these models do not capture the same kind of information and do not have the same generalization capacities, as they roughly have the same number of parameters (69M parameters for the char2char and Transformer models, 65M for the charCNN and 49M for the seq2seq model).

Finally, the impact of charOOVs on BLEU performance is shown in Figure 6.1, comparing the results for the Transformer and char2char models for different numbers of such tokens in the inputs of the PFSMB test corpus. It can be seen that the former is considerably more sensitive to occurrences of charOOVs, showing a consistent performance detriment as there are more of them in the inputs. In turn, Transformer is increasingly good when more charOOVs appear in the sources, which can be explained by the fact that most charOOVs are left unchanged between the source and the reference (e.g. emojis, # and @). In this sense, Transformer

shows a clear advantage due to its ability to manage <UNK> tokens, present during training to some extent, whereas the char2char system does not produce any <UNK>, being unable to output charOOVs, and, furthermore, these occurrences affect the rest of the input sequence, leading the model to produce wrong translations as shown in Table 6.3.

Error analysis

In order to find which kind of UGC specificities are the most difficult to translate and can explain the difference in performance between character-based and BPE-based systems, we have conducted a contrastive analysis between the predictions of the Transformer and the char2char models. For each system, we have selected the 100 source sentences with the best translation and the 100 ones with the worst translation. 1 We have manually annotated these 400 sentences, using the typology described in Finally, in Figure 6.1, it can be seen that for the PFSMB phrases that do not have any charOOVs, the char2char BLEU score is much similar than that of the Transformer in Table 6.1. Concretely, the performance gap is reduced from 3.7 BLEU points to 1.9 in the absence of charOOVs, while other UGC specificities are still present. This encourages us to further investigate the char2char behavior when charOOVs are present in the input, which we further investigate using the copy task control experiment below.

Qualitative analysis Table 6.3 reports translation hypotheses predicted by the Transformer and char2char models. These examples illustrate the difficulty of the char2char model to translate named entities: while the simple strategy of copying unknown OOVs from the source implemented by the Transformer is very effective, the char2char model tends to scramble or add letters to NE. They also illustrate the impact of phonetic writing on translation: for instance "rescend" that should be spelled "ressents" (example 4) and "joue a" that should be spelled "joue à" (example 5) both result in a wrong translation: in the former case, the meaning of the sentence is not captured, and in the latter the "a" is wrongly copied in the translation hypothesis.

Copy task control experiment

To corroborate our analysis of the impact of special characters on char2char and quantify the impact of charOOVs and of rare character sequences, we conduct a control experiment in which a char2char system with different vocabulary sizes must learn to copy the source, that is to say: we train the char2char model on an artificial parallel corpus in which the source and target sentences are identical. By varying the number of characters considered by the system, we can control the number of rare characters in our corpora (recall that with a char-vocabulary of size N all characters, but the N most frequent ones are mapped to a special <UNK> character).

Note that this task is a simplification of the problem we have highlighted in the previous section:

the model simply has to learn to always make an exact copy of the whole source sentence and not to detect some special characters (such as # or @) from the input that trigger the copy of the next characters, while the rest of the sentence should be translated.

More precisely, we built an artificial train set containing 1M random sentences with lengths between 5 and 15 characters long, keeping a 164 fixed-size character vocabulary (this corresponds to the size of the extended ASCII alphabet), and whose characters are distributed uniformly, in order to rule out the impact of rare characters and keeping only the effect of char-OOVs over the performance. We consider two test sets made of 3,000 sentences each: in-test that uses the same 164 characters as the train set and out-test that uses 705 different characters.

Source and reference are identical for every example of the train and test sets.

Results Table 6.3 reports the results achieved on the copy task with and without replacing the predicted <UNK> symbols.

Note that, in this very simple task, <UNK> characters are always replaced by their true value.

These results show that this task is not trivial for char-based systems: even when all characters have been observed during training, the system is not able to copy the input perfectly.

Above all, reducing the vocabulary size from 164 to 125 results in an increase of the BLEU score on the two considered conditions, even without replacing the <UNK> that the system has started to generate, where '%<UNK> pred.' indicates the percentage of <UNK> tokens in the prediction. Further reducing the size of the vocabulary artificially improves the quality of the systems: they generate more and more <UNK>, which are replaced during post-processing by their true value. These observations suggest that unknown or rare characters are not only difficult to copy, but they also distort the representation of the input sequence built by the system, impacting the whole prediction. ) and percentage of <UNK> characters in the prediction (%<UNK> pred.).

Robustness impact of UGC's specificities

In order to quantify and link robustness and performance impact of NMT when processing noisy UGC, we annotated 400 random-sampled PFSMB source sentences according to the typology shown in Table 6.2, including a software framework to control the type and occurrences of noise in the source, and, automatically producing a comparable reference for evaluation.

The PMUMT corpus To understand the impact of UGC peculiarities we created the Phenomena Modeling UGC Machine Translation (PMUMT) by manually annotate 400 source sentences sampled from the PFSMB: one of the authors, fluent in French and with good knowledge of UGC, has identified spans in the sentence that differ from canonical French and characterized these specificities using the fine-grained typology of [START_REF] Sanguinetti | Treebanking user-generated content: a UD based overview of guidelines, corpora and unified recommendations[END_REF] (see Table 6.2). Since the whole annotation process was done by a single person, no inter-annotator agreement can be calculated. Nevertheless, results of our pilot study for each individual UGC peculiarity (cf.

Table 6.5 for a cross-metrics analysis), show that MT performance consistently performs better on our normalized corpus than on the original noisy set.

Each span containing a UGC specificity has been 'normalized' to a form closer to canonical French.3 Table 6.4 shows some examples of annotated (source) sentences. Additionally, a normalized form of each target (i.e. English) sentence has also been produced to ensure that the target can be generated from the 'normalized' source.

In the end, the annotation of this corpus represents 200h of work, comprising an iterative improvement and debugging of the annotations to achieve the corpus' current version. 4On the other hand, to reduce the lexical variability and model the impact of UGC's named entities on translation quality, all occurrences of these have been manually mapped to a frequent named entity in the train set, 'Jean' (a French first name). A similar strategy has been proposed

by [START_REF] Marton | Improving Arabic dependency parsing with lexical and inflectional morphological features[END_REF] to reduce the diversity of number in parsing. Considering a 'real' named entity allows us to preserve the structure of the sentence, which would have been compromised

by simply erasing NE. These approaches proved to be experimentally adequate, since we can see in the column named entity in Table 6.5 that our fully-normalized version using this trick, performs better than its noisy counterpart (the original UGC corpus only keeping named entities occurrences) across our MT models and metrics. The resulting corpus contains more than 1, 310 annotations. On average, each sentence contains 2.8 UGC peculiarities; the average span length of the UGC specificities is 1.27 tokens and its maximal value 17 tokens (a full sentence in upper-case). Figure 6.4 describes the distribution of UGC peculiarities in the corpus. It appears that the most frequent kinds of specificities found in UGC are the presence of named entities and special characters (e.g. @ or #) or emoticons. Although the least frequent type is the inconsistent case changing, these have the longest span among all the UGC specificities, as some sentences are written completely in uppercase.

Additionally, in Table 6.4, we display some of the annotated examples using their corresponding typology (Table 6.2) and position in the source and reference sentences.

¿

Controlling the Number of Specificities per Sentence

Thanks to the alignment between the UGC specificities in a sentence and its normalized form, it is possible to create a normalized version of each source sentence that is closer to canonical French. Comparing the predictions of an NLP system, taking either the normalized sentences or the original non-canonical sentences as input, allows us to measure the impact of UGC on this system. However, it is impossible to perform a fine-grained analysis in which, for example, the impact of different types of specificities are compared, since UGC sentences generally contain several specificities of different types and the interactions between them cannot be easily neutralized: in our corpus, there are only 68 sentences in which a single kind of specificities occur, i.e. 17% of the original sentences.

That is why, a second version of the corpus has been automatically generated to aid analyzing the interactions between the UGC specificities in a sentence: by substituting only some of the span we have annotated, it is possible to create corpora in which the number and the kind of specificities present in each sentence is tightly controlled: we can, for instance, generate a corpus in which sentences have at least two specificities of the 1 st , 2 nd and 3 rd kind. In this framework, each original sentence can be (partially) rewritten into as many sentences as there are UGC specificities in it.

This possibility of partial substitution greatly reduces the amount of data to be annotated for the analyses: instead of having to annotate a large amount of data to find enough sentences fulfilling the requested criteria, the framework is capable of generating these sentences from our original annotation of 400 sentences. For instance, by substituting all spans but one in our original corpus, we have generated a corpus of 1,282 sentences each containing exactly one UGC specificity and by substituting all spans but two, a corpus of 1, 176 sentences containing two different specificities. We will see in Section 6.5.1 the importance of controlling the number of specificities per sentence to unravel the complex interactions between them. We have used the PMUMT corpus to evaluate the impact of UGC peculiarities on translation quality: we have reported in Table 6.5 the BLEU scores achieved by the considered systems on both the 400 original sentences and the 400 normalized sentences. As expected, translations of normalized sentences, that are more similar to the training data, are of better quality than translations of original (noisy) sentences: the BLEU scores achieved when translating normalized UGC content are close to those obtained on the in-domain test-set.

Impact of UGC specificities on translation quality

For all systems, considering the non-canonical original sentences results in a drop in translation quality of the same order of magnitude, which shows that, even if these models build sentence representations from completely different information, the presence of UGC peculiarities have a similar impact on all of them.

Individual UGC errors

To get a more precise picture of the impact of UGC on translation quality, we have computed, for each kind of peculiarities, the BLEU scores achieved on the corpus built to contain only this peculiarity and the BLEU score computed on the 'normalized' version of the same sentences. Table 6.5 reports the ratio between these two scores. Additionally, we have calculated the translation scores using more metrics, namely, CHRF and MULTI-BLEU-PERL-DETOK (MB) along with their 95% confidence interval. This is done in order to ensure that we have a large-enough data collection to obtain a stable corpus-level metric, such as BLEU. The impact of a given kind of UGC specificity on translation quality is very different from one system to another: It appears that the source sentences representation that MT systems learn to construct are not sensitive to the same kind of noise or errors in the source sentence, and even seem to be complementary. For instance, inconsistent casing strongly penalizes the seq2seq model but has only a limited impact on the char2char model (with a ratio of, respectively, 0.75 and 0.86 BLEU scores). In contrast, the presence of characters specific to online conversation, such as @ or # results in a substantial decrease in translation quality for char2char with a BLEU ratio of 0.81 , but has less impact for seq2seq or Transformer (with a ratio of, respectively, 0.88 and 0.92) , suggesting that character-based models are not able to properly model characters that hardly appear in the training set.

Interestingly, the Transformer model appears to be very robust to a wide array of UGC peculiarities, even if it was not designed specifically to handle noisy input: in particular, the presence of named entities, spelling errors (i.e. substitution, deletion, or insertion of letters), agreement error (of verb tense or in gender and number) as well as tokenization errors hardly hurt translation quality. Similarly, the char2char model succeeds in correctly translating sentences with letter addition or suppression, showing that the model actually manages to learn sentence representations that are robust to spelling errors, even if such errors are not present at training time. This result contrasts with the conclusion drawn by [START_REF] Belinkov | Synthetic and natural noise both break neural machine translation[END_REF] on artificially noisy data. However, it fails to capture regularities that are expressed in longer spans (e.g. in named entities).

It is worth noting that missing diacritics, phonetic writing and wrong tense categories give a counter-intuitive result, with a ratio larger than 1.0 (this value being within the confidence interval for the latter), implying that noisy inputs for these UGC specificities seem easier to translate for the Transformer. To further investigate this, we have evaluated these corpus partitions again using complementary metrics (discussed in Section 2.4.2), namely, MULTI-BLEU-DETOK.PERL (Koehn et al. 2007a) and CHRF [START_REF] Popović | chrF: character n-gram F-score for automatic MT evaluation[END_REF] in order to recalculate the results in question. In this regard, in Table 6.5 it can be seen that at least one of the metric results in a ratio larger than 1.0, to which we attribute a difference between both BLEU pieces of software.

Combination of specificities Table 6.6b also shows that UGC specificities seem to have less impact when sentences are selected to contain a single specificity: for instance, the char2char has an average ratio of 0.942 on the corpora used to compute the scores reported in Table 6.5, while its ratio is 0.79 on the original corpus. To better understand the impact of combinations of UGC peculiarities on translation quality, Table 6.6b reports the ratios between the BLEU scores computed on the translation of a corpus in which there are exactly N different UGC peculiarities in a sentence and on the translation of the normalized version of these sentences.

It appears that for all our systems' translation quality decreases linearly with the number of specificities, suggesting that the impacts of the different specificities are independent of each other. Surprisingly enough, the gap between the char2char and Transformer is getting smaller with the number of specificities in each sentence.

It can be seen that, initially, the ratios decrease quickly for the three models and, as there are 3 or more UGC specificities, they get to a plateau (plotted in Figure 6.6a). This suggests that the drop in performance caused by such occurrences is substantially more important as soon as noise starts appearing, and their aggregated effect does not increase linearly. Regarding the systems' performance on the normalized versions, we can see an overall downtrend for char2char and Transformer, probably due to a bias caused by a higher number of UGC specificities being contained, naturally, by longer sentences. Another important factor and a limitation of our data augmentation approach, is that there is an increasingly constrained sentence evaluation diversity, since phrases with a high number of UGC idiosyncrasies are relatively rarer and each original PFSMB sentence can generate more augmented samples, proportion is given by the 'Augment ratio' in the Additionally, it can be noticed that scores over the normalized sentences are mixed but their weighted averages by the number of phrases in each column, confirm the Transformer, seq2seq and char2char observed performance trend in the test sets in Table 6.1. Interestingly, in Table 6.5, where we present the quotient between noisy and normalized versions per UGC peculiarities within a 95% confidence interval, we can see that errors 1 -letter change/deletion/addition, 2 -diacritics omission/error and 5wrong verb tense for char2char, have a very close ration to 1.0, especially when compared to their BPE-based counterpart with the same architecture, seq2seq. Examples of the impact of such occurrences on MT performance are displayed in elements ¿, ¡ and ¬ in Table 6.6. On the other hand, 8 -inconsistent casing seems to impact the performance of char2char the least compared to the BPE seq2seq, suggesting higher robustness of the latter to this type of UGC particularity, for which examples » and … demonstrate a noticeable difference in performance. These two observations are in line with the findings of [START_REF] Niu | Evaluating robustness to input perturbations for neural machine translation[END_REF], which found subword-regularized NMT to be more robust to misspellings in general (here divided in the former 3 UGC specificities evaluated using character-level regularization of char2char)

and case-changing. Additionally, we can also notice some advantage of using char2char

when for the error 4 -faulty tokenization over seq2seq, but it is surpassed by the BPE-based Transformer, as can be seen in example ƒ and ≈ in Table 6.6.

Interestingly, on the other hand, we can notice that error 6 -special character and 9

-emojis/emoticons (usually also out-of-vocabulary) caused the highest relative performance drop when comparing char2char to seq2seq. Such observation is illustrated in example , where we notice the difficulty of the char2char model to output such domain-related characters.

This motivates our efforts in the following sections to make char2char more robust to these occurrences, which we address as out-of-vocabulary characters (charOOVs) from now on.

Improving robustness by learning to manage OOV characters

This set of experiments is motivated by observing that, even though character-level NMT systems are often praised by their word-level open vocabulary, we observed the downside that characters not presented (charOOVs) or underrepresented in the training data impose challenges to these architectures. In this respect, we hypothesize that producing the special <UNK> token, to be replaced by a source-side token during post-processing (as discussed in Section 2.3) can be a better alternative to avoid incorrect translation for some sequences that are underrepresented or not contained at all in the training data. In our framework, this aspect is especially important since the robust NMT systems we aim to conceive for noisy social media text can contain any kind of charOOVs (notably an ever-growing set of emojis) only limited by the users' will. In order to study the occurrences of abnormally short (dropped translations) or long predictions (over-translation), we display the length ratio between predictions and references for the noise UGC test set, PFSMB, and the clean corpus OpenSubTest in Figure 6.7. It can be seen that the base char2char model with 300 characters in the vocabulary incurs in 8% and 5% over-translation for such test sets, respectively. When we vary the number of characters retained in the vocabulary, we succeeded to control such behavior. However, for other sub-optimal vocabulary sizes (90, 75, 70, and 65), dropped translations were present with up to 16% and 5% of shortening on the predictions with respect to the references on PFSMB and newstest'14, respectively. It is interesting to note that the variability between the lowest and the highest prediction length is considerably bigger for the noisy UGC test. These results point to the fact that an optimal choice of the vocabulary size can alleviate excess tokens production.

Phoneme2Char translation: a character-based phonemic MT

Our method presented in Chapter 5 cannot resolve phonetic writing errors that are segmented on multiple source tokens, and that cannot generate more than one normalization tokens (we have only considered single words when looking for phonetically-closed elements in the dictionary described in Section 5.1, otherwise, the computation time becomes prohibitive). With this in mind, we intend to overcome such limitations by exploring the possibility of training MT systems that can directly translate from the French phonetization form of sentences to English text.

By doing so without using an explicit tokenization, segmentation is either determined using phoneme-based BPE units or 1-grams for comparison purposes, producing NMT systems that fully perform the complete phonetization-normalization pipeline.

Phoneticizing and translating For the data used, we phoneticized the French source-side of the OpenSubtitles dataset training, development and test corpora used to train and evaluate its respective MT baseline in this chapter, using the G2P phonetizer, as seen in Section 5.1.

Characters and sequences that are not phonetizable were detected using a regular expression and kept unchanged in their corresponding relative position with respect to neighbor phonemes.

To keep the same case-sensitive MT comparison configuration, the original pre-phonetization casing is encoded in the source phoneme sequence to learn to generate the corresponding target casing, by using two different placeholders ("<T>" to mark capital letters and "<U>" for fully-uppercased tokens) to keep the case information in the phoneticized version of the source sentences leveraging on Factored NMT [START_REF] Garcia-Martinez | Factored Neural Machine Translation Architectures[END_REF], corresponding to the inline casing approach used in [START_REF] Berard | NAVER LABS Europe's Systems for the WMT19 Machine Translation Robustness Task[END_REF].

Three models are compared: the Transformer model, either with a 16K BPE tokenization over the phonemes of IPA character-level segmentation; and the fully character-level char2char model [START_REF] Lee | Fully character-level neural machine translation without explicit segmentation[END_REF], that we refer to as phon2char, standing for the phoneticized source version. The latter, is supposed to process the input as a stream of characters, which segment (n-grams, possibly overlapped) embeddings are extracted end-to-end using a CNN block, as seen in Chapter 2. Additionally, we also show the MT results using a Transformer model to compare how self-attention NMT systems would perform on this phoneme-to-text task approach.

For the Transformer, we have chosen to explore using an IPA-level character segmentation and a 16K BPE tokenization version.

Additionally, we have experimented using two versions of the phoneticized source corpus for training: either keeping the spaces between original words after phonetization to take advantage of the original tokenization information, or removing such spaces and leaving only the phonemes.

We found that the latter consistently outperformed the former, probably because of inter-word liaisons in French pronunciation, however, since leaving the spaces leads to inherently larger source sequences, and we do not have access to enough UGC data to produce a control corpus of both approaches with comparable sequence lengths, we cannot rule out the impact of sequence length, nor it is in the scope of this work. These experiences, thus, only served the exploratory purposes of identifying the best pre-processing in order to produce the most performing phoneme-to-text MT systems.

Results

In Table 6.8, the results of using the alternative method discussed previously.

Robustness of our proposed character-based NMT models

In order to identify and characterize the robustness to phonetic writing of our phon2char model, which was the most performing system in Table 6.8, we evaluate our PMUMT corpus with it and we present these results in Table 6.10 for a single and unique UGC specificity, and in Table 6.9 for all types of UGC confounded only varying the total number of specificities occurrences (from

1 to 4+).
In the first place, in Table 6.10 we can notice that the corresponding MT system (char2char) becomes more robust to phonetic writing specificity, compared to the SB metrics in Chapter 5 Table 5.2, specifically, it increases in +13% (from 0.86 to 0.97), however, the difference in overall translation performance (lower for phon2char as discussed before), make the noisy BLEU score not comparable. Additionally, it is interesting to notice that phon2char is +8% and +4% more robust to contractions and inconsistent casing, respectively. These results

suggest that phonetization can help to recover from such UGC specificities by approaching the noise in UGC to the phoneticized form of canonical constructs.

On the other hand, phon2char showed a loss of -17% and -4% for tokenization and named entity, respectively with respect to char2char, which could be explained by losing some of the original word-level segmentation information through the phonetization process for the former and adding confusion to names, some of which are not readily french-phonetizable.

In addition, the specificities repetition and interjections, also suffer from a decreased robustness of roughly -7% comparing phon2char to char2char. This can be ex-plained due to the fact that phoneticizing such sequences will arguably always map to incorrect and rare phoneme sequences, since such specificities are not meant to be phoneticized.

Finally, regarding the impact of a given number of present UGC specificities, we can notice in Table 6.9 that phon2char have the same overall robustness profile of char2char when comparing to Table 6 0.83 (22.5) 0.81 (22.3) Figure 6.9: BLEU score ratio between pairs of normalized and noisy sentences containing N specificities for the phon2char system. BLEU scores on noisy sentences are shown in parenthesis.

Conclusions

In this chapter, we have tested the capacity of convolutional character-based NMT systems to translate noisy UGC content, and we have compared their performance to mainstream BPEbased models. We also developed a novel UGC evaluation framework to identify the advantages and caveats of our proposed models when translating UGC.

By using our proposed evaluation protocols, annotated data, and code, we were able to show that, contrary to what could be expected, this kind of system is very sensitive to UGC idiosyncrasies, and it is especially challenging to translate texts that are productive in terms of new forms, new structures, or new domains, although we did notice a generalization advantage over BPE subword units NMT for faulty or missing diacritization and letter change (to constitute a misspelling), inconsistent casing, and graphemic and punctuation repetition.

We have also investigated the effects and caveats of choosing different training vocabulary sizes in order to train character-based NMT systems to correctly treat charOOVs and rare characters during UGC evaluation, for which the results strongly point to the importance of carefully choosing the number of characters kept in the vocabulary set. Additionally, this aspect has a considerable effect on the length of the predictions and is also one factor that determines abnormal behavior. In this respect, our analysis concludes that reducing the training vocabulary and looking for its optimal size can help training more robust character-based NMT systems, which process charOOVs that can be present on OOD evaluation scenarios and real-world MT applications.

Finally, we combine our new character-based models with phonetization information of the source tokens to propose other methods to cope with UGC, which serves a twofold purpose: as a phoneme-based baseline that allowed us to quantify the performance of our phonemic MT systems in Chapter 5 and justify the extra computing overhead needed; and it also constituted the focus of our evaluation procedure to identify its robustness capabilities beyond phonetic writing.

Chapter 7

Variational inference methods for UGC

The previous chapters have highlighted the difficulties raised by the translation of UGCs and the need to consider a zero-shot scenario in which no parallel data is used during training. We have also described our first experiments to improve the robustness of translation systems to a well-identified UGC specificity. In this chapter, we describe a new contribution of our work aiming at using latent variables models to build better text representations, robust to the whole set of variations that can be found in UGC.

To address the problem raised by OOD texts, an increasing number of works explore the possibility to combine deep learning with latent variable (LV) models: the latter are indeed able to capture underlying structure information and to model unobserved phenomena. The combination of LV models with neural networks was shown to increase performance in several NLP tasks [START_REF] Kim | A tutorial on deep latent variable models of natural language[END_REF]. In this work, we focus on a specific LV model for MT, Variational NMT (VNMT), introduced by [START_REF] Zhang | Variational neural machine translation[END_REF]) which has been reported to have good performance and interesting adaptability properties [START_REF] Przystupa | Investigating the impact of normalizing flows on latent variable machine translation[END_REF][START_REF] Xiao | Wat zei je? detecting out-of-distribution translations with variational transformers[END_REF].

The goal of the work described in this chapter is twofold. First, we aim to evaluate the performance of VNMT when translating French social-media noisy UGC, a kind of OOD texts that has never been considered before in the VNMT literature. We hypothesize that, by leveraging on Variational NMT, latent models can build more robust representations able to represent OOD observations that are symptomatic of noisy UGC and automatically map them to in-distribution instances, which can be more easily translated.

Second, to account for the diversity of UGC phenomena, we introduce a new extension of VNMT that relies on Mixture Density Networks [START_REF] Bishop | Mixture density networks[END_REF] and Normalizing Flows (Rezende and Mohamed 2015). Intuitively, each mixture component extracts an independent latent space to represent the source sentence and can potentially learn different multi-modal regularization distributions, more robust to noise and capable of better translating UGC.

At the end, our contributions can be summarized as follows:

• we study the performance, in a zero-shot scenario, of VNMT models and evaluate their capacity to translate French UGC into English;

• we introduce a new model that uses Transformer as the backbone of a variational inference network to produce robust representation of noisy source sentences, and whose results outperform strong VNMT and non-latent baselines when translating UGC in a zero-shot scenario. Specifically, our model demonstrates a high robustness to noise while not impacting in-domain translation performance;

• by probing the learned latent representations, we show the importance of using several latent distributions to model UGC and the positive impact of the ability of VNMT models to discriminate between noisy and regular sentences while maintaining their representation closer in the embedding space;

• we report evidence supporting that our VNMT models act as regularizers to their backbone models, leading to more robust source embeddings that can be later transferred with a relatively high performance gain in our zero-shot UCG translation scenario.

Background

In this section, we review the methods featured in this chapter. We start by discussing variational approaches and their interest to robust NMT. We then study specific architectures used to improve performance and that we rely upon to build our proposed VNMT model.

Variational Neural Machine Translation

Variational Inference (VI) methods (Kingma and Ba 2015) are generative architectures capable, from a distributional perspective, of modeling the hidden (i.e. latent) relations that can be found in data. In a sequence-to-sequence MT tasks, where x and y are respectively the source and target sentences, VNMT assume that there exists a random variable, z (known as the latent state), modeling the implicit structure (i.e. relations) between the bilingual sentence pairs [START_REF] Zhang | Variational neural machine translation[END_REF]).

In the context of UGC translation, we believe that this latent variable can capture the variations between a source sentence and its canonical, normalized form, recovering its underlying meaning and ensuring that their representations are similar. We explore this hypothesis in Section 7.5 by assessing perturbations caused by UGC specificities.

To make computations tractable, in spite of the latent variable, VI combines a so-called variational posterior q (z|x, y) that is chosen to approximate the true posterior distribution, p(z|x), and a neural decoder generative distribution, p ✓ (y|x, z), in charge of generating the translation hypothesis conditioned on the latent variable. Once the family of densities q is chosen, the parameters of the two distributions are jointly estimated to model the output y by looking for the parameters (✓, ) that minimize the evidence lower bound objective function:

log p ✓ (y) E q (z|x,y) [log p ✓ (y|x, z)]
D KL [q (z|x, y)||p(z|x)]

(7.1)

Normalizing Flows

One of the major caveats of variational methods is that choosing the prior q(z) is a complicated process that requires some a priori knowledge of the task. In practice, a normal distribution with fixed parameters (generallyµ = 0.0 and = 1.0) is often chosen for computational reasons.

However, such an assumption can be restrictive when modeling more complex processes.

Regarding this issue, [START_REF] Jimenez | Variational inference with normalizing flows[END_REF] propose to enhance variational methods with Normalizing Flows [START_REF] Tabak | A family of nonparametric density estimation algorithms[END_REF][START_REF] Tabak | Density estimation by dual ascent of the loglikelihood[END_REF]. A normalizing flows is a series of simple bijective functions automatically chosen to extract a more suitable representation for the task at hand from a random variable.

In MT, normalizing flows were recently used to improve VNMT models: [START_REF] Setiawan | Variational neural machine translation with normalizing flows[END_REF] show that using them in an in-domain evaluation setting results in an increase of +1.3 BLEU points on the IWSLT'14 (De-En) and +0.2 BLEU points on the WMT'18 (En-De).

Mixture Density Networks

Mixture Density Networks (MDN) are another interesting generalization of variational encoding for modeling UGC. In MDN, the posterior distribution p(z|x) is no longer approximated by a single variational posterior q (z|x, y) but by a linear combination of variational posteriors qm (z|x, y):

q(z|x) = M X m=1 ↵ m (x, y) • qm (z|x, y) (7.2)
where ↵ m are known as the mixing coefficients.

Gumbel-Softmax sampling

Regarding the mixing coefficients computation, we also explore the use of a categorical probability distribution, for which probabilities are calculated by the network, such as in [START_REF] Ha | A neural representation of sketch drawings[END_REF]. Unlike theirs, our supervised end-to-end training requires backpropagating the error gradient through the variational network via reparameterized sampling (Kingma and Welling 2014) which poses optimization challenges because of the discrete random variables used as latent vector for categorical distributions. For this reason, we use the reparameterization of this distribution via the Gumbel-softmax sampling [START_REF] Jang | Categorical reparameterization with gumbel-softmax[END_REF][START_REF] Maddison | The concrete distribution: A continuous relaxation of discrete random variables[END_REF], such that, the arg max function is approximated by a softmax and generates the relaxed one-hot encoded samples corresponding to the mixing coefficients:

↵ m = exp(log(⇡ m ) + g m )/⌧ ) P M j=1 exp((log(⇡ j ) + g j )/⌧ ) (7.3)
where g m ...g M are i.i.d sampled from the Gumbel(0,1) distribution (Gumbel 1954;[START_REF] Maddison | The concrete distribution: A continuous relaxation of discrete random variables[END_REF]), ⇡ i is the probability associated to the m-th MDN's Gaussian components, jointly generated by neural networks along with the computations of the corresponding parameters (µ m , m ) for m...M ; and ⌧ is the temperature parameter, which controls variability of the sampling.

When ⌧ ! 0, the sampling exhibits a perfectly one-hot encoded output, whereas, conversely, when ⌧ ! inf, the distribution approaches a uniform one across all the MDN's components.

Extending variational methods for robust MT

The model we designed in this work adopts a variational encoder-decoder architecture inspired by SketchRNN (Section 7.1) that uses an MDN on the decoder's variational network to model multiple and independent continuous generative variational distributions. However, unlike

SketchRNN, we use a Transformer backbone for the encoder and the decoder and train our model in an end-to-end manner on canonical parallel corpora. In the following, we will first describe the general architecture of our model, denoted multi-VNMT, and then detail the encoder and decoder parameters. This combined representation is then fed to the decoder that has a similar architecture: it consists of a "standard" Transformer decoder and an MDN. The latter is sampled to obtain a prediction that will be combined with the Transformer output by (again) a gating mechanism.

General architecture

The model can be trained in an end-to-end fashion using the "reparameterization trick" (Kingma and Welling 2014). In order to ensure that the estimated standard deviations for the purpose [START_REF] Ha | A neural representation of sketch drawings[END_REF].

We also explored two methods to compute the mixing coefficient in the decoder's MDN:

either using softmax (non-latent approach), or resorting to relaxed categorical variational method that relies on a Gumbel-softmax sampling (Section 3.8).1

Encoder

According to our Transformer Base baseline architecture from Vaswani et al. (2017), the encoder is composed by a 6-layered Transformer Base encoder, which output is feed to a 128-dimensional variational network, that estimates the final latent hidden encoded vector.

In Figure 7.1b, we show the Transformer and variational encoding latent state (z) as being estimated (p(z|x)) approximating the posterior distribution parameters, learned using the reparameterization trick (Kingma and Welling 2014). On the other hand, the blue arrow in the figure ,   shows how we can introduce a source-side reconstruction loss in order to introduce mono-lingual training generative posterior (q(x|z)), such as seen applied to VNMT in Zhao et al. (2019), used as regularizer to enforce source information to be efficiently propagated and mitigate posterior collapse. In this work, we also study the impact of this source reconstruction as an accessory module for our proposed VNMT model.

In order to be comparable to the recently introduced NF-VNMT [START_REF] Setiawan | Variational neural machine translation with normalizing flows[END_REF] prove interesting for higher capacity. However, we decided to only address PF since they are the simplest solution with comparable performance improvement as other more complex flow models, according to results from [START_REF] Setiawan | Variational neural machine translation with normalizing flows[END_REF].

Similarly to NF-VNMT, we mix the last Transformer layer output to the latent vectors using a gating mechanism, and a feed-forward network in order to upscale the latent representation dimensionality and match the Transformer Base decoder number of dimensions (i.e, from 128 to 512); but unlike this model, we do so for both the encoder and decoder blocks' outputs since we introduce variational networks on both sides.

Decoder

The Transformer decoder's last layer output is passed as input to an 128-component MDN, with trainable parameters , encoding the mean and standard deviation of each one of these multivariate Gaussian kernels; and ⇡, which contains the probabilities of the categorical distribution that generates the mixing coefficient for each component, as seen in Section 7.1.

Concisely, we estimate a series of M posteriors parameterized by h ; ⇡i, i.e. q ;⇡ m (z dec |x, y 1:t 1 ), conditioned via the decoder's Transformer, on both the gated latent encoder's output and the last t 1 predicted tokens, y 1:t 1 . To compute the MDN's mixing coefficients, ↵ m (x, y 1:t 1 ), we explore either using a fully-connected layer with a softmax activation, or the relaxed categorical Gumbel distribution. Both networks computing qm and ↵ m are jointly trained in an end-to-end fashion, such that translation loss is minimal for representations sampled from the resulting mixture, obtained according to Equation 7.4. We train the MDN by variational inference using reparameterized sampling, similarly to our variational encoder network.

By using MDN, the posterior distribution of the current decoding step, p(z|x, y t ), is no longer approximated by a single variational distribution q (z|x, y 1:t 1 ) but by a linear combination of variational posteriors qm (z|x, y 1:t 1 ):

p(z|x, y t ) = M X m=1 ↵ m (x, y 1:t 1 ) • qm (z|x, y 1:t 1 ) (7.4)
where ↵ m are known as the mixing coefficients.

Regarding our choice of architecture, we also explored incorporating Normalizing Flows to the MDN's components variational inference, which resulted in poor performance and very large number of parameters. We, thus, apply these two methods independently, and we leave their simultaneous use within a single variational network for future works.

Jointly-learned MLM representations

An interesting property of our model is that it is inherently capable of being pre-trained (or jointly trained) as an auto-encoder considering only monolingual data in the source language. This pre-trained model can be later fine-tuned on other MT domains or bootstrapped to be used in other NLP tasks. Indeed, the reconstruction error can be minimized as training objective such that the network predicts one or several masked token in the source sentence.

In this work, we use the MASS approach [START_REF] Song | MASS: masked sequence to sequence pre-training for language generation[END_REF]) as masked language model (MLM) scheme, by introducing a supplementary source reconstruction loss while masking the source tokens to predict.

Using these tools, we explore a semi-supervised approach, as done in Zhao et al. (2019),

and performed experiments adding a source-side reconstruction loss term, both for MLM and reconstruction objectives:

L mono = E z⇠q (z|x) [log(p (x|z)] D KL (q (z|x)kp (z)) (7.5)
This objective is maximized by sampling the approximated posterior distribution (p ✓ (z|x)) by the means of variational inference, represented as the blue arrow in Figure 7.1b. Using the MASS MLM auxiliary loss and adding a source reconstruction term to it will, intuitively, guide the latent representation of the source sentence by modeling the inner relationships between its tokens.

In our experiments, we only use the source sentences contained by the training datasets in order to be able to unequivocally assess the advantages of this auxiliary task, ruling out the impact of supplementary monolingual data. However, using additional monolingual corpora is arguably the main interest of this approach. rate of 1.5 for the Noam schedule (Vaswani et al. 2017) (instead of 2.0 for the vanilla version)

and training for 400K training steps, as opposed to 200K used before. This was done to keep all the NMT systems comparable, and resulted in a small improvement of the Transformer Base, as we report in the following section.

MT scores

Test performances achieved by the NMT systems are reported in Table 7.1 using BLEU [START_REF] Papineni | Bleu: a method for automatic evaluation of machine translation[END_REF]) and chrF2 (Popovic 2017), both computed by SACREBLEU [START_REF] Post | A call for clarity in reporting BLEU scores[END_REF]) with the 'intl' tokenization, after detokenizing the systems outputs.

We computed the 95% statistical significance by using a 1,000-samples bootstrapping for both BLEU2 and chrF23 , as in [START_REF] Koehn | Statistical significance tests for machine translation evaluation[END_REF]. It should first be noted that the performances of the three systems we consider are identical when they are evaluated on in-domain data, whatever the evaluation measure considered (no statistically significant difference between the models). This observation highlights one of the strength of the proposed method: contrary to fine-tuning (arguably the most common method to adapt a system to a new domain) that often hurts performance on in-domain evaluation because of catastrophic forgetting [START_REF] Mccloskey | Catastrophic interference in connectionist networks: The sequential learning problem[END_REF], the improvement of the quality of UGCs by the proposed method is not at the expense of the quality of translation of canonical texts.

It also appears that, on out-of-domain text, multi-VNMT, the approach proposed in this work, outperforms the standard Transformer model as well as the state-of-the-art VNMT model, supporting our hypothesis that considering several variational inference components allows to better capture all the variations that can be found in UGC and will result in improved translation quality. Interestingly, our system also performs better than Transformer when evaluated on out-domain canonical data and not only on UGC data. It should be noted, however, that the gains of our model are consistent but small and statistically significant mainly when translation quality is assessed using chrF2.

Ablation study To better understand the impact of the different components of our model, we conduct an ablation study, whose results are reported in The † symbol denotes reported results. They have been computed using tokenized BLEU, as we reproduced the same pre-processing and vocabulary parameters.

Results using a standard experimental setup

In order to assess how multi-VNMT compares to NF-VNMT's results reported in Setiawan et al.

(2020), we have recreated4 their IWSLT'14 De-En configuration with 160K training sentence and 10K source-target joint BPE vocabulary, which in turn was adopted in [START_REF] Edunov | Understanding back-translation at scale[END_REF] and [START_REF] Wu | Pay less attention with lightweight and dynamic convolutions[END_REF].

This is an interesting alternative setup with a much smaller train set than the ones we considered in our experiments. This setup allows us to assess whether our findings on the two transformer-based VNMT architectures hold for a different language pair and a "low-resource conditions". In this regard, we report our results for multi-VNMT in this setting in Table 7.4, where it can be seen that our model has on-par (slightly better) results compared to the VNMT baseline.

Qualitative analysis

In Table 7.5, we show some examples from the PFSMB and MTNT test sets and their translation by different NMT systems. We notice a general trend of multi-VNMT (MTX in the table), outperforming the baselines and producing overall longer predictions when rare tokens or letter repetition are present in the source sentence. This is the case for Example ¿ in which some tokens have an inconsistent case ; Example ¡ contains repeated characters and words ;

Example ¬ has an out-of-vocabulary character "•", and Example √ contains a hashtag and several user mentions both identified by OOD characters ("#" and "@" respectively).

Robustness

To assess the robustness of our proposed model, we report in Figure 7.2 the noise impact ratio metric (as defined in Section 2.4.3) for each UGC specificity (Table 7.6) and with respect to the number of different UGC specificities in the input sentence (Table 7.2b). It appears that our model is, overall, more robust than all the models we have used so far, notably when

OOV and rare character are present in the source (namely, special char. and emojis.

On the other hand, some specificities seem to degrade performance for our architecture, notably wrong tense, agreement and repetition, that might be explained by the several semantically-equivalent sentence versions they represent, and that multi-VNMT has problems to disambiguate from. Overall, in Table 7.2b, it can be noticed that multi-VNMT is less sensitive to the number of noise occurrences, witnessing a less pronounced impact on performance as more specificities are present in the input. However, for a large quantity of specificities (4-7 in

Learning representations: where does the magic happen?

In this section, we analyze the neural representations, namely the variational encoder's latent space and source-side embeddings (the very same input layer in the Transformer backbone).

We intend to assess how our model behaves when translating UGC in comparison to both our latent and non-latent baselines. By doing so, we also aim to undercover if VNMT indeed learns more robust neural representations, and, confirm that variational approaches act as regularizers and are able to promote robust backbone's representations (embeddings).

Latent space analysis

To assess how VNMT builds more robust learning representations than our baselines, we report the cosine similarity distribution between the representations of the French noisy sentences and their normalized version, taking advantage of the PMUMT introduced in Chapter 6. To obtain these embeddings, we fed the 400 original noisy UGC sentences and their corresponding 400 fully normalized versions to our VNMT baseline, NF-VNMT, and to multi-VNMT.

To measure the perturbations that the model suffers when noise is present in the source, we measure the similarity between the latent representations built by our two VNMT models of the noisy sentences from PMUMT and the representations of their corresponding normalized version.

We observe that the average similarity between the representations of multi-VNMT is 0.36 compared to an average similarity of 0.26 for the representations of NF-VNMT, suggesting that the former provides more robust representations of UGC than the former. We also compare the distribution of cosine similarities of both VNMT systems in Figure 7.3, which confirms this trend by showing multi-VNMT's skewed to overall higher similarities between original and normalized sentence pairs. In Figure 7.4, we show the t-SNE (van der Maaten and Hinton 2008) visualization of both VNMT systems, displaying the latent encoding of noisy and normalized PMUMT sentences.

Latent encoding and performance

In order to visualize if the latent space is linked to MT performance, we report, in Figure 7.5 the noisy sentences and we represent their character-level edit distance metric comparing predictions to the ground truth using 4 quantiles, with cumulative probability distribution partitions of 25%, 50% and 75%, respectively. On one hand, we first considered the word-level edit distance, which gave us equal sets of bin delimiters for the quantiles [0.55, 0.64, 0.87] for our model and the VNMT baseline. On the other hand, characterlevel edit distance resulted in multi-VNMT having a better performance distribution (smaller edit distance), with [0.297, 0.433, 0.563] quantiles ' delimiters, compared to [0.304, 0.448, 0.583] of the baseline, which also hints that word-level BLEU might not uncover the real translation performance gap in our benchmarking. The bins for both plots were chosen using partitions ' delimiters [0.30, 0.44, 0.57]. This was done to compare both latent spaces with the same similarity values' bins, however, multi-VNMT has

More robust embeddings for UGC

In this section, we study the source embeddings of our models to assess whether VNMT promotes learning more robust embeddings that could prove valuable for larger-scale transfer learning models. We compare noisy and normalized versions of the FR PMUMT source side to assess whether they have a closer representation. having, at the same time, higher cosine similarity. This seems to indicate that VNMT captures some noise structure information, as a separation between noisy and normalized text becomes more evident [START_REF] Carbonnelle | Intraclass clustering: an implicit learning ability that regularizes dnns[END_REF], as opposed to our results on the latent space learning representations, discussed previously.

Noisy vs canonical data

Recovering from UGC specificities Likewise, we now report the cosine similarity between sentences with only an unique type of UGC specificity, following the isolation and study of individual and controlled UGC phenomena discussed in Chapter 6. discussion corpora. It is very interesting to notice that, although the in-domain performances for these 3 systems are very similar (between 31.4 and 31.5 BLEU in Table 7.1), the performance gap of blind UGC test sets is considerably larger, roughly +1.7 BLEU. We have also added the best transfer-learning model from the previous section (Transformer Base +FT emb.) and showed an increase of performance when translating MTNT and 4Square blind test sets by +0.3 and +0.1 BLEU, respectively. However, contrary to the VNMT models, we noticed that +FT emb.

Bootstrapping VNMT embeddings

was outperformed by the baseline, suffering a performance detriment of 0.3 BLEU.

How do MDN's components react to UGC?

We now proceed to analyze and visualize how the MDN mixture coefficients react when translating our different test sets. In order to do so, in Figure 7.8 we report results for the canonical test sets, the normalized PMUMT corpus, and its noisy original UGC version. Each bar of the Wind Rose diagram represents one of the 128 independent trained distributions' mixture weights, which have been normalized and scaled across the four graphics, and where the 7th MDN component seems to be consistently the one that drives most of the decoding for the presented experiments. Furthermore, we can notice that most mixing coefficients are, for the most part, have around 50% probability of contributing to the final inference mixture, despite not enforcing this behavior with any specific method (e.g. dropout). On the other hand, the visualization suggests that both yellow (50-60%) and blue components (30-40% of activation) are variable across test sets, being very similar between PMUMT Norm and OpenSubTest, which could indicate that the mixture components are learning to encode different types of texts, potentially working as an implicit topic modeling module. Regarding the visualization when translating PMUMT Noisy, the main MDN component identified above, seems less important even when compared to the out-of-domain newstest'14 set, which suggests that the MDN uses more

Conclusion and perspectives

Our results prove that we can achieve consistent performance improvement by using extended VNMT architectures that employ MDN, for which, N independent multi-way variational posterior approximations are produced and mixed as in Gaussian Mixture Model. Our approach adds roughly 3.5M parameters (+5%) and, systematically, outperforms previous Transformer-based VNMT using normalizing flows on MT, witnessing the largest performance gap for our very noisy PFSMB corpus.

In addition, by exploring the learning representations trained by our VNMT model, and through conducting transfer learning experiments with such, we investigate the robustness brought to UGC, and show that VNMT enforces such property to the backbone model, bringing a promising avenue for more robust pretrained neural learning representations. However, an open question arising from this work, it is currently unclear if the performance gain we observed is due to a better generalisation to distributional shift or if it corresponds to a better adaptation to noise in the input. Future works will be devoted to this question, which can be abstracted away to study whether UGC idiosyncrasies are a form of noise, some parts being learnable, or are rather points to a new domain. We report interesting but mitigated results when using an accessory source reconstruction loss to improve robustness, which we plan to study in the future using other sorts of monolingual data and training protocols, such as denoising autoencoders and monolingual source-side UGC corpora.

Using pronunciation to correct from noise As a first attempt to recover from UGC specificities, in Chapter 5 we proposed a normalization pipeline to correct phonetic writing (discussed in Section 5.1) by finding potential normalized candidates using pronunciation similarities. A ranking of such candidates by a language model (LM) provides the most likely corrected sentence. As discussed, our results showed interesting normalization cases and slightly improved results over UGC translation, without requiring any supplementary data. However, they showed limited impact on blind UGC test sets and suffered from artificial noise by confusing potentially normalized tokens. Regarding this last aspect, the identical French pronunciation of certain plural forms was shown to introduce erroneous correction of tokens that otherwise should not have been changed. Relative to the reported caveats of our methods, the study of an end-to-end trained NMT phonetic embeddings, providing both original and pronunciation information to output the best translation.

Understanding UGC specificities and fine granularity impact on MT In order to account for a wider range of UGC phenomena, we explored the use of character-based representations to assess their robustness to UGC in Chapter 6. We have highlighted the importance of the vocabulary size for char-based systems, a parameter whose importance had never been discussed until now.

These experiments also motivated the creation of a novel UGC evaluation framework with a fine-grained typology and the possibility to isolate different specificities of UGC, a necessary step to understand the mechanism at play while translating noisy UGC. This framework aims to evaluate how different models behave under different kinds of noise. Our results showed that character-level NMT is less sensitive to misspellings (in the form of letter addition or deletion, incorrect diacritics, and incorrect verb tense), as well as incorrect tokenization and inconsistent casing. However, character-based systems were generally outperformed by subword segmentation, which performed better in our experimental setup.

Improving robustness to UGC via latent variable methods

In the last part of this dissertation (Chapter 7), we explored Variational Neural Machine Translation (VNMT) and presented a novel architecture using Mixture Density Networks to perform multimodal Variational Inference.

We introduce mixture models to perform VI in a multimodal information flow, which showed improved robustness capabilities when translating UGC. Through probing and visualization methods, we investigated how latent codes can be useful to recover from noise by enforcing robust learning representations, while mitigating overfitting and thus improving out-of-domain translation performance. We conducted several experiments to evaluate how the representations of the VNMT models are less sensitive to UGC-related perturbations compared to a non-latent baseline. Our proposed approach showed improved robustness capabilities to UGC, without hurting translation quality on canonical (in-domain and out-domain) test sets.

Six years after the ParSiTi project proposal: progress on UGC translation In recent years, much of the attention in NLP has been focused on Large Language Models (LLM), which have been shown to be able to capture high-order information structures and exhibit interesting properties in zero-shot scenarios [START_REF] Kojima | Large language models are zero-shot reasoners[END_REF]. Surprisingly, these capabilities arise from a relatively simple language modeling (LM) task, i.e. masked LM, which allows the exploitation of massive amounts of text in an unsupervised manner. This property, combined with the possibility of instant access to huge and extremely diverse corpora thanks to the Internet, has led to the development of text-based systems with impressive generative power and flexibility [START_REF] Guardian | Facebook translates 'good morning' into 'attack them', leading to arrest[END_REF]. This approach makes it possible to exploit UGC data without the need of producing reference translations, thus enhancing NMT with an apparent advantage when translating UGC.

To illustrate the case, in Figure 8.1, we revisit the original UGC sample studied in Chapter 1 translated by popular translation engines, some using LLM.

In Figure 8.1 we can observe that UGC translation is still much of a work in progress to this day: in most cases, the French UGC-specific tokens (e.g. "tabuz", correctly spelled "t'abuses") are left unchanged in their English translation, apparently due to the MT systems treating them as Named Entities.

It can, however, be noticed that GPT-based [START_REF] Tom | Language models are few-shot learners[END_REF]) systems succeed in translating two noisy UGC tokens: BingGPT © correctly translates "Ki ca ?" into "Who is it ?" and "lavé rien fé" into "did nothing", correcting French phonetic writing on both; ChatGPT © is able of partially translating "lemecousonchien?" into "The guy with the dog?", recovering from the missing word segmentation, even if it mistranslates "ou" into "with" rather than "or ". six years after the ParSiTi project started. This observation highlights the inadequacies of utilizing large amounts of training data to "understand" UGC, a never productive domain, and, consequently, justify the zero-shot setting we have adopted in this work.

were proposed as a promising avenue for UGC translation, which, compared to other domain adaptation methods such as domain specialization (i.e., fine-tuning), is data-agnostic and maintains optimal performance for in-domain test sets.

Titre:

  Traduction Automatique de Contenus Générés par l'Utilisateur : une Évaluation des Systèmes de Traduction Neuronaux dans des Conditions Zero-shot Mots clés: traduction automatique, contenu généré par l'utilisateur, langue non canonique, traitement automatique de la langue.

2. 1

 1 The encoder-decoder architecture.. . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Bidirectional RNN Encoder-Decoder architecture. Reproduced from Chowdhury and Vig (2018). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Diagram of an LSTM, the RNN architecture used in this work. Reproduced from (Olah 2015). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 The Transformer Encoder-Decoder architecture. The encoder on the left side of the diagram and the decoder on the right one. Reproduced from Vaswani et al. (2017). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Attention matrix example showing the alignments between the French sentence predicted by an NMT system (y-axis) given the English sentence (x-axis). Reproduced from Bahdanau et al. (2015). . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Self-attention matrix example as produced by the Transformer model. It shows the dependencies between tokens of the same sentence that are useful to produce the current output token. The attention block colors stand for each one of the 8 multi-head attention mechanisms (here aligned horizontally). Reproduced from Vaswani et al. (2017). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Distribution of PFSMB translations length ratio with respect to ground truth translations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Attention matrix for the source sentence "Bon je veux regardé teen wolf moi mais ce soir nsm" (Ok, I do want to watch Teen Wolf tonight motherf..r) predicted by a seq2seq model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Attention matrix of a seq2seq model that exhibits the excessive token repetition problem. The sharp symbol (#) indicates spaces between words before the BPE tokenization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Example of lattice for a segment of a PFSMB UGC sample. . . . . . . . . . . . . . 5.2 Number of replacement operations of our normalizer over the PFSMB test set. The quantity of non-homophones normalizations are displayed as point labels. . 5.3 Bar plot of the BLEU score for the PFSMB test set. The translation hypotheses are divided into sentences' length groups. . . . . . . . . . . . . . . . . . . . . . . . . 6.1 BLEU score in function of the number of charOOVs in the input sentence. . . . . 6.2 Comparison of the number of UGC specificities in the best and worst translation hypotheses of char2char and Transformer. Noise categories are defined in Table 6.2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Results of the copy task, evaluated by the BLEU score before and after <UNK> replacement (+<UNK> rep.) and percentage of <UNK> characters in the prediction (%<UNK> pred.). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Distribution of UGC specificites in PMUMT. . . . . . . . . . . . . . . . . . . . . . . 6.5 BLEU scores on the original and normalized source sentences of the PMUMT corpus. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6 (a) Noisy/Clean BLEU scores' ratios for an accumulated number of UGC specificities present per sentence for each model, corresponding to the results in Table 6.6b. The 4to7 bin groups more than 4 types to provide a larger subcorpus, which weighted average is 4.34 UGC specificities per sentence. (b) BLEU score ratio between pairs of normalized and noisy sentences containing N specificities. BLEU scores on noisy sentences are shown in parenthesis. . . . . . . . . . . . . 6.7 Reference/hypothesis length ratios for different vocabulary sizes. . . . . . . . . . 101 6.8 BLEU score results for our three phoneme-to-text models on clean and noisy test sets. The best result for each test set is marked in bold, in-domain scores with a dag. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.9 BLEU score ratio between pairs of normalized and noisy sentences containing N specificities for the phon2char system. BLEU scores on noisy sentences are shown in parenthesis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1 (a) multi-VNMT architecture overview. (b) Directed graph of our encoderdecoder model variational inference. Dashed lines represent the variational approximation for the posterior distribution, and solid lines stand for the generative models. The blue arrow depicts the generative networks for source-side monolingual reconstruction distribution p(x|z). . . . . . . . . . . . . . . . . . . . 7.2 Comparison of multi-VNMT's robustness to different number of present UGC specifies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Histogram of cosine similarity for corresponding noisy and normalized PMUMT samples in the encoder's latent space of NF-VNMT and multi-VNMT. . . . . . . 7.4 T-SNE representation of the latent space for noisy and normalized versions of PMUMT sentences during evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . 7.5 T-SNE representation of the latent space for noisy PMUMT sentences during evaluation. Color portrays the quantile of character edit distance between prediction and reference. Q1 contains the best translations (lowest edit metric). . . . . . . 7.6 T-SNE representation of the latent space for noisy PMUMT sentences during evaluation. Color portrays the quantile of cosine similarity between noisy and normalized versions. Q1 contains the samples with the least distance (worse performance). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.7 T-SNE representation of the encoder embeddings for noisy and corresponding normalized PMUMT sentences during evaluation. Average cosine similarity between corresponding noisy and normalized version of the PMUMT evaluation framework are reported between parentheses for each NMT system. . . . . . . . . . . . . . 7.8 Average MDN mixture weights for test sets of different natures. . . . . . . . . . . 8.1 Typical social media thread initiated by a seed photo and its automatic translation. Inspired from a real conversation about a series of demonstrations that took place in Greece. Bing was used (in April 2016) as it was then the official MT engine for Twitter and Facebook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

3. 2

 2 scheme. Extended from Sanguinetti et al. (2020). . . . . . . . . . . . . . . . . . 3.4 Examples of translation impact due to Encoding Simplification phenomena. The references (REF) are translations by the same MT engine of the correct form, accordingly. Translations were produced on the 07/03/2022 Normalized version of the tokens is displayed in blue brackets. . . . . . . . . . . . . . . . . . . . . . 3.5 Examples of translation impact due to Marks of Expressiveness phenomena. The references (REF) are manual edits of Google's translations of the correct form to include the missing characters. Normalized version of the tokens is displayed in blue brackets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Examples of translation impact due to Boundary Shifting phenomena. The references (REF) are translations by the same MT engine of the correct form, accordingly. Normalized version of the tokens is displayed in blue brackets. . . . 3.7 Examples of translation impact due to Marks of Expressiveness phenomena. The references (REF) are manual edits of Google's translations of the correct form to include the missing tokens. Normalized version of the tokens is displayed in blue brackets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8 Examples for our considered UGC corpora. Noisy tokens and their corresponding translations are shown in bold font. . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 2 . 2 :

 22 Figure 2.2: Bidirectional RNN Encoder-Decoder architecture. Reproduced from Chowdhury and Vig (2018).

Figure 2 . 3 :

 23 Figure 2.3: Diagram of an LSTM, the RNN architecture used in this work. Reproduced from (Olah 2015).

Figure 2 . 4 :

 24 Figure 2.4: The Transformer Encoder-Decoder architecture. The encoder on the left side of the diagram and the decoder on the right one. Reproduced from Vaswani et al. (2017).

  Another type of attention mechanism, self-attention (used by the Transformer architecture), contrary to sequential source-target attention, allows processing all the (either source or target) sentence's tokens at once and model contextualized relations between each sentence's words, as seen in Figure2.6. Another particularity of the self-attention approach proposed inVaswani et al. (2017), is the multi-head mechanism, in which several self-attention matrices are computed independently, which are marked in Figure2.6 as the 8 different colors, which stand for the 8 attention heads featured in the vanilla Transformer Base architecture.

Figure 2 . 6 :

 26 Figure 2.6: Self-attention matrix example as produced by the Transformer model. It shows the dependencies between tokens of the same sentence that are useful to produce the current output token. The attention block colors stand for each one of the 8 multi-head attention mechanisms (here aligned horizontally). Reproduced from Vaswani et al. (2017).

1

  https://github.com/moses-smt/mosesdecoder

  of expressiveness: denote emotion and expressiveness of the author Emojis

  UGC translation, which constitute our first MT baselines in Chapter 4. These are later used to identify and report caveats of the main popular MT paradigms when processing UGC.[START_REF] Bentivogli | Neural versus phrasebased machine translation quality: a case study[END_REF] showed that NMT predictions need less post-edition processing and present lower error rates than PB-SMT in English ! German translation of TED talks. On the same train of thought,[START_REF] Bojar | Findings of the 2016 conference on machine translation[END_REF]'s results also indicate that NMT is significantly better than PB-SMT, since less post-editing is needed to output a correct translation. On the other hand, Castilho et al.(2017) reports were somewhat ambiguous, showing that NMT has an edge over PB-SMT in terms of fluency and translation errors, while some results reflect an improvement over PB-SMT in terms of post-editing needs. However, none of the aforementioned works were conducted using experimental setups with a purposely high mismatch between training and test corpora. In this line of research,[START_REF] Haddow | Analysing the effect of out-of-domain data on SMT systems[END_REF];[START_REF] Koehn | Six challenges for neural machine translation[END_REF] found that NMT performs relatively poorly when translating OOD texts, as well as under low-resource conditions.[START_REF] Dowling | SMT versus NMT: preliminary comparisons for irish[END_REF] also showed a significant improvement of PB-SMT's BLEU score over NMT on -typically-low-resource English ! Irish translation. More recently,[START_REF] Anastasopoulos | An analysis of source-side grammatical errors in NMT[END_REF]

Further

  during this dissertation, in Chapter 5, we proposed an automatic normalization pipeline to approach noisy versions of text sequences to canonical constructs, which are ultimately expected to perform better, since the inherent OOD-drift nature of UGC gets minimized. In this order of ideas, several works have focused on using lattices to model uncertain inputs or potential processing errors that occur in the early stage of the translation pipeline. For instance,[START_REF] Su | Lattice-based recurrent neural network encoders for neural machine translation[END_REF] proposed lat2seq, an extension of sequence-to-sequence models[START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF], which are capable of encoding several possible input possibilities by conditioning the RNN output to multiple predecessors' paths. More recently, Sperber et al. (2017) introduced a model based on Tree-LSTMs (Tai et al. 2015), to correct the output of an Automatic Speech Recognition (ASR) system. On the other hand, Le et al. (2008) use lattices composed of written subword units to improve the recognition rate in ASR.

  combined several ASR systems to improve the detection of OOVs. More recently, van der Goot and van Noord (2018) achieved state-of-the-art performance on dependency parsing of UGC using lattices.Closely related to this work,[START_REF] Baranes | Spelling Normalisation of Noisy Text[END_REF] explored several normalization techniques on French UGC. In particular, to recover from typographical errors, they considered a rule-based system, SxPipe[START_REF] Sagot | SxPipe 2: architecture pour le traitement pré-syntaxique de corpus bruts[END_REF], which produced lattices encoding the alternative spelling of OOVs and used a language model to select the best correction.Several works have explored different approaches to normalize noisy UGC in various languages. For instance,[START_REF] Stymne | Spell checking techniques for replacement of unknown words and data cleaning for haitian creole SMS translation[END_REF] use Approximate String Matching, an algorithm based on a weighted Levenshtein edit distance to generate lattices containing alternative spelling of OOVs.[START_REF] Wang | A beam-search decoder for normalization of social media text with application to machine translation[END_REF] employ a Conditional Random Field and a beam-search decoding approach to address missing punctuation and words in Chinese and English social media text.More recently,[START_REF] Watson | Utilizing character and word embeddings for text normalization with sequence-to-sequence models[END_REF] proposed a neural sequence-to-sequence embedding that improves FastText[START_REF] Bojanowski | Enriching word vectors with subword information[END_REF] representations with word-level information, which achieved state-of-the-art on the QALB Arabic normalization task[START_REF] Mohit | The first QALB shared task on automatic text correction for arabic[END_REF]).

  to explain translation errors in a black-box query-based fashion to ultimately characterize the difficulties involved in this task. Additionally, examples of translations for each UGC phenomenon we consider are displayed and discussed to illustrate how they impact MT quality, notably when using very popular and public translation engines.Finally, after stating the motivation and translation problems at hand, we reviewed the literature on UGC translation related to each of the approaches studied in this work.After reviewing the main methods and techniques for MT and discussing the challenges that UGC poses for automatic translation; in the next chapter, we present our first out-of-the-box MT baselines. This set of experiences has a twofold purpose: it helped us identify and characterize the main translation quality problems caused by UGC on the most used MT approaches; and these baseline results are useful to keep track of the impact of possible methods to cope with UGC investigated throughout this dissertation.In this chapter, we describe the main experimental protocols, data and MT systems that we use as baselines and architecture backbone for our methods aiming to enhance the quality of MT systems on UGC. In order to keep consistency and ensure a fair comparison between all the methods presented in this work, we have systematically considered translating French UGC to English and identical training and test corpora. All the datasets and experimental protocols correspond to those of our first publication(Rosales Núñez et al. 2019). 1

  used encoder-decoder sequence-to-sequence techniques, i.e. a Bidirectional Long-Short Term Memory Recursive Neural Networks (Bi-LSTM) encoder with attention-based decoding[START_REF] Luong | Effective approaches to attention-based neural machine translation[END_REF][START_REF] Michel | MTNT: A testbed for machine translation of noisy text[END_REF], and a Transformer architecture that relies on a multi-head attention mechanism(Vaswani et al. 2017). The training, development, and test corpora are those described in Section 3.5.

seq2seq(

  [START_REF] Koehn | Six challenges for neural machine translation[END_REF] and Transformer[START_REF] Passban | Revisiting robust neural machine translation: A transformer case study[END_REF]) NMT architectures, by studying cases in which, as opposed to the PB-SMT system, the attention mechanism fails to provide a correct translation.The results presented in this chapter have been published in(Rosales Núñez et al. 2019).

Figure 4 . 1 :

 41 Figure 4.1: Distribution of PFSMB translations length ratio with respect to ground truth translations.

Figure 4 . 2 :

 42 Figure 4.2: Attention matrix for the source sentence "Bon je veux regardé teen wolf moi mais ce soir nsm" (Ok, I do want to watch Teen Wolf tonight motherf..r) predicted by a seq2seq model.

Figure 4 . 3 :

 43 Figure 4.3: Attention matrix of a seq2seq model that exhibits the excessive token repetition problem. The sharp symbol (#) indicates spaces between words before the BPE tokenization.

Figure 5 . 1 :

 51 Figure 5.1: Example of lattice for a segment of a PFSMB UGC sample.

  Figure 5.1 displays an example of such a lattice, showing the normalization candidates. In these lattices, a path between the initial and final nodes represent a (possible) normalization of the input sentence.

  Figure 5.2: Number of replacement operations of our normalizer over the PFSMB test set. The quantity of nonhomophones normalizations are displayed as point labels.

  Figure 5.3: Bar plot of the BLEU score for the PFSMB test set. The translation hypotheses are divided into sentences' length groups.

  , des que j'ouvre le magasin je commence a avoir le vertige mon coeur ce met a battre a 200 et je sens que je vais faire un malaise, norm src arrivé au taff, dès que j'ouvre le magasin je commence à avoir le vertige mon coeur se met à battre à 200 et je sens que je vais faire un malaise, ref once at work, as soon as I open the store I'm starting to feel dizzy my heart starts racing at 200 and I feel I'm gonna faint, raw MT I start to get dizzy. My heart starts to beat at 200 and I feel like I'm going to faint. norm MT When I get to the taff, as soon as I open the store, I start to get dizzy. My heart starts pounding at 200 and I feel like I'm gonna get dizzy.¡ src c un peu plus que mon ami qui faite son annif, norm src c'

Figure 6 . 1 :

 61 Figure 6.1: BLEU score in function of the number of charOOVs in the input sentence.

  Figure 6.4: Distribution of UGC specificites in PMUMT.

Figure 6

 6 Figure 6.5: BLEU scores on the original and normalized source sentences of the PMUMT corpus.

Figure 7 .

 7 Figure 7.1a shows the architecture of our model. The source sentence is processed by a standard Transformer encoder, whose output is passed as input to a VI network enhanced with NF to predict a latent representation of the input sentence. This vector and the output of the Transformer encoder's last layer are combined using the gating mechanism of Setiawan et al. (2020).

  Figure 7.1: (a) multi-VNMT architecture overview. (b) Directed graph of our encoder-decoder model variational inference. Dashed lines represent the variational approximation for the posterior distribution, and solid lines stand for the generative models. The blue arrow depicts the generative networks for source-side monolingual reconstruction distribution p(x|z).

Figure 7 . 2 :

 72 Figure 7.2: Comparison of multi-VNMT's robustness to different number of present UGC specifies.

Figure 7 . 3 :

 73 Figure 7.3: Histogram of cosine similarity for corresponding noisy and normalized PMUMT samples in the encoder's latent space of NF-VNMT and multi-VNMT.

First

  , we can notice that the NF-VNMT latent representations present a series of outliers when noisy sentences abound, contrary to the multi-VNMT representations, which also have a more compact support. In this set of 43 outlier observations (roughly 5% of the 800 plotted sentences' representation), 88% (37) are the original -noisy -UGC samples of PMUMT. Secondly, multi-VNMT embeddings have higher sparsity (0.190 compared to 0.185 of NF-VNMT), which suggests that our model has more structured representations[START_REF] Francis | Structured sparsity through convex optimization[END_REF]), which we link to MDN multiple components in Section 7.7, seemingly contributing to enforce sparsity.

Figure 7

 7 Figure 7.4: T-SNE representation of the latent space for noisy and normalized versions of PMUMT sentences during evaluation.

  Figure 7.5 does not show clear clusters of performance, however, multi-VNMT seems to perform worse for vectors with larger norm, whereas performances appear to be more uniformly-distributed for the baseline. Latent space recovering from noise To measure the perturbations that the model suffers when source-side noise is present, we compare how the latent representations built by our two VNMT models of the noisy PMUMT match those of their corresponding normalized version. In Figure 7.6, we plot the same dimensional reduced latent space and we encode color for their bins of cosine similarity of the hereby shown noisy sentences to their corresponding normalized version.

Figure 7

 7 Figure 7.6: T-SNE representation of the latent space for noisy PMUMT sentences during evaluation. Color portrays the quantile of cosine similarity between noisy and normalized versions. Q1 contains the samples with the least distance (worse performance).

  Figure 7.7: T-SNE representation of the encoder embeddings for noisy and corresponding normalized PMUMT sentences during evaluation. Average cosine similarity between corresponding noisy and normalized version of the PMUMT evaluation framework are reported between parentheses for each NMT system.

  As discussed above, in Figure7.7 we noticed that VNMT seems to enforce noisy morphology modeling to the Transformer's embeddings in an implicit fashion. This motivated us to study whether the information in such learning representations can be used by the Transformer Base backbone model and benefit from improved robustness without the direct latent space contribution, i.e. removing all the VI blocks, thus using only the backbone model. This is also computationally favorable, since the number of parameters of the final model matches the backbone architecture, avoiding all the complexity overhead introduced by the VI modules. Thus, we report BLEU scores for the Transformer Base model trained on OpenSubtitles, by either initializing the VNMT-pretrained embeddings or fine-tuning (FT) the system. We have performed FT using the same data configuration as in OpenSubtitles 5 and continued training for 3 epochs from the Transformer Base model in Table7.1 while replacing the embeddings by their VNMT-learned version's weights. PFSMB (Blind)MTNT (Blind)

  (a) PMUMT norm (b) PMUMT noisy (c) OpenSubTest (d) newstest'14

Figure 7

 7 Figure 7.8: Average MDN mixture weights for test sets of different natures.

Figure 8 . 1 :

 81 Figure8.1: Typical social media thread initiated by a seed photo and its automatic translation. Inspired from a real conversation about a series of demonstrations that took place in Greece. Bing was used (in April 2016) as it was then the official MT engine for Twitter and Facebook.

(Chapter 4).

	the Impact of UGC Specificities on Translation Quality. In Proceedings of the Seventh
	Workshop on Noisy User-generated Text (W-NUT 2021), November 11th, Punta Cana,
	Dominican Republic. (Rosales Núñez et al. 2021a) (Chapter 6).
	• José Rosales Núñez, Djamé Seddah, Guillaume Wisniewski (2023). Multi-way Variational
	NMT for UGC: Improving Robustness in Zero-shot Scenarios via Mixture Density Networks.
	To appear in Proceedings of the 22nd Nordic Conference on Computational Linguistics
	(NoDaLiDa 2023), May 22nd-24th, Tórshavn, Faroe Islands. (Chapter 7).
	• José Rosales Núñez, Djamé Seddah, Guillaume Wisniewski (2019). Comparison between
	NMT and PBSMT Performance for Translating Noisy User-Generated Content. In Pro-
	ceedings of the 22nd Nordic Conference on Computational Linguistics (NoDaLiDa 2019),
	October 1-2, Turku, Finland. (Rosales Núñez et al. 2019) • José Rosales Núñez, Djamé Seddah, Guillaume Wisniewski (2019). phonetic Normal-
	ization for Machine Translation of User Generated Content. In Proceedings of the 5th
	Workshop on Noisy User-generated Text (W-NUT 2019), November 4th, Hong Kong.
	(Núñez et al. 2019) (Chapter 5).
	• José Rosales Núñez, Guillaume Wisniewski, Djamé Seddah (2021). Noisy UGC Transla-
	tion at the Character Level: Revisiting Open-Vocabulary Capabilities and Robustness of
	Char-Based Models. In Proceedings of the Seventh Workshop on Noisy User-generated
	Text (W-NUT 2021), November 11th, Punta Cana, Dominican Republic. (Rosales Núñez
	et al. 2021b) (Chapter 6).
	• José Rosales Núñez, Djamé Seddah, Guillaume Wisniewski (2021). Understanding

.1 The encoder-decoder architecture for MT

  , called the decoder, that is in charge of generating the T d -length sequence of target tokens y = (y 1 , y 2 , ..., y n ). An overview of this type of architectures is shown in Figure2.1.

	Input	Encoder	State (h)	Decoder	Output
		Figure 2.1: The encoder-decoder architecture..
	(https://d2l.ai/chapter_recurrent-modern/encoder-decoder.html)
	(Way 2019) for target sentences with an arbitrary length (Jurafsky and Martin 2009); second, it
	allows designing and train a single, end-to-end model directly from a parallel corpus.
	The underlying idea of this paradigm is to have a first -possibly stack of -neural networks,
	known as the encoder, which outputs, for each token of the T S -length source sentence x = ⇣ ⌘
	(x 1 , x 2 , ..., x T S ), a contextualized representation h =	h D 1 , h D 2 , ..., h D t , ..., h D T S	each of these
	representations being a vector of D dimensions, where D is generally called the "hidden size".
	These vectors (or only some of them) are then 'fed' to another neural network (i.e. considered
	as its input)
	attention-based models (i.e. Transformers), which are the two most popular NMT architectures
	nowadays. Both architectures generate a hidden representation, either sequentially in recurrent
	networks or in parallel in Transformers. This representation is used to encode the meaning of the
	source sentence, and passed in cascade through the network to generate the target sequence
	of target language tokens.
	2.2Most of the modern NMT models have adopted the encoder-decoder paradigm, introduced
	by Sutskever et al. (2014) and Cho et al. (2014). Using this paradigm in MT has two main
	advantages over previous approaches: first, it generates contextually appropriate translations

8 ans après, je viens de percuter.... :o 'tai je

Cr#pbank src ma TL se vide après j'me fais chier puis jme sens seule mais c surtout pdt les vacs mais c pas le cas dc ça compte pas trop ref my TL is emptying then I get bored then I feel alone but mostly during holidays but it's not the case so it's not so important google my TL is empty after I'm pissing then I feel alone but c especially pdt vacs but it does not matter that it does not count too much src Dans Flappy Bird quand mon Bird il fait 10 jsuis trop contente #TeamNul Mdddr ref At Flappy Bird when my Bird has 10 Im so happy #TeamFail Loool google In Flappy Bird when my Bird is 10, I'm too happy #TeamNul Mdddr src Boooooooooooooooooooooonnne Annniversaure Ma viiiiiiiiiiiiiie jtm plus que tout profite bien de tes 22ans moaaaaaaaaa ref Haaaaaaaaaaaaaaaaaaaaaapppy Biiirthday My liiiiiiiiiiiiiife luv you more than anything enjoy your 22years mwaaaaaaaaah google Boooooooooooooooooooooonnne Annniversaure My viiiiiiiiiiiiiie jtm more than anything enjoy your 22 years moaaaaaaaaaa MTNT src AJA que le XV de France féminin est un train de faire un grand chelem, OKLM ref TIL that the XV of Feminine France is doing a grand chelem, FRESH google AJA that the XV of France feminine is a train of a grand slam, OKLM src Je pensais mais c'est le même ident et mdp que mon compte "normal", et il détecte même la profile pic et le nom ref I thought so but it's the same username and password as my 'normal' account, and it detects even the profile pic and the name google I thought but it's the same ident and mdp as my "normal" account, and it even detects the pic profile and the name src Aaaaaaaah.... me disais bien que je passais à côté d'un truc vu les upvotes. ref Aaaaaaaah.... 8 years later, I've just realized.... :o damn I had the feeling that I was missing something considering the upvotes. google Aaaaaaaah .... 8 years later, I just collided ....: oh well I was telling myself that I was missing something seen the upvotes.

Table 3

 3

.2: Examples from both UGC showing the source phrase, reference translation, and Google Translate output. UGC specificities are highlighted using bold characters. Translation site accessed on 28-01-2021.

Table 3

 3

	UGC's specificities
	Encoding simplification
	• Letter change/deletion/addition
	• Diacritic omission/error
	• Phonetization
	• Contraction over-splitting
	• Wrong verb tense
	• Wrong gender/grammatical number
	• Abbreviation
	Marks of expressiveness
	• Inconsistent casing
	• Emoticons/smileys
	•	Punctuation reduplication and Graphemic stretching
	• Interjections
	Boundary shifting
	• Tokenization errors
	UGC Context-dependent
	• UGC-specific characters
	• Named Entity

.3, which is based on the typology of

[START_REF] Sanguinetti | Treebanking user-generated content: a UD based overview of guidelines, corpora and unified recommendations[END_REF]

. These categories represent distinct lexical and grammatical variability that can be

Table 3

 3

.3: Typology of UGC specificities used in this work and as our manual annotation scheme. Extended from

[START_REF] Sanguinetti | Treebanking user-generated content: a UD based overview of guidelines, corpora and unified recommendations[END_REF]

.

Table 3

 3

		Encoding simplification
		¨Letter change/deletion/addition		≠ Abbreviation
	SRC	Je suis alé[allé] à la bibliothèque SRC	Merci bn[bien], les amis.
	REF	I went to the library	REF	Thank you very much, friends.
	Google	I am random at the library	Google	Thank you bn, friends.
		AE Phonetization/spelling mistake		Ø Wrong verb tense
	SRC	Il est nez[né] pour sa[ça].	SRC	Je suis allez[allé] à l'école.
	REF	He was born for it.	REF	I went to school.
	Google	He is nose for his	Google	I am going to school

.4: Examples of translation impact due to Encoding Simplification phenomena. The references (REF) are translations by the same MT engine of the correct form, accordingly. Translations were produced on the 07/03/2022 Normalized version of the tokens is displayed in blue brackets.

  In this work, we divide them into two types, with corresponding examples in Table 3.7:Table 3.7: Examples of translation impact due to Marks of Expressiveness phenomena. The references (REF) are manual edits of Google's translations of the correct form to include the missing tokens. Normalized version of the tokens is displayed in blue brackets.

		UGC Context-dependent specificities
		¨Special characters		≠ Named entities
	SRC	#Tenez bon ça va durer.	SRC	c'est excellent de jouer à fish fins.
	REF	#Hold on it's going to last. REF	It's excellent to play fish fins.
	Google	#Get good it will last.	Google	It is excellent to play fine fish.

  highlighted the disadvantages of fine-tuning, which requires knowing the test distribution, especially variable for UGC; and training a different model or module that specializes in targeting a domain distribution. These limitations ultimately pose an overhead for MT systems and require a priori understanding of the test distribution. The authors also assessed that using text outside the main target distribution, such as in UGC-augmented training corpora, actually degrades performance on the original target distribution (canonical text used during training).

In this regard, we focus our research on exploring methods that can generalize over a wide range of noisy phenomena when testing, which, in turn, aim to produce more robust MT systems without using any specific UGC text. Concretely, we have systematically chosen training corpora constrained to canonical texts and zero-shot evaluation of UGC test sets, these being considerably outside the distribution of our training datasets, as we will quantify and discuss in further detail in Section 3.6 At this point, it is worth noting that, since we want to conceive MT architectures that can adapt and generalize over new forms and noisy variations of the canonical training datasets, we are particularly interested in Neural Machine Translation (NMT) approaches. In this train of thought, our work mainly focuses on proposing methods that can enhance neural learning representations of noisy constructs, either by pre-processing pipelines or architecture modifications that translate into performance improvement when processing UGC.

  Table 3.8. On the other hand, to evaluate whether our methods keep their generalization properties and, thus, are still able to perform well when processing clean test sets, our evaluation policy is to always display the performance of our MT systems over two canonical test sets: newstest'14 and OpenSubTest. Both of these test corpora cover the domain of our training data respectively: formal linguistic forms featured in Je sais mais au final c'est moi que le client va supplier pour son offre et comme Jsui un gars cool, jfai au mieux. EN(ref) I don't know but in the end I am the one who will have to deal with the customer begging for his offer and because I'm a cool guy, I do whatever I can to help him. EN (ref) if you understand me leave the same thing or have gotten over it I take everything that can help me. Table 3.8: Examples for our considered UGC corpora. Noisy tokens and their corresponding translations are shown in bold font.WMT and considerably more colloquial ways of expression in OpenSubtitles.

	UGC Corpus		Example
	MTNT FR (src) Corpus #sentences #tokens	ASL TTR #chars	Corpus	#sents #tokens	ASL TTR #chars
	train set					UGC test
	WMT	2.2M	64.2M	29.7 0.20	335	PFSMB	777	13,680 17.60 0.32	116
	OpenSubtitles	9.2M	57.7M	6.73 0.18	428	MTNT	1,022	20,169 19.70 0.34	122
	test set					UGC blind
	OpenSubTest	11,000	66,148	6.01 0.23	111	PFSMB	777	12,808 16.48 0.37	119
	newstest'14	3,003	68,155 22.70 0.23	111	MTNT	599	8,176 13.62 0.38	127

PFSMB FR (src) si vous me comprenez vivé la mm chose ou [vous] avez passé le cap je pren tou ce qui peu m'aider.

Table 3 .

 3 10 reports the noise level of our test sets introduced in Section 3.5 with respect to# Metric / Test set ! PFSMB † MTNT † Newstest OpenSubsTest

	3-gram KL-Div	1.563	0.471	0.406	0.006
	%OOV	12.63	6.78	3.81	0.76
	avg. BPEstab	0.018	0.024	0.049	0.13
	PPL	599.48	318.24	288.83	62.06

Table 3 .

 3 10: Domain-related measure on the source side (FR), between Test sets and LARGE OP E NSU B T I T L E S training set. Dags indicate UGC corpora. our largest training set, LARGE OP E NSU B T I T L E S. The measures show how divergent our UGC corpora are from our largest training set. As shown by its OOV ratio, PPL, and KL-divergence value, the PFSMB corpus is considerably noisier than the MTNT corpus, making it a more difficult target in this translation scenario. We can also notice that both UGC test sets are considerably dissimilar to other canonical OOD set (newstest'14), especially in terms of OOV ratio and BPE piece stability (BPEstab). This metric, proposed by us in Rosales Núñez et al. (2019), quantifies the average lexical diversity of a test set for a given BPE tokenization model, i.e. low

average BPE stability points to a more variable BPE neighborhood for the average BPE piece, and thus, higher average vocabulary complexity. Specifically, if N is the total number of BPE tokens in a tokenized corpus, j each of the words of the BPE vocabulary (set to 16K) and freq j

Table 4 .

 4 1 reports the results of the different models that we have considered. It is important to note that, as stated in Section 2.3, we have implemented an alignment-based method to substitute any possible <UNK> token into the predictions by copying their corresponding aligned match from the source phrase. Alignments have been predicted by FastAlign[START_REF] Dyer | A simple, fast, and effective reparameterization of IBM model 2[END_REF]) trained on the same training data as the corresponding MT system.

			PB-SMT			seq2seq			Transformer
		PFSMB	MTNT	News	Open	PFSMB	MTNT	News	Open	PFSMB	MTNT	News	Open
	WMT	20.5	21.2	22.5 †	13.3	17.1	24.0	29.1 †	16.4	15.4	21.2	27.4 †	16.3
	Small	28.9	27.3	20.4	26.1 †	26.1	28.5	24.5	28.2 †	27.5	28.3	26.7	31.4 †
	Large	30.0	28.6	22.3	27.4 †	21.8	22.8	17.3	28.5 †	26.9	28.3	26.6	31.5 †

Table 4 .

 4 1: BLEU score results for our three models for the different train-test combinations. The best result for each test set is marked in bold, best result for each system (row-wise) in blue, and score for in-domain test sets with a dag. 'News' and 'Open' stand, respectively, for the newstest'14 and OpenSubtitlesTest test sets.

			PB-SMT			seq2seq			Transformer
		PFSMB	MTNT	News	Open	PFSMB	MTNT	News	Open	PFSMB	MTNT	News	Open
	Large 32K	22.7	22.1	16.1	27.4 †	25.3	27.2	21.9	28.4 †	27.8	28.5	27.1	31.9 †

Table 4

 4

.2: BLEU scores for the Large training configuration using a 32K BPE vocabulary.

Table 4 .

 4 3: Examples from our noisy UGC corpus translated by our MT systems, displaying source (src) and reference translation (ref ).

seq2seq Stop Tenwolf, please. Please, please, please, please, please, please, please, yes, yes, yes, yes, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, Transformer Please stop!

Table 4 .

 4 4: Noise added by the MT system estimated by TSNR for the PFSMB corpus, the lower, the better. Small and Large refer to the small and large instances of the OpenSubtitles training set.

	.4, we

Table 5 .

 5 1 reports the most common normalization changes performed by this method in the PFSMB test set (described in Chapter 3). It can be noticed that they are mainly grammatical corrections of badly used in-vocabulary tokens with identical French pronunciation.

	Normalization	a!à sa!ça et!est la!là à!a tous!tout des!de regarder!regardé ils!il prend!prends
	Number of app.	87	16	15	13	12	11	8	7	6	6

Table 5

 5

.1: Most frequent normalization replacements on the PFSMB test corpus.

  Table 5.3: BLEU score results comparison on the MTNT and PFSMB blind test sets. The G2P phonetizer has been used for normalization. M&N18 stands for (Michel and Neubig 2018)'s baseline system.

						Blind Tests
		System				MTNT PFSMB
		Large -PB-SMT Raw		29.3	30.5
		Large -PB-SMT Phon. Norm	26.7	26.9
		Small -Transformer Raw	25.0	19.0
		Small -Transformer Phon. Norm 24.5	18.3
		M&N18 Raw			19.3	13.3
		M&N18 UNK rep. Raw		21.9	15.4
							Transformer
		PFSMB MTNT News Open	PFSMB MTNT News Open
		WMT 20.5	21.2	22.5 †	13.3	15.4	21.2	27.4 †	16.3
	Baseline	Small 28.9	27.3	20.4	26.1 †	27.5	28.3	26.7	31.4 †
		Large 30.0	28.6	22.3	27.4 †	26.9	28.3	26.6	31.5 †
		WMT 20.4	20.2	21.9 †	13.4	15.0	20.4	26.7 †	16.2
	G2P	Small 28.4	26.2	19.9	26.1 †	29.0	28.3	25.7	31.4 †
		Large 29.0	27.6	21.8	27.4 †	28.5	28.2	25.9	31.5 †
		WMT 20.4	20.4	21.7 †	13.4	14.6	20.7	26.5 †	16.1
	Espeak	Small 28.0	26.3	19.8	26.2 †	28.5	28.8	25.6	31.4 †
		Large 28.3	27.7	21.6	27.4 †	27.5	28.6	25.8	31.5 †

Table

5

.2: BLEU score results for our three benchmark models on baselines (without normalization) and normalized test sets using G2P and Espeak phonetizers. The best result for each test set is marked in bold, in-domain scores with a dag.

  est un peu plus que mon amie qui fête son annif Cartes bancaires de Zlatan retrouvés dans un taxi... On en parle ou pas WWW44 norm src Carte bancaire de Zlatan retrouvé dans un taxi... On en parle ou pas WWW44 ref Zlatan's bank cards found in a cab... we talk about it or not WW44 raw MT Zlatan's bank cards found in a cab... we talk about it or not WW44 norm MT Zlatan bank card found in a taxi... we talk about it or not WWW44

	ref	it's a bit more than a friend to me who celebrate his birthday,
	raw MT	It's a little more than my friend doing his birthday,
	norm MT It's a little more than my friend celebrating her birthday,
	¬ src	zlatan est nez pour marqué
	norm src zlatan est né pour marquer
	ref	Zlatan was born to score
	raw MT	Zlatan's nose is for marking
	norm MT Zlatan was born to score
	√ src
	Table

  Table6.1: BLEU score for our models for the different train-test combinations. In-domain test sets are marked with a dag. 'News' and 'Open' stand, respectively, for the WMT and OpenSubtitles test sets. WMT and OpenSubtitles are the training corpora, described in Section 3.5

				WMT			OpenSubtitles
		PFSMB MTNT News † OpenSubTest	PFSMB MTNT News OpenSubTest †
	BPE-based models
	seq2seq	9.9	21.8	27.5	14.7	17.1	27.2	19.6	28.2
	+ <UNK> rep.	17.1	24.0	29.1	16.4	26.1	28.5	24.5	28.2
	Transformer	15.4	21.2	27.4	16.4	27.5	28.3	26.7	31.4
	Character-based models
	charCNN	6.2	12.7	17.2	9.2	13.3	16.3	10.1	21.7
	+ <UNK> rep.	16.1	18.2	22.1	11.5	18.6	20.2	14.6	23.9
	char2char	7.1	13.9	18.1	8.8	23.8	25.7	17.8	26.3

Table 6

 6

		30	char2char Transformer	29.7	30.2
	BLEU score	20 25	24.8 25.8		19.8
					16.9
		15
			0	1 # charOOVs	2

.1 reports the BLEU scores of the different character-and BPE-based models that we consider. As observed in Section 4.3, the simple replacement strategy for the <UNK> token substantially increases the BLEU score, which also benefits the word-level decoding of charCNN, whereas char2char, with a character-by-character output, and using the original implementation parameter, does not output any <UNK>.

  Table 6.2, to identify which UGC specificities were the hardest to translate. Examples of annotations are given in Table 6.3.For instance, the char2char model only outputs 8 sharp symbols when translating the test set of the PFSMB, whereas the reference, contains 345 hash tags starting with a '#'.While the Transformer model is less sensitive to this problem (it produces 105 sharp symbols when translating the PFSMB test), its translation quality also suffers from the presence of hashtags as it often tries to translate part of the hashtags relying on the sub-word units identified by the BPE tokenization rather than simply copying them from the source. Additionally, it can be noticed that errors 2 and 12 (diacritization and graphemic/punctuation stretching) are treated somewhat better by the char2char model than by the Transformer model, being less frequent in the worst translations of the former.

	Figure 6.2 shows the number of UGC specificities in
	each of the 100 worst and best translations of the two	code kind of specificities
	considered models.	(1)	Letter deletion/addition
		(2)	Missing diacritics
		(3)	Phonetic writing
		(4)	Tokenization error
		(5)	Wrong verb tense
		(6)	Special chars. (#, @, URL)
		(7)	Wrong gender/number
		(8)	Inconsistent casing
		(9)	Emoji
		(10)	Named Entity
		(11)	Contraction
		(12)	Graphemic/punct. stretching
		(13)	Interjection
		Table 6.2: Typology of UGC specificities used
		in our manual annotation scheme. Refer to
		Section 3.3 for further details.

2

For both models, the most difficult specificities appear to be the presence of NE (category 10) and the inconsistent casing (category 8) often corresponding to several words written in full uppercase to denote emotions or excitement. Interestingly, these two types of noise have more impact on the char2char model than on the Transformer model, even if it could be expected that the character embeddings learned by the former would not be sensitive to the case. Another important category of errors is category 6 that corresponds to hashtags, mentions, and URLs, for which the char2char model is not capable of producing characters or a sequence of characters that are rare in the training set (namely, #, @ or http://www).

  Comparison of the number of UGC specificities in the best and worst translation hypotheses of char2char and Transformer. Noise categories are defined in Table6.2.

		(a) Worst translation hypotheses	(b) Best translation hypotheses
		250
		200
	Counts	100 150
		50
		0
		ch ar de l/a dd m is si ng di ac rit ic s ph on et ic w rit in g	to ke ni za tio n w ro ng te ns e	sp ec ia l ch ar	ag re em en t	ca si ng	em oj i na m ed en tit y	co nt ra ct io n	re pe tit io n	ch ar de l/a dd m is si ng di ac rit ic s ph on et ic w rit in g to ke ni za tio n w ro ng te ns e sp ec ia l ch ar ag re em en t ca si ng	em oj i na m ed en tit y co nt ra ct io n re pe tit io n
								char2char Transformer
	Figure 6.2:

  Results of the copy task, evaluated by the BLEU score before and after <UNK> replacement (+<UNK> rep.

			Vocabulary Size
		164 125 100	80	60
	in-test
	%<UNK> pred.	0	0.2	5	17 29.5
	BLEU	92.9 95.8 77.6 24.9	1.9
	+<UNK> rep.	92.9 96.6 98.4 98.5 98.7
	out-test
	%<UNK> pred.	0	9.2 13.8	25	36
	BLEU	54.5 63.7 52.3 15.3	0.9
	+<UNK> rep.	54.4 96.6 98.7 99.1 99.5
	Figure 6.3:

  ItAnnoysMeWhen in the moring at 7 am someone talks to me although I didn't wake up yet.N. src le matin à 7h on me parle alors que je suis pas encore réveillé. N. ref in the moring at 7 am someone talks to me although I didn't wake up yet.

	src	JohnDoe389 (10) qui n'arrive pas a (2) depasser (2) 1 a (2) FlappyBird ... ptddddr (12,13)
	ref	JohnDoe389 who can't score more than 1 at FlappyBird ... lmaooooo
	N. src Jean qui n'arrive pas à dépasser 1 à Jean ...
	N. ref Jean who can't score more than 1 at Jean...
	¡ src	#CaMeVénèreQuand (6) le matin a (2) 7h on me parle alors que je suis pas encore réveiller. (5)
	ref #¬ src vu sa tete (2) c (3) normal kon (3) est (3) jms (11) parler (5) d'elle !
	ref	in light of her face it's normal no one ever spoke about her!

N. src vu sa tête c'est normal qu'on a jamais parlé d'elle ! N. ref in light of her face it's normal no one ever spoke about her! √ src y a ma cousine qui joue a (2) flappy bird (10) mdrrrrrrrrrrr (12, 13) elle et plus nuuul (12,7) que moi ref my cousin plays flappy bird loooooooooool she's more hopeless than me N. src y a ma cousine qui joue à Jean Jean elle et plus nulle que moi N. ref my cousin plays Jean Jean she's more hopeless than me Table 6.4: Examples from our annotated noisy UGC corpus. Source sentences have been annotated with UGC specificities of Table 6.2 (in blue) according to their numerical code. For each example, the original source and reference (src and ref ) and their corresponding normalized version (N. src and N. ref ) are shown.

Table 6 .

 6 5: BLEU score ratios between pairs of noisy and normalized sets of sentences, containing only one UGC specificity. BLEU scores on noisy sets are shown in parenthesis. Three different metrics are shown for comparison: MULTIBLEU-DETOK.PERL (MB) , CHRF and SACREBLEU (SB). Error for 95% confidence intervals (CI Err).

  table. Noisy/Clean BLEU scores' ratios for an accumulated number of UGC specificities present per sentence for each model, corresponding to the results in Table 6.6b. The 4to7 bin groups more than 4 types to provide a larger subcorpus, which weighted average is 4.34 UGC specificities per sentence. (b) BLEU score ratio between pairs of normalized and noisy sentences containing N specificities. BLEU scores on noisy sentences are shown in parenthesis.

	0.9			s2s c2c tx	# sents.	1 1,306 1,776 1,439 1,089 2 3 4+
	Ratio				Augment ratio s2s	3.26 0.90 (30.1)	5.82 0.83 (27.0)	8.51 0.77 (24.2)	13.04 0.75 (23.2)
	0.8				c2c	0.92 (29.5)	0.87 (26.6)	0.83 (24.3)	0.83 (23.2)
	1	2	3	4to7	TX	0.96 (32.8)	0.89 (30.0)	0.86 (28.3)	0.84 (26.5)
	Number of present UGC specificities per sent.		(b)
		(a)
	Figure 6.6: (a)

Table 6 .

 6 toux ceux qui ont l'esprit disons, un peu fermé de regarder sur les "Français d'origine contrôlée" norm Je conseille à tous ceux qui ont l'esprit disons, un peu fermé de regarder sur les "Français d'origine contrôlée" ref I advise everyone with a, let's say a little narrow mind to watch about the "Français d'origine contrôlée" s2s I suggest cough those who have minds say, a little closed to look at the "frances of controlled origins" c2c I counsel those who have the mind, a little close to looking at the French original controlled original controlled. Tx I advise anyone with a mind, say, a little closed to look at the controlled French. ƒ src le côté suis tro cool au quotidien et je relach tout quan j'ai bu norm les gens qui m'aiment me détestent quand j'ai bu ref my side very cool in everyday life and loosen everything when I've been drinking s2s I've been drinking all the time and I've been drinking everything quan I've been drinking. c2c the side of the daily cool side and relacing everything when I've been drinking Tx I'm the cool side. I'm the cool one. mais de voir la joie des grands joueurs comme Zlatan, Motta, Verratti je trouve ça magnifique norm Je sais pas vous, mais de voir la joie des grands joueurs comme Jean, Jean, Jean je trouve ça magnifique ref I don't know about you, but seeing the joy of great players like Zlatan, Motta, Verratti I think it's wonderful s2s I don't know you, but seeing the joy of the great players like Zlatan, Motta, Verratti, I think it's beautiful. c2c I don't know about you, but to see the joy of great players like Zlatan, Motta, Varratt, I think it's beautiful. Tx I don't know about you, but seeing the joy of big players like Zlatan, Motta, Verratti, I think it's beautiful. 6: Examples from our noisy UGC corpus showing the Transformer, char2char and seq2seq predictions. Present UGC specificities of Table 6.2 (in blue) are marked in bold.We have noticed that the special <UNK> token management and generation are highly affected by the training vocabulary size for character-based models, i.e. if there are not enough characters excluded from the vocabulary (and consequently, replaced by <UNK>) at train time, the translation system does not learn how to correctly manage such tokens when present during test time. For the sake of this observation, we trained character-based systems with different vocabulary sizes, for which were observed <UNK> being generated during evaluation only when retaining 90 characters or fewer. In terms of configuration, our new char2char systems have 6 vocabulary size variations: 90, 85, 80, 75, 70 and 65. In this way, we hypothesize that the choice of optimal vocabulary size, often overlooked in the NMT character-based literature, can help to improve generalization over out-of-domain (OOD) test sets, while keeping optimal performance on in-domain test evaluation. Furthermore, regarding our two noisy UGC test sets, as we previously mentioned, the PFSMB contains an especially high overlap between source and reference compared to the MTNT, which serve us to evaluate whether our strategy can benefit the most from copying from the source without compromising translation of noisy texts that, change much more between source and reference.To follow, we report results for variations of the base char2char[START_REF] Lee | Fully character-level neural machine translation without explicit segmentation[END_REF]) model (originally with 302 characters kept in the vocabulary) with considerably less number of characters in an effort to correctly train the model to produce <UNK> when needed. In Table6.7 we can see that the best char2char for our noisy UGC test sets are those with a similar vocabulary size of 70 and 75, which improves the performance of the base char2char in Table6.1 by +0.8 and +0.6 BLEU points on the PFSMB and MTNT test sets respectively. Furthermore, these models do not lose any performance on in-domain evaluation, having the same or higher BLEU performance for newstest'14 and OpenSubTest clean test set for all the reduced vocabulary size systems.It is also worth noticing that the vocab-85 system also improves the newstest'14 by +2.1 BLEU points, which is a different domain to the training corpus (OpenSubtitles).Additionally, in Table6.9, we show results for the noisy UGC blind tests, for which we can observe a BLEU performance improvement on both tests, outperforming the seq2seq model with <UNK> replacement. In turn, with these results, we noticed that the BPE-based Transformer continues to be an upper-bound of performance for the character-base models considered,

			Letter deletion/addition/change
	¿ src	j'arrive pas à boir normalemen
	norm j'arrive pas à boire normalement
	ref	I can't drink normally
	s2s	I can't drink normal.
	c2c	I can't drink normally.
	Tx	I can't drink normal men.
	¡ src	Je conseille à Tokenization
	≈ src J'sais pas vous, ∆ src pendant que vous me laissez en chien à l'atelier mon score de flappy bird fait que d augmenter
	norm pendant que vous me laissez en chien à l'atelier mon score de Jean fait que d'augmenter
	ref	while you're bailing out on me at the workshop my flappy bird score is just increasing
	s2s	when you leave me as a dog when you leave me as a dog at the workshop.
	c2c	while you leave me dog at the workshop my flappy bird score is that increasing
	Tx	while you leave me as a dog at the workshop my flappy bird score is just up.
	« src	il ma dit que c'était normal aussi et que ça allait redescendre,
	norm il m'a dit que c'était normal aussi et que ça allait redescendre,
	ref	he told me it was normal too and that it would come down,
	s2s	He said it was normal, too, and it was going to go down,
	c2c	He told me it was normal, too, and it was going back,
	Tx	He told me it was normal, too, and it was gonna come down,
			Inconsistent casing
	» src	Jean DANS VOS YEUX	… src	JE VIENS DE VOIR Jean ET Jean JE PEUX PLUS
	norm Jean dans vos yeux	norm Je viens de voir Jean je peux plus
	ref	Jean IN YOUR EYES	ref	I JUST WATCHED Jean AND Jean CAN'T TAKE IT
	s2s	Jean D in VOSY	s2s	I'm going to kill Jean and Jean I can't believe it.
	c2c	Jean in your eyes	c2c	I'm here to see Jean And Jean I can no longer.
	Tx	Jean in your eyes	Tx	I just saw Jean and Jean again.
		Domain-specific characters and emojis
	src	Avec mes magnifiques jumeaux Jean et Jean @maxcarver @Charlie_Carver ⇤
	norm Avec mes magnifiques jumeaux Jean et Jean
	ref	With my wonderful twins Jean and Jean @maxcarver @Charlie_Carver ⇤
	s2s	with my gorgeous Jean and Jean @maxarver @Carlie_Carver @Carlie_Carver #
	c2c	with my beautiful Jean twins, Jean Jean and Jean Charlier Charlier Carver.
	Tx	with my beautiful twins Jean and Jean imexcarver Charlie_Charver @Charver

  It is veryinteresting to notice that. for this task, the char2char model achieved a notable performance for UGC and OOD test (newstest'14) compared to the Transformer, which contrasts to our previous text-to-text MT experimental results. However, we can see that the latter performs better in in-domain translation, which points to low robustness properties on this MT task. In the second place, it can be noted that the finer-grained IPA-level Transformer, is systematically outperformed by its BPE tokenization counterpart, probably because BPE helps the model to disambiguate between very similar pronunciations by containing larger phoneme tokens. PFSMB) test sets respectively for the char2char model. This suggests that conceiving methods that exploit both phoneticized and original text information are worth it over purely phoneme-to-text training data, and justifies up to some extent the processing over-head needed for our normalization pipeline in Chapter 5. BLEU score results for our three phoneme-to-text models on clean and noisy test sets. The best result for each test set is marked in bold, in-domain scores with a dag.

	Finally, we can notice that using only the phoneticized source information is detrimental to MT, at least in overall translation quality compared to our phoneticized correction systems in Chapter 5, which achieves +5.1 and +3.1 BLEU over in-domain (OpenSubTest) and UGC PFSMB MTNT News Open phon2char 20.7 20.1 13.5 21.7 † Transformer 16K Phon. BPE 10.4 12.1 4.4 22.5 † IPA-level 5.2 7.3 2.2 18.8 † (Small Training Corpus Figure 6.8:

  .6b, but the former seems more robust to a relatively high number of occurring UGC specificities.

	System	c h a r d e l / a d d m i s s i n g d i a c r i t i c s p h o n e t i c w r i t i n g t o k e n i z a t i o n w r o n g t e n s e s p e c i a l c h a r a g r e e m e n t c a s i n g	e m o j i	n a m e d e n t i t y c o n t r a c t i o n r e p e t i t i o n i n t e r j e c t i o n s
	phon2char	0.96 (31.3)	1.00 (27.0)	0.97 (27.2)	0.81 (25.5)	0.99 (27.4)	0.79 (22.0)	0.99 (28.4)	0.88 (25.3)	0.79 (22.9)	0.90 (27.4)	0.95 (30.0)	0.88 (25.5)	0.87 (26.9)

Table 6 .

 6 10: BLEU score ratios between pairs of noisy and normalized sets of sentences for the phon2char system, containing only one UGC specificity. BLEU scores on noisy sets are shown in parenthesis.

		1	2	3	4+
	# sents.	1,306 1,776 1,439 1,089
	phon2char	0.91 (27.0)	0.86 (24.2)

  Intuitively, an MDN is a combination of M variational encoders. Our intuition is that UGCs contain several different kinds of variations covering very different aspects ranging from morphology to phonemics, including lexicon and sentence structure (see Chapter 3) and it is illusory to hope that a single latent variable will be able to capture all of them. But, with an MDN, it is possible that each component of the encoder

is able to model different UGC specificities, allowing us to better model UGC. In the past, MDN has been used to address sequence-to-sequence generative tasks, such as SketchRNN

[START_REF] Ha | A neural representation of sketch drawings[END_REF]

and modeling of sequential environment states in reinforcement learning

[START_REF] Ha | Recurrent world models facilitate policy evolution[END_REF]

.

  , we also report results for our VNMT model extending the encoder's variational inference mechanism with a 4-flows Normalizing Planar Flows (PF) (Rezende and Mohamed 2015). Other autoregressive normalizing models, such as Sylvester Flows (van den Berg et al. 2018), are available and could

  Table7.2. Overall, we obtain the best BLEU scores across all test sets for the "full" multi-VNMT model. In Table7.2 we can notice that, overall, we obtain the best BLEU scores across all test sets for the full multi-VNMT version. As interesting mixed results, we can highlight the cases for static latent representation (z static), where instead of sampling from the learned distributions, we retrieve their mean as output, and which showed slightly better BLEU scores when translating the MTNT and newstest'14 test sets, with +1.2 and +0.1 BLEU points improvement, respectively. However, results are inconsistent for UGC test sets and otherwise worse than those of the full model for both in-domain and canonical OOD test sets for our two training configurations. This might be explained by the lack of stochastic perturbations provided by the sampling step during training, leading the model to lose generalization during evaluation.It is also interesting to note that using a categorical variational version of the mixing coefficients rather than the usual choice of computing them with a softmax improves translations Posterior collapse Comparing multi-VNMT and its ablated version system removing the MDN module, both trained on OpenSubtitles and when evaluating the corresponding indomain test set (OpenSubTest), we have calculated the average KL divergence of the variational decoder's MDN, which resulted in 0.21 and 0.15, respectively. Performing the same analysis for the WMT training and evaluation configuration, KL divergence resulted in 0.38 for the full multi-VNMT and 0.33 for its version removing the MDN block. These results suggest that our proposed architecture is less prone to suffer from the posterior collapse phenomenon, and this could be explained by the use of several independent posterior distributions when including MDN in our model. This could also explain why, in Table7.1, our systems employing MDN have an overall higher BLEU results than the aforementioned ablated system where we remove this component. Table 7.4: Translation performance of our considered VNMT models on the De-En IWSLT'14 experimental setup.

				WMT			OpenSubtitles
		PFSMB †	MTNT †	News ⇧	OpenTest	PFSMB †	MTNT †	News OpenTest ⇧
	multi-VNMT	16.0	21.8	27.9	16.7	28.4	29.2	26.4	31.5
	Mono-multi-VNMT	15.8	21.8	28.0	16.5	29.3	28.7	26.2	31.6
	MLM-multi-VNMT	15.6	21.8	27.9	16.5	28.0	28.9	26.4	31.5
	MonoMLM-multi-VNMT	15.6	21.6	28.0	16.5	27.8	29.0	26.3	31.5
	Table 7.3: Best system using MLM and source monolingual variational reconstruction loss.
					in-dom test
		NF-VNMT (Z ~4-PF) (Setiawan et al. 2020)	36.1 †
		multi-VNMT (Z ~4-PF)	36.3

quality: the latter is only performing better for the newstest'14 test set when training on the OpenSubtitles corpus (⇡ non-latent). Following the same trend, the WMT training data configuration also shows improvements when using the Gumbel-Softmax version, for which +0.8 and +0.3 BLEU point increment were obtained for both the PFSMB and MTNT UGC test sets, respectively.

Table 7 .

 7 2b), the assessed robustness of our proposed architecture diminishes, matching that of the Transformer baseline.

		c h a r d e l / a d d m i s s i n g d i a c r i t i c s p h o n e t i c w r i t i n g t o k e n i z a t i o n w r o n g t e n s e s p e c i a l c h a r a g r e e m e n t c a s i n g	e m o j i	n a m e d e n t i t y c o n t r a c t i o n r e p e t i t i o n i n t e r j e c t i o n s
		0.80	0.95	0.93	0.96	0.94	0.88	0.95	0.75	0.91	0.86	0.95	0.90	0.93
	s2s	(28.7)	(33.9)	(27.3)	(30.8)	(30.7)	(26.1)	(27.1)	(27.7)	(31.0)	(31.7)	(30.8)	(30.2)	(29.2)
		0.99	0.99	0.86	1.00	0.97	0.81	0.96	0.86	0.83	0.94	0.91	0.95	0.91
	c2c	(32.5)	(29.6)	(25.2)	(31.9)	(28.8)	(24.6)	(28.9)	(28.0)	(26.2)	(32.7)	(30.4)	(26.2)	(28.7)
		0.98	1.02	1.03	0.98	1.02	0.92	0.97	0.90	0.75	0.99	0.93	0.89	0.86
	TX	(35.3)	(34.0)	(33.2)	(32.9)	(33.7)	(29.2)	(33.8)	(26.9)	(28.3)	(35.4)	(31.1)	(30.8)	(30.2)
		1.00	0.99	0.95	0.96	1.00	0.92	1.00	0.88	0.83	0.98	0.96	0.86	0.83
	multi-VNMT	(35.5)	(34.7)	(34.0)	(33.1)	(32.5)	(30.3)	(30.2)	(26.8)	(30.6)	(35.1)	(31.1)	(28.3)	(28.2)

Table 7 .

 7 6: BLEU score ratios between pairs of noisy and normalized sets of sentences, containing only one UGC specificity. BLEU scores on noisy sets are shown in parenthesis.

  BLEU score ratio between pairs of normalized and noisy sentences containing N specificities. BLEU scores on noisy sentences are shown in parenthesis.

						s2s		1	2	3	4+
						c2c	# sents.	1,306 1,776 1,439 1,089
	Ratio	0.9				tx mdn-tx	s2s c2c	0.90 (30.1) 0.92 (29.5)	0.83 (27.0) 0.87 (26.6)	0.77 (24.2) 0.83 (24.3)	0.75 (23.2) 0.83 (23.2)
		0.8					TX	0.96 (32.8)	0.89 (30.0)	0.86 (28.3)	0.84 (26.5)
								0.97	0.93	0.88	0.84
		1	2	3	4	7	multi-VNMT	(33.0)	(30.3)	(28.4)	(26.6)
		# of present UGC specificities per sent.	(b)
			(a)

  Table 7.8: BLEU scores of our best systems on UGC blind test sets.

Table 7 .

 7 MTNT † JESC ⇧ 9: BLEU scores of our translation systems trained on OpenSubtitles Ja-En. The † symbol indicates the UGC test sets, and ⇧ in-domain test sets.

	Transformer	5.90	11.50
	multi-VNMT	6.30	11.80
	(Michel and Neubig 2018)	6.65	-
	(Michel and Neubig 2018) (+tuning)	9.82	-
	(Pryzant et al. 2018)	-	6.30

  These examples show that, despite relying on large amounts of text, most of them gathered from the Internet, state-of-the-art MT engines have made little progress in UGC translation, Did you see he deserved it wsh #AperoChezRicard Did you see he was asking for it, wsh #AperoChezRicard ! +10000, sheh! ! +10000, sheh! ! tabuz, did nothing ! Tabuz, didn't wash anything ! kica? lemecousonchien?! Kica? The guy with the dog? ! Wtf is wrong with him ? #PETA4EVER ! What's wrong with him? #PETA4EVER. ! who is it ? the dog ? looool ! Ki ca? The dog? Looool.

	(@rigolboche)
	Original source
	T'as vu il l'a bien cherché wsh #AperoChezRicard
	! +10000, shah! ! tabuz, lavé rien fé ! ki ca ? lemecousonchien ? ! Wtf is wrong with him ? #PETA4EVER ! ki ca ? le chien ? looool
	Bing © (2016)	Bing © (2023)
	Did you see he looked for it wsh #AperoChezRicard	Did you see he looked for it wsh #AperoChezRicard
	! +10000, shah! ! tabuz, washed nothing fe ! ki ca ? theguyorhisdog? ! Wtf is wrong with him ? #PETA4EVER. ! ki ca ? the dog? looool	! +10000, sheh! ! tabuz, washed nothing fe ! Kica? Lemecousonchien? ! Wtf is wrong with him ? #PETA4EVER. ! ki ca? The dog? looool
	Google Translator © (2023)	DeepL © (2023)
	Did you see he looked for it wsh #AperoChezRicard	You see he had it coming wsh #AperoChezRicard
	! +10000, sheh! ! tabuz, washed nothing fe ! Kica? Lemecousonchien? ! Wtf is wrong with him ? #PETA4EVER ! ki ca? The dog? looool	! +10000, shah! ! tabuz, washed nothing fe ! kica? lemecousonchien? ! Wtf is wrong with him ? #PETA4EVER. ! ki ca? the dog? looool
	BingGPT © (2023)	ChatGPT © (2023)

Cf. the long standing series of workshops on Noisy User-Generated Text http://noisy-text.github.io/

2022/.

www.statmt.org/europarl/

www.statmt.org/wmt15/training-parallel-nc-v10.tgz

Popular French websites devoted respectively to video-games and health.

http://statmt.org/paracrawl

The main goal of this thesis is to improve MT performance when translating noisy UGC. Nevertheless, it is worth noting that we started to explore the impact of UGC on the performance of NLP system with a Part-of-Speech task. This first experience, resulted in a publication at TALN in

To compute the average performance gap, we consider the WMT and Small trained systems, defining newstest'14 and OpenSubTest and in-domain and OOD canonical test sets accordingly

"_" indicates BPE pieces

marked vs. mark-INFINITIVE in English.

The translation quality was simply defined as the edit distance between the translation hypothesis and the reference translation.

Similar conclusions can be drawn from the observation of the 100 best translations (see Figure1in supplementary material for details).

To ensure that this normalization has actually made our corpus closer to a canonical corpus, we have computed the perplexity of the original sentences and of the normalized sentences estimated by a 5-gram Kneser-Ney language model trained on the OpenSubtitles corpus: the normalized version has a perplexity of 2,214 (and 11.60% of its token are OOV) far lower than the original version (with a perplexity of 8,546 and an OOV ratio of 19.60%).

The annotated corpus and code collection can be found in https://github.com/josecar25/PMUMT_anno tated_UGC_corpus/

The model has been implemented in OpenNMT(Klein et al.

2018).

SACREBLEU signature: nrefs:1|bs:1000|seed:12345|case:mixed|eff:no|tok:intl|smooth:exp|version:2.1.0.

SACREBLEU signature: nrefs:1|bs:1000|seed:12345|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0.

We have re-implemented our models using the Fairseq toolkit[START_REF] Ott | fairseq: A fast, extensible toolkit for sequence modeling[END_REF] to keep the same NMT framework. We focus on their system using 4 Planar NF as ours use the same approach.

It is worth highlighting that there is no difference between the corpus used for any of the transfer-learning MT systems and the one used for training the benchmark in Table7.1.

Acknowledgements

First and foremost, I want to thank my supervisors, Guillaume Wisniewski and Djamé Seddah, for the countless hours that they dedicated to our discussions and their deep involvement in this work. I am very fortunate for the opportunity to benefit from their expertise throughout my Ph.D., for their exceptional guidance and their continuous support, always looking out for me. This dissertation would not have been possible without them.

I would also like to thank the members of my dissertation committee, Thierry Poibeau, Christophe Cerisara, Joseph Le Roux and Anne Vilnat, for their invaluable feedback. Their expertise and diverse perspectives provided rich insights into this work.

Chapter 6

Character-level for noisy UGC MT After exploring our first automatic normalization method tailored to one UGC specificity (phonetic writing), we decided to study and improve the generalization properties of NMT systems by leveraging fine-grained translation granularity. This approach, compared to the one introduced in the previous chapter, aims to address a wider range of UGC specificities that the MT system could correctly translate by relying on models that operate at the character level. Indeed, character-based models are open-vocabulary models designed specifically to learn n-grams representations (and therefore translations) of tokens. Segmentation at the character-level is one of the standard approaches to deal with the OOV problem in NLP systems as it is, in contrast to handling OOVs using an a unique <UNK> token (discussed in Section 2.3), capable of producing a learning representation for any sequence of seen characters.

In order to investigate the impact of these models on UGC translation, we compare the robustness of characters-based systems and that of BPE-based systems. We do not consider coarser granularities, such as words, since it has been proven that BPE tokenization consistently outperforms word-level segmentation applied to MT, as reviewed in Section 2.3. Our intuition is that, by conceiving robust-enough NMT systems that take advantage of character-level compositions, the architecture could associate neural representations of noisy OOVs (e.g. spelling errors) to 'clean' tokens, therefore automatically recovering from noise.

In a second step, building on the results of our first experiments, in Section 6.3 we show, that despite their word-level open-vocabulary properties, the presence of out-of-vocabulary characters (charOOVs), a characteristic of UGC (refer to Chapter 3), hurts translation quality of character-based models and questions our intuition about these models. In order to elaborate There's my cousin who plays a flappy bird mdrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr c2c There's my cousin that plays a flappy bird might bring her and nouse than me. Table 6.3: Examples from our noisy UGC corpus showing the Transformer (denoted as Tx) and char2char (denoted as c2c) predictions. Source sentences have been annotated with UGC specificities of Table 6.2 (in blue) according to their numerical code. Parts of the reference that were correctly translated are underlined and noise occurrences and their translations are marked in bold.

Experimental Evaluation

Evaluation Protocol

We have conducted experiments on our VNMT systems and the NF-VNMT baseline using the data and protocols discussed in Chapter 3. In this regard, we present an ablation test of our proposed multi-VNMT system and show the impact of our architecture's design over MT performance across our different test sets. We also performed experiments by incorporating MLM and reconstruction objective functions, as reviewed in Section 7.1. In addition, we evaluate the impact of temperature ⌧ in order to assess whether the sparsity of mixing coefficients favors a given type of texts among our test sets. Concerning this, as an initial experimental configuration, we chose ⌧ = 1.0, which was selected mainly aiming to avoid artificial gradient scaling during backpropagation, directly caused by this coefficient being relatively larger than 0 in order to introduce variability to the sampling process.

We quantify the robustness of multi-VNMT by employing the PMUMT evaluation framework discussed in Section 6.5. Finally, we present a series of visualizations and metrics to characterize how multi-VNMT behaves when processing UGC during evaluation, in order to give further insights of its robustness capabilities compared to the NF-VNMT and Transformer baselines.

Training models All systems are trained using a batch size of 4,096 tokens using the Adam optimizer (Kingma and Ba 2015) accumulating gradients every two steps, and the Noam learning rate schedule (Vaswani et al. 2017) with 8,000 warm-up steps. Throughout training, learning rate reaches a maximum value of 0.0009 and minimal value of 0.0001. Both encoder and decoder Transformers are trained using 0.1 dropout, and we used 0.1 label smoothing [START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF]. Training for, at most, 300K training iterations on a single Nvidia V100 took about 50 hours to converge for the multi-VNMT models. In order to avoid posterior collapse, we use C -VAE [START_REF] Prokhorov | On the importance of the Kullback-Leibler divergence term in variational autoencoders for text generation[END_REF], with values = 1 and C = 0.1, as done in [START_REF] Setiawan | Variational neural machine translation with normalizing flows[END_REF]. Additionally, we used a Kullback-Leibler (KL) annealing schedule of 80K iterations, i.e. scaling the KL divergence term in Equation 7.1, which allows the model to mostly focus on the translation loss before starting regularizing the KL divergence.

We have also retrained the Transformer Base baseline system presented in previous chapters with the optimal hyperparameters found for the VNMT systems, namely, a peak learning

Impact of source-side monolingual joint training

In Table 7.3 we report results with our proposed multi-VNMT system when using source-side monolingual corpora MLM and reconstruction loss terms. √ src Vient de perdre une grosse heure a #flappybird cc @JohnDoe533 @JohnDoe534 @JohnDoe535 ref Just lost a big hour on #flappybird cc @JohnDoe533 @JohnDoe534 @JohnDoe535 TX Results in Table 7.7 provide evidence that VNMT enforces more robust embeddings, which perform significantly better over the PFSMB UGC test set compared to the baseline, the system Frozen embs. giving the most consistent results over UGC. This system also keeps good performance over the newstest'14 canonical OOD test set, while taking advantage of an increased robustness to UGC. Such an improvement alleviates the loss of performance over newstest'14 in our previous results, which was the only test set for which multi-VNMT underperformed the non-VNMT baseline in Table 7.1. However, since the embeddings are less prone to over-fitting, the in-domain OpenSubTest's results are -0.1 BLEU points less than Transformer Base. Only the Fine-tuned model was able to maintain the same in-domain performance. These results indicate that VNMT promotes robustness to the NMT backbone and could be useful for achieving more robust pretrained embeddings.

Blind test sets scores

We now evaluate our best performing model (multi-VNMT trained on OpenSubtitles) on the blind test sets used previously during this dissertation, translating never-seen UGC tests, to assess whether our approach proves valuable for generalization over a larger spectrum of UGC types. We have also included the 4Square corpus (Berard et al. 2019a) to validate our VNMT system on other domain of UGC (restaurant reviews). We also display the results when using the NF-VNMT baseline and the Transformer Base model to assess improvement of our proposed architecture for such test sets. We report such results in Table 7.8, where we can see that multi-VNMT consistently outperforms the baselines for our blind UGC test sets, including the 4Square corpus, which it UGC domain differs from the PFSMB and MTNT social-media dense representations when processing noisy texts. Comparing the mixture coefficients in the figure, we can notice that the noisy UGC PMUMT and the out-of-domain newstest'14, diverge from the in-domain OpenSubTest and normalized UGC PMUMT corpus.

VNMT for other languages: Japanese UGC

Datasets and experiments In order to assess whether our results on UGC translation hold for different languages, we trained the same OpenSubtitles data configuration, i.e. 9.2M sentences with 16K BPE tokenization, using the Ja-En language pair. In addition, we segmented Japanese using Kytea [START_REF] Neubig | Pointwise prediction for robust, adaptable japanese morphological analysis[END_REF]) and the Transformer Base with unchanged training hyperparameter is used for both multi-VNMT and baseline. For evaluating these systems, we report results using the JESC [START_REF] Pryzant | JESC: japanese-english subtitle corpus[END_REF]) subtitles corpus and KFTT [START_REF] Neubig | Pointwise prediction for robust, adaptable japanese morphological analysis[END_REF]) as both in-domain and OOD canonical corpora, respectively, as well as,

MTNT's Ja-En UGC corpus.

Discussion of UGC Ja-En MT task

In order to assess whether our results on UGC translation hold for different languages, we trained the same OpenSubtitles data configuration using the Ja-En language pair (2.1 M sentences) with 16K BPE tokenization. In addition, we segmented Japanese using Kytea as pre-processing. We train for 120K iterations and use a target word drop rate of 0.4. For evaluating these systems, we report results using the JESC [START_REF] Pryzant | JESC: japanese-english subtitle corpus[END_REF]) subtitles corpus as in-domain test set, as well as the MTNT's Ja-En UGC corpus [START_REF] Michel | MTNT: A testbed for machine translation of noisy text[END_REF]. Results are reported in Table 7.9, which showed that multi-VNMT performs better than the non-latent baseline when translating the MTNT and the in-domain JESC test sets with a +0.4 and +0.3 BLEU improvement respectively. Despite the different experimental conditions, it can be interesting to have an overview of the level of performance to expect in different language conditions. Table 7.9 shows that our results are on par with previously reported scores, even if the settings are not comparable (we have trained a transformer-based model with fewer data compared to [START_REF] Michel | MTNT: A testbed for machine translation of noisy text[END_REF] and a different architecture than [START_REF] Pryzant | JESC: japanese-english subtitle corpus[END_REF], as well.)

Chapter 8

Conclusions and perspectives

The goal of this work is to improve the robustness of Neural Machine Translation (NMT) when processing noisy social media user-generated content (UGC). Such texts pose challenges due to their extensive forms of expression and multi-domain nature, as explained in Chapter 3. We have focused on the zero-shot translation scenario, in which the training data is restricted to publicly available canonical corpora, and developed methods that are agnostic to the target distribution of UGC, and can better generalize over a wide range of UGC sources.

Identification and formulation

We started by describing NMT and phrase-based statistical machine translation baselines in Chapter 4 and elaborated on the differences in performance on test sets of different nature. Particularly, we focused on the type of errors observed when using NMT to translate UGC and assessed how robust our baselines are. In this regard, neural-based translation methods provided overall best results for canonical test data, while being relatively brittle to UGC noise than PB-SMT.

We therefore focused our research (mainly on the French-English language pair) on NMT, which turned out to be better at translating in-domain and out-domain canonical corpora, but not robust to UGC compared to PB-SMT. An analysis of the attention mechanism provides a first explanation for NMT's inadequacy to translate UGC: NMT indeed predicts translations with a less similar length distribution than the references, caused either by omission of translations, over-translation due to failure of the attention mechanism, or hallucinations.

General conclusions

In this work, we have addressed the automatic translation of UGC and the challenges it poses, both in terms of target distribution drift and robustness to noise. Indeed, a very rich productivity lies behind the amazing nature of UGC: a rich hub for emergent communication derived from user interactions. This observation motivates our focus on the zero-shot scenario, i.e., we intended to study and improve the generalization capabilities of the models without any or minimal effort to try to match the target distribution, nor any attempt to tailor the data source to the task throughout this dissertation.

Regarding our efforts to improve UGC MT performance, we started our work by producing a lexical normalizer based on specific heuristics (i.e., aiming to correct phonetic spelling), in order to later broaden our scope and analyze a larger set of UGC specificities, thus directing our research towards improving and evaluating the robustness of NMT for UGC. To this end, we found our proposed UGC evaluation framework helpful, allowing us to both compare different NMT architectures and assess the impact of well-defined UGC phenomena on performance.

Regarding our research on NMT, popular character-level NMT architectures showed interesting robustness properties, being relatively more robust than their subword BPE counterparts, but lacking overall in-domain performance in our experiments.

In the final stage of this research, we obtained promising results when exploring Transformerbased VNMTs, which provide consistently better results than their non-latent backbones and improved robustness to UGC. As part of our experiments, we proposed a VNMT model with Mixture Density Networks, which results in better translation quality, especially when when processing UGC. In addition, by exploring these models, we show evidence of VNMT's robustness capabilities to UGC and, Perhaps more interestingly, we report experiments and visualizations suggesting that VNMT forces more robust learning representations on the backbone model, which can later be exploited without the VI blocks and retained after transfer (e.g., for fine-tuning).

In summary, our research has demonstrated the complex and extremely variable nature of UGC, requiring us to expect the improbable during evaluation, motivating our studies on generalization and robustness. As a complementary framework, we developed evaluation protocols and resources to characterize the impact of UGC on performance, making our best efforts to avoid reducing UGC specificities to simple noise. Finally, variational architectures