Nearest Neighbor AD Average Edit Distance APE Automatic Post-Editing AR Autoregressive ASR Automatic Speech Recognition AXE Aligned Cross-Entropy BERT Bidirectional Encoder Representations from Transformers Bi-sync Bilingual Synchronization BLEU Bilingual Evaluation

Our preliminary experiments showed that using top-k sampling instead of beam search for decoding increased the diversity of the generated ẽsub .

Titre: Ecrire en deux langues : la traduction automatique neuronale au service d'aide à la rédaction bilingue Mots clés: Rédaction bilingue, Aide à la rédaction, Traduction automatique neuronale, Traduction automatique non-autorégressive, Mémoire de traduction Résumé: Dans un monde de plus en plus globalisé, il est de plus en plus courant d'avoir à s'exprimer dans une langue étrangère ou dans plusieurs langues. Cependant, pour de nombreuses personnes, parler ou écrire dans une langue étrangère n'est pas une tâche facile. Les outils de traduction automatique peuvent aider à générer des textes en plusieurs langues. Grâce aux progrès récents de la traduction automatique neuronale (NMT), les technologies de traduction fournissent en effet des traductions utilisables dans un nombre croissant de contextes. Pour autant, il n'est pas encore réaliste d'attendre des systèmes NMT qu'ils produisent des traductions sans erreur. En revanche, les utilisateurs ayant une bonne maîtrise d'une langue étrangère donnée peuvent trouver des aides auprès des technologies de traduction aidée par ordinateur.

Lorsqu'ils rencontrent des difficultés, les utilisateurs écrivant dans une langue étrangère peuvent accéder à des ressources externes telles que des dictionnaires, des terminologies ou des concordanciers bilingues. Cependant, la consultation de ces ressources provoque une interruption du processus de rédaction et déclenche une autre activité cognitive. Pour rendre le processus plus fluide, il est possible d'étendre les systèmes d'aide à la rédaction afin de prendre en charge la composition de textes bilingues. Cependant, les études existantes se sont principalement concentrées sur la génération de textes dans une langue étrangère. Nous suggérons que l'affichage de textes correspondants dans la langue maternelle de l'utilisateur peut également aider les utilisateurs à vérifier les textes composés à partir d'entrées bilingues. Dans cette thèse, nous étudions des techniques pour constru-ire des systèmes d'aide à la rédaction bilingues qui permettent la composition libre dans les deux langues et affichent des textes monolingues synchronisés dans les deux langues. Nous présentons deux types de systèmes interactifs simulés.

La première solution permet aux utilisateurs de composer des textes dans un mélange de langues, qui sont ensuite traduits dans leurs équivalents monolingues. Nous étendons le modèle Transformer pour la traduction en ajoutant un décodeur duel: notre modèle comprend un encodeur partagé et deux décodeurs pour produire simultanément des textes en deux langues. Nous explorons également le modèle de décodeur duel pour plusieurs autres tâches, telles que la traduction multi-cible, la traduction bidirectionnelle, la génération de variantes de traduction et le sous-titrage multilingue.

La deuxième contribution vise à étendre les systèmes de traduction commerciaux disponibles en ligne en permettant aux utilisateurs d'alterner librement entre les deux langues, en changeant la boîte de saisie du texte à leur volonté. Dans ce scénario, le défi technique consiste à maintenir la synchronisation des deux textes d'entrée tout en tenant compte des entrées des utilisateurs, toujours dans le but de créer deux versions également bonnes du texte. Pour cela, nous introduisons une tâche générale de synchronisation bilingue et nous implémentons et expérimentons des systèmes de synchronisation auto-régressifs et non-autorégressifs. Nous étudions également l'utilisation de modèles de synchronisation bilingue pour d'autres tâches spécifiques, telles que le nettoyage de corpus parallèles et la NMT avec mémoire de traduction, afin de mieux évaluer la capacité de généralisation des modèles proposés.

Abstract: In an increasingly global world, more situations appear where people need to express themselves in a foreign language or multiple languages. However, for many people, writing in a foreign language is not an easy task. Machine translation tools can help generate texts in multiple languages. With the tangible progress in neural machine translation (NMT), translation technologies are delivering usable translations in a growing number of contexts. However, it is not yet realistic for NMT systems to produce error-free translations. Therefore, users with a good command of a given foreign language may find assistance from computer-aided translation technologies.

In case of difficulties, users writing in a foreign language can access external resources such as dictionaries, terminologies, or bilingual concordancers. However, consulting these resources causes an interruption in the writing process and starts another cognitive activity. To make the process smoother, it is possible to extend writing assistant systems to support bilingual text composition. However, existing studies mainly focused on generating texts in a foreign language. We suggest that showing corresponding texts in the user's mother tongue can also help users to verify the composed texts with synchronized bitexts. In this thesis, we study techniques to build bilingual writing assistant systems that allow free composition in both languages and display synchronized monolingual texts in the two languages. We introduce two types of simulated interactive systems.

The first solution allows users to compose mixed-language texts, which are then translated into their monolingual counterparts. We propose a dual decoder Transformer model comprising a shared encoder and two decoders to simultaneously produce texts in two languages. We also explore the dual decoder model for various other tasks, such as multi-target translation, bidirectional translation, generating translation variants, and multilingual subtitling.

The second design aims to extend commercial online translation systems by letting users freely alternate between the two languages, changing the texting input box at their will. In this scenario, the technical challenge is to keep the two input texts synchronized while taking the users' inputs into account, again with the goal of authoring two equally good versions of the text. For this, we introduce a general bilingual synchronization task and implement and experiment with autoregressive and non-autoregressive synchronization systems. We also investigate bilingual synchronization models on specific downstream tasks, such as parallel corpus cleaning and NMT with translation memories, to study the generalization ability of the proposed models.

Firstly and most importantly, I would like to express my deepest gratitude to my advisor François Yvon and my supervisor from SYSTRAN, Josep Crego. This thesis would not have been made possible without their endless support and help. It is one of the two most important things in my life until now to have François as my Ph.D. advisor. I have learned so much from him during the last three years. His guidance will have a big influence on my entire research career. He said he was lucky to have me, but I am more than lucky to have him. Words cannot express my appreciation for him. It is also my pleasure to meet Josep, not only as a supervisor but also as a very good friend. I will not forget our pleasant collaboration and the nice trip we had in Hong Kong. I would also like to thank my committee members, Pierre Zweigenbaum, Philippe Langlais, Qun Liu, Jan Niehues, and Rachel Bawden, for their insights during the defense and for their detailed reports that helped me to improve my thesis.

I am also grateful to my friends at LISN, Shu Okabe, Maxime Bouthors, Alban Petit, Paul Lerner, MinhQuang Pham, François Buet, Léa-Marie Lam-Yee-Mui, Rémi Uro, Yajing Feng, and Dávid Javorský. I appreciate our daily coffee break and interesting discussions. A special thank goes to Alina Karakanta, who dragged me out of depressed emotions when I had bad experimental results. I would also like to thank my intern Ruiyang Zhou, who helped me a lot during my defense.

Moreover, I would like to thank my colleagues from SYSTRAN for always being nice to me, especially Dakun Zhang, who was very supportive when I wrote my thesis. My thank also goes to Jean Senellart, ex-CEO of SYSTRAN, for helping to create this collaborative project between LISN and SYSTRAN.

Last but not least, I would like to thank my parents and my girlfriend, Yunbei Zhang, for their support and love during the last three years. I will always love you.

This work was granted access to the HPC resources of IDRIS under the allocation 2022-[AD011011580R2] made by GENCI. [START_REF] Li | Universal conditional masked language pre-training for neural machine translation[END_REF]. In the first step, the French word "Je" is aligned to "I" in English and to "我" in Chinese. Therefore, "Je" is replaced with "我", and "I" is masked. In the second step, tokens from both sentences are randomly masked. . where we progressively inject more En segments; for the dash-dot curve, En is the matrix language, with a growing number of Fr segments. (b) Direction MXL→Fr. The target BLEU is always much higher than the source BLEU, with about a 20point difference. The gap between the dash-dot and solid curves is due to the basic sentence structure of the matrix language (see Section 5.5.3). As dash-dot curves represent insertion in the reference target sentence, the corresponding BLEU score is always higher than the solid curve and actually reaches 100 (in the absence of any embedded language). . 7.1 A design of an online bilingual writing system. Users can freely choose the language to write, and the system automatically updates the texts in the other box. . . . . . 7.2 Methods for generating synthetic initial translations for each edit type. Rectangle purple boxes refer to separate models used to generate the desired operations. Differences in artificial initial translations (in blue boxes) are marked in bold. Initial translations ẽins for insertion are generated by randomly removing segments in the reference sentence e. For ẽsub , e is first back-translated into an intermediate sentence f * using top-5 sampling, then translated back to ẽsub with LCD. The first method to generate ẽdel randomly inserts [gap] tokens into e and decodes with a GAP insertion model (Xiao et al., 2022a). The ẽdel 1 is obtained by replacing [gap] with the predicted extra segments. The second method uses a model trained with WikiAtomicEdits data to translate e. . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Confusion matrices for predicted tags of Edit-MT for both directions. . . . . . . . . 7.4 Example attention matrix of Edit-MT when translating without forced oracle tag.

List of Figures

The model generates a copy tag and then copies the initial translation as output. . . 7.5 Example attention matrix of Edit-MT when translating with a forced substitution tag. The model properly replaces "que" with "auxquels" by paying more attention to "to" in the source and adds "accès" (access) to generate a good translation following the oracle tag.

List of Tables

5.1 Summarization of the model architecture and data of systems used in this chapter.

2 Transformers implies using two separate models, one for each direction. 1 Transformer refers to one model that can perform translation in two directions. 2 indep decoders indicates two independent decoders. . . . . . . . . . . . . . . . . . . . . . 5.2 BLEU scores when translating the monolingual newstest data and the synthetic mxl-newstest data for two language pairs En-Fr and En-Es. copy refers to a donothing baseline that simply copies the input. Small numbers contain BLEU scores computed separately when the target language is the embedded language (left) and the matrix language (right) in MXL source. For the monolingual tests (left part), these correspond to scores computed on the same sentences that are also included in the MXL test sets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Analysis of the copy constraint.

Exclusive refers to the percentage of test tokens appearing in only one hypothesis. Both and Punc are for tokens and punctuations+digits appearing in both hypotheses, and Lost is for tokens not found in either. . . . . . .

5.4

Analyzing the copy rate of tokens evaluated on mxl-newstest2014 for En-Fr and mxl-newstest2013 for En-Es. We report the number of (pre-translated) tokens that should be copied and the corresponding ratios. . . . . . . . . . . . . . . . . .

5.5

Percentage of sentences for which all target words have been precisely copied without and with order changes (Copy and Swap, respectively) and some target words in the MXL source have been changed by the model (Word-change), evaluated on mxl-newstest2014 (En-Fr) and mxl-newstest2013 (En-Es). We separately report numbers for cases where English is the matrix language (Matrix En) or embedded language (Matrix Fr/Es). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Results of

6.

3 BLEU scores for multi-target models. Pseudo (ps) refers to models trained from scratch with partial synthetic reference data. FT indicates models fine-tuned from the pre-trained multilingual (multi) models. FT+ps refers to models fine-tuned using partial synthetic reference data. . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Similarity scores (SIM) for multi-target models. The similarity scores are computed as the cross-lingual similarity between sentence embeddings of the two target translations computed with LASER. Pseudo (ps) refers to models trained from scratch with partial synthetic reference data. FT indicates models fine-tuned from the pretrained multilingual (multi) models. FT+ps refers to models fine-tuned using partial synthetic reference data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 BLEU scores for asynchronous decoding: sequential decoding on the dual models trained from scratch (top), wait-3 models fine-tuned on the pre-trained models, and sequential decoding on the dual FT models (bottom). Results using 2-round sequential decoding for one decoder are obtained in a second decoding pass using either automatic (2-round) or reference (ref) translations. . . . . . . . . . . . . . . 6.6 BLEU scores for bidirectional decoding models trained with actual data (top) and synthetic data (bottom). Pseudo (ps) refers to models trained with partial synthetic reference data. ps-dup systems are trained with twice as much data as ps settings. . 6.7 Consistency scores (Cons) for bidirectional decoding models trained with actual data (top) and synthetic data (bottom). The consistency score is an averaged BLEU score between the forward and backward translations BLEU(ê f w , êbw ) and BLEU(ê bw , êfw ). Pseudo (ps) refers to models trained with partial synthetic reference data. ps-dup systems are trained with twice as much data as ps settings. . . . 6.8 Results of politeness MT models. Tags are used for the pre-train model to generate the desired styles. Decoders of indep and dual compute two translations in one decoding step, while the results using sequential asynchronous decoding for one decoder are obtained with the 2-round procedure of Section 6.2.3. . . . . . . . . . . 6.9 BLEU scores for captions (En) and subtitles (Fr), with structural and lexical consistency scores between the two hypotheses. The consistency scores are in percentage.

The base and pipeline settings are trained from scratch with original data. +FT indicates models fine-tuned from the pre-trained translation model. share refers to tying all decoder parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.10 BLEU scores for multiple initialization settings when fine-tuning dual decoder models.

1-decoder refers to only initializing the subtitling decoder with pre-trained parameters. 6.11 Performance of various decoding methods. Models fine-tuned with the original data are denoted with w/real. All BLEU scores are averaged over the two outputs. for the two generation methods (Del 1 and Del 2 ). + tag refers to decoding with the oracle tag as a forced prefix. The best performance is in bold. . . . . . . . . . . . 7.4 BLEU scores of Edit-MT on Bi-sync for Fr-En. . . . . . . . . . . . . . . . . . . . . 7.5 BLEU scores of multilingual Edit-MT on Bi-sync for both En-Fr and Fr-En. The multilingual model denoted with multi is the same model used for both directions. . 7.6 Data used for experiments in Section 7.5. FM ratio is the ratio of sentences with at least one matched similar translation. FM train is the actual number of examples with a similar translation used for training after setting aside 1,000 test sentences. Each train sentence is matched with up to 3 similar translations. GNO and Ubu refer to GNOME and Ubuntu, respectively. . . . . . . . . . . . . . . . . . . . . . . . . . 7.7 BLEU scores by performing standard translation without using TMs on test sets from multiple domains. All is computed by concatenating test sets from all domains, with 11k sentences in total. Copy refers to copying the retrieved similar translations to the output. + FT refers to the model fine-tuned on the multi-domain TM data. . 7.8 BLEU scores on test sets from multiple domains when translating with TMs. + R implies using the related segments instead of a full initial sentence for inference. + FT refers to the model fine-tuned on the multi-domain TM data. The best performance in each block is in bold. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.9 TER scores on test sets from multiple domains when translating with TMs. . . . . 7.10 BLEU scores on unseen domains when translating with TMs. . . . . . . . . . . . . D.1 BLEU scores for each domain, the task is standard MT with FM ≥ 0.6. All is computed by concatenating all test sets (11k sentences in total). Copy refers to copying the TM match into the output. . . . . . . . . . . . . . . . . . . . . . . . . D.2 COMET scores for each domain, the task is standard MT with FM ≥ 0.6. . . . . D.3 BLEU scores for each domain, the task is standard MT with FM ∈ [0.4, 0.6). . . D.4 COMET scores for each domain, the task is standard MT with FM ∈ [0.4, 0.6). . D.5 BLEU scores for each domain, the task is MT with TMs with FM ∈ [0.4, 0.6). . D.6 COMET scores for each domain, the task is MT with TMs with FM ∈ [0.4, 0.6).

-Introduction

. Context and Motivation

In an increasingly global world, more and more situations appear where people need to write, speak, or, more generally, express themselves in a foreign language or multiple languages. For instance, researchers from different countries often write research articles or theses in English, research institutions often write emails in both the original language and English, countries or supranational bodies with multiple official languages compose official documents in all official languages, travelers fill out administrative documents in the language of the host country, audio-visual contents come with subtitles in multiple languages to widen the potential audience, etc. However, for many people, writing in a foreign language (L2) is not an easy task.

Machine translation (MT) tools, which translate texts written in one language into other languages, can help generate texts in multiple languages. With the significant advances seen in MT, especially the tangible progress in neural machine translation (NMT) in recent years, MT systems are delivering usable translations in a growing number of contexts. Nevertheless, relying on this technology to produce high-quality documents is not yet realistic since the current state-of-the-art NMT systems have not reached the level where systems can produce error-free translations, and also because fully automatic translation does not provide the necessary control over the produced translation. Therefore, users with a good command of L2 but not at a professional level may find assistance from various Computer Assisted Language Learning tools or Computer Aided Translation (CAT) systems. Users can access external resources such as dictionaries, terminologies, or bilingual concordancers [START_REF] Bourdaillet | TransSearch: from a bilingual concordancer to a translation finder[END_REF] to help with writing. To this end, several studies have focused on developing systems to help users compose text in L2 [START_REF] Koehn | Enabling monolingual translators: Post-editing vs. options[END_REF][START_REF] Huang | Transahead: A computer-assisted translation and writing tool[END_REF][START_REF] Curran Associates | An SMT-driven authoring tool[END_REF][START_REF] Chen | FLOW: A first-language-oriented writing assistant system[END_REF]Liu et al., 2016b).

Nevertheless, consulting external resources causes an interruption in the writing process and starts another cognitive activity, even when writing in the users' native language (L1) [START_REF] Leijten | Writing in the workplace: Constructing documents using multiple digital sources[END_REF]. Besides, L2 users tend to rely on L1 [START_REF] Wolfersberger | L1 to L2 writing process and strategy transfer: A look at lower proficiency writers[END_REF] to prevent a breakdown in the writing process [START_REF] Cumming | Writing expertise and second-language proficiency*[END_REF]. Therefore, it has been proposed to integrate bilingual text composition into writing tools, where input in L2 can guide the writing process. For instance, the system presented by [START_REF] Chen | FLOW: A first-language-oriented writing assistant system[END_REF] can be illustrated in the following example. A French writer composing the English sentence [I return home because I am tired.] may be unsure about how to formulate the notion of "return home." An input in French, such as [I rentre à la maison because I am tired.], can be used by the system to propose alternative formulations in English. This is certainly a more acceptable solution than using fully automatic translation. First, the user actively participates in the writing process, using L2 as much as possible, without needing heuristic searches for external sources that interrupt the writing process. Second, the existing text fragments in L2 that the user is able to produce with high confidence can be used as valuable contextual clues to improve the system's suggestions. The shared task on L2 writing assistant that translates L1 fragments in an L2 context, proposed by [START_REF] Maarten Van Gompel | SemEval 2014 task 5 -L2 writing assistant[END_REF], attempts to simulate this scenario and evaluate its difficulty.

However, existing studies mainly focused on generating texts in L2, leaving the users to decide whether the provided texts precisely conveyed what they wanted to express, while it is not easy to evaluate L2 texts for users who are not at a professional level. On the other hand, researchers have explored using round-trip translation that translates the MT output written in L2 back to L1 to evaluate the quality of translation [START_REF] Moon | Revisiting round-trip translation for quality estimation[END_REF]. Therefore, we suggest that it is also helpful to show the corresponding texts in L1 to help users verify the composed texts with synchronized bitext. Taking the previous example, when a user composes [I rentre à la maison because I am tired.], he/she can verify the French sentence [Je rentre à la maison parce que je suis fatigué.] displayed together with the English translation [I return home because I am tired.] to make sure that the English sentence is precise. Furthermore, users can obtain synchronized texts in two languages while only making an effort to compose approximately one sentence. Such systems should allow users to write freely in both languages and always provide synchronized monolingual texts in the two languages. However, existing systems do not support both functionalities at the same time. Systems like [START_REF] Chen | FLOW: A first-language-oriented writing assistant system[END_REF] allow free writing in both languages but only show the final texts in L2, as illustrated in Figure 1.1a (top). Commercial translation systems like Google, DeepL, SYSTRAN, etc., always display texts in both languages. However, users can only edit the source side while the target side is computed by the systems and cannot be changed freely, as shown in Figure 1.1a (bottom). CAT systems assume the given source sentence to be fixed and only allow edits on the target side.

In this thesis, we study techniques to build bilingual writing assistant systems that allow free composition in both languages and simultaneously display synchronized monolingual texts in the two languages. We propose two types of simulated interactive systems as possible solutions. The first design contains three text boxes. Users can compose texts with a free mix of both languages in only one box. The other two boxes display the corresponding texts in each language generated by the system. An illustration

Type to translate French

Je rentre à la maison parce que je suis fatigué.

English

I return home because I am tired.

Type to translate

I rentre à la maison because I am tired.

English

I return home because I am tired.

French

Je rentre à la maison parce que je suis fatigué.

(b) A three-box design allowing free bilingual composition in the left box and displaying monolingual texts on the right.

Type to translate English

I return home because I am tired.

Type to translate French

Je rentre à la maison parce que je suis fatigué.

(c) A two-box design allowing bilingual composition by changing input boxes. Users can freely choose the language to write, and the system automatically updates the texts in the other box. of this design is shown in Figure 1.1b. Such interactions require the system to simultaneously translate a mixed-language sentence into both languages.

The second idea extends commercial translation systems with only two text boxes, as illustrated in Figure 1.1c. Each box shows texts in one language, and users can freely choose to write in either box at their will. Once texts in one box are changed, the system automatically updates texts in the other language so that texts in the two boxes always remain synchronized. This system requires a bilingual synchronization ability that does not specify a particular source or target language but always keeps the two texts as translations of each other while taking the users' input into account.

This thesis targets the study of new methods for generating high-quality synchronized texts in two languages in the two simulated situations described above using NMT techniques. The developed methods can be further integrated into well-designed interactive systems to build real bilingual writing assistant tools. We have not explored the interactive systems in real-world scenarios in this thesis but aim to explore this more in the future.

. Contributions

Our contributions of this thesis are as follows:

• We study bilingual writing assistance in two different scenarios. The first situation reads sentences written in a mixed language and generates monolingual sentences in both languages, while the second solution aims to synchronize texts in the two languages when either language is changed. • We propose a dual decoder NMT model which is able to simultaneously disentangle a mixed-language sentence into both component languages, and we conduct thorough analyses of the task of translating mixed-language sentences. • We apply the dual decoder model to various other tasks, like multitarget translation, bidirectional decoding, multi-style decoding, and multilingual subtitling, to evaluate the challenges and rewards of dual decoding. • We propose a general bilingual synchronization task that aims to synchronize text in one language to another so that both texts are always parallel to each other. We also propose novel non-autoregressive architectures to perform this task. • We study ways to generate various types of synthetic data that can simulate user-generated texts during bilingual writing and show that the generated artificial data is effective in training the corresponding models. • We also apply bilingual synchronization models to more specific tasks, like parallel corpus cleaning and translation with translation memories, to study the adaptation capacity of our proposed models.

. Outline

This thesis is organized as follows:

• Chapter 2 gives an introduction to conventional NMT systems. We discuss text tokenization methods, classical NMT model architectures used in this thesis, training and inference procedures, and evaluation metrics.

• Chapter 3 reviews several NMT tasks incorporating augmented sources into the translation process. Conventional NMT produces the translation solely based on the source sentence, while these tasks all use some information other than the source when producing the translation. This is also the case in our bilingual writing scenario, where segments composed in a bilingual way contain information from both languages. We focus on interactive MT, automatic post-editing, NMT using translation memories, incorporating lexical constraints in NMT, and multi-source/multi-target NMT. • In Chapter 4, we introduce non-autoregressive NMT, an emerging subdomain of NMT which aims to generate all target words in one single step instead of one by one from left to right. The key advantage of non-autoregressive NMT is the speedup in translation time compared to conventional autoregressive methods, which can be a good fit for real-world interactive scenarios, where the translation latency needs to be very low so that the models can quickly update translations following users' input. • In Chapter 5, we study the task of simultaneously translating mixedlanguage sentences into both component languages. We propose a novel dual decoder Transformer model and a mixed-language data generation method for this task. • Chapter 6 applies our proposed dual decoder model to various tasks, including multi-target translation, bidirectional decoding, multi-style decoding, and multilingual subtitling. • Chapter 7 describes a new framework targeting the bilingual synchronization task, which generalizes MT. We propose autoregressive approaches to perform this task. We study various methods to generate synthetic training data in order to simulate this task. We also adapt models performing generic bilingual synchronization to two specific downstream tasks: NMT with translation memories and parallel corpus cleaning. • Chapter 8 focuses on studying bilingual synchronization with nonautoregressive methods. We also apply the non-autoregressive generic bilingual synchronization model to the NMT with translation memories task. We conduct an in-depth study of non-autoregressive NMT with translation memories, which has not been explored in the literature. • Finally, Chapter 9 concludes this thesis with a summarization of our contributions and an outlook on future research directions.

. List of Publications

This thesis draws contributions from some of the articles published as follows:

• Jitao Xu

-Neural Machine Translation

Neural machine translation (NMT) is currently the most widely used technique in the area of machine translation (MT). Thanks to the introduction of the first encoder-decoder architecture [START_REF] Allen | Several studies on natural language and backpropagation[END_REF][START_REF] Pollack | Recursive distributed representations[END_REF][START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF]Cho et al., 2014a), then completed with an attention mechanism [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF][START_REF] Vaswani | Attention is all you need[END_REF], the performance of NMT systems is now good enough for a growing number of services, both for the general public and the translation industry. Given a source sentence f and the corresponding target sentence e, NMT systems model the conditional probability P (e|f ) of e given f , with f = f 1 , f 2 , . . . , f S and e = e 1 , e 2 , . . . , e T two sequences of discrete units. This chapter provides a brief background introduction to the conventional NMT systems used in this thesis. A more detailed review of NMT can be found in [START_REF] Stahlberg | Neural machine translation: A review[END_REF].

. Vocabulary and Byte Pair Encoding

. The Open Vocabulary Problem

When processing texts with NMT models, raw texts need to be firstly tokenized into unique tokens, which then construct the vocabularies of the source and target languages. Tokens may contain words, subwords, characters, numbers, and punctuations. NMT models generally operate with a fixed vocabulary, while MT is an open-vocabulary problem. Due to computational constraints, it is impossible to include every word in a language in the vocabulary, especially for morphologically rich languages. Early approaches (Cho et al., 2014a;[START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF] only consider the N most frequent words of each language in the vocabulary and treat all rare and unknown words as a special token <unk>. As a result, unknown words become a major challenge for word-based NMT models since the translation quality degrades vastly as the number of unknown words increases (Cho et al., 2014a).

In order to deal with the open vocabulary problem, some studies propose to perform character-based NMT [START_REF] Chung | A characterlevel decoder without explicit segmentation for neural machine translation[END_REF][START_REF] Luong | Achieving open vocabulary neural machine translation with hybrid word-character models[END_REF][START_REF] Costa-Jussà | Character-based neural machine translation[END_REF][START_REF] Lee | Fully characterlevel neural machine translation without explicit segmentation[END_REF][START_REF] Cherry | Revisiting character-based neural machine translation with capacity and compression[END_REF], which considers characters instead of words as unique tokens. In this way, all words can be decomposed into sequences of characters, and unknown words only happen when an unknown character appears in the test sentence, which is relatively rare. Therefore, character-based models can handle rare words from particular domains better as they will not be considered as un-known words. These approaches successfully reduce the vocabulary size of each language to the number of unique characters that occur in the language. However, one downside is that character-based approaches have to deal with longer sequences and largely increase the translation time, as each word is separated into a character sequence. Recent work [START_REF] Libovický | Why don't people use character-level machine translation?[END_REF] finds that character-based models do not perform better on domain robustness or morphological generalization, despite often having this as a motivation. They only show robustness to source input noise. Besides, the segmentation choice has different effects according to different languages. Ideographic languages like Chinese and Japanese have different segmentation effects compared to alphabetic languages like English and French [START_REF] Kreutzer | Learning to segment inputs for NMT favors character-level processing[END_REF][START_REF] Zhang | Neural machine translation of logographic language using sub-character level information[END_REF]. Languages with different morphological categories also perform differently across segmentation methods [START_REF] Zhang | Neural machine translation of logographic language using sub-character level information[END_REF].

. Byte Pair Encoding

Both word-based and character-based NMT models have significant flaws, which has led researchers to propose an intermediary solution based on subwords. In such systems, words are segmented into subword sequences based on their occurrence frequency in the training corpus. Frequent words are often left unchanged, while rare words are generally split into a sequence of subword units. Sennrich et al. (2016c) introduce subword-level tokenization for NMT using the Byte Pair Encoding (BPE, [START_REF] Gage | A new algorithm for data compression[END_REF] algorithm. Each word in a training corpus is first split into a sequence of characters plus a special end-of-word symbol to mark word boundaries. All existing unique characters are considered as the initial subword unit set, also called the vocabulary of subwords. At each iteration, the most frequent pair of units that appears in the training corpus is merged into a new unit, which is then added to the vocabulary. The merge operation is recorded and will be used to apply BPE tokenization. The training corpus is also updated according to the merge operation. This merging step is performed iteratively until a predefined maximum number of merge operations is achieved. The frequency count of unit pairs does not include pairs that cross word boundaries, assuring that the merge operations only happen within original words. The final BPE model thus consists of an ordered set of unique merge operations.

BPE tokenization first splits all words into sequences of individual characters, then iteratively applies the merge operations according to their appearing order in the trained BPE model. Rare words that are not merged into their original form remain as a sequence of subword units or a sequence of characters in some extreme cases. For instance, in the WMT14 English-French corpus that is used in Chapters 5, 6, and 7, the word attribution is a common word in a training corpus, while Attribution and attribute are relatively rarer. As a result, attribution remains unsegmented after applying BPE tokenization, while Attribution and attribute are split into [Attribu@@ tion] and [attribu@@ te], respectively, with "@@" as a special joiner to mark a split in a word.

BPE tokenization thus handles the open-vocabulary problem better by using a vocabulary built with common words, subwords, and characters. NMT systems using BPE can operate with a vocabulary size of about 30-50k and almost covers all words, even though unknown words may still appear in some extreme cases where new characters are introduced.

BPE operations can be learned separately or jointly for the source and target language. Sennrich et al. (2016c) demonstrate that languages written in the same alphabet can benefit from a joint BPE vocabulary to increase the consistency between source and target language. Joint BPE also gives the possibility to use a shared source-target vocabulary which is necessary for parameter sharing of the word embedding matrices in NMT models [START_REF] Press | Using the output embedding to improve language models[END_REF][START_REF] Inan | Tying word vectors and word classifiers: A loss framework for language modeling[END_REF], which we discuss in Section 2.2.1.

. Other Methods

As the merge operations of BPE are fixed once constructed, the resulting segmentations for given words are unique, despite multiple segmentations being possible. [START_REF] Provilkov | BPE-dropout: Simple and effective subword regularization[END_REF] propose BPE-dropout, which applies stochastic corruptions in the segmentation procedure of BPE by randomly dropping merges at each merging step. Therefore, the resulting segmentations thus vary for given words, helping NMT models to learn the compositionality of words better and to be robust to segmentation errors. [START_REF] Kudo | Subword regularization: Improving neural network translation models with multiple subword candidates[END_REF] introduce sentence-piece tokenization, which models the segmentation of a complete sentence rather than separate words with a unigram language model. It can yield different segmentations for the same sentence by performing random subword sampling based on the unigram language model.

. Neural Machine Translation Models

State-of-the-art NMT systems generate translations in an autoregressive way, in which the probability of generating a target token is conditioned on the previously generated target tokens and the source sentence. Therefore, the probability P (e|f ) is decomposed as a chain rule:

P (e|f ) = T i=1 P (e i |e <i , f ; θ), ( . )
where T is the length of the target sequence, and θ represents the model parameters.

Most NMT systems follow an end-to-end encoder-decoder architecture. The encoder encodes the source sequence into high dimension vector representations. The decoder takes the source representations and the representations of previously generated target tokens computed by itself as input to generate new tokens. At each time step i, the decoder computes a probability distribution over the output target vocabulary by mapping the hidden state s i for time step i to a vector space R

|V | where |V | is the size of the target vocabulary:

P (⋅|e <i , f ) = softmax(s T i W p ), ( . )
where W p is a linear transformation that maps s i to R |V | . The softmax function normalizes the output vector to a probability distribution, in which the probability of each element n is:

softmax(z) n = e z n ∑ J j=1 e z j for n = 1, . . . , J, ( .
)
where J is the dimension of the vector. The generation often selects the token with the maximum probability as the predicted token at time step i.

With the advances in the field of NMT in recent years, NMT model architectures have evolved from approaches based on Recurrent Neural Network (RNN, Cho et al., 2014a;[START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF] to approaches based on Convolutional Neural Network (CNN, [START_REF] Gehring | Convolutional sequence to sequence learning[END_REF] and achieved the current state-of-the-art Transformer-based approaches [START_REF] Vaswani | Attention is all you need[END_REF]. This section mainly describes the Transformer architecture, which is used for all models developed in this thesis. RNN and CNN-based approaches are also briefly introduced.

. Transformer Model

The Transformer model is a sequence-to-sequence model based on the self-attention mechanism proposed by [START_REF] Vaswani | Attention is all you need[END_REF]. It is first introduced for the NMT task and further explored as the state-of-the-art backbone architecture for large pre-trained language models like BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF], XLM [START_REF] Conneau | Cross-lingual language model pretraining[END_REF], GPT-3 (Brown et al., 2020), etc. The Transformer model also follows the encoder-decoder architecture as discussed below.

Transformer Encoder

The Transformer encoder contains several identical encoder layers. Each layer consists of a multi-head self-attention sub-layer followed by a positionwise fully connected feed-forward sub-layer. The attention function maps a query and a set of key-value pairs to an output, where the queries, keys, values, and outputs are all vector representations. By packing a set of queries, keys, and values into matrices Q, K, and V, respectively, the attention mechanism is computed as:

Attention(Q, K, V) = softmax( QK T d k )V, ( . )
where d k is the dimension of query and key vectors. In practice, the dimension of Q, K, and V are always set identical to d k . In a self-attention sub-layer, Q, K, and V are identical, and all come from the output of the previous layer. The multi-head attention mechanism linearly projects Q, K, and V into different subspaces, performs attention separately in parallel, then concatenates the outputs from all subspaces back to dimension d k . More precisely, it is computed as:

MultiHead(Q, K, V) = Concat(head 1 , . . . , head h )W O head i = Attention(QW Q i , KW K i , VW V i ), ( . )
where

W Q i , W K i , W V i ∈ R d h ×d k , W O ∈ R hd h ×d k , h × d h = d k ,
with h as the number of heads, and d h the dimension of each projected vector. The feed-forward sub-layer consists of a single hidden layer network with a ReLU activation, applied to each position separately and identically:

FFN(x) = ReLU(xW 1 + b 1 )W 2 + b 2 , ( . )
where [START_REF] He | Deep residual learning for image recognition[END_REF] is applied to each sub-layer, followed by layer normalization [START_REF] Lei Ba | Layer normalization[END_REF]. These two operations together are referred to as AddNorm:

W 1 ∈ R d k ×d hidden , W 2 ∈ R d hidden ×d k , b 1 ∈ R d hidden and b 2 ∈ R d k . A residual connection
AddNorm(x) = LayerNorm(x + SubLayer(x)), ( . )
with SubLayer(x) applying a sub-layer function (MultiHead or FFN) to x.

Transformer Decoder

The Transformer decoder also contains several identical decoder layers. Different from the encoder layer, each decoder layer consists of three sublayers: a decoder self-attention sub-layer, an encoder-decoder cross-attention sub-layer, and a feed-forward sub-layer. In order to preserve the autoregressive property, the decoder is prevented from seeing information from future positions by applying a causal mask to the decoder self-attention sub-layer. For each position i, attention scores for all positions greater than i, in the input of the softmax function in Equation ( . ) in the decoder self-attention layer, are masked out (setting to -∞). The encoder-decoder cross-attention layer takes the output of the last encoder layer as keys and values, performing attention as MultiHead(Q, H, H) with H as the encoder output and Q as the output of the previous decoder self-attention sub-layer. The feedforward sub-layer performs similarly to the encoder layers, and each sub-layer is also surrounded by an AddNorm operation. The final decoder layer output states are passed to a linear output transformation to compute the output probability distributions.

Embeddings

In NMT models, a token sequence is represented as a sequence of integers. Each integer refers to the index of the corresponding token in the vocabulary. Both encoder and decoder require an embedding matrix, which transforms a token integer into a high dimension real-valued vector. This vector, also called word embedding, possesses semantic properties (Mikolov et al., 2013a,b). The embedding matrix E contains a vector for each token in the vocabulary and is of size |V | × d with |V | as the corresponding vocabulary size and d as the dimension of the embedding vector. In sequence transduction models, the embedding matrices are often randomly initialized and learned together with other parameters so that the learned embeddings fits the specific task better. When the vocabularies of the source and target language are identical, which is often the case when performing a joint BPE tokenization, the parameters of the embedding matrices E l S and E l T , together with the linear output matrix W p can be shared [START_REF] Press | Using the output embedding to improve language models[END_REF][START_REF] Inan | Tying word vectors and word classifiers: A loss framework for language modeling[END_REF] without losing translation quality.

Positional Encoding

As the computations of attention and feed-forward layers are independent and identical across all positions, the Transformer model does not include the order information of sequences and considers the distance between every two positions as identical. In order to inject positional information of each token into the sequences, [START_REF] Vaswani | Attention is all you need[END_REF] propose to use an absolute positional encoding, which is added element-wise to the word embedding vector of each token and is defined as:

PE(pos, 2i) = sin( pos 10000 2i/d k ) PE(pos, 2i + 1) = cos( pos 10000 2i/d k ), ( .
)
where pos is the position in the input sequence, and i is the dimension in a vector of dimension d k . The resulting embedding vectors are taken as input by the first encoder or decoder layer.

. RNN-based Models

Early architectures before the Transformer model mainly follow the RNN encoder-decoder architecture with attention proposed by [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF]. The encoder contains a bidirectional RNN, consisting of a forward and a backward RNN, to encode information not only from preceding tokens but also from future tokens. The forward RNN reads the input sequence from left to right and computes a sequence of forward hidden states ( -→ h 1 , . . . , -→ h I ), while the backward RNN reads the input from right to left, generating a sequence of backward hidden states (

← - h 1 , . . . , ← - h I ).
The representation of a token at position j of the bidirectional RNN is the concatenation of forward and backward hidden states at this position:

h j = [ - → h j ; ← - h j ],
where [⋅; ⋅] denotes the concatenation operation. The RNN module can also be replaced by Long Short Term Memory units [START_REF] Hochreiter | Long Short-Term Memory[END_REF] as in [START_REF] Wu | Google's neural machine translation system: Bridging the gap between human and machine translation[END_REF], or by Gated Recurrent Unit (Cho et al., 2014b).

The RNN decoder generates target tokens from left to right, following the autoregressive property. Therefore, the RNN decoder does not use a bidirectional module. At each time step i, the hidden state of the decoder is computed as:

s i = g(e i-1 , s i-1 , c i ), ( . )
where c i is a context vector for time step i. It is computed as a weighted sum of the encoder hidden states h j :

c i = S j=1 α ij h j , ( . )
where α ij is the attention weight describing how well the input tokens around position j match the output at position i. The attention weight α ij of h j is computed by:

ij = f attn (s i-1 , h j ) α ij = exp( ij ) ∑ S k=1 exp( ik ) , ( .
)
where f attn is an alignment model which computes the soft alignment between the source and target token representations. The decoder output state s i is used to compute the output probability distribution. The alignment model is defined as a feed-forward network as in [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF] and jointly trained with the other components of the RNN model. [START_REF] Luong | Effective approaches to attention-based neural machine translation[END_REF] find that simply using dot-product as f attn also yielded good performance.

. CNN-based Models

The CNN-based model [START_REF] Gehring | Convolutional sequence to sequence learning[END_REF] uses a stacked convolution layer with gated linear units (GLU, Dauphin et al., 2017) to compute the hidden states of the source and target sentences. For each convolutional layer, the output hidden state h l i at position i of layer l contains information about k input elements around i. The convolution kernel is computed as:

Y = W l [h l-1 i-k/2 ; . . . ; h l-1 i+k/2 ] + b l , ( .
)

which maps k concatenated input states [h l-1 i-k/2 ; . . . ; h l-1 i+k/2 ] ∈ R kd into an intermediate vector Y ∈ R 2d with parameter W l ∈ R 2d×kd and b l ∈ R 2d . The intermediate vector is separated into two vectors A, B ∈ R d where Y = [A; B],
and then fed into the GLU for non-linearity:

v([A; B]) = A ⊗ σ(B), ( .
)
where ⊗ is the element-wise multiplication and σ is the sigmoid function.

Residual connections also apply to each layer so that the actual output state at position i is:

h l i = v(W l [h l-1 i-k/2 ; . . . ; h l-1 i+k/2 ] + b l ) + h l-1 i .
( . )

For the decoder, the convolution kernel is shifted by k/2 to the left to prevent the decoder from seeing future information.

g l i = v(W l [g l-1 i-k+1 ; . . . ; g l-1 i ] + b l ) + g l-1 i .
( . )

Note that the parameters W l and b l are different from the encoder parameters. [START_REF] Gehring | Convolutional sequence to sequence learning[END_REF] also use an attention mechanism in every decoder layer similar to the Transformer. They combine the current decoder state g l i with the embedding vector g 0 i :

d l i = W l d g l i + b l d + g 0 i , ( . )
and apply dot-product to compute the attention weight between d l i and the encoder output state h u j as:

a l ij = exp(d l i ⋅ h u j ) ∑ S t=1 exp(d l i ⋅ h u t )
.

( . )

The context vector c l i is computed as the weighted sum of both the encoder states and the embeddings for all positions:

c l i = S j=1 a l ij (h u j + h 0 j ), ( . )
which is directly added to the decoder state g l i to make the actual output state of the attention mechanism s l i = g l i + c l i . The final probability distribution is computed similarly to the Transformer and RNN models.

The CNN model is similar to Transformer as it cannot capture the distance information very well and requires a positional embedding to be added to the word embedding vector.

. Training Neural Machine Translation Models

NMT models can be trained with Maximum Likelihood Estimation (MLE). Given a training set D = {(f , e) i , i = 1, . . . , N }, the set of all parameters θ of an NMT model is estimated by solving the following optimization problem: . ) which is equivalent to minimizing the cross-entropy (CE) loss:

θ * = arg max θ N i=1 log P (e (i) |f (i) ; θ), (
θ * = arg min θ - N n=1 T i=1 log P (e (n) i |e (n) <i , f (n) ; θ).
( . )

For each training example, the CE loss in formally defined as: ) where the ground truth label distribution q(k|f ) = δ k,e is a Dirac delta function, with q(k = e|f ) = 1 and q(k|f ) = 0 for all k ≠ e. However, training directly on the standard CE loss may result in over-fitting, as the model becomes too confident and learns to give full probability to the ground truth label. To alleviate this problem, [START_REF] Curran Associates | Rethinking the inception architecture for computer vision[END_REF] propose the label smoothing strategy, which uniformly assigns small probabilities to tokens other than the ground truth label to make the model less confident. With a smoothing parameter , the ground truth distribution is modified as:

L CE = - T i=1 k∈V log(p(k|e <i , f )) ⋅ q(k|f ), ( .
q ′ (k|f ) = (1 -) ⋅ δ k,e + |V | .
( . )

The label-smoothed CE loss is thus defined as:

L LSCE = - T i=1 (1 -) ⋅ log P (e i |e <i , f ) + |V | ⋅ k∈V log P (k|e <i , f ) . ( . )
In practice, label smoothing is often applied with = 0.1. The parameters can be optimized using the gradient descent method with backpropagation [START_REF] Rumelhart | Learning representations by back-propagating errors[END_REF]. As NMT models are often very large and require a great amount of parallel data to train, gradient descent is thus computationally infeasible. The common practice uses stochastic gradient descent with mini-batches. The most widely used gradient-based optimization algorithm is Adam [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF], which is based on adaptive estimates of lower-order moments of the gradients.

Standard NMT models only perform bilingual translation from one source language into one target language, with the training dataset D containing only one pair of languages. Thanks to the sequence-to-sequence architecture of NMT models, the dataset D can be easily extended to have multiple language pairs without significant modifications to the model architecture, training one single NMT model on several language pairs to perform multilingual translation. The encoder side of a multilingual NMT model often remains similar to a bilingual model. However, the multilingual decoder requires an extra signal to specify the desired target language. The most commonly used solution is to prepend an extra token tag indicating the output language either to the source or target sentence during training [START_REF] Johnson | Google's multilingual neural machine translation system: Enabling zero-shot translation[END_REF]. This tagging mechanism is very simple yet effective. It can provide additional information to the model from various aspects, such as distinguishing real and synthetic parallel training data [START_REF] Caswell | Tagged backtranslation[END_REF][START_REF] Marie | Tagged backtranslation revisited: Why does it really work[END_REF], controlling translation style (Sennrich et al., 2016a), providing specific domain indications [START_REF] Kobus | Domain control for neural machine translation[END_REF], etc.

. Teacher Forcing and Scheduled Sampling

During training, the ground truth prefix e <i is used for every time step i to compute the output probability distribution. NMT models often do not make predictions at the training stage. The distribution vector is only used to compute the CE loss. This procedure is also called teacher forcing. For the Transformer decoder, the training through time steps can be parallelized by preparing the prefix e <i for each time step and batch all steps into one matrix. This is implemented by adding a begin-of-sentence token to the beginning of each sentence, thus shifting the reference target sequence by 1 step to the right so that each position can only attend to preceding tokens thanks to the decoder self-attention causal mask described in Section 2.2.1. However, NMT models generate tokens from scratch during inference and can only condition on their own previous predictions, which can be possibly incorrect. NMT models thus need to generate words based on noisy prefixes containing previous decoding errors (also known as cascading errors). Therefore, errors may accumulate as the decoding continues. This issue is known as the exposure bias problem, meaning that the inference stage is exposed to wrong predictions which have never been encountered during training.

In order to mitigate the exposure bias problem, [START_REF] Daumé | Search-based structured prediction[END_REF] first advocate the idea of incorporating the model predictions at training time for structured prediction problems. [START_REF] Bengio | Scheduled sampling for sequence prediction with recurrent neural networks[END_REF] introduce the scheduled sampling strategy, which is a curriculum learning strategy. During training, for each time step, the sampling strategy decides with probability ε to use either the ground truth or previously predicted tokens. The latter can be a token randomly sampled according to the probability distribution P (⋅|e <i , f ) or taken as the arg max P (⋅|e <i , f ). It can also be a weighted average of all word embeddings, with P (⋅|e <i , f ) as weights [START_REF] Goyal | Differentiable scheduled sampling for credit assignment[END_REF]. To increase robustness, one can also add noise to the probability distribution [START_REF] Zhang | Bridging the gap between training and inference for neural machine translation[END_REF]. When ε = 1, the model always uses ground truth tokens as in conventional training, while when ε = 0, the model is trained with only predictions as in inference. [START_REF] Bengio | Scheduled sampling for sequence prediction with recurrent neural networks[END_REF] propose using a decay schedule of ε to go from the former to the latter during the training procedure. The model uses ground-truth tokens more frequently at the beginning of training as it is not well-trained to make reasonable predictions and samples more from its own prediction at the end of training.

Scheduled sampling suits the RNN-based model well as the RNN decoder always proceeds step by step. However, this is not straightforward for the Transformer model as the training procedure is highly parallelized. Forcing the Transformer model to proceed step by step as RNN models do yields significant computation overheads. [START_REF] Mihaylova | Scheduled sampling for transformers[END_REF] successfully apply scheduled sampling to the Transformer models by performing a two-pass decoding strategy. The first decoding pass is like the conventional training of the Transformer, generating the output probability distributions for all positions. The second pass takes a mixture of the reference target sequence and the model predictions from the first pass as input to the decoder, in which each position decides to use the reference token with a probability ε. Liu et al. (2021a,b) further improve scheduled sampling for Transformers by taking into account model prediction confidence and by applying schedules not only based on the number of training steps but also on decoding steps, as errors are likely to appear as the decoding step accumulates.

. Neural Machine Translation Inference

The ultimate objective of NMT models is to generate a target sentence given the source. Therefore, during inference, NMT models aim to search for the optimal target sequence ê, which delivers the highest conditional probability given the source f as: ê = arg max e P (e|f ; θ).

( . )

However, the theoretical search space grows exponentially with respect to the target sequence length as it is possible to select all tokens in the vocabulary as output at each time step, making the exact search intractable. For instance, a target sequence with 10 tokens yields a space of |V | 10 possible sequences. An intuitive alternative is greedy search, which simply takes the most probable token at each time step: êi = arg max e i ∈V P (e i |e <i , f ; θ).

( . )

Nevertheless, greedy search is a single path with no way back. It cannot make any changes if errors are made at certain steps and have to continue searching with wrong predictions.

The most practical approach for standard NMT inference is beam search. Instead of always taking the most probable token, beam search keeps the top k most probable candidates at each time step. For time steps i > 1, each of the top k hypotheses generates a probability distribution over the vocabulary. Beam search only keeps the top-k hypotheses with the highest cumulative probability among k ⋅ |V | candidates. After each time step, the k hypotheses are re-ranked according to their cumulated probability. The searching procedure stops when the end of sentence symbol <eos> is generated or when the decoding reaches a predefined maximum step, and the hypothesis with the highest probability is considered as the final translation.

However, beam search tends to generate translations that lack diversity [START_REF] Ott | Analyzing uncertainty in neural machine translation[END_REF], which is sometimes problematic for data augmentation methods like back-translation (Sennrich et al., 2016b). [START_REF] Edunov | Understanding back-translation at scale[END_REF] propose an alternative top-k sampling strategy to increase translation diversity. Unlike beam search, which keeps the top k candidates at each time step, the sampling strategy randomly samples a candidate from the k most probable candidates following their probabilities to increase the diversity of final translation. Even though the quality of translation generated via topk sampling decreases compared to beam search, the increased diversity is helpful for applications like back-translation [START_REF] Edunov | Understanding back-translation at scale[END_REF].

. Evaluation Metrics

Evaluating translation quality is not a simple task, even for humans and much more so for machines. Besides, human evaluations are always expensive and time-consuming. Therefore, automatic evaluation metrics, which correlate with human evaluations and are inexpensive, are required to quickly evaluate translation hypotheses generated by different systems. In this section, we briefly discuss three metrics based on word-level matching and two other metrics based on sentence representation similarity comparison using recently proposed large pre-trained language models.

. BLEU

The most commonly used MT evaluation metric is the Bilingual Evaluation Understudy (BLEU, [START_REF] Papineni | Bleu: a method for automatic evaluation of machine translation[END_REF]. It measures the corpus-level n-gram match precision, factored by a brevity penalty for short hypotheses. BLEU computes a modified n-gram precision for a candidate corpus by counting n-gram matches over the corpus:

p n = ∑ ê∈C ∑ w n min(count(w n |e), count(w n |ê)) ∑ ê∈C ∑ w n count(w n |ê) , ( . )
where w n is an n-gram, C is the candidate corpus, and ê and e refer to a hypothesis and a reference translation, respectively. count(w n |e) computes the number of times n-gram w n appears in a sequence e.

The n-gram precision penalizes translations that are longer than the references. However, it cannot reflect hypotheses with shorter lengths well. [START_REF] Papineni | Bleu: a method for automatic evaluation of machine translation[END_REF] introduce a brevity penalty (BP) score, which is a factor that penalizes short hypotheses:

BP = 1 if | Ĉ| > |C| exp(1 - |C| | Ĉ| ) if | Ĉ| ≤ |C|, ( . )
where | Ĉ| and |C| refer to the number of tokens in the hypothesis and reference corpus. In practice, BLEU is usually computed with N = 4, thus counting precisions from uni-gram to 4-grams. The exact BLEU score takes the geometric average of n-gram precisions factored by BP:

BLEU(ê, e) = BP ⋅ exp( 1 4 4 n=1 log p n ).
( . )

BLEU score thus ensures that a high-scoring translation must match the reference in length, word choice, and word order. However, BLEU score and other exact word matching metrics are highly influenced by the tokenization of sentences. For example, [Thank you.] and [Thank you .] are semantically identical but are considered as two different sentences with different lengths. Besides, "you." is considered as one single word, which is neither matched to "." nor to "you". [START_REF] Post | A call for clarity in reporting BLEU scores[END_REF] propose SacreBLEU, a standard implementation of the BLEU metric, which always requires translations and references to be of raw text as input and applies internal tokenization to both texts so that the tokenization will not influence the final BLEU score anymore. The use of a standardized tool as Sacre-BLEU is very important but is still far from being widely adopted by the MT community [START_REF] Marie | Scientific credibility of machine translation research: A meta-evaluation of 769 papers[END_REF].

. METEOR

The METEOR score [START_REF] Banerjee | METEOR: An automatic metric for MT evaluation with improved correlation with human judgments[END_REF][START_REF] Denkowski | Meteor universal: Language specific translation evaluation for any target language[END_REF], standing for the Metric for Evaluation of Translation with Explicit ORdering, considers not only exact word matches but also matches of stems and synonyms. It first computes a unigram matching alignment between the translation and reference. The alignment here is defined as each unigram in one sequence being matched to at most one unigram in the other sequence. Unigram matches take into account not only exact word matches but also when the two words are the same after being stemmed and if they are synonyms. With the unigram alignment, it then obtains the number of matched unigrams m, the number of unigrams in the translation t, and the reference r. The unigram precision P = m/t and recall R = m/r are used to compute the F mean as:

F mean = P ⋅ R αP + (1 -α)R . ( . )
METEOR also incorporates a fragmentation penalty. It computes n-grams of unigram matches so that it can obtain a minimum number of matched n-grams ck. The penalty is then computed as:

P en = γ ⋅ ( ck m ) β , ( . )
and the final METEOR score is therefore:

METEOR(ê, e) = (1 -P en) ⋅ F mean .

( . )

The hyperparameters α, β, and γ need to be tuned for different languages so that METEOR can maximize the correlation with human judgments. The stemming and synonyms are also language-specific, which limits the ability to compute and compare METEOR scores across languages. et al. (2006) propose Translation Edit Rate (TER), which is a metric that measures the amount of editing required to transform a translation into an exact reference. The TER score computes the number of edits normalized by the reference length:

. TER

Snover

TER(ê, e) = #Edits #Reference words , ( . )
where the number of edits computes the number of insertions, substitutions, deletions, and segment shifts. The segment shift is defined as moving a contiguous sequence of words within the hypothesis to another location. The number of shifts is obtained by repeatedly selecting the shift that reduces the number of insertions, substitutions, and deletions at most until no more beneficial shifts remain within the hypothesis.

. Metrics Based on Pre-trained Language Models

With the advance of large pre-trained language models, several metrics incorporate pre-trained models to obtain high-quality sentence embeddings that are used to compute the similarity between the hypotheses and the reference. Pre-trained model-based metrics show a better correlation with human judgments than word-level matching-based metrics. [START_REF] Rei | COMET: A neural framework for MT evaluation[END_REF] propose the COMET score, standing for the Crosslingual Optimized Metric for Evaluation of Translation. COMET uses a pre-trained cross-lingual language model to encode the hypothesis translation, the reference, and also the source sentence. It obtains a vector representation of a sentence with a mean pooling operation to compute the averaged embedding over tokens. The states are computed as the weighted sum of the pre-trained model's output at each encoder layer. With the obtained sentence embeddings, [START_REF] Rei | COMET: A neural framework for MT evaluation[END_REF] further train a small network on quality estimation data to either directly regress to a quality score or measure the distance between the hypothesis and both source and reference. Zhang et al. (2020b) propose BERTScore. They compute the similarities between all tokens in the hypothesis and all tokens in the reference. The similarity is defined as the cosine similarity between contextualized token embeddings. They use pre-trained BERT models to encode the hypothesis and the reference to obtain the output states of each token as the token embedding. They further compute the token match precision and recall by taking the token with the highest similarity score as a match and the F1 score as the harmonic mean of precision and recall.

. Conclusion

In this chapter, we introduced the conventional NMT systems used in this thesis. We have reviewed well-known text tokenization methods, various NMT model architectures, conventional training and inference procedures, and several evaluation metrics. This chapter has not covered all works for each topic we discussed. We refer to [START_REF] Mielke | Between words and characters: A brief history of openvocabulary modeling and tokenization in NLP[END_REF] for a more complete review of text tokenization methods; [START_REF] Dabre | A survey of multilingual neural machine translation[END_REF] for more developments of multilingual NMT; and [START_REF] Wiher | On decoding strategies for neural text generators[END_REF] for more discussions about decoding strategies.

There are also many aspects of NMT that are not covered in this chapter. We will review NMT tasks using augmented sources other than only the source sentence in Chapter 3. These tasks are highly related to building bilingual writing systems, which incorporate bilingual information in the translation process. We will introduce non-autoregressive NMT, which generates multiple tokens in one decoding step, in Chapter 4. Non-autoregressive models greatly accelerate the inference speed, which is suitable for interactive systems that need to update translations frequently. We refer to further readings for some subdomains of NMT research that are not discussed in this thesis, like data augmentation methods to enlarge the training corpus (Wang et al., 2018b;[START_REF] Edunov | On the evaluation of machine translation systems trained with backtranslation[END_REF]; low-resource language translation [START_REF] Sennrich | Revisiting low-resource neural machine translation: A case study[END_REF][START_REF] Wang | A survey on low-resource neural machine translation[END_REF][START_REF] Haddow | Survey of Low-Resource Machine Translation[END_REF]; parallel data cleaning and selection (van der [START_REF] Marlies Van Der Wees | Dynamic data selection for neural machine translation[END_REF][START_REF] Koehn | Findings of the WMT 2018 shared task on parallel corpus filtering[END_REF][START_REF] Koehn | Findings of the WMT 2019 shared task on parallel corpus filtering for lowresource conditions[END_REF][START_REF] Koehn | Findings of the WMT 2020 shared task on parallel corpus filtering and alignment[END_REF]; unsupervised NMT [START_REF] Artetxe | Unsupervised neural machine translation[END_REF]Lample et al., 2018a,b); speech translation and simultaneous translation [START_REF] Anastasopoulos | Tied multitask learning for neural speech translation[END_REF][START_REF] Dalvi | Incremental decoding and training methods for simultaneous translation in neural machine translation[END_REF][START_REF] Arivazhagan | Monotonic infinite lookback attention for simultaneous machine translation[END_REF][START_REF] Elbayad | Efficient Waitk Models for Simultaneous Machine Translation[END_REF]; domain adaptation of NMT [START_REF] Chu | A survey of domain adaptation for neural machine translation[END_REF][START_REF] Pham | Revisiting multi-domain machine translation[END_REF][START_REF] Saunders | Domain adaptation and multi-domain adaptation for neural machine translation: A survey[END_REF]; document-level NMT [START_REF] Abdul | Document level contexts for neural machine translation[END_REF][START_REF] Maruf | A survey on document-level neural machine translation: Methods and evaluation[END_REF]; robust NMT [START_REF] Cheng | Towards robust neural machine translation[END_REF][START_REF] Cheng | Robust neural machine translation with doubly adversarial inputs[END_REF], etc.

-Neural Machine Translation with Augmented Sources

Standard neural machine translation (NMT) systems take only a source sentence as input and generate output translations from scratch on the target side. However, NMT systems can incorporate augmented sources other than the source sentence to help improve the translation quality. This is also the case of bilingual writing, where text segments are composed in both languages, therefore containing bilingual information that may help both text segments when generating monolingual texts. In this chapter, we describe several applications integrating augmented sources.

. Interactive Machine Translation

Interactive machine translation (IMT), also known as interactive translation prediction [START_REF] Knowles | Neural interactive translation prediction[END_REF][START_REF] Santy | INMT: Interactive neural machine translation prediction[END_REF]), text prediction [START_REF] Foster | User-friendly text prediction for translators[END_REF], etc., is an important branch of Computer Aided Translation (CAT). IMT systems are designed to help human translators produce high-quality translations. They repeatedly produce translation suggestions or completions while the human translators type a translation [START_REF] Langlais | TransType: a computer-aided translation typing system[END_REF]. Human translators can decide to accept the suggestion provided by IMT systems. Otherwise, they can type a translation if the suggestion is not satisfactory. IMT systems will then generate new suggestions based on the translation segments entered by human translators.

. Prefix Decoding

In IMT, human translators are always in control of the entire translation process. Moreover, IMT systems should respect the texts typed or accepted by human translators when generating new suggestions [START_REF] Langlais | TransType: a computer-aided translation typing system[END_REF]. Therefore, the core task for IMT systems is to provide translation suggestions based on specific constraints imposed by human translators. Most works consider a left-to-right translation mode, where human translators generate or accept translations step by step from the beginning. In this situation, the constraint is a prefix of a translation. IMT systems thus perform a prefix-constrained prediction to generate suffix translations by taking both the source sentence and the constrained prefix as input [START_REF] Langlais | TransType: a computer-aided translation typing system[END_REF][START_REF] Koehn | Refinements to interactive translation prediction based on search graphs[END_REF][START_REF] Alabau | CASMACAT: A computer-assisted translation workbench[END_REF][START_REF] Green | Human effort and machine learnability in computer aided translation[END_REF][START_REF] Wuebker | Models and inference for prefix-constrained machine translation[END_REF][START_REF] Knowles | Neural interactive translation prediction[END_REF][START_REF] Santy | INMT: Interactive neural machine translation prediction[END_REF]. Thanks to the autoregressive nature of NMT systems, prefix decoding can be straightfor- wardly implemented by conditioning on the given prefix e * ≤t and the source sentence to generate the target tokens: p(e i |e * ≤t , ê[t+1,i-1] , f ; θ) [START_REF] Knowles | Neural interactive translation prediction[END_REF][START_REF] Santy | INMT: Interactive neural machine translation prediction[END_REF]. As illustrated in Figure 3.1a, tokens predicted by IMT systems at the initial positions i ≤ t are ignored, and the beam search strategy only starts to consider various candidates from the position t + 1.

. Relaxing Constraint Order

The strict left-to-right translation may not be an ideal writing mode for human translators, as some may prefer to first translate difficult words and then easy ones [START_REF] Huang | Transmart: A practical interactive machine translation system[END_REF]. Some studies have explored several alternatives to the strict prefix-constrained decoding.

One of the directions is to allow human modifications or indications of a complete initial translation. Note that this approach is more related to postediting, which is another important branch of CAT tools. Post-editing requires human translators to edit a translation produced by a machine translation (MT) system instead of starting to translate from scratch with generated suggestions. However, since this approach involves more substantial interaction between humans and systems than post-editing and proposes translation suggestions, we also consider it as another type of IMT. [START_REF] Marie | Touch-based pre-post-editing of machine translation output[END_REF] initially attempt to revise an initial translation in a touch-based scenario. In their proposed system, users only need to select usable translation segments, which do not require to be a prefix, while the IMT system automatically uses the selected spans to retranslate the source sentence. This process can be iteratively repeated until there are no more improvements. [START_REF] Grangier | QuickEdit: Editing text & translations by crossing words out[END_REF] study a similar approach by crossing-out undesired words in a proposed translation. They use a dual-encoder sequence-to-sequence architecture where the second encoder encodes the marked translation as input. Using a dual encoder to integrate the augmented sources is a common practice. More details about the dual-encoder architecture are discussed in Sections 3.2 and 3.5. [START_REF] Wang | Touch editing: A flexible one-time interaction approach for translation[END_REF] further explore this idea by marking a sequence of actions to a given translation and then using a dual-encoder model to perform translation by taking the marked translation as a second input.

Another direction focuses on giving more freedom to human translators when performing translations. [START_REF] Lee | Intelli-CAT: Intelligent machine translation post-editing with quality estimation and translation suggestion[END_REF], [START_REF] Qian | Glancing transformer for non-autoregressive neural machine translation[END_REF], [START_REF] Huang | Transmart: A practical interactive machine translation system[END_REF][START_REF] Yang | WeTS: A benchmark for translation suggestion[END_REF] allow users to perform edits at any position in a hypothesis sequence and provide translation suggestions or autocompletion based on context information from both prefixes and suffixes. Xiao et al. (2022a) extend these approaches by considering a text infilling task to generate multiple missing segments from a source sentence concatenated with pre-translated segments. Figure 3.1b illustrates this approach. Their method allowed the translation segments approved by human translators to be interspersed among gaps. They use a special token to mark the gaps between two pre-translated segments to construct an initial target sequence. Instead of using dual-encoder architectures, they concatenate the source sequence with the initial target sequence, with a separator token in between, to make a long input sequence on the source side. The approach to concatenate the extended sources with the source sentence has been widely explored across various applications, which we also discuss in the following sections. On the target side, instead of producing the entire target sentences, Xiao et al. (2022a) only output segments that are predicted to infill the gaps in the initial sequence. The final translation is constructed by replacing gaps in the initial target sequence with predicted segments.

. Automatic Post-Editing

The automatic post-editing (APE) task focuses on automatically editing a machine-generated translation to improve the translation quality and correct translation errors to reach a good translation [START_REF] Simard | Rulebased translation with statistical phrase-based post-editing[END_REF]. Therefore, unlike MT systems which only require source-target parallel data, APE systems need triplets of examples consisting of a source sentence, a target sentence generated by an MT system, and a human post-edited version of the machine-generated target sentence. The last element should not be a reference translation, which is different from regular MT tasks, as APE systems need to learn from human post-editing to perform edits to the translation [START_REF] Félix Do Carmo | A review of the state-of-the-art in automatic post-editing[END_REF]. Using reference translations may misrepresent the editing patterns between MT outputs and human post-editing results. However, such triplets are challenging to collect as human post-edited translations are often proprietary and difficult to get access to. To this end, several works studied exploiting artificial APE triplets to alleviate the data scarcity problem. [START_REF] Junczys | Log-linear combinations of monolingual and bilingual neural machine translation models for automatic post-editing[END_REF] use round-trip translation to create synthetic source and MT output by first translating monolingual target language sentences into the source language, then retranslating them back to the target language. [START_REF] Negri | ESCAPE: a large-scale synthetic corpus for automatic post-editing[END_REF] approximate the actual APE triplets by treating parallel corpora from MT tasks as the source and artificial human post-edits and generating MT outputs from the source sentences with separate MT systems.

Neural APE methods mainly focus on two general directions: dualencoder models and single-encoder models with a concatenated long input sequence. As we mentioned in Section 3.1 and will also discuss in the following sections, these two directions are also widely used in other applications integrating augmented sources, like IMT, NMT with translation memories and lexical constraints, multi-source and bidirectional translation, etc. et al. (2016) and [START_REF] Hokamp | Ensembling factored neural machine translation models for automatic post-editing and quality estimation[END_REF] perform APE by concatenating the source sentence and the MT output into a long sequence. They separate the two sentences with a special token, as illustrated in Figure 3.2a. [START_REF] Niehues | Pretranslation for neural machine translation[END_REF] propose a similar approach, which concatenates a pre-translation generated by a statistical machine translation (SMT) system to the source. They mark tokens from the source and target languages with different characters to distinguish the two languages. In this single-encoder approach, the encoder can process inputs with a mixture of language. The advantage of this approach is that it does not require modifications to the model architecture, meaning that it can be easily applied to various implementations. Correia and Martins (2019) and Lopes et al. ( 2019) incorporate pre-trained multilingual BERT models [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] into an encoder-decoder architecture. BERT, referring to Bidirectional Encoder Representations from Transformers, is a large-scale language model trained on a very large dataset. It is able to output high-quality sentence representations that could be di- rectly used or fine-tuned for downstream tasks. Correia and Martins (2019) and [START_REF] António | Unbabel's submission to the WMT2019 APE shared task: BERT-based encoder-decoder for automatic post-editing[END_REF] initialize their encoder and decoder with the multilingual BERT parameters and used the concatenated sequence as input to perform APE.

. Single Encoder Model

Crego

Despite various approaches developed for APE, with the great and rapid advance in NMT, it has become more difficult for APE models to improve NMT model outputs which are already of high quality [START_REF] Chollampatt | Can automatic post-editing improve NMT?[END_REF].

. Dual Encoder Model

The dual-encoder model involves two encoders, one for the source sentence and the other for the MT output. Figure 3.2b shows the dual-encoder approach. We mainly focus on Transformer-based approaches. A typical dual-encoder Transformer was proposed by [START_REF] Junczys | MS-UEdin submission to the WMT2018 APE shared task: Dual-source transformer for automatic post-editing[END_REF]. They use two separate encoders with the same architecture to map the source sentence and the MT output. In their decoder, they use two stacked encoder-decoder cross-attention layers in each decoder block, with one layer attending to the representation of the source encoder and the other attending to the hidden state generated from the MT output encoder. [START_REF] Shin | Multi-encoder transformer network for automatic post-editing[END_REF] study a similar approach, with an additional crossattention layer in the decoder to compute the attention between the output of the two encoders. [START_REF] Tebbifakhr | Multi-source transformer with combined losses for automatic post editing[END_REF] also apply two encoders. However, they concatenate the outputs from the two encoders as one sequence to perform cross-attention on the decoder side. The decoder side of their approach remains unchanged compared to the basic Transformer architecture. [START_REF] Pal | A transformer-based multi-source automatic post-editing system[END_REF] further incorporate a third encoder which maps the concatenated output sequence from the two encoders to a new representation for the decoder.

. Neural Machine Translation with Translation Memories

A Translation Memory (TM) is a database that contains segments in a source language with the corresponding translations in a target language, which are recorded during the regular activities of human translators. It is mainly used to match up previously generated translations to new content similar to the contents translated in the past. Many domains, like technical domains, contain highly repeated segments, terms, and phraseology, which are more efficient for reusing existing translations rather than translating from scratch. We assume that this pair of sentences exists in the TM, which translates the following English sentence into French: [How long does the flight last?] ↝ [Combien de temps dure le vol?]. If the same source sentence appears in a future document (an exact match), the TM will suggest reusing the translation that has been saved. In addition to exact matches, TMs are also useful with fuzzy matches, which can help to reduce translation effort and to increase consistency when a new sentence is similar to a previously translated sentence but not identical. For example, when translating the input sentence: [How long does a cold last?], the TM may also suggest the previous translation since only two replacements (un rhume (a cold) → le vol (the flight)) are needed to achieve a correct translation.

For decades, the localization industry has proposed TM technologies in CAT tools to allow human translators to visualize one or several similar or usable translations extracted from the TM when translating a sentence, leading to higher productivity and consistency [START_REF] Yamada | The effect of translation memory databases on productivity[END_REF]. Hence, even though the retrieval methods of TM differ among CAT tools [START_REF] Bloodgood | Translation memory retrieval methods[END_REF], human translators generally accept discounted translation rates for sentences with high fuzzy matches.

1

The idea of using TMs to help translation is also explored in the MT area, where extracted translations are used to help improve the translation quality. In the context of MT, TM simply refers to parallel bilingual data. In earlier SMT approaches, the TM retrieval shares the same idea with SMT about managing and retrieving the optimal combination of longest translated ngrams. This property leads to the development of several techniques like the use of TM in SMT decoding [START_REF] Koehn | Convergence of translation memory and statistical machine translation[END_REF][START_REF] Wang | Integrating translation memory into phrase-based machine translation during decoding[END_REF], adaptive MT [START_REF] Zaretskaya | Integration of machine translation in cat tools: State of the art, evaluation and user attitudes[END_REF] or fuzzy match repairing [START_REF] Ortega | Fuzzymatch repair using black-box machine translation systems: what can be expected[END_REF][START_REF] Knowles | A comparison of machine translation paradigms for use in black-box fuzzy-match repair[END_REF].

In the context of NMT, methods using TMs to help improve translation quality are also known as retrieval-based approaches. They mainly contain two stages: the retrieval of similar translation segments and the integration of similar translations into NMT systems. Integrating TMs into NMT is structurally identical to the APE task. However, as Sánchez-Gijón et al. ( 2019) indicate, processing TMs and MT outputs requires different efforts in human post-editing. The edition of automatic (noisy) translation is quite different from the edition of a good translation of a similar source sentence.

. Retrieving Similar Translations

Consider a TM as a set of N sentence pairs {(f , e) i , i = 1, . . . , N } where f i and e i are mutual translations. A TM must be conveniently stored to allow fast access to the pair (f i , e i ) that shows the highest similarity between f i and a given new sentence f . Many methods to compute sentence similarity have been explored, mainly falling into two broad categories: lexical match (i.e., fuzzy match) and distributed representation match. The former relies on the ratio of overlaps between the source sentences taken into account, while the latter counts on the generalization power of neural networks when building sentence representations.

Fuzzy Matching

Fuzzy matching is a lexicalized matching method aimed at identifying sentences that are similar to a given sentence f . The fuzzy matching score between two sentences f i and f j is defined as:

FM(f i , f j ) = 1 - ED(f i , f j ) max(|f i |, |f j |) , ( . )
where ED(f i , f j ) is the edit distance between f i and f j , and |f | is the length of f . Many variants have been proposed to compute the edit distance, generally https://signsandsymptomsoftranslation.com/2015/03/06/fuzzy-matches/ performed on normalized sentences (ignoring, for instance, case, number, punctuation, spaces, or inline tag differences that are typically handled at a later stage). Also, inverse document frequency and stemming techniques are used to give more weight to significant words or less weight to morphological variants [START_REF] Bloodgood | Translation memory retrieval methods[END_REF][START_REF] Vanallemeersch | Assessing linguistically aware fuzzy matching in translation memories[END_REF]. Fuzzy matching delivers excellent performance under large overlapping conditions. However, it requires two sentences to be similar in terms of length. When it is not the case, sentences with large overlaps may receive low FM scores. Consider, for instance, the input: [How long does the flight arriving in Paris from Barcelona last?] and the TM entry of the previous example: [How long does the flight last?] ↝ [Combien de temps dure le vol?]. Even though the TM entry may be of great help when translating the input sentence, it receives a low FM score (1 -5/12 = 0.583) because of multiple insertion/deletion operations needed. Therefore, a second lexicalized similarity measure that focuses on finding the longest n-gram overlap between sentences is also used.

N -gram Matching

N -gram matching is also called subsequence matching or chunk matching in CAT tools and is usually combined with source-target alignment to help human translators find translation fragments. The n-gram matching score between f i and f j is defined as:

NM(f i , f j ) = max N (f i ) ∩ N (f j ) , ( . )
where N (f ) denotes the set of n-grams in sentence f , max(q) returns the longest n-gram in the set q, and |r| is the length of an n-gram r.

Sentence Representation Matching

The current research on sentence-level similarity measures has made tremendous advances thanks to distributed word representations computed by neural networks (Mikolov et al., 2013a,b;[START_REF] Pennington | GloVe: Global vectors for word representation[END_REF][START_REF] Joulin | Bag of tricks for efficient text classification[END_REF][START_REF] Bojanowski | Enriching word vectors with subword information[END_REF] and large pre-trained language models. These methods are used to compute the vector representation of a given sentence, also denoted as sentence embedding. Well-known sentence embedding methods include LASER [START_REF] Artetxe | Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond[END_REF] and LaBSE [START_REF] Feng | Language-agnostic BERT sentence embedding[END_REF]. The similarity score between sentences f i and f j is computed as the cosine similarity of their distributed representations h i and

h j : EM(f i , f j ) = h i ⋅ h j ||h i || × ||h j || , ( . )
where ||h|| denotes the magnitude of vector h. There exists fast retrieval implementations like the faiss 2 toolkit [START_REF] Johnson | Billion-scale similarity search with gpus[END_REF], which can quickly compute the similarities between a query vector and a set of vectors, providing fast retrieval of similar translations.

Sentence similarity methods based on distributional representations typically outperform lexicalized methods in capturing semantic similarity, particularly under small lexical overlap conditions. To this end, [START_REF] Cai | Neural machine translation with monolingual translation memory[END_REF] and [START_REF] Khandelwal | Nearest neighbor machine translation[END_REF] use the distributed representation computed by the NMT model itself to incorporate similar monolingual target sentences or contextualized target tokens instead of always retrieving sentences based on source side similarities.

. Integrating Translation Memories

NMT models incorporating TMs can be clustered into three main groups of approaches: single-encoder model with concatenated input, dual-encoder model, and nearest neighbor-based approaches [START_REF] Khandelwal | Nearest neighbor machine translation[END_REF]. The first two groups are similar to the APE models described in Section 3.2, using complete TM sentences. The third group tends to extract and integrate similar translation segments or tokens in a similar context instead of the entire sentence.

Bulte and Tezcan (2019) experiment with concatenating retrieved similar translations to the source sentence as a long input sequence without modifying the model architecture. Therefore, the model can learn to directly copy useful segments from the concatenated similar translations. They explore using up to 3 similar translations and concatenate them all together with the source sentence. We extend this approach in [START_REF] Xu | Boosting neural machine translation with similar translations[END_REF] by distinguishing tokens in a similar translation that are aligned to source tokens from those "unrelated" tokens that cannot find counterparts in the source. We add a second input feature to the encoder embedding that marks source tokens, aligned target tokens, and unrelated tokens. More details about detecting unrelated tokens are presented in Chapter 8. We also propose "Priming NMT" in [START_REF] Pham | Priming neural machine translation[END_REF], in which we use both the source f and the target ẽ of the retrieved examples. Different from Bulte and Tezcan (2019), we concatenate the similar source f and the input f to make the source sequence and train the model to translate the corresponding target sequence by concatenating the similar translation ẽ and the reference e. In Priming NMT, the encoder does not contain target sequences. The model learns to use f from the source side and ẽ as a previously generated sequence on the target side to produce the translation. During inference, the source side remains the same as training, and we prefix the target side with ẽ using prefix decoding to only output the desired target translation. Gu et al. (2018b) use an external black-box search engine to search for similar translations and integrate TM by rerunning the NMT model to obtain context vectors of similar translations. To some extent, this approach is equivalent to using a second encoder with a shared parameter. [START_REF] Xia | Graph based translation memory for neural machine translation[END_REF] extend this idea by incorporating TMs into a compact graph as a second encoder. [START_REF] Cai | Neural machine translation with monolingual translation memory[END_REF] jointly train the retrieval module together with the NMT model to extract translation-specific monolingual target sentence and incorporate it with an extra encoder. This dual-encoder approach is similar to the approach of [START_REF] Junczys | MS-UEdin submission to the WMT2018 APE shared task: Dual-source transformer for automatic post-editing[END_REF] for APE. He et al. (2021b) encode retrieved similar translations using the decoder embedding matrix and integrate them on the decoder side by performing cross-attention between the decoder input sentence and the similar translation. This approach is more like a dual "decoder" architecture. Zhang et al. (2018a) incorporate cross-lingual alignment information to retrieved TMs and extract only similar target segments whose aligned source pieces were present in the input source sentence. The extracted similar pieces are used to guide NMT in terms of extra rewards during decoding. [START_REF] Khandelwal | Nearest neighbor machine translation[END_REF] further extend this idea by performing k-nearest neighbor (kNN) MT. For each decoding step, they search for k target tokens with similar contexts in the TM and use them to help generate the token prediction probability. [START_REF] Zheng | Adaptive nearest neighbor machine translation[END_REF] relax using fixed kNN for each target token by dynamically deciding the number of neighbors to use among the k neighbors for each decoding step. However, the kNN-based approaches require extensive computing efforts as it needs to compute and store the contextualized vector representation of each token in each target sentence of the TM and perform a kNN search of vectors in a very large data store for each beam at each decoding step.

Retrieval-based approaches can improve translation quality by using TMs. Therefore, they are often applied to perform domain adaptations by searching for similar translations in specific domains [START_REF] Xu | Boosting neural machine translation with similar translations[END_REF][START_REF] Khandelwal | Nearest neighbor machine translation[END_REF]. Sentence-level retrieval is often very fast and only yields negligible time overheads. However, it often requires a good match to help the translation. For some domains, especially more general domains, it is difficult to find a proper translation similar to the source at the sentence-level, and using poor matches even degrades the translation quality [START_REF] Xu | Boosting neural machine translation with similar translations[END_REF]. On the contrary, token-level retrieval can search for more accurate matches for each token that cannot be found only by searching for entire sentences but with a significant retrieval time overhead. However, the performance gains of kNN MT on more general domains are still much smaller than those on specific domains.

. Lexical Constraints for Neural Machine Translation

NMT models typically operate in an end-to-end fashion. Therefore, it is not obvious to introduce additional knowledge or add controls during the translation process. However, it is useful to introduce external sources of knowledge in some scenarios, like incorporating domain-specific terminologies or respecting human-entered translation segments like in IMT to generate higher quality translations. The extra knowledge often appears as terminology words or pieces of translation segments that are considered as lexical constraints and are required to be present in the output translation. Studies on NMT with lexical constraints focus on generating a good translation that properly includes given constraints.

. Lexically Constrained Decoding

Lexically constrained decoding (LCD) aims to force NMT models to produce all required constraints in their exact form in the translation. [START_REF] Hokamp | Lexically constrained decoding for sequence generation using grid beam search[END_REF] propose using grid beam search (GBS) for LCD. Instead of maintaining only one beam at each time step, they explore a grid of beams, in which the hypotheses of each beam were filled with both the token generated by the model and force-predicted constrained tokens. The GBS procedure only stops if all constrained tokens are predicted, and the model predicts the end-of-sentence symbol. The hypothesis sequence that covers all constraints with the highest probability in the beam set is selected as the final translation. In this way, GBS assures that all constrained tokens are present in the output translation. However, the expanded beams also cost a large runtime overhead. The complexity increase is linear to the number of token constraints compared to standard beam search decoding. [START_REF] Post | Fast lexically constrained decoding with dynamic beam allocation for neural machine translation[END_REF] and [START_REF] Hu | Improved lexically constrained decoding for translation and monolingual rewriting[END_REF] improve the GBS by a dynamic beam allocation. They expand candidates with constrained tokens and keep only one single beam of hypotheses at each time step as regular beam search decoding to make the complexity independent to the number of constraints. [START_REF] Hasler | Neural machine translation decoding with terminology constraints[END_REF] also study similar approaches to incorporate terminology constraints via constrained beam search [START_REF] Anderson | Guided open vocabulary image captioning with constrained beam search[END_REF]. However, even with the improvements, LCD still encounters a large decoding overhead compared to standard beam search. Additionally, LCD sometimes hurts the overall quality of generated translations, as it breaks the semantic assessment of NMT models by forcing constraints to occur in the final translation, even when they do not fit in the context or when a synonym is selected by the NMT model.

. Soft Lexical Constraints

LCD methods mainly focus on the decoding step by injecting hard constraints with a significant efficiency loss. On the contrary, other approaches have focused on training the NMT model to use additional terminology translations. Crego et al. (2016) first propose to replace domain-specific source terminologies with abstract placeholder tokens. They train NMT models with pre-processed data containing the placeholder on both the source and target sides so that the model learns to predict the corresponding placeholder token on the target side. The final translation is obtained by replacing target placeholders with the translated terminologies. [START_REF] Dinu | Training neural machine translation to apply terminology constraints[END_REF] first propose to inject the terminology translation into the source sentence. They experiment with two approaches. The first one appends the terminology translation to the source word. Taking the example from Section 3.3, the source sentence is augmented as: [How long does the flight vol last?], making it a sentence mixing two languages. The second approach directly replaces the source terminology by its translation: [How long does the vol last?]. In both settings, the target side remains unchanged. [START_REF] Dinu | Training neural machine translation to apply terminology constraints[END_REF] train NMT models with the augmented source sentence to learn a copy behavior of the terminology translation. This is somewhat similar to Bulte and Tezcan (2019), even though the source sequence structures are different. [START_REF] Song | Code-switching for enhancing NMT with pre-specified translation[END_REF] also apply the "replace" setting of [START_REF] Dinu | Training neural machine translation to apply terminology constraints[END_REF] and incorporate a pointer network [START_REF] Gulcehre | Pointing the unknown words[END_REF] to enhance the copy behavior.

All these approaches do not involve any modifications to the decoding stage and use a standard beam search strategy. Therefore, these approaches do not incur any overhead during decoding. The terminology translations injected on the source side are soft constraints, as there is no guarantee that they will always appear on the target side during decoding. [START_REF] Dinu | Training neural machine translation to apply terminology constraints[END_REF] show that this soft terminology constraint obtained a terminology usage rate greater than 90% for both settings. et al. (2016) and [START_REF] Dinu | Training neural machine translation to apply terminology constraints[END_REF] focus on direct copies of the terminology constraints, while LCD methods involve hard constraints that might not suit the grammatical context of the translation. The above approaches lack the flexibility to accommodate terminological variation, e.g., due to morphological inflection, as terms may not always be in the correct form and are even provided as lemmas in some real-world scenarios.

. Morphological Inflection

Crego

To address this issue, [START_REF] Michon | Integrating domain terminology into neural machine translation[END_REF]) extend Crego et al. (2016) by replacing terms with placeholders indicating part-of-speech and morphological information on both source and target sides. Their model can learn to translate dedicated source placeholders into proper target ones with the correct morphological change. This helps the translation by generating placeholders with morphological information and facilitates the post-processing to use the correct form of target terminologies.

Bergmanis and Pinnis (2021) and Niehues (2021) extend Dinu et al. ( 2019) by annotating the source side word with the target lemma instead of an exact translation with correct morphological inflection. Their models do not learn to simply copy the target lemma but to perform copy-and-inflect behavior instead. Xu and Carpuat (2021b) introduce a cross-lingual rulebased inflection module to predict the target inflected form of each lemma constraint solely based on the source sentence. The predicted inflected target constraints can be integrated into NMT models as in [START_REF] Dinu | Training neural machine translation to apply terminology constraints[END_REF] or in other models.

. Multi-source/Multi-target Translation

Another common situation for MT applications that involve auxiliary resources is multi-source/multi-target translation. Multi-source translation [START_REF] Franz | Statistical multi-source translation[END_REF][START_REF] Josep | Local lexical adaptation in machine translation through triangulation: SMT helping SMT[END_REF] aims to translate multiple source languages into a single target language, while multi-target translation [START_REF] Dong | Multi-task learning for multiple language translation[END_REF] does the opposite, translating one source language into multiple target languages.

. Multi-source Translation

The principal purpose of multi-source translation is to use multi-parallel corpora to improve the translation quality. [START_REF] Zoph | Multi-source neural translation[END_REF] use a multi-encoder NMT model to encode two source languages at the same time. They combine the hidden states from both encoders into a single hidden state before passing to the decoder. Firat et al. (2016b) and [START_REF] Garmash | Ensemble learning for multisource neural machine translation[END_REF] explore a mixture of NMT experts ensemble approach. They adopt an encoder-decoder NMT model for each source-target language pair and combine the probability distribution of separate NMT models at each time step to make the prediction. Probability distributions are combined as a weighted sum where a gated network computes the weights to control the contributions of each model. [START_REF] Dabre | Enabling multisource neural machine translation by concatenating source sentences in multiple languages[END_REF] propose a different way to concatenate parallel sentences in multiple source languages into a long input sentence, similar to [START_REF] Curran Associates | Neural fuzzy repair: Integrating fuzzy matches into neural machine translation[END_REF]. The only difference is that they do not use any separator tokens to indicate individual source sentences and rely on the model to learn to distinguish sentence boundaries.

All the above methods require multi-parallel corpora, in which all languages are aligned among each other. Even though the separate models in the ensemble approach can be pre-trained on different bilingual data for each language pair, they still need multi-parallel data to train the gated network. However, multi-parallel data is relatively scarce compared to bilingual data, thus limiting the possible use cases of large-scale multi-source NMT to restricted domains (e.g., multilingual institutions or bodies). [START_REF] Nishimura | Multi-source neural machine translation with missing data[END_REF] extend the above two approaches to perform translation when some multi-source inputs are missing. They use a special token to replace the missing input sentence and train multi-source models to ignore the input when the missing token is present. This approach makes a step towards exploring incomplete multi-parallel data. [START_REF] Choi | Improving a multi-source neural machine translation model with corpus extension for low-resource languages[END_REF] create synthetic multi-parallel data by translating sentences from a pivot language into missing languages to improve translation quality for low-resource language pairs. Xu et al. (2021c) further study using auxiliary languages in a multi-source setting. Similarly, they create synthetic sentences in the auxiliary languages by translating from bilingual training data using a pre-trained multilingual NMT model. This idea is also similar to [START_REF] Niehues | Pretranslation for neural machine translation[END_REF], as discussed in Section 3.2, in which the SMT output acts like an artificial auxiliary source in the target language. Xu et al. (2021c) enable a multi-source NMT model to handle single-source situations by randomly dropping the auxiliary language during training. They study the concatenated multi-source and multiencoder approaches. They also randomly drop the auxiliary language during training so that the model can perform standard translation without auxiliary sources. This is similar to Bulte and Tezcan (2019), who combine original parallel data and TM augmented data to train NMT models, which can translate with and without a TM.

. Multi-target Translation

As Och and Ney (2001) point out, the ultimate goal of multi-source translation is to use manually translated texts to help translate them into more languages. Another variant considers translating multiple target languages simultaneously, assuming weak target languages can benefit from the strong target languages. Although the multi-target concept can be generalized to translating into more than two target languages, most work has focused on generating two languages. [START_REF] Neubig | Multi-target machine translation with multi-synchronous context-free grammars[END_REF] first study simultaneously generating two targets under the SMT scenario, with the objective that a strong target language model can help the other language. [START_REF] Dong | Multi-task learning for multiple language translation[END_REF] explore multi-target NMT by performing multi-task learning. Their model contains a shared encoder and a separate decoder for each target language. The source sentence representation is identical across translation directions so that source language data across language pairs can all be used. Even though [START_REF] Dong | Multi-task learning for multiple language translation[END_REF] aim to perform multi-target translation, their model is trained only with bilingual data for different language pairs. The multi-task objective is achieved in a general term by jointly optimizing the loss from different language pairs across mini-batches. In other words, this multi-task training scheme only handles one language pair at a time, even though it can perform simultaneous decoding by only encoding the source sentence once and translating with multiple decoders at the same time. [START_REF] Wang | Synchronously generating two languages with interactive decoding[END_REF] propose a synchronous self-attention framework to handle two target languages simultaneously. In this framework, the generation of a target language relies not only on the source sentence but also on the previously generated tokens from the other target language. The synchronous self-attention model consists of a shared encoder and two decoders. The two decoders interact with each other at the decoder self-attention layer of the Transformer model. [START_REF] Wang | Synchronously generating two languages with interactive decoding[END_REF] use an additional decoder cross-attention module so that each decoder attends not only to itself but also to the other decoder. The hidden states from the two attention modules are then combined into one to pass through other layers.

The multi-target approach has also been applied to simultaneously generate a transcript and a translation from spoken input. [START_REF] Anastasopoulos | Tied multitask learning for neural speech translation[END_REF] propose a triangle model to perform simultaneous automatic speech recognition (ASR) and translation. The speech translation task aims at translating an audio input into a target text in a different language. In this triangle model, the encoder computes vector representations of the audio source. The ASR decoder performs as a regular decoder, attending only to source representations, while the translation decoder attends to the hidden states of both the encoder and the ASR decoder to generate translations. [START_REF] Le | Dual-decoder transformer for joint automatic speech recognition and multilingual speech translation[END_REF] study the same task with a dual-decoder model similar to [START_REF] Wang | Synchronously generating two languages with interactive decoding[END_REF]. They additionally perform a decoder cross-attention in the encoder-decoder cross-attention sublayer, which enhances the interaction between the two decoders. The multi-target translation is also applied to the subtitling task to jointly generate consistent subtitles in various languages from an audio source [START_REF] Karakanta | Between flexibility and consistency: Joint generation of captions and subtitles[END_REF]. Our own work with dual decoding and its applications to multi-target translation is in Chapters 5 and 6.

. Bidirectional Decoding Translation

The concept of multi-target translation generalizes to produce several outputs at the same time, which also suits the bidirectional decoding translation setting. Regular MT tasks generate a target sentence from left to right, while bidirectional decoding MT aims to generate another target sequence in the inverse direction from right to left. As the left-to-right generation cannot see predicted information in the future, bidirectional decoding MT tends to make use of information generated from the other direction to improve the translation quality.

Under the NMT scenario, Liu et al. (2016a) explore jointly training two separate NMT models, one for each direction, with a combined loss from both models. During decoding, the two models generate k-best hypotheses each, which are then re-scored to select the best translation. Zhang et al. (2018b) and [START_REF] Su | Exploiting reverse target-side contexts for neural machine translation via asynchronous bidirectional decoding[END_REF] propose making use of the target side's future information by performing asynchronous bidirectional decoding. This architecture resembles the triangle model of [START_REF] Anastasopoulos | Tied multitask learning for neural speech translation[END_REF]. The backward right-to-left decoder performs as regular NMT decoders, only in a reverse direction, generating hidden states from right to left. The forward left-to-right decoder, which performs similarly to the translation decoder of [START_REF] Anastasopoulos | Tied multitask learning for neural speech translation[END_REF], attends both to the source hidden states and the backward hidden states during decoding. Therefore, the forward decoder can make use of future context on the target side to generate better translations. [START_REF] Zhou | Synchronous bidirectional neural machine translation[END_REF] and Zhang et al. (2020a) explore synchronous bidirectional decoding NMT. They use the same model as [START_REF] Wang | Synchronously generating two languages with interactive decoding[END_REF] to generate in both directions at the same time. The final translation is the direction with a higher probability. The backward translation is reversed to make a proper sentence from left to right. More recently, He et al. (2021a) combine bidirectional decoding MT and multi-target MT together and managed to generate two languages in two directions simultaneously. For each target among the four, the prediction depends on the source as well as on all four targets. They extend [START_REF] Zhou | Synchronous bidirectional neural machine translation[END_REF], [START_REF] Wang | Synchronously generating two languages with interactive decoding[END_REF], and [START_REF] Zhou | Synchronous bidirectional neural machine translation[END_REF] by making each decoder attend to all four decoders.

. Conclusion

In this chapter, we reviewed several NMT tasks incorporating augmented sources besides the source sentence. The augmented sources can be of various types, such as human-corrected translation segments, pre-translated MT outputs, similar translations extracted from TMs, lexical constraints and terminologies, translations from auxiliary languages, etc. Despite all these varieties, the internal methods to incorporate these augmented sources do not vary much. They can be clustered into two general approaches: the single-encoder approach that mixes the augmented sources with the source sentence in the same sequence, and the dual-encoder approach that uses an additional encoder to encode the augmented sources. Among these two approaches, the single-encoder approach does not require specific modifications to model architectures and allows different types of language mix in the input sequence. Augmented sources can be concatenated to the source sentence or interspersed among source tokens, providing more flexibility for language mixing. Our works in Chapters 5, 7, and 8 are all based on a single-encoder that incorporates bilingual information.

-Non-autoregressive Neural Machine Translation

Conventional neural machine translation (NMT) systems perform translations in an autoregressive (AR) way, as described in Chapter 2. The generation of one token is conditioned on the source sentence and the previously generated target sequences, as in Equation ( . ). Therefore, the decoding procedure only generates one token at each decoding step. The AR characteristic suits Recurrent Neural Network (RNN)-based NMT models well, as RNN models cannot be parallelized and must proceed step by step. However, the AR decoding is slow as the decoding time increases linearly with the length of the target translation, and the computationally expensive NMT decoder is used at each time step to generate only one token. Besides, AR generation also suffers from the exposure bias problem, as introduced in Section 2.3.1. When performing offline translations, AR decoding can be largely accelerated with batch decoding to translate many sentences simultaneously [START_REF] Helcl | Nonautoregressive machine translation: It's not as fast as it seems[END_REF]. However, in the case of interactive translation scenarios, NMT systems often only have a single sequence to deal with that cannot be batched. For interactive systems like bilingual writing tools, it is important to generate translations within a low latency so that users will not wait long to see the translation suggestions once they make changes to the texts.

Thanks to the Transformer model [START_REF] Vaswani | Attention is all you need[END_REF], which allows more parallelizable architectures, an important variant of NMT, namely nonautoregressive neural machine translation (NAT), has emerged (Gu et al., 2018a). NAT is able to generate multiple tokens in parallel in one single decoder step, as it removes the AR connection by making a conditional independence assumption that all tokens on the target side are independent of each other given the source sentence:

P NAT (e|f ) = P L (T |f ; θ) ⋅ T i=1 P (e i |f ; θ), ( . )
where P L (T |f ; θ) measures the probability that the target sentence is of length T given the source. Compared to Equation ( . ), the conditioning history tokens e <i are removed for NAT models. Unlike AR models, which gradually increase the output sequence step by step, NAT models aim to generate all tokens at the same time. Therefore, NAT models do not have any target information in the translation process and generate each token independently and solely based on the source. The independence assumption and missing target information make NAT a very challenging task, as NAT models fail to capture the dependency within target tokens (Xiao et al., 2022b). The first NAT model of Gu et al. (2018a) ]. This phenomenon is called the multimodality problem of NAT (Gu et al., 2018a). From another aspect, this is similar to a one-to-many multilingual NMT model, in which the model is trained to translate the source language into multiple target languages, as also experimented with by [START_REF] Zhou | Understanding knowledge distillation in non-autoregressive machine translation[END_REF]. If no indication about the desired target language is given to the model, it may generate sentences in arbitrary target languages. However, the indications in multilingual NMT models like using a tag token to claim the desired target language [START_REF] Johnson | Google's multilingual neural machine translation system: Enabling zero-shot translation[END_REF] is a relatively strong constraint in AR approaches, while it is more difficult for NAT models to respect such constraints like target sentence length.

In this chapter, similar to Chapter 2, we briefly introduce methods in NAT and consider the following aspects:

• Model architecture • Objective function • Learning paradigm • Knowledge distillation of training data A more detailed review can be found in (Xiao et al., 2022b) and [START_REF] Gu | Non-autoregressive sequence generation[END_REF],

1 from which we borrow inspirations for this chapter. We also introduce methods using NAT with augmented sources, as in Chapter 3 for AR models.

. Non-autoregressive Translation Architectures

Many NAT model architectures have been proposed in recent years. We only briefly introduce some examples in this section, with a particular focus on iterative refinement-based models.

. Fully Non-autoregressive Model

The definition of NAT indicates that it is fully non-autoregressive, generating all target tokens of a translation at once with a time complexity of https://nar-tutorial.github.io/acl2022/ O(1). Fully NAT models mainly follow the Transformer model. In order to predict multiple tokens, the decoder needs to be initialized with an input sequence to indicate the number of tokens it is required to generate. Therefore, most NAT models incorporate a length predictor, which aims to estimate the target sentence length based on the source. The length predictor also helps NAT models to alleviate the multimodality problem by providing an estimated length that includes certain contextual target information. For instance, it is much less likely that a NAT model will generate [Thank .] if given a length of 2 tokens, as Thank comes from [Thank you .], which is of length 3. Gu et al. (2018a) implement the length predictor as a fertility predictor, which takes the encoder output hidden states as input and predicts a fertility value for each input position, indicating how many target tokens should be generated from each source token. The decoder input length is thus the sum of all fertility values. The encoder of a fully NAT model is identical to the Transformer encoder. For the decoder, since the AR property is removed, the causal mask in the decoder self-attention layer (cf., Section 2.2.1) is no longer necessary. However, contrary to AR models, which use teacher forcing during training (cf., Section 2.3.1), the decoder input for NAT models requires an initial input sequence. Gu et al. (2018a) propose to copy the source token embeddings as decoder input by following the fertility information or uniformly, where decoder input t is the copy of the [St/T ]-th encoder input. To enhance the positional information, Gu et al. (2018a) add a positional attention layer in each decoder layer, in which the query and key of Equation ( . ) are the positional encoding vectors, and the value is the decoder states, respectively.

During inference, Gu et al. (2018a) propose noisy parallel decoding in which they sample multiple fertility predictions instead of taking the candidates with the best probability. In this way, they can generate multiple translations of different lengths. However, the noisy parallel decoding further requires an AR model to score and re-rank the translations to find the best candidate. An alternative to the noisy parallel decoding is the length beam decoding [START_REF] Guo | Non-autoregressive neural machine translation with enhanced decoder input[END_REF], which uses the best length prediction L and expands it to [Lk, L + k] with a beam size of 2k + 1.

. In Between Autoregressive and Non-autoregressive

Several approaches have been proposed to mix AR and non-autoregressive translations to better capture the dependencies on the target side. Semiautoregressive translation (SAT) model is AR at the sequence level but can produce a group of tokens in parallel. Wang et al. (2018a) consecutively group K target tokens into one element by concatenating the K vectors into a higher dimension, then compute the probability of the target sequence as:

P (e|f ) = [(T -1)/K]+1 i=1 P (G i |G 1 . . . G i-1 , f ), ( . )
where K is the number of tokens to predict in parallel, and G i is the concatenated vector of e (i-1)⋅K+1 , . . . , e i⋅K . The model can therefore predict K tokens at once. SAT model lies in between AR models and fully NAT models. The model becomes a conventional AR model when K = 1 and turns into a fully NAT model like the one presented in (Gu et al., 2018a) when K is greater than the maximum sequence length. Wang et al. (2018a) also propose to relax the strict decoder causal mask by letting tokens in a group e (i-1)⋅K+1≤j≤i⋅K be able to attend to all tokens in the group G i . [START_REF] Stern | Blockwise parallel decoding for deep autoregressive models[END_REF] propose blockwise parallel decoding, similar to Wang et al. (2018a), but dynamically select the number of tokens predicted in parallel. At each time step, they first generate k consecutive tokens in parallel, then select the first k tokens that match the AR model's output and keep the k tokens for the next step.

Contrary to SAT, the locally AR translation model (LAT, [START_REF] Kong | Incorporating a local translation mechanism into non-autoregressive translation[END_REF] is non-autoregressive at the sequence level but generates a short sequence of tokens at each position autoregressively. It uses a small RNN-based model to predict a short sequence of K tokens at each prediction position. The predicted sequence is supposed to be fluent and well-translated since the AR module measures the local target dependency. However, LAT requires an extra merging algorithm to align and merge the output sequences to obtain the final translation. [START_REF] Kong | Incorporating a local translation mechanism into non-autoregressive translation[END_REF] compute the Longest Common Subsequence (LCS) between two adjacent generated sequences, select matched tokens, and resolve conflicts for other tokens by taking the ones with a higher probability. If no LCS match is found, they simply concatenate the two sequences.

. Iterative Refinement Based Models

Instead of producing all tokens in parallel only once, several methods have been proposed to iteratively adjust the translation to obtain better performance by conducting multiple decoding rounds. We introduce some typical examples below.

Conditional Masked Language Model

The Conditional Masked Language Model (CMLM, [START_REF] Ghazvininejad | Mask-predict: Parallel decoding of conditional masked language models[END_REF] borrows the idea from the masked language model (MLM) training of large pre-trained language models [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Conneau | Cross-lingual language model pretraining[END_REF]. MLM masks out certain tokens e mask in an input sequence by replacing them with a special [mask] token and aims to recover the masked tokens on the output by conditioning on the other unmasked tokens e obs and predicting the individual probabilities P (e|e obs ) for each e ∈ e mask . CMLM extends this idea under the translation scenario by conditioning on both the source sentence and the unmasked tokens, thus computing the probability P (e|f , e obs ). The model architecture again follows the Transformer model and removes the decoder causal mask. During training, CMLM randomly masks certain tokens. The ratio of masked tokens is uniformly randomly sampled between 0 and 100%. This differs from MLM models, which use a constant masking rate of 15%. Similar to Gu et al. (2018a), CMLM needs to know the target sequence length to place [mask] tokens. Once again, CMLM follows [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] to prepend a special [length] token to the source sentence and use the hidden states of this [length] token from the encoder output to estimate the length of the target sentence.

The decoding procedure of CMLM does not finish in one single decoding step. Instead, [START_REF] Ghazvininejad | Mask-predict: Parallel decoding of conditional masked language models[END_REF] propose a mask-predict method to generate the final output within a constant number of cycles. At the initial iteration, CMLM predicts the target sentence length from the encoder and initializes the decoder with all [mask] tokens to generate all tokens together. For the later iterations, it selects n tokens with the lowest output probabilities, masks these tokens with [mask], then restarts another iteration to predict the masked tokens until the predefined number of iteration I is reached. The number of tokens n to be masked varies across iterations with a linear decay:

n = L ⋅ I -i I , ( .
)
where L is the predicted target length, and i is the iteration number. The decoding methods, such as mask-predict, which repeatedly adjust the output translations, are also known as iterative refinement [START_REF] Lee | Deterministic non-autoregressive neural sequence modeling by iterative refinement[END_REF]. CMLM is able to perform NAT in constant decoding rounds, despite the length of generated sentences, and delivers better translation quality compared to fully NAT models. To explore more candidates during inference, CMLM can sample multiple target lengths or use the top-k predictions to generate several translations. The translation with the best average log probability is finally picked.

Insertion Transformer

CMLM must know the target sequence length in advance to predict tokens, and the length cannot be changed once it is predicted. The Insertion Transformer [START_REF] Stern | Insertion transformer: Flexible sequence generation via insertion operations[END_REF] instead generates translations via insertion operations. It jointly produces pairs of a target token and the corresponding location in the sequence to insert the token. The target sequence always contains a begin-of-sentence and an end-of-sentence token, and the actual sequence length of a target with n tokens is thus n+2. The number of possible insertion slots is thus n + 1. Insertion Transformer first concatenates the decoder output hidden states to the left and right of each insertion slot as one vector to build the slot representation. The slot matrix H ∈ R (n+1)×d con-

taining n + 1 slot

Levenshtein Transformer

The masking step in CMLM decoding, which masks tokens with the lowest probabilities, can be seen as a kind of deletion operation to cross out unconfident predictions. The Levenshtein Transformer (LevT, [START_REF] Gu | Levenshtein transformer[END_REF] can be viewed as a generalization of CMLM and the Insertion Transformer, as it models possible insertion and deletion operations on target sequences. LevT is able to edit the target sequence during generation to arbitrarily change the target length and iteratively refine the output sequence in a dynamic way.

Placeholder Insertion Unlike the Insertion Transformer, which models the joint probability of insertion slot and content token, LevT separates it into two steps. The first step predicts how many tokens should be inserted for each possible insertion slot. Similar to the Insertion Transformer, LevT also concatenates the decoder state to the left and right of each slot as a single vector, then passes it through a placeholder classifier, which simply predicts the number of tokens to insert for each insertion slot. LevT then modifies the initial target sequence by adding predicted corresponding placeholder tokens to each slot, as illustrated in Figure 4.1. Thanks to this placeholder insertion step, LevT is able to insert multiple tokens in one slot within the same decoding step.

Token Prediction Once placeholders are inserted, LevT then performs similarly to CMLM to predict an output token for each placeholder, conditioned on the source and existing target tokens.

Deletion After token prediction, LevT incorporates a deletion operation in which the target sequence decoder states are passed through a deletion classifier to decide whether each token should be deleted. Figure 4.1 also illustrates the deletion operation.

LevT encoder

Un chat dort . Training With the above operations, LevT is able to make edits to the target sequence by iteratively performing the insertion and deletion operations. It does not need an extra length predictor as the length of the target sequence is dynamically adjusted during generation. However, training the LevT model requires proper training data for each operation. Specifically, the placeholder insertion module needs to learn how to insert tokens, while the deletion module needs to detect errors from a target sequence. All three operations take the same decoder architecture but only differ in the output linear classification layer. However, since the decoder input for each operation is different, each operation needs to recompute the input through the decoder. The decoder parameters for each operation can be shared or separately trained.

The training process is performed sequentially, as illustrated in Figure 4.1. Similar to CMLM, [START_REF] Gu | Levenshtein transformer[END_REF] first randomly drop tokens from the reference e to build the input sequence e

′ for placeholder insertion. The only difference is that CMLM replaces the dropped tokens with a mask token and keeps the target sequence length unchanged, while LevT actually deletes those tokens, yielding a subsequence e

′ of e. The reference placeholder numbers for each position can be obtained by computing the edit distance between e

′ and e. The reference for the token prediction operation can also be computed at the same time. The input e ′′ thus must always be the oracle placeholder insertions. However, for the deletion operation, [START_REF] Gu | Levenshtein transformer[END_REF] do not generate any synthetic sentences but directly use the prediction made by token prediction operation as input. Once again, reference deletion labels are obtained by computing the edit distance between e

′′′ and e.

Inference During inference, LevT starts with an empty target sequence and generates the translation by alternatively performing deletion and insertion (placeholder insertion + token prediction) operations until convergence or a maximum decoding round is reached. In the first iteration, the deletion is omitted as nothing can be deleted from an empty sequence. The iterative refinement converges when the input and output of one iteration are the same, as it predicts nothing to delete and nothing to insert in an iteration, or it encounters a loop where the deleted tokens are reinserted in a single iteration.

. Training Objectives

Most NAT models are trained with the standard cross-entropy (CE) loss as:

L CE = - T i=1 log P (e i |f ; θ), ( . )
where the probability P (e i |f ; θ) is computed independently for each token given the source f . CMLM and LevT measure the prediction CE loss only on the masked segments but not the entire sequence. These models only predict tokens for positions where the input is a mask or a placeholder. Therefore, computing loss on other unmasked positions is meaningless. The CE loss is also the objective function used to train placeholder insertion and deletion for LevT. The CE loss measures the exact match at each token position, thus giving high penalties to output sequences with small order shifts. However, it is very difficult for NAT models to correctly predict the exact token at each position due to the multimodality issue as several word orders may be equally correct. To alleviate this issue, several approaches propose to consider latent alignments between the output target sequence and the reference using alternative training objectives, aiming to ease the strict match at each position of the CE loss.

. Connectionist Temporal Classification

Libovický and Helcl (2018) apply the Connectionist Temporal Classification (CTC, [START_REF] Graves | Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks[END_REF]) loss for NAT. CTC is first applied in automatic speech recognition systems. Assuming an input sequence x and a reference sequence y, in which x is longer than y (|x| ≥ |y|), CTC aims to label the input sequence x with tokens in y and empty tokens and find all valid monotonic alignments that can recover the reference sequence y after removing empty tokens and collapsing consecutive repeated tokens. CTC suits speech recognition tasks well, as the input sequence of phonemes is always longer than the output sequence of lexical tokens, and many phonemes may align to the same token or to nothing when there are pauses in the speech. For NAT, CTC allows NAT models to freely choose the best prediction regardless of strict position constraints. As CTC allows predicting empty tokens, the number of possible combinations of reference and empty tokens of an output length L and a reference length T is thus L T . CTC is able to efficiently compute the CE loss summed over all possible alignments using dynamic programming:

L CTC = - a∈β(e) L i=1 P (a i |f ; θ), ( . )
where a is a possible alignment, and β(e) refers to all possible alignments. However, unlike speech signals, the source sentences in translation may be shorter than the target. To match the CTC requirement, [START_REF] Libovický | End-to-end non-autoregressive neural machine translation with connectionist temporal classification[END_REF] upsample the source sentence k times of length S by slicing each encoder output state into k vectors. The obtained sequence of length L = k ⋅ S is used to initialize the decoder. In practice, NAT models often use k = 2 or k = 3.

CTC-based model [START_REF] Saharia | Non-autoregressive machine translation with latent alignments[END_REF][START_REF] Gu | Fully non-autoregressive neural machine translation: Tricks of the trade[END_REF]) is now one of the state-of-the-art models for fully NAT systems.

. Aligned Cross-entropy

Ghazvininejad et al. (2020) focus on penalizing lexical errors, which indicate wrong tokens, rather than word order errors. The latter refers to correct tokens appearing in the wrong positions. Given a reference sequence e = e 1 , . . . , e T and a predicted sequence of probability distributions P = P 1 , . . . , P L , [START_REF] Ghazvininejad | Aligned cross entropy for non-autoregressive machine translation[END_REF] aim to find a monotonic alignment between these two sequences a ∶ {1, . . . , T } → {1, . . . , L} which minimize the aligned cross-entropy (AXE) loss:

L AXE = - T i=1 log P a(i) (e i ) - k∉a log P k ( ), ( . )
where is an empty token allowed during training but removed in the output translation. The first term is an aligned cross-entropy between e and P, and the second term is a penalty for unaligned predictions. The search for the optimal alignment is performed using dynamic programming.

. Order-agnostic Cross-entropy

CTC and AXE losses focus on the monotonic alignment between prediction and reference sequences. [START_REF] Du | Order-agnostic cross entropy for non-autoregressive machine translation[END_REF] extend the AXE loss to allow reordering in the alignment to find the overall best possible alignment. They propose the Order-agnostic cross-entropy (OAXE) loss, which almost removes word order errors and focuses on lexical matching. OAXE assumes the prediction sequence has the same length as the reference, similar to CE loss. However, instead of measuring the standard CE loss, OAXE aims to find a permutation of the reference tokens so that the reordered reference sequence can obtain a minimum CE loss:

L OAXE = arg min O i ∈O (-log P (O i |f )), ( . )
where O i is a possible reordering or permutation of the original reference,

-log P (O i |f )
is the CE loss of a reordered reference. [START_REF] Du | Order-agnostic cross entropy for non-autoregressive machine translation[END_REF] apply the Hungarian matching algorithm [START_REF] Kuhn | The hungarian method for the assignment problem[END_REF] to efficiently search for the optimal permutation.

. Learning Paradigms

Directly learning to generate the complete translation is a difficult task for NAT. Some works have studied training NAT models with a better curriculum so that models gradually learn to produce more and more complicated sequences. et al. (2021) propose the Glancing Transformer (GLAT). The idea is similar to [START_REF] Mihaylova | Scheduled sampling for transformers[END_REF], who perform two-pass decoding to mix model predictions and ground-truth tokens. GLAT aims to reveal certain reference words in the decoder input to help the model focus on generating segments instead of the entire sentence. It initializes the decoder input by a uniform copy of the encoder output states for the first decoding round. In this step, no target token is revealed to the decoder. GLAT then computes the distance d(ê, e) between the predicted sequence ê and the reference e and randomly selects S = λd(ê, e) reference tokens, with λ a hyperparameter. The selected tokens are then revealed in the decoder input by replacing the corresponding encoder output states with the target token embeddings. The mixed decoder input sequence is then passed through a second decoding pass to compute losses only for unrevealed segments and update the model parameters.

. Glancing Transformer

Qian

The number of revealed tokens is thus dynamically adjusted through the training process, guiding the model to learn from an easier task to a harder one. At an early stage, the model is weak and will generate predictions that have a high distance to the reference. Therefore, more ground-truth tokens are revealed, so the model only focuses on predicting small segments with a larger context. As training proceeds, the model becomes more powerful and predicts translations that are closer to the reference. Therefore, fewer tokens are revealed, and the model is trained to predict longer segments.

. Multi-granularity Training

. Pre-training

Large pre-trained language models benefit AR NMT models [START_REF] Conneau | Cross-lingual language model pretraining[END_REF][START_REF] Edunov | Pre-trained language model representations for language generation[END_REF][START_REF] Clinchant | On the use of BERT for neural machine translation[END_REF][START_REF] Weng | Acquiring knowledge from pre-trained model to neural machine translation[END_REF]Yang et al., 2020a;[START_REF] Zhu | Incorporating bert into neural machine translation[END_REF]Yang et al., 2020b). Several studies have also explored how NAT benefits from pre-trained language models. [START_REF] Guo | Incorporating bert into parallel sequence decoding with adapters[END_REF] use two BERT models, one for the encoder initialization and the other for the decoder. They add two feed-forward networks as an adapter layer [START_REF] Bapna | Simple, scalable adaptation for neural machine translation[END_REF] in each BERT encoder layer and a cross-attention layer in each decoder layer to perform CMLM and show improvements with respect to CMLM models trained from scratch on both high-resource and low-resource language pairs. Su et al. ( 2021) use a single BERT model and add a conditional random field output layer to better model the target dependency. Their model only contains an encoder without a decoder. [START_REF] Li | Universal conditional masked language pre-training for neural machine translation[END_REF] pre-train a multilingual model by jointly training on MLM task for the encoder and CMLM task for the decoder. The intuition is to pretrain a multilingual encoder and a bidirectional decoder to benefit from bilingual and monolingual data of various languages. A pre-trained bidirectional decoder may achieve better results on AR translation [START_REF] Conneau | Cross-lingual language model pretraining[END_REF]. As illustrated in Figure 4.2, they first create mixed-language (MXL) source sentences by randomly replacing source tokens, which have aligned tokens on the target side, with the aligned words from a third language. The aligned words are obtained from a multilingual translation dictionary. At the same time, target tokens aligned to the replaced source tokens are masked with a [mask] token. After this, they also apply random dual-masking to both source and target sequences, similar to MLM. The resulting data consists of a mixed-language source sequence with masks on its original language tokens and a target sequence with masks from both steps. [START_REF] Li | Universal conditional masked language pre-training for neural machine translation[END_REF] also use monolingual data, setting the initial target sentence identical to the source. The source sequence passes through the encoder and is trained on the MLM task like BERT. The target sequence passes through the decoder like CMLM. Both tasks are jointly trained with a combined loss during the pre-training stage. The pre-trained model is suitable for fine-tuning on both standard AR translation and NAT with the CMLM approach and obtains performance gains for both tasks under both low-resource and high-resource scenarios. ). In the first step, the French word "Je" is aligned to "I" in English and to "我" in Chinese. Therefore, "Je" is replaced with "我", and "I" is masked. In the second step, tokens from both sentences are randomly masked.

. Knowledge Distillation

Knowledge Distillation [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF] is first proposed to distill knowledge from a strong teacher model into a relatively weak student model. The student model is trained to match the output probability distribution of the teacher model by minimizing the CE loss and the distance between the output probability distributions of the teacher and student model. Sequencelevel knowledge distillation (KD, [START_REF] Kim | Sequence-level knowledge distillation[END_REF] works differently. The teacher model translates all training source sentences f into the target language via beam search. The obtained higher-quality translation ê is called distilled data. Student models are then trained on the new dataset D = {(f , ê) i , i = 1, . . . , N } by considering the distilled data ê as reference. In this thesis, we abuse the term KD to denote sequence-level knowledge distillation since we are not concerned with the original knowledge distillation. KD is almost similar to the so-called "forward translation" or "self-training" [START_REF] Zhang | Exploiting source-side monolingual data in neural machine translation[END_REF] as they all use translated data as synthetic references to train new models.

KD is first applied to NAT models by Gu et al. (2018a), who train the NAT models only with KD data and found that KD significantly improves the NAT performance. Since then, KD has been almost the default technique applied to most NAT systems. Several works have targeted analyzing why KD can improve NAT models. [START_REF] Zhou | Understanding knowledge distillation in non-autoregressive machine translation[END_REF] show that KD reduces the complexity of training data, which is also observed by Crego and Senellart (2016). KD serves as a translation selection process by generating only one of the possible translations with increased consistency. Therefore, KD helps to mitigate the multimodality problem of NAT by providing more consistent translations. For example, if [Merci .] is always translated into [Thank you .] during training, then NAT models will be less likely to generate tokens that belong to the sentence [Thanks .]. Zhou et al. ( 2020) also find that distilled data is more monotonic with respect to the source sentence. In other words, the target tokens are more likely to appear in the same relative order as the source tokens, and distilled data contains fewer reorderings than the real reference. Xu et al. (2021b) further demonstrate that reducing the amount of reordering helps NAT models to better learn the alignment between source and target tokens. They also find that if source tokens are pre-reordered to match the target order [START_REF] Qiu Ran | Guiding nonautoregressive neural machine translation decoding with reordering information[END_REF], KD makes no improvements compared to real parallel data. Lastly, Xu et al. (2021b) also find that KD reduces the lexical diversity on the target side, which also eases the learning of source-target alignment.

However, KD is time-consuming and strongly affected by the capacity of teacher AR models. Errors made by the AR models will be propagated to NAT models during training, especially for low-frequency words, which the AR models often fail to properly generate, as pointed out by Ding et al. (2021b). They study how to expose the raw data lexical knowledge to NAT models during training by incorporating extra alignment information between source tokens and real target tokens in order to help NAT models with better lexical choices. Huang et al. (2022) aim to correct the training-inference mismatch in CMLM by replacing certain tokens in the initial target sequence e obs with predictions made by the model itself in another decoding round, starting with an all-masked input to better simulate the inference scenario. This strategy is similar to the scheduled sampling method [START_REF] Mihaylova | Scheduled sampling for transformers[END_REF], as introduced in Section 2.3.1. It exposes the model to repeated tokens produced by the model in e obs during training so that the model can learn to refine its prediction, therefore improving the performance when trained on both real and distilled data. Most existing works studying the integration of augmented sources into NAT focus on incorporating lexical constraints for NAT. Surprisingly, all these works are based on the LevT model (cf., Section 4.1.3), which supports edit operations during the generation process. [START_REF] Susanto | Lexically constrained neural machine translation with Levenshtein transformer[END_REF] first experiment by simply using the lexical constraints as the initial sequence for the LevT inference. Instead of the standard LevT, which always starts with an empty target sequence, the initial target input to the decoding is a sequence containing all constraints. This approach does not change the training of LevT and is only applied for inference. Only initializing the decoding with constraints implies soft constraints, as the model may delete these tokens during decoding. It is also possible to perform hard constraints by disallowing deleting constraint tokens and preventing insertions in the middle of consecutive constraint tokens. Xu and Carpuat (2021a) propose EDITOR, which replaces the deletion operation in the LevT model with a repositioning operation. It simultaneously deletes a token and inserts it at another position. EDITOR also takes the lexical constraints as initial input. However, it allows the initial constraints to be provided in arbitrary order as the repositioning operation is able to adjust constraints to better fit a translation. Xu and Carpuat (2021b) further incorporate a rule-based morphological inflection module, allowing the initial constraints to be in lemma format without explicit inflection. [START_REF] Zeng | Neighbors are not strangers: Improving non-autoregressive translation under low-frequency lexical constraints[END_REF] pay particular attention to low-frequency constraints as they are likely to be deleted with soft constraints decoding. They simulate training with rarer words when generating initial subsequences for the LevT. They first sample several tokens from the reference as pseudo constraints based on TF-IDF scores (term frequency-inverse document frequency). These tokens are then prevented from being deleted when generating the initial subsequence. [START_REF] Zeng | Neighbors are not strangers: Improving non-autoregressive translation under low-frequency lexical constraints[END_REF] further provide the alignment information of target constraints (pseudo constraints for training and real constraints in inference) with the aligned source tokens by adding an extra alignment embedding to the source tokens. The alignment embedding values indicate for each source token the relative order index of the aligned target constraints.

The above approaches focus on lexical constraints for translation, while [START_REF] Wan | Incorporating terminology constraints in automatic postediting[END_REF] explore lexical constraints for automatic post-editing. They propose a multi-source LevT with a second encoder taking an external MT output as input, and they initialize the decoder with target constraints, as proposed by [START_REF] Susanto | Lexically constrained neural machine translation with Levenshtein transformer[END_REF]. They also explore a single-encoder approach where the source is processed as [START_REF] Dinu | Training neural machine translation to apply terminology constraints[END_REF] to incorporate target constraints and initialize the LevT decoding by MT output.

. Non-autoregressive Translation for Word-level Quality Estimation

Ding et al. (2021c) adapt the LevT to perform the word-level quality estimation. The task is to predict: (a) whether each predicted target word is correct or not; (b) whether there are missing words between each pair of predicted words. These tasks perfectly suit LevT: predicting target word correctness is somewhat equivalent to the deletion operation and predicting missing words is similar to the placeholder insertion module. Therefore, Ding et al. (2021c) fine-tune a pre-trained LevT model specifically on these tasks with some task adaptations and achieve good performance when the finetuning data is limited.

. Conclusion

In this chapter, we reviewed NAT from various aspects like model architectures, training objectives, learning paradigms, the use of KD, and NAT with augmented sources. However, there are still some approaches like NAT with latent variable and variational auto-encoder [START_REF] Ma | FlowSeq: Non-autoregressive conditional sequence generation with generative flow[END_REF][START_REF] Shu | Latent-variable non-autoregressive neural machine translation with deterministic inference using a delta posterior[END_REF][START_REF] Lee | Iterative refinement in the continuous space for non-autoregressive neural machine translation[END_REF][START_REF] Bao | latent-GLAT: Glancing at latent variables for parallel text generation[END_REF] that are not covered as we are not concerned with these methods in this thesis. NAT is still a new research area in NMT, as many unresolved problems exist. The key problem is the conditional independence assumption that makes NAT models difficult to model the target dependencies. Besides, despite several analyses about the effectiveness of KD for NAT, we think this issue still requires further studies to thoroughly understand why real references perform worse than KD. We believe that the eventual purpose of these studies is to get rid of KD and solely use real references to train NAT models.

In this chapter, we paid particular attention to the LevT-based approaches, as these models can perform edits to an initial target sequence during the translation process. We will also explore LevT-based methods to synchronize bitexts by editing one of the two languages in Chapter 8.

-Disentangling Mixed-Language Sentences

with Dual Decoding

. Introduction

As introduced in Chapter 1, one possible way to perform bilingual writing is to write a sentence in mixed-language (MXL), then translate the MXL sentence into monolingual sentences in both component languages. Showing texts in both a foreign language (L2) and the users' mother tongue (L1) may help the users to verify the composed texts. Besides, the existing text segments in an MXL sentence produced by the users with high confidence can provide valuable contextual information to control the translation into L2 and produce better translations. Therefore, in this chapter, we study the task of translating MXL sentences into monolingual sentences in both languages.

MXL sentences are similar to Code-Switching, which denotes the alternation of two languages within a single discourse, utterance, and sometimes words [START_REF] Poplack | Sometimes i'll start a sentence in spanish y termino en español: toward a typology of code-switching 1[END_REF][START_REF] Sitaram | A survey of code-switched speech and language processing[END_REF]. We mostly consider sentencelevel language mixing in this thesis. Code-Switching is a common and essential communicative phenomenon that occurs in multilingual communities during informal spoken and written interactions. It generally consists of short inserts of a secondary language that are embedded within larger fragments in the primary language. Code-Switching is still an understudied domain in natural language processing [START_REF] Aguilar | From English to code-switching: Transfer learning with strong morphological clues[END_REF], and most work to date has focused on token-level language identification (LID, [START_REF] Samih | Multilingual code-switching identification via LSTM recurrent neural networks[END_REF] and language modeling for automatic speech recognition [START_REF] Indra Winata | Code-switched language models using neural based synthetic data from parallel sentences[END_REF]. More tasks have been considered lately, such as named entity recognition [START_REF] Aguilar | Named entity recognition on codeswitched data: Overview of the CALCS 2018 shared task[END_REF], part-of-speech tagging [START_REF] Ball | Part-of-speech tagging for codeswitched, transliterated texts without explicit language identification[END_REF], or sentiment analysis [START_REF] Patwa | SemEval-2020 task 9: Overview of sentiment analysis of codemixed tweets[END_REF]. Even though similar to Code-Switching, the MXL sentences we deal with in this chapter are slightly different, as Code-Switching often refers to informal texts while MXL sentences are relatively more formal. Moreover, we intend to study interactive systems under a simulated situation. In our case, MXL sentences are artificial and not realistic. Therefore, we distinguish the term MXL sentences from Code-Switching and use MXL sentences in this chapter.

Little attention has been paid to the task of machine translation (MT) for MXL input. [START_REF] Johnson | Google's multilingual neural machine translation system: Enabling zero-shot translation[END_REF] mention using a multilingual neural machine translation (NMT) system to translate an MXL sentence to a third target language by showing only one example. [START_REF] Dinu | Training neural machine translation to apply terminology constraints[END_REF] perform some sort of MXL translation in their replacement setting, as introduced in Section 3.4.2. However, they mainly focus on translating target terminologies, thus only translating the MXL sentence into one of the two languages. [START_REF] Menacer | Machine Translation on a parallel Code-Switched Corpus[END_REF] release a parallel Arabic-English MXL corpus dedicated to MT applications. This MXL data is extracted from the United Nations data: while translations into English are readily available, the purely Arabic side of the corpus is obtained using Google Translate to fill the missing Arabic bits. [START_REF] Gupta | Training data augmentation for code-mixed translation[END_REF] manage to generate synthetic MXL data from parallel bilingual corpora. They first fine-tune a multilingual BERT model on an MXL dataset to perform token-level LID. To create synthetic MXL sentences, they use the fine-tuned LID model to predict possible positions to inject target words, then replace the source words with the aligned target ones. Very recently, a shared task on MT in Code-Switching settings has been held [START_REF] Chen | Calcs 2021 shared task: Machine translation for code-switched data[END_REF].

As discussed in Section 3.5.1, the multi-source way [START_REF] Franz | Statistical multi-source translation[END_REF][START_REF] Schwartz | Multi-source translation methods[END_REF][START_REF] Zoph | Multi-source neural translation[END_REF] to handle this task of translating one source language into two target languages generates the first translation in a target language l T 1 , which, once revised, can be used in conjunction with the original source to generate the translation in the other language l T 2 . The expected benefit of this approach is to facilitate word disambiguation. However, this approach risks cascading errors to the second translation unless the translation in l T 1 is thoroughly revised before the second translation round. The advent of NMT technologies [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF][START_REF] Vaswani | Attention is all you need[END_REF] has made it possible to design multilingual models capable of translating from multiple source languages into multiple target languages (Firat et al., 2016a;[START_REF] Johnson | Google's multilingual neural machine translation system: Enabling zero-shot translation[END_REF].

I return home because I am tired.

Je rentre à la maison parce que je suis fatigué.

I rentre à la maison because I am tired. In this chapter, we conduct dual decoding to simultaneously translate an MXL input, denoted as a new language l S , into both component languages, l T 1 and l T 2 . Figure 5.1 gives an illustration. While the same goal of generating monolingual sentences from MXL inputs can be achieved with a multilingual system translating an MXL sentence independently into l T 1 and l T 2 , we expect several payoffs from a dual decoding:

• Obtaining two translations in l T 1 and l T 2 via a single decoding process;

• Better translation qualities in both languages;

• More consistent translations in l T 1 and l T 2 than if they are performed independently.

We also perform several analyses to better understand the task of translating MXL texts into monolingual sentences. [START_REF] Dinu | Training neural machine translation to apply terminology constraints[END_REF] suggest that target tokens injected into the source sentence can be considered as lexical constraints. In our case, when simultaneously translating MXL texts into two languages, it can be considered that lexical constraints in both languages are present at the same time. Therefore, we expect the following "copy" constraint to be satisfied: every word in the MXL source text should appear in at least one of the two outputs. In the bilingual writing scenario, the copy constraint also indicates that texts composed by the users in either language should be preserved in the output translations.

Our main contributions are the following:

• We conduct a comparative study of architectures for MT of MXL input and propose a dual decoder model. • We propose a synthetic MXL data generation method and demonstrate the artificial data's effectiveness in training NMT models. • In our systems, the translation of MXL texts is almost as good as the translation of monolingual texts -a performance that regular systems are unable to match. • MXL translation systems achieve a near-deterministic ability to recopy in the output target words found in the input, suggesting that they are endowed with some LID abilities. • We are also able to obtain competitive results on the SemEval 2014 Task 5: L2 Writing Assistant with our model, which we see as one potential application area of MXL translation.

The contributions in this chapter are published in (Xu and Yvon, 2021a,b).

. An Architecture for Dual Decoding

In our setting, we consider the simultaneous translation of sentence f in source language l S into two target sentences e

1 and e 2 in languages l T 1 and l T 2 . In this situation, various modeling choices can be entertained [START_REF] Le | Dual-decoder transformer for joint automatic speech recognition and multilingual speech translation[END_REF]: where

P (e 1 , e 2
T = max(|e 1 |, |e 2 |
), and we use placeholders whenever necessary. The factorization in Equation ( . ) implies a joint event space for the two languages and a computational cost we deem unreasonable. Instead, we resort to the second (dual ) formulation that we contrast with the third one (independent generation) in our experiments. Taking e 1 as an example, in dual decoding, the probability of e [START_REF] Le | Dual-decoder transformer for joint automatic speech recognition and multilingual speech translation[END_REF]. The asynchronous decoding is explored in Chapter 6.

. Dual Decoder Model

We implement Equation ( . ) with a synchronous coupling of two decoders sharing the same encoder. Our dual decoder model is inspired by [START_REF] Wang | Synchronously generating two languages with interactive decoding[END_REF]. Figure 5.2 illustrates this design. Compared to a standard Transformer, we use the same encoder, but we add a decoder cross attention layer in each decoder block to capture the interaction between the two decoders. We denote the output hidden states of the previous layer for each decoder as H 1 l and H 2 l . The decoder cross-attention is computed as:

H 1 l+1 = MultiHead(H 1 l , H 2 l , H 2 l ) H 2 l+1 = MultiHead(H 2 l , H 1 l , H 1 l ), ( . )
where MultiHead is the multi-head attention described in Equation ( . ). For simplicity, we omit the other sub-layers (self-attention, encoder-decoder cross attention, feed-forward, and layer normalization). The two decoders are thus fully synchronous as each requires the hidden states of the other in each block to compute its own hidden states. Similar to the original decoder self-attention layer in the Transformer model (cf., Section 2.2.1), we also apply the attention mask in the decoder cross-attention layer to prevent the decoders from seeing future information from the other decoder. The decoder cross-attention can be inserted before or after the encoder-decoder attention block. Our preliminary experiments with these variants showed that they were performing similarly. We thus only report results obtained with the decoder cross-attention as the last attention component of a block, as illustrated in Figure 5.2.

. Synchronous Beam Search

Our decoding algorithm uses a dual beam search. Assuming each decoder uses its own beam of size k, the cross-attention between decoders can be designed and implemented in multiple ways: for instance, in each decoder layer, one can have each hypothesis in decoder 1 attend to all hypotheses in ? ? decoder 2 and vice versa, creating k intermediate representations for each hypothesis. In this way, each hypothesis at each decoder layer yields k candidates, which will, however, create an exponential blow-up of the search space in the final decoder layers. Following [START_REF] Zhou | Synchronous bidirectional neural machine translation[END_REF], we only compute the attention between the 1st-best candidates of each decoder, the 2nd-best candidates of each decoder, etc. This heuristic ensures that the number of candidates in each decoder beam remains fixed, as illustrated in Figure 5.3. However, there is an added complexity since the ranking of hypotheses in each decoder beam evolves over time: the best hypothesis in decoder 1 at time step t may no longer be the best at time step t + 1. Preserving the consistency of the decoder states, therefore, implies recomputing the entire prefix representation for each hypothesis and each decoder at each time step, thus creating a significant computing overhead. In our experiments, the dual decoder is about twice as slow as the two independent ones. We also explored other implementations where all candidate prefixes in one beam always attend to the best candidate in the other beam or the averaged hidden states of all candidates from the other beam. These variants deliver very similar results, as also found in [START_REF] Zhou | Synchronous bidirectional neural machine translation[END_REF]. For simplicity reasons, we use the first scheme in all experiments.

. Training

Training the dual decoder model requires triplets of instances comprising one source and two targets. Given a set of such examples D = {(f , e 1 , e 2 ) i , i = 1 . . . N }, the training maximizes the combined log-likelihood for the two tar-get sequences:

L(θ) = D ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ |e 1 | t=1 log P (e 1 t |e 1 <t , e 2 <t , f ; θ) + |e 2 | t=1 log P (e 2 t |e 2 <t , e 1 <t , f ; θ) ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ , ( .
)
where θ represents the set of parameters.

. Mixed-language Data Generation

There exists large-scale bilingual parallel data in the community. However, parallel corpora aligned with natural code-switched or MXL data are very scarce [START_REF] Menacer | Machine Translation on a parallel Code-Switched Corpus[END_REF]. Therefore, similar to [START_REF] Song | Code-switching for enhancing NMT with pre-specified translation[END_REF], we generate artificial MXL sentences from regular translation data.

The Matrix Language Frame (MLF) theory (Myers-Scotton, 1997) defines the concept of matrix and embedded languages where the matrix language is the primary language that the sentence structure should conform to and notably provides the syntactic morphemes, while the influence of the embedded language is less evident and is mainly manifested in the insertion of content morphemes. We abuse the terms "matrix" and "embedded" language here, as we do not attempt to generate linguistically realistic code-switched sentences matching the constraints of the MLF theory. We use these terms in a much looser sense where the "matrix" language in an MXL sentence is the one that receives arbitrary insertions from the "embedded" language. This means that our artificial MXL sentences will contain insertions of unconstrained fragments containing both content and function words, which the theory would generally consider ungrammatical.

Taking a bilingual parallel corpus, we first compute word alignments between parallel sentences using fast_align 1 [START_REF] Dyer | A simple, fast, and effective reparameterization of IBM model 2[END_REF]. We then extract so-called minimal alignment units following the approach of Crego et al. (2005). These units correspond to small bilingual phrase pairs (f , e) extracted from symmetrized word alignments such that all alignment links outgoing from words in f reach a word in e, and vice-versa. In Appendix A, we briefly introduce this extraction method.

For each pair of parallel sentences, we first randomly select one language as the matrix language; then randomly sample the number of replacements r to appear in a derived MXL sentence with an exponential distribution as:

P (r = k) = 1 2 k+1 ∀k = 0, . . . , R, ( . )
where R is a predefined maximum number of replacements. We repeat the sampling process with a probability P (redo) = 1 2 R+1 to ensure the probabilities sum to 1. We also make sure that the number of replacements does not exceed half of either the original sentence length from both parallel sentences, adjusting the actual number of replacements as:

n = min( S 2 , T 2 , r), ( . )
where S and T are the lengths of the parallel sentences. We finally choose uniformly at random r alignment units and replace these fragments in the matrix language with their counterpart in the embedded language.

. Experiments

. Datasets

We conduct experiments with two language pairs: English-French (En-Fr) and English-Spanish (En-Es). We use WMT14 En-Fr 3 and WMT13 En-Es 4 data for MXL data generation and training NMT systems. We discard sentences that do not possess the correct language using the fasttext LID model 5 [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF]. We use Moses tools (Koehn et 2007) to normalize punctuation, remove non-printing characters and discard sentence pairs with a source/target ratio higher than 1.5, with a maximum sentence length of 250. We tokenize all WMT data using the Moses tokenizer.

6 For each language pair, we use a shared source-target vocabulary built with a joint Byte Pair Encoding (BPE) of 32K merge operations, using the implementation published by Sennrich et al. (2016c).

7

The actual data we use after preprocessing contains 33.9M parallel sentences for En-Fr and 14.5M for En-Es. The corresponding test sets are newstest2014 (En-Fr) and newstest2013 (En-Es). For each pair of sentences in the training data, we generate an MXL sentence using the method described in Section 5.3 and consider the generated MXL data as a new source language for translation. We also generate an MXL version of each test set, which we name as mxl-newstest2014 and mxl-newstest2013, respectively, and use it for evaluation. Approximately half of the test sentences are mostly English with inserts in French or Spanish, and mostly French or Spanish with inserts in English for the other half.

. Machine Translation Systems

We implement the dual decoder model using fairseq 8 (Ott et al., 2019), 9 with a hidden size of 512 and a feed-forward size of 2,048. We optimize with Adam (Kingma and Ba, 2015), set up with an initial learning rate of 0.0007 and an inverse square root weight decay schedule, as well as 4,000 warm-up steps. All models are trained with mixed precision and a batch size of 8,192 tokens for 300k iterations on 4 V100 GPUs. For standard Transformer models, we share the decoder input and output matrices [START_REF] Press | Using the output embedding to improve language models[END_REF][START_REF] Inan | Tying word vectors and word classifiers: A loss framework for language modeling[END_REF]. For dual decoder models, each decoder contains an input and an output matrix. Therefore, we share all four input and output decoder matrices.

We call the dual decoder models dual. To study the effectiveness of dual decoding, we compare dual models to several approaches:

• The base setting, where we train two separate Transformer models, one translating MXL into English, and the other translating MXL into Spanish or French. • The multi setting, where we train one model able to generate either pure matrix or embedded language in the output. To this end, similar to a multilingual NMT model [START_REF] Johnson | Google's multilingual neural machine translation system: Enabling zero-shot translation[END_REF], we add a tag at the beginning of each MXL sentence to specify the desired target language. Taking En-Fr as an example, we add a <EN> tag for MXL-En and a <FR> tag for MXL-Fr. We use the combination of MXL-En and MXL-Fr data for training, which implies that each source side (MXL sentence) is duplicated in the training data once for each possible output.

• The indep setting, which extends multi by using one encoder and two separate decoders and training on the two output languages simultaneously. indep is a simplified multi-task version [START_REF] Dong | Multi-task learning for multiple language translation[END_REF] of the dual model, implementing the independent decompositon of Equation ( . ) without the decoder cross-attention layer. For indep, the only interaction between the two outputs is thus a shared loss function: for each training instance, the loss function sums the two prediction terms for the embedded and the matrix language. The training data remains the same. Note that since the number of replacements described in Equation ( . ) has P (r = 0) = 0.5, informing that half of the generated data remains unchanged, all systems described above also have the ability to translate monolingual source data in either direction. Table 5.1 summarizes the architecture and data of different models used in this chapter.

For comparison purposes, we also use the original parallel data to train two baselines systems:

• The base-mono setting, which consists of regular Transformer systems for the considered language pairs (En↔Fr and En↔Es), similar to base. • The bilingual setting, which is one model capable of translating from and into both two languages, similar to multi. The selection of the desired target language relies on the same tagging mechanism as multi, which means that both types of models see exactly the same examples on the target side. The only difference with multi is thus that this system does not see any MXL texts on the source side during training. All resulting baseline Transformer models have exactly the same hyperparameters and use the same training scheme as base and multi approaches. Performance is measured with BLEU, computed with SacreBLEU [START_REF] Post | A call for clarity in reporting BLEU scores[END_REF].

. Results and Analysis

. Main Results

We run tests using the artificial MXL datasets, as described in Section 5.4.1, as well as on the original test sets, in order to evaluate our models' ability to translate both MXL and monolingual sentences. Results are reported in Table 5.2, where we also report scores separately computed for the "matrix" and "embedded" parts of the test sets. We also report a "do-nothing" baseline (copy) which simply copies the input as output. As is obvious in the copy line, the "embedded" part contains mostly source language and corresponds to an actual translation task, whereas the "matrix" part mostly contains target words on the source side and is much easier to translate.

On the left part of this table, we see that the baseline systems, either with two (base-mono) or one single (bilingual) model(s), do better on monolingual test sets than their counterparts trained on MXL data (respectively base and multi). For both language pairs, the observed differences are in the range of 1-1.5 BLEU points. Conversely, when translating MXL sentences, models trained with MXL data perform significantly better than monolingual baseline models, which have never seen MXL on the source side.

Moreover, we note the marked differences between BLEU scores obtained by these models when the matrix language of the MXL source is the target language and when the embedded language is the target. In the former case, translation is nearly perfect, while in the latter case, they nonetheless use the little information available to improve the monolingual scores (about 1-1.5 BLEU points), nearly matching the performance of the baseline systems on monolingual sentences. As illustrated, for instance, for En-Fr, the dual model improves from 33.7 to 35.1; in the same condition, the bilingual system only improves by 0.4 points.

Among the models trained with MXL data, multi is the weakest, while the other three models achieve comparable performance. With joint training (indep) and dual decoding (dual), we can recover and even surpass the performance of the two separate systems used in the base condition with one single system. On the monolingual tests, these systems also match the multilingual baseline (bilingual) performance. Table . : BLEU scores when translating the monolingual newstest data and the synthetic mxl-newstest data for two language pairs En-Fr and En-Es. copy refers to a do-nothing baseline that simply copies the input. Small numbers contain BLEU scores computed separately when the target language is the embedded language (left) and the matrix language (right) in MXL source. For the monolingual tests (left part), these correspond to scores computed on the same sentences that are also included in the MXL test sets.

. Preserving Copy Constraint

We also measure how well the copy constraint expressed in Section 5.1 is satisfied. It stipulates that every token in an MXL sentence should be either copied into one language and translated into the other or copied to both, which mostly happens for punctuations, numbers, or proper names. No tokens should be dropped out. This is because these tokens act as strong constraints entered by the user, which therefore need to be preserved in the output.

We conduct analysis by computing the percentage of output tokens in four categories: (a) tokens that are copied exclusively, which appear only in one of the two translations; (b) tokens that are copied to both translations, which include regular tokens and (c) punctuations and digits; (d) Lost tokens that appear in neither translations. Our analysis in Table 5.3 shows that the base model is more likely to reproduce the patterns observed in the reference and notably is less likely to generate two copies for a token than the other systems. However, indep and, to a larger extent, dual, are able to reduce the rate of lost tokens, i.e., of source tokens that are not found in any output for En-Fr. This shows that the interaction between the two decoders helps to increase the consistency between the two outputs. As we mainly focus on dual decoding of MXL sentences, we use the dual model for further analyses in this chapter.

En-Fr

En

. Effect of Mixing Languages

In order to better study the effect of mixing languages, we modify the synthetic data generation method to keep one language as the matrix language, in which matrix segments are incrementally replaced by translations of the embedded language. We relax the constraint on the maximum number of replacements and generate new test sets with an increasing number of replacements, ranging from 1 to 20, resulting in twenty versions of the MXL test sets (in each direction).

10 In Figures 5.5 and 5.6, we plot the BLEU scores of both source MXL sentences and their translations produced

We perform as many replacements for sentences that could not accommodate replacements as possible. The solid curve takes Fr as the matrix language, where we progressively inject more En segments; for the dash-dot curve, En is the matrix language, with a growing number of Fr segments. (b) Direction MXL→Fr. The target BLEU is always much higher than the source BLEU, with about a -point difference. The gap between the dash-dot and solid curves is due to the basic sentence structure of the matrix language (see Section . . ). As dash-dot curves represent insertion in the reference target sentence, the corresponding BLEU score is always higher than the solid curve and actually reaches (in the absence of any embedded language). by the dual model for both language pairs by using each language as the matrix language to visualize the impact of progressively introducing more target fragments into the source.

The same behavior is observed for both language pairs and directions: on average, inserting random target fragments boosts the translation performance, with a larger payoff for the first few target segments. There exists a significant gap in the output BLEU scores when MXL source sentences with different matrix languages reach the same (input) BLEU scores. For an MXL sentence, even though we generate a large number of replacements, the basic grammar structure of the matrix language is still maintained. Therefore, taking the target language as the matrix gives the model a pre-translated sentence structure that is much easier to reproduce.

. Implicit Language Identification in Translation

A second question concerns the ability of the translation system to identify target fragments in the source and to copy them to the target, even though these fragments are indistinguishable from genuine source segments. We use labels computed during the MXL data generation procedure to sort out pre-translated (target) segments from actual source segments to be translated. For instance, only tokens with a label [eng] denoting English are expected to be translated when translating into French. All other tokens corresponding to French words are expected to be copied. As reported in Table 5.4, our dual models are able to copy almost all pre-translated tokens for both language pairs and directions. for En-Fr and mxl-newstest2013 for En-Es. We report the number of (pretranslated) tokens that should be copied and the corresponding ratios.

Refining the analysis, we also study whether the relative order of target words changes or is preserved during the translation. Table 5.5 reports the percentage of exact copies and switched orders and the percentage of sentences where the dual models change the pre-translated target segments in the MXL source. We observe large differences with respect to the position of the matrix language again. When the matrix language is the target language, the model almost always preserves the observed token order since it indicates a correct sentence structure for the hypothesis. When translating into the embedded language, we observe a larger number of word order changes: in this case, inserted target segments may not appear in their correct order in the MXL sentence, an issue the model tries to fix.

An example of this order change is in Figure 5.7, where we observe a swap between the input ("différent choix") and output ("choix différent") word orders. There are also many word changes in this latter case, indicating that the model is making proper changes to make a better translation instead by the model (Word-change), evaluated on mxl-newstest2014 (En-Fr) and mxl-newstest2013 (En-Es). We separately report numbers for cases where English is the matrix language (Matrix En) or embedded language (Matrix Fr/Es).

of simply reusing existing fragments. MXL translation models thus seem to perform some language identification, as they almost perfectly sort out target language tokens (which are almost always copied) from the source language tokens (which are always translated).

. Correcting Morphological Errors

The last issue concerns morphological errors: when inserting foreign words into a matrix source, one cannot always expect to also introduce the right inflection marks, some of which can only be determined once the target context is known. Another interesting phenomenon that we do not simulate here is when the embedded (target) lemma is adapted to bear a morphological inflection that only exists in the matrix language. For example, the English word "tokenized" should be adapted into a French sentence [Les textes doivent être tokenisés .] to follow the French morphological inflection. This means that two linguistic systems are mixed within the same word, thereby posing more difficulties for MT [START_REF] Manandise | Morphology to the rescue redux: Resolving borrowings and code-mixing in machine translation[END_REF].

To illustrate the ability to correct grammar errors in input fragments, we manually add noises to an MXL sentence and display its translation in Figure 5.7. Where the input contains the lemma of the French word "tenter" (to try), the model inserts a modal "peuvent" to fix the context. Another illustration is the adjective "différent" which is moved into the post-nominal position, and an article ("un") is inserted. The model also corrects crosslingual syntactic errors. The correct singular French word for "automobilist" should be "automobiliste" in the noisy input. The model corrects it by using a proper plural expression as in the reference. This indicates that the model not only copies what already exists but also tends to adjust translations whenever necessary.

. Computing Translations in Context

In this section, we evaluate MXL translation on a more realistic task: the SemEval 2014 Task 5: L2 Writing Assistant [START_REF] Maarten Van Gompel | SemEval 2014 task 5 -L2 writing assistant[END_REF]. This task involves translating L1 fragments within an L2 context, where the test set design is such that there is exactly one L1 fragment inserted in each utterance. This task can be handled as an MT task from MXL to monolingual, which suits our MXL translation models well. Therefore, we are able to perform direct inference on this task using our model trained in the previous sections. We evaluate two language pairs: French-English and English-Spanish, which contain 495 and 498 sentences, respectively. Below we give an example test segment provided by the organizers for each language pair (the inserted and reference segments are in boldface):

• Input (L1=English,L2=Spanish): "Todo ello, in accordance con los principios que siemprehemos apoyado." Reference: "Todo ello, de conformidad con los principios que siempre hemos apoy-ado." • Input (L1=French,L2=English): "I rentre à la maison because I am tired." Reference: "I return home because I am tired." The official metric for the SemEval evaluation is the word-based accuracy of the L1 fragment translation, which means that the L2 context of each sentence is not taken into account in scoring. We do not use any marks nor special tokens to locate the L1 fragment in the source text and treat it as a regular MXL sentence as in previous sections. Since our systems are full-fledged NMT systems, there is no guarantee that our systems generate translations that always contain the exact reference L2 prefix and suffix. Therefore, we explore two options to compute these scores. The first is to post-process the output HYP and align it with the L2 reference context in REF via edit distance. This alignment allows us to retrieve and only score the relevant fragment in HYP. We refer to this option as free-dec.

The second option is to ensure that the L2 context will be present in the output translation. To this end, we use the constrained decoding option of fairseq, implementing the methods of [START_REF] Post | Fast lexically constrained decoding with dynamic beam allocation for neural machine translation[END_REF]; [START_REF] Hu | Improved lexically constrained decoding for translation and monolingual rewriting[END_REF]. We explore two different ways to express the L2 context as decoding constraints. The first method treats every token in the L2 context as a separate constraint (token-cst). Continuing with the previous example, the L2 context "I, because, I, am, tired." yields five constraints. The second way uses the prefix and suffix of the L2 context as two multi-word constraints (presuf-cst). In this case, "I " and "because I am tired." yield just two constraints. In both cases, constraints are required to be present in the prescribed order in the output. Note that we only perform constrained decoding with multi models as they use a standard beam search during decoding.

. Results

Scores are computed with the SemEval evaluation tool,

11 which enables a comparison with other submissions for this task. Results are in Tables 5.6 and 5.7. For En-Es, our MXL translators (indep and dual) outperform the best system in the official evaluation [START_REF] Maarten Van Gompel | SemEval 2014 task 5 -L2 writing assistant[END_REF] and achieve state-of-the-art performance. Note that these models are not specifically designed nor tuned in any way for the SemEval task. We only perform zeroshot inference on the SemEval test set. For Fr-En, our system achieves better performance than the fourth best participating system, with a clear gap with respect to the top results. In both cases, constraint decoding hurts performance: given that the automatic copy of target segments is already nearly perfect, introducing more constraints during the search has an apparent detrimental effect on this task.

To better study the performance gap between these language pairs, we additionally score the development and test data of this task with BLEU scores. The development sets contain 500 sentences for each language pair. We evaluate BLEU scores on full sentences generated by our systems with the complete reference sentence. Results in Table 5.8 show that we achieve performance in the same ballpark for the two language pairs, suggesting that the observed difference in the SemEval metric is likely due to a mismatch between references and system outputs. The official metric is word accuracy which may exclude acceptable translations.

. Conclusion

In this chapter, we studied the task of disentangling MXL texts into monolingual sentences of its component languages to simulate one kind of bilingual writing situation. We have conducted experiments with several NMT architectures and have proposed the dual decoder model to simultaneously generate translations in both languages within a single model. We have also presented a data augmentation method to generate artificial MXL data. Our experiments demonstrated that artificially generated MXL data could be used to train NMT systems to translate both monolingual and MXL sentences (into one or even two languages). With dual decoding of the two languages, we were able to build systems that were as good as a baseline bilingual system on monolingual texts and much better on MXL texts. Our system does not need any explicit language identification and almost perfectly sorts out source tokens from target tokens in an MXL utterance. Another interesting feature of our system is that it is able to correct syntactic errors in the source text and make necessary modifications to produce proper translations. We have finally reported state-of-the-art results for the SemEval L2 Writing Assistant task for Es-En, while the related results for Fr-En were still somewhat lagging behind the best scores.

In the future, we would like to generate more realistic MXL texts from monolingual sentences using translation models [START_REF] Gupta | A semisupervised approach to generate the code-mixed text using pre-trained encoder and transfer learning[END_REF][START_REF] Tarunesh | From machine translation to code-switching: Generating high-quality code-switched text[END_REF][START_REF] Song | switch-GLAT: Multilingual parallel machine translation via code-switch decoder[END_REF]. Another line of research would be to continue experimenting with more realistic language data containing other linguistic phenomena. Finally, we also intend to study the somewhat more realistic condition where a mixture of languages A and B is translated into a third language C. We believe that the artificial MXL generation methods developed in our work would also be effective for this task. Note that this setting would, however, require 3-way parallel data, a situation that is considered in the next chapter, where we study other applications of dual decoding.

-More Applications of Dual Decoding

. Introduction

The dual decoding scheme for computing synchronized translations presented in Chapter 5 can also be used for several other purposes. In this chapter, we consider the following scenarios in which to apply dual decoding:

• Multi-target translation, where we simultaneously translate a source language into two target languages (Zhou et Our main contributions are the following:

• Mitigating the data scarcity problem of multi-parallel data by finetuning the dual decoder model from standard translation models; • A new parameter sharing scheme, where the two decoders share all their parameters and achieve comparable performance at a reduced model size; • Concrete solutions to mitigate the exposure bias problem between two decoders; • Quantitative evaluations of the increased consistency incurred by a tight interaction between decoders. The contributions in this chapter are published in (Xu and Yvon, 2021b;Xu et al., 2022a).

. More Techniques of Dual Decoding

. Fine-tuning Dual Decoder Model

In this chapter, we continue to use the dual decoder model proposed in Chapter 5. As introduced in Section 5.2.3, training the dual decoder model requires triplets of instances associating one source with two targets. However, multi-parallel corpora are relatively scarce compared to bilingual ones. Unlike Chapter 5, generating large-scale synthetic data is not trivial for applications like multi-target translation and multilingual subtitling.

We would like to take advantage of large-scale bilingual parallel data to mitigate the data scarcity issue. Therefore, we consider a two-step procedure that combines bilingual and trilingual data. In a first step, we pre-train a standard multilingual translation model (one monolingual encoder, one bilingual decoder), in which we use tags to select the target language [START_REF] Johnson | Google's multilingual neural machine translation system: Enabling zero-shot translation[END_REF]. This only requires bilingual data {(f , e 1 ) i , i = 1 . . . N 1 } and {(f ′ , e 2 ) j , j = 1 . . . N 2 }. We then initialize the dual decoder model with pretrained parameters and fine-tune it with a trilingual dataset. Both decoders thus start with the same pre-trained decoder. The decoder cross-attention layers cannot benefit from pre-training and are initialized randomly. Tags are no longer necessary during the fine-tuning stage as both target translations are required.

. Sharing Decoders

One weakness of the dual decoder model is that it contains two separate decoders, yielding an increased number of parameters (×1.6 in our models w.r.t. standard translation models). Inspired by the idea of tying parameters in embedding matrices [START_REF] Inan | Tying word vectors and word classifiers: A loss framework for language modeling[END_REF][START_REF] Press | Using the output embedding to improve language models[END_REF], we extend the dual decoder model by sharing all parameters in the two decoders. In this way, the two decoders become identical after sharing. The total number of parameters remains close to that of a standard translation model (×1.1) since the only increase comes from the additional decoder cross-attention layer. However, the two decoders are expected to perform decoding in different targets simultaneously. To distinguish tasks for each decoder, we prepend a tag at the beginning of each target sentence to indicate the desired output. During inference, the tags are given as a forced prefix for each decoder to decode entire sentences. Sharing decoder is applied for multilingual subtitling in Section 6.6.

. Asynchronous Dual Decoding

Simultaneously generating tokens in two languages is a very strong requirement and may not bring out all the benefits of dual decoding, especially when the two target languages have different word orders. Therefore, we also propose relaxing this assumption by allowing one decoder to start generating tokens before the other. This is implemented by having the delayed decoder generate dummy tokens for a fixed number of steps before generating meaningful words, a strategy similar to the wait-k policy, which is largely used in simultaneous translation [START_REF] Elbayad | Efficient Waitk Models for Simultaneous Machine Translation[END_REF].

?

Figure . : A graphical view of the asynchronous dual beam search process generating translation ê2 with a reference translation in e 1 as forced prefix.

A more extreme case of delayed processing is when one decoder can access a complete translation in another language. In our implementation, this is simulated with partial forced decoding, where one translation is predefined while the other is being computed. We explore the asynchronous decoding in two settings: (a) within a 2-round, sequential procedure, where the output of the first pass for decoder 1 is fully known and fixed when computing the second-pass output for decoder 2; (b) using a reference translation instead of a hypothesis generated in the first pass in one of the two decoders as the forced prefix to perform controlled decoding where the output in l T 2 not only translates the source but is also consistent with the reference translation in l T 1 . The second situation is very close to multi-source translation [START_REF] Franz | Statistical multi-source translation[END_REF], as discussed in Section 3.5.1. Figure 6.2 illustrates the asynchronous decoding setting with a reference translation as the prefix.

Note that with the attention mask in the decoder cross-attention layer, as described in Section 5.2.1, future information from the prefixed decoder cannot be seen during the second-round decoding, even though it is available.

. Multi-target Machine Translation

. Data

We first evaluate our dual decoder model on the multi-target machine translation (MT) task for three directions: English to German/French (En→ De/Fr), German to English/French (De→En/Fr), and English to Chinese/ Japanese (En→Zh/Ja). The difference between the two settings with En, Fr, and De is that when De is in target, the word orders are very different between De and Fr. When De is the source language, the word order differences between Fr and En are not so large. The word orders are also very different between Zh and Ja, and both languages are very distinct from En. Similarly to [START_REF] Wang | Synchronously generating two languages with interactive decoding[END_REF]He et al., 2021a), we use the IWSLT17 dataset 1 [START_REF] Cettolo | WIT3: Web inventory of transcribed and translated talks[END_REF] as our test bed. The original data is not entirely multi-parallel. We thus extract the common English sentences from En-De and En-Fr data with the corresponding translation to build a truly trilingual corpus. The En→Zh/Ja trilingual data is built similarly. The pre-training experiments, as mentioned in Section 6.2.1, additionally use WMT20

2 En-De, De-Fr, En-Zh, En-Ja, and WMT14 3 En-Fr bilingual datasets. We use the IWSLT tst2012 and tst2013 as development sets and test our model on tst2014.

4 More than 90% of the test sentences are already multi-parallel, so we follow the same procedure to build a reduced optimize with Adam, set up with a maximum learning rate of 0.0007 and an inverse square root decay schedule, as well as 4,000 warm-up steps. For finetuning models, we use Adam with a fixed learning rate of 8e-5. We share the decoder input and output matrices for standard Transformer models, while for dual decoder models, we share all four input and output decoder matrices. All models are trained with mixed precision and a batch size of 8,192 tokens on 4 V100 GPUs. Pre-training standard translation models last for 300k iterations, while all other models are trained until no improvement is found for 4 consecutive checkpoints on the development set. Performance is computed with SacreBLEU [START_REF] Post | A call for clarity in reporting BLEU scores[END_REF].

Similar to experiments in Chapter 5, we compare our dual decoder models dual to a simplified multi-task model (indep) without decoder crossattention and to baseline standard Transformer models (base), which contain two separate models trained for each direction. All these models are trained using the IWSLT trilingual data. The pre-trained multilingual model using WMT data is denoted as multi.

. Results

We evaluate the performance of models trained only with trilingual data and the pre-trained models in a multilingual way. Table 6.3 shows that the indep model outperforms the base model in all directions, demonstrating the benefits of jointly training two independent decoders. The same gain is not observed for the dual model. For En→De/Fr and En→Zh/Ja, dual slightly outperforms the baseline models, while for De→En/Fr, the performance is even worse than the baseline. One explanation is that dual decoding suffers from a double exposure bias problem, as errors in both decoders jointly contribute to derailing the decoding process. The effect seems to be more severe for target languages with similar word orders like En and Fr. We will get back to this exposure bias issue in Section 6.4. The pre-trained multi models do not perform as well as base. This is expected as WMT data are out-of-domain for IWSLT test sets.

As discussed in Section 2.3.1, translation models are exposed to their previous predictions during inference, and errors may accumulate and derail the translation. In our dual decoder model, this exposure bias problem is amplified as the model conditions on predictions from both decoders and errors from both decoders are cascaded to future predictions. To test the possible exposure bias issue, we use the base model to translate source texts f into targets ê1 and ê2 , which are then merged with the original data to build a pseudo-trilingual training set. This strategy is a form of knowledge distillation that is widely used for non-autoregressive translations, as introduced in Section 4.4. Taking En→De/Fr as an example, we use the base En→De model to translate half of the English source f 1/2 into Table . : BLEU scores for multi-target models. Pseudo (ps) refers to models trained from scratch with partial synthetic reference data. FT indicates models fine-tuned from the pre-trained multilingual (multi) models. FT+ps refers to models fine-tuned using partial synthetic reference data.

German ê1/2 and use the base En→Fr model to translate the other half of the English source f 2/2 into French ê2/2 , thus obtaining the pseudo-trilingual dataset

{(f 1/2 , ê1 1/2 , e 2 1/2 ), (f 2/2 , e 1 2/2 , ê2 2/2
)} that is as large as the original data. Pseudo datasets for De→En/Fr and En→Zh/Ja are generated similarly. For fair comparisons, we only use half of the translations as pseudo references for each target language. We see in Table 6.3 (ps) that dual models trained with these artificial references almost close the gap between the independent and dual decoder models.

Another approach we study is to fine-tune the dual decoder model on a pre-trained multilingual translation model. Initializing our trilingual models with pre-trained models, as described in Section 6.2.1, brings an additional improvement for both the indep and dual models, as shown in Table 6.3 (FT), which validates that our dual decoder model can benefit from pretrained multilingual models even though the decoder cross attention layer is randomly initialized. This confirms that the dual decoder model can be effectively trained, even without large-scale tri-parallel data. These results also highlight the large gains of fine-tuning on a tri-parallel corpus, which improves our baseline multilingual models by nearly 5 BLEU points on average.

We additionally experiment with fine-tuning pre-trained models with the synthetic pseudo-trilingual data. However, this setting (FT+ps in Table 6.3) does not bring any gain in translation quality. For the indep models we see a small loss due to training with noisy references, while for dual, it seems that mitigating exposure bias is less impactful when starting from well-trained models.

. Further Analyses

The value of dual decoding is to ensure that translations ê1 and ê2 are more consistent than performing independent decoding. To evaluate this, we compute the similarity scores between these two translations by computing the cosine similarity of sentence embeddings of the two translations using LASER,

12 a library to calculate and use multilingual sentence embeddings. As shown in Table 6.4, the dual models always generate translations that are more similar to each other on average than the indep models in all conditions. Note that both approaches are meant to translate the same source texts into the same languages. Therefore, the similarity scores are always quite high in all cases. Figure 6.3 shows some examples that dual generates more consistent translations than indep for the direction De→En/Fr. Table . : Similarity scores (SIM) for multi-target models. The similarity scores are computed as the cross-lingual similarity between sentence embeddings of the two target translations computed with LASER. Pseudo (ps) refers to models trained from scratch with partial synthetic reference data. FT indicates models fine-tuned from the pre-trained multilingual (multi) models. FT+ps refers to models fine-tuned using partial synthetic reference data.

As explained in Section 6.2.3, the dual decoder model is not limited to strictly synchronous generation and accommodates relaxed variants (as well as alternative dependency patterns) where one decoder can start several steps after the other. We fine-tune the "wait-k" dual models from the pre-trained models with k = 3 for the three directions and evaluate the effects. The wait-k approach requires extra training as the dual decoder model needs to be trained to consider the dummy tokens prepended in one of the two decoders. As shown in Table 6.5 (wait-3), the BLEU scores are slightly improved for both targets in some directions when either side is delayed by 3 steps. These results suggest that depending on language pairs, the information flow between decoders can be beneficial from a small amount of asynchronicity. It can also be harmful to language pairs like De→En/Fr, as word orders in En and Fr are better synchronized than other language pairs. Our implementation also enables us to have one decoder completing its translations before the other one begins. We thus experiment with a 2-round sequential decoding strategy (see Section 6.2.3), in which we first compute the complete translation ê1 in one target language with the dual model, then decode the other one ê2 with ê1 as a forced prefix in the other decoder. One can also use the reference e 1 as the prefix when available. In this case, the second decoding step has access to both the source and the other target sequence. Compared to the wait-k approach, the 2-round decoding strategy does not require any additional training and can be applied directly during inference.

We decode both dual and dual FT models with this strategy. Results in Table 6.5, obtained with both automatic and reference translations in one language, show that this technique is able to improve the dual models in almost all directions trained from scratch. 2-round sequential decoding with reference in one language provides the other decoder with the ground truth, therefore alleviating the exposure bias problem that hurts the dual models. However, combining the results of FT models in Tables 6.3 and 6.5, we see that fine-tuned models are less sensitive to errors made during decoding. This again shows the benefits that dual models actually get from using pre-trained models.

. Bidirectional Decoding

Bidirectional decoding [START_REF] Finch | Bidirectional phrase-based statistical machine translation[END_REF][START_REF] Zhou | Synchronous bidirectional neural machine translation[END_REF]Liu et al., 2020b) aims to integrate future information in the decoder by jointly translating in the forward (L2R) and the backward (R2L) directions. Another expectation is that the two decoders, having different views of the source, will deliver complementary translations. Dual decoding readily applies in this setting, with one decoder for each direction, with the added benefit of generating more coherent outputs than independent decoders. We evaluate this added consistency by reusing the experimental setting (data, implementation, and hyperparameters) of Section 6.3 and by training 4 bidirectional systems from En into De, Fr, Zh, and Ja. Similar to [START_REF] Zhou | Synchronous bidirectional neural machine translation[END_REF], the selected output translation (forward or backward) is the one that has a higher probability. We invert the translation if the R2L output is picked. We measure the consistency between translations in the two directions by computing the averaged BLEU score between the two translations.

We first train models using the same IWSLT corpora as in Section 6.3 for each language pair. To build the tri-parallel training data for this task, we add an inverted version of the target sentence to each training sample to build the R2L reference. We reuse the base models from Section 6.3 for the corresponding language pairs. Results are reported in Table 6.6. In this setting, the dual models again suffer a clear drop in BLEU scores as compared to the indep models. We again attribute this gap to the impact of the exposure bias. As seen in Table 6.7, the loss in BLEU scores of the dual systems is accompanied by a very large increase in consistency of the outputs (+31.1). This indicates that the dual models learn to directly use tokens from the other direction as if they were always correct during training. However, during inference, once the prediction made by one direction is incorrect, the other direction simply copies the wrong token, which yields severe cascading errors.

To mitigate this problem, we again introduce pseudo-parallel targets, as in Section 6.3.3, where one of the two targets is automatically generated with the base model, as also performed by [START_REF] Zhou | Synchronous bidirectional neural machine translation[END_REF]; [START_REF] Wang | Synchronously generating two languages with interactive decoding[END_REF]; Zhang et al. (2020a); He et al. (2021a). We generate the synthetic references using individual En→De L2R and En→De R2L Transformer models, for instance. The En→De R2L system is trained on En-De trilingual data with the German reference simply inverted. Similar to the pseudo-data described in Section 6.3.3, we generate a pseudo dataset

{(f 1/2 , e f w 1/2 , êbw 1/2 ), (f 2/2 , êfw 2/2 , e
bw 2/2 )}, in which each original source sentence occurs just once. This pseudo dataset assures that the forward and backward training target sentences are not always deterministically related, which forces each decoder to put less trust in tokens from the other direction. We also consider a duplicated pseudo dataset {(f , e f w , êbw ), (f , êfw , e bw )}, in which each source sentence is duplicated, occurring once with the reference data in each direction. Results in Table 6.6 show that both indep and dual models can benefit from the pseudo data (ps). The dual models again close the gap with indep and yields systems that surpass the baseline by about 1 BLEU point. Systems trained with duplicated pseudo data (ps-dup) can further improve the performance by 0.6 BLEU points on average.

By computing the averaged BLEU score between the two output translations BLEU(ê f w , êbw ) and BLEU(ê bw , êfw ), we can also evaluate the increment of consistency incurred in dual decoding. These scores are reported in Table 6.7. Our dual models always produce more consistent translations than the indep models, with a +13.4 to +17.9 increment in consistency when averaged over language pairs, thereby demonstrating the positive impact of dual decoding.

. Multi-style Decoding

As another application of dual decoding, we study the generation of pairs of consistent translation alternatives, using variation in "politeness" as our test bed. Some languages like German, French, Chinese, and Japanese contain several levels of formalities expressed by using honorifics that English does not possess. When translating from English to these languages, it is difficult to predict the appropriate honorific, while users may want to control the politeness level of the translation (Sennrich et al., 2016a). The principal purpose of this study is to provide deterministic control over the generation to produce consistent variants with the possibility to choose which version is produced.

. Datasets and Experimental Settings

We borrow the experimental setting and data of Sennrich et al. (2016a).

13

The training data consists of OpenSubtitles2012 En-De data with 5.58M sentence pairs, out of which 0.48M of German references are annotated as polite and 1.06M as informal. The rest is deemed neutral. The annotation tool is based on the ParZu dependency parser 14 and an annotation script released with the data. Polite/Informal tags are based on an automatic analysis of the German side according to rules described in (Sennrich et al., 2016a). The test set, which we use as the development set, is a random sample of 2,000 sentences from OpenSubtitles2013. We use the testyou set as our main test set, which consists of 2,000 random sentences also extracted from OpenSubtitles2013 where the English source contains a second-person pronoun you(r(s(elf))) in the English source. We built a shared vocabulary with a joint BPE of 32K merge operations.

The German reference sentences are only annotated with politeness labels, but alternative translations do not exist. Therefore, we manage to generate tri-parallel data to fit our dual decoder model. We first pre-train a tag-based MT system with politeness control as in (Sennrich et al., 2016a) and use it to predict the polite counterpart of each informal sentence and vice-versa. We label the source polite sentences with an informal tag and use the pre-trained model to generate corresponding synthetic informal translations. The polite translations are generated similarly. We also randomly extract an equivalent number of neutral sentences as the polite and informal ones, i.e., 1.54M. References of neutral sentences are identical for both polite and informal targets. The resulting tri-parallel corpus thus contains 3.07M sentences. Similar to the multi-target task (Section 6.3), we fine-tune the pre-trained model with this data until convergence. During fine-tuning, we omit the control tags used in pre-training as both polite and informal translations are required. The annotation tool distributed with the data is used to assess the politeness of the output translations.

. Results

Table 6.8 (top) reports the performance of the pre-trained model. Reference refers to the annotation result of the German reference sentences. None is translated without adding any tags to the source text, while polite and informal are translated with all sentences tagged respectively as polite and informal. Oracle is obtained by prefixing each source input with the reference tag. These results show the effectiveness of side constraints for the generation of variants. For both polite and informal categories, the pre-trained model generates translations that mostly satisfy the desired requirement.

Results of the fine-tuned dual decoder models are in Table 6.8 (bottom): we see that indep and dual models are very close and tend to generate more neutral translations. Compared to the pre-trained model, they also slightly improve the BLEU scores.

As discussed in Section 6.2.3, our dual decoder model can delay one decoder until the other is finished. We redo the same 2-round decoding procedure as in Section 6. 3.4. Results in Table 6.8 (bottom) indicate that given the full translation of informal variations, the dual model tends to generate less neutral sentences but more polite ones. The same phenomenon is also observed in the other direction. This implies that the output variations can be better controlled with 2-round sequential decoding.

. Multilingual Subtitling

As the amount of online audio-visual content continues to grow, the need for captions and subtitles in multiple languages also steadily increases, as it widens the potential audience of these contents. In this section, we use "caption" to refer to a text written in the same language as the audio and "subtitle" when translated into another language. Captions, which are often meant for viewers with hearing difficulties, and subtitles, which are produced for viewers with an imperfect command of the source language, may have slightly different traits that we ignore here.

Both activities are closely related: human subtitle translators often generate subtitles directly based on the original captions without viewing or listening to the original video/audio files. This strategy, however, runs the risk of amplifying simplifications or errors present in the captioning in the subtitle approximations. It may even happen that both texts need to be simultaneously displayed on the screen, for instance, in countries with several official languages or to help foreign language learners. This means that captions and subtitles need to be consistent not only with the video content but also with each other. It also implies that they should be synchronized [START_REF] Karakanta | Between flexibility and consistency: Joint generation of captions and subtitles[END_REF]. Finally, even in scenarios where only subtitles would be needed, simultaneously generating captions may still help better check the correctness of subtitles.

Early approaches to automatic subtitling (e.g., [START_REF] Piperidis | Multimodal, multilingual resources in the subtitling process[END_REF]) also assumed a pipeline architecture (Figure 6 translated from captions derived from ASR transcripts. A recent alternative (Figure 6.4 (a)), which mitigates cascading errors, is to independently perform captioning and subtitling in an end-to-end manner (Liu et al., 2020a;Karakanta et al., 2020a). The risk, however, is to generate inconsistencies (both in alignment and content) between the two textual streams. This approach might also be limited by the lack of appropriate training resources (Sperber and Paulik, 2020). Various ways to further strengthen the interactions between these tasks by sharing parameters or loss terms are evaluated by Sperber et al. (2020). Figure 6.4 (c) illustrates these approaches.

In this section, we explore simultaneously generating both captions and subtitles from ASR transcripts using dual decoding, 15 as illustrated in Figure 6.4 (d). Note that our work is not completely end-to-end from audio signals to subtitles. We focus on text-to-text experiments. We also explore sharing the decoder parameters as described in Section 6.2.2. Generally speaking, automatically turning ASR transcripts into full-fledged captions involves multiple changes, depending on the specification of the captioning task. In our case, this transformation comprises four main aspects: the segmentation for display (via tag insertion), the removal of certain features from spoken language (e.g., fillers, repetitions, or hesitations), ASR error correction, and punctuation prediction. The transcript-to-subtitle task involves the same transformations, with an additional translation step to produce text in another language. Figure 6.5 illustrates the various transformations between input transcripts and the corresponding output segments. As our experiments suggest, a tighter integration not only improves the quality and consistency of captions and subtitles but also enables better use of all available data, with hardly any impact on model size.

. Datasets and Resources

For our experiments, we use MuST-Cinema 16 (Karakanta et al., 2020b), a multilingual Speech-to-Subtitles corpus compiled from TED talks, in which subtitles contain additional segmentation tags indicating change-of-screen ([eob]) or line ([eol]). Our experiments consider the translation from English into French. Our tri-parallel data also includes a pre-existing unpunctuated ASR output generated by Karakanta et al. (2020a), which achieves a word error rate (WER) score of 39.2% on the MuST-Cinema test set speech transcripts. In order to emulate a real production scenario, we segment these transcripts as if they were from an ASR system performing segmentation based on prosody. As this kind of ASR system tends to produce longer sequences compared to typical written text [START_REF] Cho | Segmentation and punctuation prediction in speech language translation using a monolingual translation system[END_REF], we randomly concatenate the English captions into longer sequences, to which we align the ASR transcripts using the conventional edit distance, thus adding a subsegmentation aspect to the translation task. This concatenation method was proposed and implemented by François Buet. More details about the data processing procedure are in Appendix C.1. The resulting training, development, and test data actually used for this experiment contains 133k, 499, and 255 lines, respectively.

Similar to Section 6.3, we continue to apply the pre-training fine-tuning scheme to take advantage of large-scale bilingual data. As we work only on the En-Fr direction, the pre-trained model is not a multilingual model and thus does not use any tags in pre-training. For pre-training, we use the https://ict.fbk.eu/must-cinema/ WMT14 En-Fr data as in Section 6.3.1. For fine-tuning, we use synthetic tri-parallel data similar to Sections 6.3 and 6.4, in which we alternatively replace one of the two target side references with hypotheses generated from a baseline system for the corresponding direction. We tokenize all data with Moses scripts and use a shared source-target vocabulary of 32K BPE (Sennrich et al., 2016c).

. Experimental Settings

We pre-train the En→Fr translation model for 300k iterations. All other models are trained until no improvement is found for 4 consecutive checkpoints on the development set. We mainly measure performance with BLEU. TER [START_REF] Snover | A study of translation edit rate with targeted human annotation[END_REF] and BERTScore (Zhang et al., 2020b) are also reported in Appendix C.3. Segmentation tags in subtitle texts are also taken into account, and BLEU scores are computed over full sentences. In addition, measuring the consistency between captions and subtitles is also important. We reuse the structural and lexical consistency score proposed by [START_REF] Karakanta | Between flexibility and consistency: Joint generation of captions and subtitles[END_REF].

17 Structural consistency measures the percentage of utterances having the same number of blocks in both languages, while lexical scores count the proportion of words in the two languages that are aligned in the same block (refer to Appendix C.2 for additional details).

We call the dual decoder model dual. Models sharing the parameters of the two decoders are denoted as share. Baseline translation models separately trained for each direction (T en →C en , T en →S fr ) are denoted by base.

To study the effectiveness of dual decoding, we mainly compare dual with a pipeline system. The latter uses the base model to produce captions which are then translated into subtitles using an independent system trained to translate from caption to subtitle (T en →C en →S fr ).

Like the dual model, base and pipeline systems can also benefit from pre-training. For the former, we pre-train the direct transcript-to-subtitle translation model (T en →S fr ); for pipeline, the caption-to-subtitle model (C en →S fr ) is pre-trained, while the first step (T en →C en ) remains as in the base system since both source and target sides are English. Note that all fine-tuned systems start with the same model pre-trained using WMT En-Fr data, even though fine-tuned systems need to predict segmentation tags that are not seen during pre-training.

. Main Results

We only report in Table 6.9 the performance of the two baselines and fine-tuned (+FT) models, as our preliminary experiments showed that train- ing the dual decoder model with only tri-parallel data was not optimal. The BLEU score of the do-nothing baseline, which copies the source ASR transcripts to the output, is 28.0, suggesting that the captioning task actually involves much more transformations than simply inserting segmentation tags. We observe that fine-tuning improves subtitles (Fr) generated by base and pipeline systems by ∼1 BLEU. Once fine-tuned with synthetic tri-parallel data, our dual decoder model outperforms base+FT by 0.7 BLEU and pipeline+FT by 1.4 BLEU in the subtitling direction. Fine-tuning also delivers a positive effect on the captioning direction, as dual is able to improve 1.2 BLEU over base. Sharing all parameters of both decoders yields a further increase of 0.2 BLEU for subtitling, with about one-third fewer parameters. This indicates that sharing parameters between decoders is effective.

We also measure the structural and lexical consistency between captions and subtitles generated by our systems in Table 6.9. As expected, pipeline settings always generate very consistent pairs of captions and subtitles, as subtitles are direct translations of the captions. All other methods generate both outputs directly from the ASR transcripts. dual models do not perform as well but are still able to generate captions and subtitles with a much higher structural and lexical consistency between the two outputs than the base systems. Experiments in previous sections show that dual decoder models generate translations that are more consistent in content. We further show here that our dual models generate hypotheses that are also more consistent in structure. Some examples of output captions and subtitles produced by the share model are shown in Figure 6.6.

. The Effect of Fine-tuning

As the pre-trained En→Fr model has never seen sentences in the source language (En) on the target side, we first only use it to initialize the subti- tling decoder and use a random initialization for the captioning decoder. To study the effect of initialization, we conducted an ablation study by comparing three settings: initializing only the subtitling decoder (1-decoder), both decoders, and the shared decoder. As illustrated in Table 6.10, initializing both decoders brings improvements in both directions, with a gain of 1.6 BLEU for captioning and 0.3 BLEU for subtitling. Moreover, sharing parameters between decoders further boosts the subtitling performance by 0.2 BLEU. As it seems, the captioning decoder also benefits from a decoder pre-trained in another language.

. Mitigating Exposure Bias

We also analyze the influence of the exposure bias issue in this application scenario. To this end, we compare fine-tuning the dual model with original vs. artificial tri-parallel data. For simplicity, we only report the averaged captioning and subtitling BLEU scores in Table 6.11. Results show that finetuning with the original data (w/real) strongly degrades the BLEU scores for the generated text, resulting in worse performance than the baseline.

Model

Decoding Setting

Baseline 2-round Ref base In another set of experiments, we perform asynchronous 2-round sequential decoding, as in Section 6.3 and Section 6.5. We first decode the dual models to obtain hypotheses in both languages ê1 and ê2 . During the second decoding round, we use the output English caption ê1 as a forced prefix when generating the French subtitles ê2 . The final English caption ê1 is obtained similarly. The 2-round decoding scores are shown in Table 6.11 and compared with the optimal situation where we use references instead of model predictions as forced prefixes in the second round (in column Ref).

Results in Table 6.11 suggest that dual decoder models fine-tuned with original data (w/real) are pretty sensitive to exposure bias, which can be mitigated with artificial tri-parallel data. Their performance can be improved by ∼1.5 BLEU when using 2-round decoding, thereby almost closing the initial gap with models using synthetic data. Fine-tuning with synthetic triparallel data is slightly better overall and more stable across various decoding configurations, which is also found in Section 6.3.4.

. Conclusion

In this chapter, we have explored dual decoding in several application scenarios as a way to generate pairs of consistent translations. The dual decoder model can be effectively trained using actual or partly artificial data. It can also directly benefit from pre-trained models, therefore making the best use of large-scale bilingual data. Considering the four applications, we have observed that dual decoding is prone to exposure bias in the two decoders, and we have proposed practical remedies. Using these, we have achieved BLEU scores that match those of simple multi-task learners and display increased consistency between the two outputs, further confirming the findings in Chapter 5. Additionally, we showed that parameter sharing on the decoder side is viable and effective, at least for related languages. A limitation of the dual decoder model is that the decoding time is twice as slow as regular translation models, as mentioned in Section 5.2.2. However, with shared decoders, our model yields almost no additional cost in space.

In our future work, we plan to consider other strategies, such as scheduled sampling [START_REF] Bengio | Scheduled sampling for sequence prediction with recurrent neural networks[END_REF][START_REF] Mihaylova | Scheduled sampling for transformers[END_REF], to mitigate the exposure bias problem. Another area we seek to improve is the relaxation of strict synchronicity in decoding. We finally wish to study more applications of this technique, notably to generate controlled variation: controlling gender variation [START_REF] Zmigrod | Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology[END_REF] or more complex forms of formality levels, as in [START_REF] Niu | Controlling neural machine translation formality with synthetic supervision[END_REF], are obvious candidates.

-Bilingual Synchronization

. Introduction

In this chapter, we study another scenario of bilingual writing. As introduced in Chapter 1, this scenario extends online commercial translation systems. Unlike the proposition in Chapter 5, we assume that texts in two languages are separated into two boxes, while users can freely choose to compose in either language at their will, as illustrated in Figure 7.1. Once texts in one language are edited, the other language is then automatically synchronized so that the two sentences are always mutual translations. To this end, the technical challenge is to keep the two input texts synchronized while taking the users' input into account. Our scenario is a joint bilingual edition and translation process, requiring multiple decodings with small changes. However, neural machine translation (NMT) is generally viewed as a one-shot activity process that generates the target sentence based on the sole source language input.

Type to translate English

I return home because I am tired.

Type to translate French

Je rentre à la maison parce que je suis fatigué.

Figure . : A design of an online bilingual writing system. Users can freely choose the language to write, and the system automatically updates the texts in the other box.

Here We focus on the revision part of the translation process and consider bilingual synchronization (Bi-sync), which we define as follows. Given an existing pair of a source (f ) and a target (ẽ) sentences, which may or may not be mutual translations, the task is to compute a revised version e of the initial translation ẽ, such that e is an actual translation of the source sentence f . Our definition also implies the detection of parallel sentences, where no edition is needed. Bi-sync is necessary for the bilingual writing scenario when the source side of an existing sentence pair is edited, requiring updating the target accordingly so that both sides remain synchronized.

In principle, Bi-sync is more general than standard machine translation (MT). The latter corresponds to the synchronization with an initially empty target (ẽ = [ ]). More interesting situations in Bi-sync occur when parts of the initial target text can be reused, meaning that the synchronization model only needs to make a few changes to synchronize the translation. In this situation, Bi-sync mainly acts to edit an initial translation ẽ rather than to generate a translation from scratch.

Bi-sync also encompasses several existing tasks incorporating auxiliary information in the translation process, as we introduced some of them in Chapter 3: synchronization is needed in interactive MT [START_REF] Knowles | Neural interactive translation prediction[END_REF] and bilingual editing [START_REF] Bronner | Cosyne: Synchronizing multilingual wiki content[END_REF], where ẽ is the translation of a previous version of f ; in MT with lexical or terminological constraints [START_REF] Hokamp | Lexically constrained decoding for sequence generation using grid beam search[END_REF][START_REF] Post | Fast lexically constrained decoding with dynamic beam allocation for neural machine translation[END_REF], where ẽ initially contains target-side constraints [START_REF] Susanto | Lexically constrained neural machine translation with Levenshtein transformer[END_REF]Xu and Carpuat, 2021a); in translation memory (TM) based approaches [START_REF] Curran Associates | Neural fuzzy repair: Integrating fuzzy matches into neural machine translation[END_REF], where ẽ is a retrieved similar translation for a source sentence; in automatic postediting (APE, do Carmo et al., 2021), where ẽ is the output of an MT system; in a newly proposed translation error correction task [START_REF] Lin | Automatic correction of human translations[END_REF], where ẽ is a human translation that contains errors made by human translators when producing the translation; as well as in parallel corpus cleaning and fixing [START_REF] Negri | Semeval-2013 task 8: Cross-lingual textual entailment for content synchronization[END_REF][START_REF] Carpuat | Detecting cross-lingual semantic divergence for neural machine translation[END_REF][START_REF] Pham | Fixing translation divergences in parallel corpora for neural MT[END_REF], where ẽ is a possibly noisy translation that should be filtered out or fixed.

We develop and evaluate methods to implement sequence-to-sequence models that perform Bi-sync. In this chapter, we focus on autoregressive approaches that are similar to [START_REF] Curran Associates | Neural fuzzy repair: Integrating fuzzy matches into neural machine translation[END_REF], where the source sentence and the initial translation are concatenated into one input sequence, as discussed in Section 3.3.2. We also study various ways to artificially generate the triplets of examples (f , ẽ, e) that are needed in training. We will study non-autoregressive approaches to performing Bi-sync in Chapter 8.

We also evaluate the Bi-sync model on two specific downstream tasks: parallel corpus cleaning and fixing and NMT with TMs. The former attempts to synchronize noisy segments in a parallel corpus to improve the corpus quality. This is arguably more difficult than the Bi-sync task, as many initial translations are already correct and need to be kept unchanged. As introduced in Section 3.3, NMT with TMs tends to make use of similar translations extracted from TM to help improve the translation quality. We study how to apply fine-tuning of general Bi-sync models to both tasks to evaluate the adaptation capability of Bi-sync models to more specific tasks.

Our main contributions are the following:

• We propose a new Bi-sync task and several methods to generate synthetic editing data for different types of editing operations. • We design a specific training procedure for autoregressive edit-based model and show that our model can be trained with artificial data. • We conduct various analyses to better understand our model under the new Bi-sync task. • We perform empirical validations of the Bi-sync model on several spe-cific tasks. Some of the contributions in this chapter are published in [START_REF] Xu | Boosting neural machine translation with similar translations[END_REF](Xu et al., , 2022b)).

. Generating Editing Data

We consider a general scenario where, given a pair of sentences f and ẽ, assumed to be related but not necessarily parallel, we aim to generate a target sentence e that is parallel to f . We would also like ẽ and e to be close, as, in our settings, ẽ is often a valid translation of a sentence that is close to f . Training such models requires triplets of examples (f , ẽ, e). While large amounts of parallel bilingual data are available for many language pairs, they are however rarely associated with related translations ẽ.

1 Therefore, we study ways to simulate synthetic ẽ from e, while making sure to preserve large portions of the reference sentence e when generating samples. As string editing operations can be decomposed into a sequence of atomic operations among insertions, substitutions, and deletions, we design our artificial training samples so that edits from ẽ to e only involve one single type of operation. Figure 7.2 illustrates our data generation methodology for each type of edit, along with a generated example.

. Insertions

We mainly follow Xiao et al. (2022a) to generate initial translations ẽins for insertion by randomly dropping segments from the original target sentence e. For each e, we first randomly sample an integer k ∈ [1, 5], then randomly remove k non-overlapping segments from e. The length of each removed segment is also randomly sampled with a maximum length of 5 tokens. We also impose that the overall ratio of removed segments does not exceed 0.5 of the original target e. Unlike Xiao et al. (2022a), the initial translations ẽins we generated do not contain any placeholders to locate the positions of removed segments (see Figure 7.2). This makes ẽins a more realistic starting point as the insertion positions are rarely known in practical settings. Our preliminary experiments to reproduce Xiao et al. (2022a) also show that identifying the insertion positions in ẽins makes the text infilling task much easier than when they are unknown. Models trained with explicit insertion positions possess a higher precision and recall score for predicting the expected segments.

A situation where such triplets actually exist is APE. However, we do not consider APE since the MT output depends on the original MT system and is sometimes difficult to improve (Chollampatt et al., ). The post-editions also depend on human post-editors and are often unique, inconsistent, and difficult to generalize (Wisniewski et al., ).

increase diversity, ẽ′ ins (used as lexical constraints to create ẽsub ) is sampled with a different random seed than ẽins (used for the insertion task).

As shown in Figure 7.2 (Substitution), the target sentence e = [Cela n' arrivera pas .] is first translated into the source language f * = [That will not happen .] using top-5 sampling. The lexical constraints we generated from e is ẽ′ ins = [Cela pas .]. By translating f * back to the target language with ẽ′ ins , we obtain ẽsub = [Cela ne se produira pas .], where the segment [n' arrivera] is substituted with the equivalent [ne se produira].

. Deletions

Simulating deletions requires the initial translation ẽdel to be an extension of the original target e. We propose two strategies to generate ẽdel .

Method 1

The first method uses a GAP insertion model proposed by Xiao et al. (2022a). This approach randomly replaces word segments with a placeholder [gap] to generate an initial translation ẽgap . The task is to predict the missing segments based on a long input made as the concatenation of f and ẽgap . This task is different from our insertion task, as (a) insertion positions are identified as [gap] symbols in ẽgap , and (b) generation only computes the sequence of missing segments e seg , rather than a complete sentence. An interesting finding of Xiao et al. (2022a) is that the GAP model is mostly able to match the number of predicted segments to the number of placeholders in ẽgap , even though there are no specific hard constraints for this requirement.

This finding inspired our first strategy. We use the GAP insertion model to generate extra segments for a pair of parallel sentences as follows. We randomly insert k ∈ [1, 5] [gap] tokens into the original target sentence e, concatenate it with the source f and feed the input to the GAP model to predict extra segments, yielding the synthetic target sentence ẽdel 1 , as illustrated in Figure 7.2 (Deletion 1). Since the starting point is a pair of parallel sentences, introducing [gap] tokens forces the GAP model to predict extra segments. This method always extends parallel sentences with additional segments on the target side. However, these segments are arbitrary and may not contain any valid semantic information nor be syntactically appropriate in their contexts.

Method 2

We thus consider a second strategy based on actual edit operations collected in the WikiAtomicEdits dataset 3 [START_REF] Faruqui | Wiki-AtomicEdits: A multilingual corpus of Wikipedia edits for modeling language and discourse[END_REF]. This dataset https://github.com/google-research-datasets/wiki-atomic-edits contains edits of an original sequence x and the resulting sequence x ′ , with exactly one insertion or deletion operation for each training example, collected from Wikipedia edit history. This notably ensures that both versions of each utterance are syntactically correct. We treat the deletion operation data of WikiAtomicEdits as "reversed" insertions and use both to train a sequence-to-sequence wiki model (x short → x long ), generating longer sentences from shorter ones. The wiki model is then used to expand the target sentence e into a longer version ẽdel 2 . Compared to ẽdel 1 , ẽdel 2 is syntactically more correct. However, it is also, by design, very close to e, as there is only one edit away to convert ẽdel 2 to e.

As both simulation methods for the deletion task have merits and flaws, we randomly select examples from both ẽdel 1 and ẽdel 2 to build the final synthetic initial translation samples for the deletion operation ẽdel .

. Copy and Translate Operations

In order to handle parallel sentences that would not require any changes, we add a fourth copy operation, where the initial translation ẽcopy is identical to the target sentence (ẽ copy = e). For the copy operation, the model is expected to generate the exact initial target ẽcopy as output without making changes. Therefore, the data eventually used to learn edit operations is built with triplets (f , ẽ, e) where ẽ is uniformly randomly sampled from ẽins , ẽsub , ẽdel , and ẽcopy . The training data thus contain the same amount of data for each edit operation. Finally, we consider training examples where ẽ is empty to also maintain the capacity to perform standard MT from scratch.

. Edit-MT

We implement Bi-sync with a Transformer-based [START_REF] Vaswani | Attention is all you need[END_REF] autoregressive model called Edit-MT. In this model, the initial translation ẽ we generated is simply concatenated to the source f , with a special token to separate the two sentences. This technique has been used, e.g., in (Crego et al., 2016;[START_REF] Niehues | Pretranslation for neural machine translation[END_REF][START_REF] Hokamp | Ensembling factored neural machine translation models for automatic post-editing and quality estimation[END_REF] for APE, in [START_REF] Dabre | Enabling multisource neural machine translation by concatenating source sentences in multiple languages[END_REF] for multi-source MT, or in (Bulte and Tezcan, 2019) for translation with TMs, as introduced in Chapter 3. The input side of the editing training data is thus f [sep]ẽ, as shown in Figure 7.2 (top). Note that we do not reset the positional encoding for the second part ẽ, as it does not yield significant effects on the overall performance [START_REF] Yang | WeTS: A benchmark for translation suggestion[END_REF].

On the target side, we add a categorical prefix to indicate the type of edit operation(s) associated with a given training sample, as is commonly done for multi-domain [START_REF] Kobus | Domain control for neural machine translation[END_REF] or multilingual MT [START_REF] Johnson | Google's multilingual neural machine translation system: Enabling zero-shot translation[END_REF] and is also applied in Sections 5.4.2 and 6.2.2. For each basic edit (insertion, substitution, and deletion), we use a binary tag to indicate whether this operation is required. For instance, an initial translation needing insertions ẽins would have tags [ins] [!sub] [!del] prepended to the target e, indicating that only insertion is required while substitution and deletion are not. The copy operation corresponds to the case where all three tags are set to negative as [!ins] [!sub] [!del]. Note that we use tags on the target side, rather than on the source side, as [START_REF] Johnson | Google's multilingual neural machine translation system: Enabling zero-shot translation[END_REF]. This tagging scheme on the target side provides us with various ways to perform edit-based MT:

• We can perform direct inference without knowing the editing type associated with ẽ. In this case, Edit-MT evaluates itself the required edition types. The decoding procedure starts by first generating editing tags for the predicted edition type, then producing the corresponding edited translation. • When the operations are known, we can use the corresponding tags as a forced prefix to generate a translation with desired edits. Edit-MT thus is indicated by a required edition type. We can generate more controllable edited translations by forcing Edit-MT with certain editions. In a bilingual writing system, we can obtain the required operations by storing edit histories and computing the text differences in one human revision. • The inference procedure can also only generate the output editing tags and predict the relationship between the source f and the initial translation ẽ. For example, we can predict parallelism when the system outputs the copy tag

[!ins] [!sub] [!del].
In this situation, the decoding is truncated to a maximum length of 3 to ignore the translation part. For Edit-MT, the ability to perform standard MT is also preserved by training with a balanced mixture of artificial editing data and regular parallel data, as in [START_REF] Curran Associates | Neural fuzzy repair: Integrating fuzzy matches into neural machine translation[END_REF]Xu et al., 2021c). The regular translation data corresponds to an empty initial translation ẽ = [ ]. For these examples, the target side does not contain any tags.

. Bilingual Synchronization

. Datasets

We first evaluate Edit-MT on the general Bi-sync task where ẽ is assumed to be the translation of a former version of the source sentence f , and only a limited number of edits is sufficient to restore the parallelism. We conduct experiments on WMT14 English-French data Fr-En) and evaluate on two test sets. The first is an artificial derivation of the standard newstest2014 set, and the second is a small parallel sentence compression dataset 5 of [START_REF] Ive | Parallel sentence compression[END_REF]. For the artificial News task, we generate the required ẽ for each editing operation according to the methods used in Section 7.2, resulting in four versions (Ins, Sub, Del 1 , Del 2 ) of newstest2014, with each containing 3,003 sentences. The sentence compression dataset contains a subset of documents also selected from newstest2014, where sentences from both languages are manually compressed by human annotators while remaining parallel in the two languages. We only retain utterances where the compressed and uncompressed versions actually differ on both sides, resulting in 526 test sentences. We discard training examples with invalid language tags as computed by fasttext language identification model 6 [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF], yielding a training corpus of 33.9M examples. The parallel training data is identical to what we use in Chapter 5. We tokenize all data using Moses tokenizer and build a shared source-target vocabulary with 32K Byte Pair Encoding (BPE) units (Sennrich et al., 2016c) learned with subword-nmt.

7 Since we use both parallel and artificial editing data to train Edit-MT models, the total training data contains about 68M utterances.

. Experimental Settings

We implement our Edit-MT model and conduct experiments using the fairseq 8 toolkit [START_REF] Ott | fairseq: A fast, extensible toolkit for sequence modeling[END_REF]. Edit-MT relies on the Transformer-base model of [START_REF] Vaswani | Attention is all you need[END_REF]. We use a hidden size of 512 and a feedforward size of 2,048. We optimize using Adam with a maximum learning rate of 0.0007, an inverse square root decay schedule, and 4,000 warm-up steps. All input and output embedding matrices are tied [START_REF] Press | Using the output embedding to improve language models[END_REF][START_REF] Inan | Tying word vectors and word classifiers: A loss framework for language modeling[END_REF], and Edit-MT is trained with mixed precision and a batch size of 8,192 tokens on 4 V100 GPUs for 300k iterations. We save checkpoints for every 3,000 iterations and average the last 10 saved checkpoints for inference.

For comparison, we report a "do-nothing" baseline by simply copying the initial translation ẽ as the output. Performance is computed with SacreBLEU [START_REF] Post | A call for clarity in reporting BLEU scores[END_REF].

. Main Results

We first evaluate the ability of Edit-MT to perform standard translation. We use the original newstest2014 for both directions. The source side for Edit-MT contains an empty ẽ. We copy the base-mono results from Section 5.5 as baselines since these models are trained and tested on the same WMT14 data as in this section. From Table 7.2, we can observe that Edit-MT only performs slightly worse than the baseline, with -1.9 BLEU for En-Fr and -0.5 BLEU for Fr-En, which is expected, as also pointed out by [START_REF] Curran Associates | Neural fuzzy repair: Integrating fuzzy matches into neural machine translation[END_REF]. These results suggest that Edit-MT can perform standard translations with an acceptable loss of performance compared to baseline translation models. Therefore, in the rest of this section, we focus on the Bi-sync task.

Model

En-Fr Fr-En base-mono . . Edit-MT . .

Table . : BLEU scores of Edit-MT performing standard translation on newstest2014 for both En-Fr and Fr-En.

We separately evaluate the learnability of each editing operation. For this, we generate an initial version ẽ for each test sentence and each editing operation so that we have 3,003 test samples per operation. For deletion, we test the performance of both generation methods introduced in Section 7.2.3. We also derive two tasks from the compression test set: parallel sentence compression (comp) and extension (ext). For compression, the task consists of generating the compressed target sentence e comp , given the compressed source f comp and the original target e. For extension, the model is expected to produce e with the original source f and the compressed target e comp . These two tasks are respectively similar to the deletion and the insertion settings. There are slight differences, in any case. The initial translation ẽ for these settings is always syntactically correct, and the removed or inserted segments are selected for their lower informativeness. In this respect, these tasks are more about restoring an adequate rather than a fluid translation.

As mentioned in Section 7.3, the generation in Edit-MT models is conditioned on predicted or oracle editing tags that are prefixed to the output. We evaluate both situations. The oracle tag setting (+ tag) uses forced-prefix decoding with the correct tags. For the compression and extension tasks, we use the deletion and insertion tags, respectively. Tables 7.3 and 7.4 report results for both directions, to be contrasted with a baseline score corresponding to simply copying ẽ as the output. For this experiment, Edit-MT is able to edit the given ẽ for all types of required edits. It obtains large gains over the copy baseline for insertion, 9 substitution, and deletion for both translation directions. When tested on the more realistic compression and extension tasks, which have different edit distributions to the artificial training data, Edit-MT can also improve ẽ by 1.2-4.7 BLEU for En-Fr and 2.4-7.9 BLEU for Fr-En. Note that in the above settings, Edit-MT predicts the required editing type for each test sentence. By prefixing Edit-MT with the oracle editing type tags (+ tag), we can further increase the performance on almost every task in both directions. The effect of editing tags is further analyzed in Section 7.4.5.

. Multilingual Edit-MT

The general purpose of Bi-sync is to restore parallelism between two sentences. In the previous section, we have decomposed the synchronization directions into two separate models. The En-Fr model only edits initial French translations, and the Fr-En model only performs editing in the other direction. Ideally, we would like our model to perform Bi-sync in both directions within a single model, treating the two languages without any difference. In this section, we thus study the multilingual setting of our model.

BLEU gains in insertion are artificially high. This is because the copy baseline is hindered by a high brevity penalty. We swap the source sentence f fr and the initial translation ẽen for the Fr-En direction so that English sentences always appear first, regardless of being expected to be edited. The source side is therefore composed of training examples as f en [sep] ẽfr and ẽen [sep] f fr . For the target side, we further prefix a language tag to indicate the desired output language, i.e., the language to be edited. For instance, a French sentence needing substitutions will be prefixed with [fr] [!ins] [sub] [!del] to the target sentence. During inference, we always prefix the language tag to require the desired languages. The language tag is necessary for multilingual Edit-MT, as we can arbitrarily choose to edit either language to make it parallel to the other. We double the batch size when training the multilingual Edit-MT model.

As can be seen in Table 7.5, multilingual Edit-MT delivers similar results compared to its monolingual counterparts in both directions within a single model. This implies that we can effectively treat both languages equally when performing Bi-sync. In addition, multilingual models can save half of the deployment resources in real-world applications, which is always preferred over deploying two separate models.

. The Effect of Editing Tags

We discussed in Section 7.4.3 that simply prefixing Edit-MT with the oracle editing tags during inference can improve the performance in all cases. In this section, we conduct analyses to better understand the effect of editing tags.

We first evaluate the accuracy of editing tag prediction by taking the tags predicted during the synchronization process of Edit-MT for the basic editions tasks (Ins, Sub, Del 1 , Del 2 ). The two deletion test sets are combined as one, [ as they share the same editing tag. To gain a better understanding of the editing tag prediction, we also consider a copy operation, as we train Edit-MT with four different tags. The copy test set is built by simply concatenating the source sentence f with the reference target e as input. Edit-MT is thus supposed to predict the copy tag for this case, then produce the reference sentence unchanged. We use tags predicted by the monolingual models for each direction and plot the confusion matrix of the predicted tags for each edition type in Figure 7.3. Results show that Edit-MT generally predicts correctly for insertion, deletion, and copy operations. However, it fails to make precise predictions for sentences requiring substitution and wrongly considers half of the sentences as already parallel.

We further analyze the effect of using forced oracle tags by showing the attention weights computed by Edit-MT during inference. We use the encoder-decoder cross-attention weights of the last decoder layer. The attention weights are averaged over all heads (cf., Section 2.2.1). Figure 7.4 illustrates the attention weights when editing an initial translation requiring substitutions. We do not prefix anything, and Edit-MT incorrectly predicts the copy tag and leaves the output unchanged. The attention weights are mostly concentrated on the initial translation part ẽ of the input. In this case, Edit-MT almost ignores the original source f and only looks at ẽ to simply copy the output.

When prefixing Edit-MT with the correct substitution tag for the same input, the model is thus able to make proper changes. As shown in Figure 7.5, Edit-MT correctly substitutes "que" with "auxquels" and adds "accès", which corresponds to "access" in the source f but does not appear in ẽ. In this case, Edit-MT performs actual translation by looking at both the source tokens and the initial target tokens at the same time, even when a token should be directly copied.

. Translating with Translation Memories

As explained in Section 7.1, Bi-sync encompasses TM-based MT, whereby an existing similar translation retrieved from a TM is turned into an adequate translation of the source. As introduced in Section 3.3, here TM simply refers to parallel bilingual data. Edit-MT actually uses the same architecture as the retrieval-based model of [START_REF] Curran Associates | Neural fuzzy repair: Integrating fuzzy matches into neural machine translation[END_REF]. In this section, we study the performance of our Edit-MT model in this practical scenario.

. Related Words in Translaton Memory Matches

Given a source sentence f and a TM match pair ( f , ẽ) in which f is similar to f , apart from directly using the complete sequence of the retrieved similar translation ẽ as input, we also propose an alternative where we remove segments in ẽ that are not aligned with the input source f . Considering the example in Section 3.3, the segments [the flight] and [a cold ] are not related to each other on the source language side. From that, the target segment [le vol ] is thus irrelevant to translating the source sentence f . In this section, we discuss an algorithm capable of identifying the set of target tokens T ∈ ẽ that are related to tokens in the source sentence f . We define the related target tokens set T as:

T = ⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩ e j ∈ ẽ ∶ ∃ fi ∈ LCS | ( fi , ẽj ) ∈ A ∧ ∀ fi ∉ LCS | ( fi , ẽj ) ∉ A ⎫ ⎪ ⎪ ⎪ ⎬ ⎪ ⎪ ⎪ ⎭ , ( .
)
where A is the set of word alignments between words in f and ẽ, and LCS is the set of words in f , which belong to the Longest Common Subsequence (LCS) between f and f . The LCS is computed as a by-product of the edit distance [START_REF] Paterson | Longest common subsequences[END_REF]. Word alignments are computed by fast_align 10 ( [START_REF] Dyer | A simple, fast, and effective reparameterization of IBM model 2[END_REF]. Figure 7.6 illustrates the set of related words T , along with the LCS between f and f and the alignments between f and ẽ. The TM source sentence f has an LCS set of 5 tokens {How, long, does, last, ? }. The set of related target words T is also composed of 5 tokens {Combien, de, temps, dure, ? }, all aligned to at least one word in LCS and to no other word. Note that both LCS and T consist of collections of indices (word positions in their corresponding sentences) while word strings are used in the examples to facilitate reading. When applying this related setting, the actual sequence used as a similar translation is thus ẽ = [Combien de temps dure ? ]. This is structurally very similar to the insertion data introduced in Section 7.2.

? • ? ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ■ last • last ⋅ ⋅ ⋅ ■ ⋅ ⋅ ⋅ cold • flight ⋅ ⋅ ⋅ ⋅ ⋅ ■ ⋅ a • the ⋅ ⋅ ⋅ ⋅ ■ ⋅ ⋅ does • does ⋅ ⋅ ⋅ ■ ⋅ ⋅ ⋅ long • long ⋅ ■ ■ ⋅ ⋅ ⋅ ⋅ How • How ■ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅

Combien

. Datasets

Our experiments use a multi-domain corpus. This corpus contains 11 different domains for the En-Fr direction, collected from OPUS 11 (Tiedemann, 2012): documents from the European Central Bank (ECB); from the European Medicines Agency (EME); Proceedings of the European Parliament (Epp); legislative texts of the European Union (JRC); News Commentaries (News); TED talk subtitles (TED); parallel sentences extracted from Wikipedia (Wiki); localization files (GNOME, KDE, and Ubuntu) and manuals (PHP). All these data were deduplicated prior to training. To evaluate the ability of our models to actually make use of TMs instead of memorizing training examples, we also test on two unseen domains: OpenOffice (Office) from OPUS and the PANACEA environment corpus 12 (ENV).

For each source sentence, we retrieve from the same domain the top 3 TM matches according to the fuzzy match score described in Equation ( . ) based on the Fuzzy-Match implementation.

13 As we are mainly interested here in the editing behavior, we only keep TMs having a sufficiently similar translation but without exact matches on the source side, i.e., requiring a fuzzy match score FM ∈ [0.6, 1). We then split the data by keeping 1,000 sentences with at least one match as the test set for each domain. The remaining data is used for training. Note that the ratio of sentences with at least one similar translation greatly varies across domains. Detailed statistics about these corpora are shown in Table 7.6. We use all retrieved (up to 3) TM matches for training and only the best match for testing. The initial set of 4.4M parallel sentences (para) is thus augmented with about 2.6M examples for which a good TM match is available (similar ) or just the related segments (related ). Data preprocessing is the same as in Section 7.4.1, and we build a shared source-target vocabulary with 32K BPE units (Sennrich et al., 2016c).

. Experimental Settings

We consider two baseline settings for TM-based MT: the FM setting of Bulte and Tezcan (2019) and the FM # setting described in Section 7.5.1. Both settings concatenate the source sentence f and the similar translation or similar segments ẽ as the source input, while the model architecture is an unchanged Transformer model. The FM model is trained using para + similar data, and the FM # uses para + related. These two baselines are trained with the same configuration as in Section 7.4.2. We also report the "do-nothing" scores obtained by simply copying the retrieved similar translations as the output, as also done in Section 7.4.3.

The Edit-MT models trained in Section 7.4 differ from FM and FM # in both the task and the domains used for training. Hence, we consider reusing the generic Edit-MT model and fine-tuning it on this task to evaluate the capability of our model to generalize to this MT with TMs task. We use para + similar + related data and fine-tune the Edit-MT En-Fr model for only 1 epoch with a fixed learning rate of 8e

-5 . As we do not have any information about the expected editing operations required to change a similar translation into the reference, we set all editing tags as positive for similar data and prefix the output with [ins] [sub] [del]. We conjecture that mostly insertions are required for the related data as the irrelevant parts of ẽ have already been removed. We thus simply activate the insertion tag. Note that our fine-tuned model can perform translations with both similar sentences and related segments within the same model. We evaluate both BLEU and TER scores using SacreBLEU. We first perform standard translation on the multi-domain test sets without using the retrieved similar translations. As shown in Table 7.7, both FM and FM # yield strong performance as they are specifically trained on the indomain data. Our generic Edit-MT model performs much worse in general when performing zero-shot translations on these test sets. However, the performance greatly varies across domains. For instance, the generic Edit-MT performs similarly to FM on the News domain, as it is closer to the WMT data. The overall poorer performance of Edit-MT indicates the important domain mismatch between the generic WMT data and the in-domain data. However, by performing fine-tuning, Edit-MT can vastly improve its performance. Note that there is still a gap of about 5 BLEU points between the fine-tuned Edit-MT model and FM model, indicating that fine-tuned Edit-MT approaches specifically trained models but are still weaker. In the rest of this section, we focus on performing translations with TMs.

Results in Tables 7.8 and 7.9 reproduce the overall good performance of FM and show that FM # also significantly improves the copy baseline, even though the gains are smaller than FM. The generic Edit-MT performs much worse and does not even match the copy results in BLEU. TER results, however, show that even this model actually identifies useful edits, as we see improvements with respect to the copy baseline. When prefixed with the editing tag 14 (+ tag), we observe minor improvements (+0.8 BLEU on average) that are further increased in the related scenario (+ R + tag). Finetuning gives us a much more significant performance boost (+13.4 BLEU, -0.142 TER points on average). The differences between generic and finetuned models highlight the effect of the task and domain mismatches on our

We conjecture with a substitution tag for zero-shot inference as we have no information about the required edit type. Fine-tuned model uses the same tag as finetuning data.

initial results. The related setting also benefits from fine-tuning, albeit by a smaller margin (+9 BLEU). Our best results overall, using fine-tuning, are superior to FM # and close to that of FM. Note that our model is much worse than the two baseline models when performing standard translations. This has practical implications since FM # and FM are specifically trained on the in-domain data to transform a retrieved translation, whereas the generic Edit-MT is initially trained with artificial edits and then only slightly fine-tuned on the in-domain data. The results show that fine-tuned Edit-MT can better use retrieved TMs than baseline models under the same architecture, as Edit-MT is pre-trained on large-scale data that may contain more varieties of editions than the indomain data, which is relatively small.

To appreciate this difference, we directly test our models on the two unseen domains (Office and ENV). These domains are neither used to train FM and FM # nor to fine-tune Edit-MT. Results in Table 7.10 unambiguously show that in this setting, the fine-tuned Edit-MT even outperforms the strong FM model. This suggests that our Edit-MT model has adapted not only to the domain but also the task, as it can effectively perform zero-shot TM-based translation on unseen domains. The improvements indicate that Edit-MT has the potential to perform translations with TMs as a pre-trained model. The FM model requires large amounts of training examples with retrieved similar translations, which is difficult to find for some domains like News. However, training data for Edit-MT are synthetic and can be generated for all training data in arbitrary amounts. With pre-trained Edit-MT models, even a small amount of similar translations can benefit from a fine-tuning procedure, yielding an increase in performance.

Model

Office ENV copy . . To better understand Edit-MT on TM-based translation, we further analyze its performance grouped by the difference ∆ between ẽ and e (computed as edit distance): ∆ measures the editing effort needed to turn ẽ into e. For the results in Table 7.11, all 11k test sentences are merged into one test set, further broken down by values of ∆. Since translating with a complete TM sentence performs better than the related setting, we only take the complete TM setting for the following analyses. Apart from computing the BLEU scores, we also compute the average edit distance (AD) between systems' outputs and the references. The generic Edit-MT improves the similar translation in terms of both BLEU and AD for ∆ ≥ 3. However, it is difficult for Edit-MT to detect minor changes (∆ < 3) without fine-tuning. Once fine-tuned, Edit-MT performs similarly to FM for small changes, which further confirms that Edit-MT adapts to the TM-based translation task. We also break down the aggregate test set by editing operations required to transform ẽ to e. Table 7.12 shows that the generic Edit-MT struggles to perform well when substitutions are needed. Fine-tuning vastly improves the ability to substitute and delete from ẽ, separately or even in combination. Fine-tuned Edit-MT even outperforms FM when only deletions are required.

FM

. Parallel Corpus Cleaning

The objective of our models is to restore bilingual parallelism between a pair of sentences. This ability is also helpful for parallel corpus cleaning tasks, which we define as follows. Given a source sentence f and a possibly incorrect translation ẽ, we would like to detect cases of non-parallelism and perform appropriate edits to obtain a parallel sentence pair. We study how Edit-MT deals with this new problem on two publicly available datasets: first on the SemEval 2012&3 Task 8: Cross-lingual Textual Entailment (CLTE) for Content Synchronization [START_REF] Negri | Divide and conquer: Crowdsourcing the creation of cross-lingual textual entailment corpora[END_REF][START_REF] Negri | Semeval-2012 task 8: Cross-lingual textual entailment for content synchronization[END_REF][START_REF] Negri | Semeval-2013 task 8: Cross-lingual textual entailment for content synchronization[END_REF], then with the OpenSubtitles corpus [START_REF] Lison | OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles[END_REF].

. Cross-Lingual Text Entailment

The CLTE task aims to identify multi-directional entailment relationships between two sentences x 1 and x 2 , written in different languages. Each (x 1 , x 2 ) pair in the dataset is annotated with one of the following relation-ships:

• Bidirectional (x 1 ⇔x 2 ): the two fragments entail each other (semantic equivalence).

• Forward (x 1 ⇒x 2 & x 1 ⇍x 2 ): unidirectional entailment from x 1 to x 2 . • Backward (x 1 ⇏x 2 & x 1 ⇐x 2 ): unidirectional entailment from x 2 to
x 1 . • No Entailment (x 1 ⇎x 2 ): no entailment between x 1 and x 2 . The dataset contains a training set of 500 sentence pairs and two test sets of the same size (test-2012 and test-2013). Our evaluation is for the Fr-En direction, where x 1 is in French and x 2 is in English.

The tagging mechanism of Edit-MT introduced in Section 7.3 can readily be used for this classification task, subject to some adjustments as represented in Table 7.13. We then treat x 1 as f and x 2 as ẽ to match the input format of Edit-MT and perform zero-shot inference reusing the same Edit-MT models trained in Chapter 7. The source side is the concatenation of x 1 and x 2 , while the target sequence is truncated by only taking the first three edit tags as the predicted label for the corresponding input pair, treating our Edit-MT as a mere classification model as mentioned in Section 7.3. We also slightly fine-tune Edit-MT with the 500 examples of CLTE training data for 5 epochs with a fixed learning rate of 8e-5. Results are in Table 7.14, together with the best scores reported in [START_REF] Negri | Semeval-2013 task 8: Cross-lingual textual entailment for content synchronization[END_REF] for both test sets and the scores reported in [START_REF] Carpuat | Detecting cross-lingual semantic divergence for neural machine translation[END_REF], which are the best performance reported we could find. As can be seen, out-of-the-box Edit-MT fails to clearly detect the entailment relationships. This is not surprising, as there is a significant difference between our editing training data and the CLTE test sets. For instance, the initial translation for insertion is always grammatically incorrect, while all sentences in CLTE are syntactically correct. Nevertheless, after slight fine-tuning with the CLTE data, Edit-MT for both directions can quickly adapt to the task, achieving state-of-the-art performance.

15 This hints at the fact that Edit-MT actually learns to identify various cases of non-parallelism and generate appropriate edits.

. Fixing OpenSubtitles Corpus

We further evaluate the ability of Edit-MT to detect parallel sentences and to fix noisy data. We experiment with the OpenSubtitles 16 data [START_REF] Lison | OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles[END_REF] for the En-Fr direction. The French side is translated from English in this corpus but contains noisy segments. A standard approach is to filter out these noisy sentences from the training material when building systems. Our goal in this experiment is to see whether Edit-MT can automatically identify and edit, rather than discard, noisy sentence pairs so that training can use the entire set of available parallel data. We evaluate the performance on the 10,159 segments of the En-Fr Microsoft Spoken Language Translation (MSLT) task [START_REF] Federmann | Microsoft speech language translation (MSLT) corpus: The IWSLT 2016 release for English, French and German[END_REF], which offers an MT scenario motivated by real-world applications.

The OpenSubtitles data is first deduplicated and then processed similarly to Section 7.4.1. We first apply the fine-tuned classification model used for the CLTE task to predict the relation for all sentence pairs in OpenSubtitles data. Approximately 60% of the data is classified as parallel, indicating that no edit operation was predicted for those segments. Models trained on these 60% clean data are denoted below as filtered. For the other 40% of presumably noisy data, we reuse the generic Edit-MT En-Fr model trained in Section 7.4 to edit the translations, using the predicted edit tag as a prefix on the target side (cf., Section 7.3). Models trained using the edited data are denoted as fixed. We train NMT models with all data (full) or just the 40% noisy data as baselines. For comparison, we also train a model containing the same number of sentences (15.8M) as the noisy subset, randomly selected from the filtered subset.

Results are in Table 7.15. Aggressively filtering the noisy data improves over using the entire training corpus (+2 BLEU) more than revising it (+1 BLEU). The second set of results yields similar conclusions with smaller datasets: here, the effect of automatically fixing a set of initially noisy data improves the BLEU score by 7.2 points and closes half the gap with a completely clean corpus of the same size. Note that these results were obtained without adaptation, simply reusing the pre-trained generic Edit-MT model. This suggests that editing-based strategies may provide an effective alternative to filtering in situations where the training data is small and noisy.

. Conclusion

In this chapter, we introduced another possible solution for bilingual writing, which simulates a revision scenario where updates in one language need to be propagated to another language. We proposed a new Bi-sync task that corresponds to this situation. We also proposed various ways to create artificial initial translations for different editing types needed to train Bi-sync models. We have explored autoregressive architectures for this task. Experiments showed that our Edit-MT models trained with a mixture of real parallel data and artificial triplets were able to perform Bi-sync tasks even in real-world scenarios while also maintaining the ability to perform standard translation. We have also shown that a multilingual edit-based model could perform Bi-sync in both directions within a single model. This led to more realistic application scenarios like bilingual writing, where we can treat both languages as completely equivalent.

We also explored Bi-sync on two specific downstream tasks. Our experiments demonstrated that Edit-MT could be applied to fix TMs by detecting parallel sentences with a slight amount of fine-tuning and correcting imperfect translations without any adaptations. We have also shown that Edit-MT could be quickly adapted to multi-domain MT with TMs tasks, where it could outperform dedicated models when tested on unseen domains.

In our future work, we would like to study ways to reduce the computational cost of fully re-decoding the input sequence for Edit-MT, especially in contexts where small changes that need to be reproduced in the target are repeatedly applied to the source sequence. We also want to experiment with actual bilingual writing interactive scenarios where such techniques apply.

-Non-autoregressive Bilingual Synchronization

. Introduction

In Chapter 7, we studied bilingual synchronization (Bi-sync) for autoregressive (AR) architectures. Bi-sync models are expected to edit an initial target sequence to restore parallelism between the source and target languages. However, AR neural machine translation (NMT) systems do not really edit an initial target sequence. Due to the AR generation, these systems have to generate a complete target sentence from scratch in order to make changes to an initial target sequence present on the source side. Even though only a few edits are required, AR models still need to regenerate the entire target sequence through a standard decoding process. In an interactive bilingual writing scenario, always recomputing the entire target sequence is time-consuming, especially for the AR generation.

On the contrary, some non-autoregressive machine translation (NAT) models have been proposed to perform iterative refinement decoding [START_REF] Lee | Deterministic non-autoregressive neural sequence modeling by iterative refinement[END_REF][START_REF] Ghazvininejad | Mask-predict: Parallel decoding of conditional masked language models[END_REF][START_REF] Gu | Levenshtein transformer[END_REF], where the model generates translations by iteratively editing the outputs of previous iterations, starting with a possibly empty initial hypothesis, as we introduced in Section 4.1.3. Among these NAT approaches, edit-based models like the Levenshtein Transformer (LevT, [START_REF] Gu | Levenshtein transformer[END_REF] seem to be a natural fit for the Bi-sync task. It has recently been applied to incorporate lexical constraints in machine translation (MT) [START_REF] Susanto | Lexically constrained neural machine translation with Levenshtein transformer[END_REF]Xu and Carpuat, 2021a), as discussed in Section 4.5.

Therefore, in this chapter, we study Bi-sync with non-autoregressive approaches. We show that the vanilla LevT cannot perform well on this task and explain why this is the case. We propose Edit-LevT, a new NAT model based on the LevT with an improved training procedure to incorporate empty and non-empty initial target sentences. Edit-LevT is able to perform Bi-sync much better than LevT. We also apply Edit-LevT to the MT with translation memories (TM) task, as also considered in Section 7.5 for AR Edit-MT.

We pay special attention to the NAT with TMs task and study it more in-depth, as it is a realistic application of Bi-sync but has never been studied to the best of our knowledge. We propose TM-LevT for this task, which is adapted from Edit-LevT with a different source input format. We also improve the training procedure in two ways: (a) by also including the initial candidate translation on the source side, as done in the AR decoding with TMs (Bulte and Tezcan, 2019); (b) by simultaneously training with empty and non-empty initial target sentences, which is also applied to Edit-LevT. In our experiments, TM-LevT achieves comparable performance to the AR approach on various domains when translating with and without TMs. It also yields a faster decoding speed with a reduced decoding load. We also observe that incorporating an initial translation on both source and target sides makes the widely used sequence-level knowledge distillation (KD, Kim and Rush, 2016) useless, even when performing translation from scratch. This contrasts with standard NAT models, which resort to KD to alleviate the multimodality issue (Gu et al., 2018a), as discussed in Section 4.4. As far as we know, we are the first to perform NAT with TMs and match AR model performance in this setting.

Our main contributions in this chapter are the following:

• We propose a new NAT model based on the LevT that is able to perform Bi-sync and achieves similar results to AR Edit-MT models from Chapter 7. • We perform empirical validations of the non-autoregressive Bi-sync model on the MT with TMs task to validate the generalization ability of our non-autoregressive Bi-sync model. • We improve our proposed NAT model to perform the NAT with TMs task and achieve comparable results to a strong AR approach. Some of the contributions in this chapter are published in (Xu et al., 2022b,c).

. Edit-LevT

Even though the edit-based nature of LevT seems to suit Bi-sync, it is still designed to perform standard MT. The original LevT model initializes the decoder with a randomly noised version of the target reference during training and starts with an empty sentence in inference (cf., Section 4.1.3). This means that the input to the LevT decoder e ′ is always a subsequence of e, and the deletion operation in LevT is only trained to detect prediction errors made by the model itself. In Bi-sync, we would instead initialize the target side with the given initial translation ẽ in both training and inference so that the model can directly edit the given target sentence. However, the original training scheme does not suit Bi-sync, as the initial translation ẽ may contain tokens that are not present in the reference e but correspond to a previous source f and should be removed. Figure 8.1 shows an example of an initial translation ẽ containing the word "asleep" that should be replaced. The distribution of these tokens may greatly differ from token prediction errors made by LevT, as they are not really "errors" but just irrelevant to the actual reference e. To make this point, we have experimented with original LevT models by simply replacing the noised decoder input e ′ with the initial translations ẽ generated in Section 7.2 for training. However, this model cannot make proper edits and almost considers ẽ as the final translation. This is further discussed in Section 8.3.2. Therefore, we propose Edit-LevT, a new NAT model aiming to fix this issue and adopt the LevT model for Bi-sync. To do so, we modify the training regime and add an extra deletion operation (init-del) before the insertion operation to the original LevT. As illustrated in Figure 8.1, init-del is trained to detect irrelevant tokens from the initial sequence ẽ, while the final deletion (final-del) aims to delete prediction errors made by the model, therefore making Edit-LevT able to detect both irrelevant tokens and its own errors. We apply the initial deletions to e ′ during training by taking the union of reference and predicted deletions in the init-del operation, resulting in a subsequence e ′′ which is then passed to the insertion operation. We use the same deletion classifier for both the init-del and the final-del operations, which means that Edit-LevT does not contain any extra parameters com-pared to the original LevT. During inference, Edit-LevT performs as LevT, iteratively applying deletions and insertions to an initial candidate translation until convergence or a maximum decoding round is reached.

To adapt Edit-LevT to Bi-sync, for each triplet of samples (f , ẽ, e), we use the initial translation ẽ to initialize the decoding (e ′ = ẽ). Different from Edit-MT, the source side of Edit-LevT only contains the source sentence f . Finally, in order to preserve the ability to perform standard MT, our training data is prepared as follows: with a tunable probability p, we decide either to train with an initial translation ẽ or to train from scratch. In the former case, the decoder is initialized with ẽ, while in the latter case, we use a noised subsequence e ′ generated as in [START_REF] Gu | Levenshtein transformer[END_REF]. Edit-LevT is then jointly trained on both tasks. The probability p controls the proportion of each sample type. Taking p = 0 is equivalent to training an Edit-LevT model with only parallel data. We use p = 0.5 in our experiments, making it equivalent to mixing parallel and editing data as in the Edit-MT model (cf., Section 7.3). The probability p can be carefully designed with a schedule or a curriculum during training to optimize the behavior of Edit-LevT, which we leave for future work. Note that for Edit-LevT, we do not use any tags on the target side, as Edit-LevT already includes an internal mechanism to predict the editing operation(s). During inference, we initialize the target sequence with initial translations ẽ, using a setting akin to that of [START_REF] Susanto | Lexically constrained neural machine translation with Levenshtein transformer[END_REF] for LevT with lexically constrained decoding, which does not involve any changes in the iterative refinement decoding algorithm.

. Bilingual Synchronization

. Datasets and Experimental Settings

We reuse the same training and test data for En-Fr and Fr-En in Section 7.4.1 in this experiment. Edit-LevT is implemented using the fairseq 1 toolkit [START_REF] Ott | fairseq: A fast, extensible toolkit for sequence modeling[END_REF]. We use a hidden size of 512 and a feed-forward size of 2,048. We follow [START_REF] Gu | Levenshtein transformer[END_REF] to optimize using Adam with a maximum learning rate of 0.0005, an inverse square root decay schedule, and 10,000 warm-up steps. We also share all embedding matrices and use a shared decoder for the placeholder insertion, token prediction, and the two deletion operations. Each operation thus only differs in the linear classifier. However, the init-del and final-del operations also share the same deletion classifier. Edit-LevT models are trained with mixed precision and a batch size of 16,384 tokens on 4 V100 GPUs for 300k iterations. We save checkpoints for every 3,000 iterations and average the last 10 saved checkpoints for inference. For inference, we set a maximum decoding round of 10. Sequence-level KD is widely used in NAT model training (Gu et al., 2018a). However, we use real target references instead of distilled targets from an AR model for training, as our editing data ẽ are generated based on real target sentences e, and also to be fully comparable with Edit-MT in Chapter 7 trained on the same data.

We also train a vanilla LevT model which does not contain the initdel module. Similar to Edit-LevT, the vanilla LevT model also takes ẽ as initialization for the decoder with a probability of p = 0.5 during training. Therefore, the only difference between LevT and Edit-LevT is the init-del operation. Table 8.13 summarizes the architecture and data format of different models used in this chapter. We report the "do-nothing" copy baseline and compare Edit-LevT to the vanilla LevT and the AR Edit-MT model under the same test settings in Section 7.4.3. Performance is computed with SacreBLEU [START_REF] Post | A call for clarity in reporting BLEU scores[END_REF].

. Main Results

We first experiment with LevT and Edit-LevT performing standard translation for both directions. In this setting, the decoder of both NAT models is initialized with empty. Results are shown in Table 8.1. For NAT models, the overall performance lags behind AR approaches. This is not surprising, especially when trained without KD. However, our Edit-LevT can outperform the vanilla LevT model trained with the same data. This implies that training with init-del has positive effects for standard translation. In the following of this section, we focus on performing the Bi-sync task with Edit-LevT. Results of the Bi-sync task for both directions are shown in Tables 8.2 and 8.3. The original LevT model can slightly improve ẽ in most test situations. However, the performance gains are much lower, and it does not make many differences compared to the copy baseline except for the insertion task. The main reason behind this phenomenon is that, in the design of LevT training, even if we initialize the target side with ẽsub or ẽdel , which are not a subsequence of the reference e with certain tokens to be deleted, the training procedure always starts with the insertion operation to evaluate if it should insert placeholders in ẽsub or ẽdel . Therefore, tokens to be substituted or deleted in ẽsub or ẽdel are never seen by the deletion operation, which makes the LevT model almost perform the standard NAT task rather than make use of initial translations ẽ.

On the contrary, our Edit-LevT model obtains more significant gains compared to LevT in all situations. Since the only difference between Edit-LevT and LevT is the init-del operation, the results demonstrate that adding init-del effectively helps Edit-LevT learn to detect and delete tokens from an initial translation. Nevertheless, there is still a clear performance gap between Edit-LevT and Edit-MT (without oracle tags). This is not surprising as NAT models generally suffer from a performance drop compared to autoregressive approaches due to the independent assumption of target tokens, especially in our case where Edit-LevT is trained without using the distilled data. However, the performance gap is narrowed compared to the gap when performing standard translation in Table 8.1. Edit-LevT can even outperform Edit-MT without oracle tags on the extension task for En-Fr. Note that we do not compare Edit-LevT to Edit-MT with oracle tags, as the latter incorporates extra oracle information that Edit-LevT does not possess.

. Multilingual Edit-LevT

Similar to Section 7.4.4, we also explore Edit-LevT under the multilingual setting. Since the source side of Edit-LevT only contains the source sentence f , we do not use language tags to distinguish the target language and simply concatenate data from both directions for training. We also double the batch size when training the multilingual Edit-LevT. As shown in Table 8.4, the multilingual Edit-LevT performs similarly to its monolingual counterparts on the synthetic test sets (Ins, Sub, Del 1 , Del 2 ) in both directions. On the more realistic compression (comp) and extension (ext) test sets, multilingual Edit-LevT even outperforms monolingual models in both directions for both tasks. This implies that for Edit-LevT, training with data from both directions can even help to boost the performance to some extent. Note again that, similar to the previous section, we do not compare Edit-MT models with oracle tags.

. Simplifying Editing Data for Edit-LevT

The editing training data generated in Section 7.2 is composed of ẽins , ẽsub , ẽdel , and ẽcopy . However, the original training scheme of the LevT model uses a subsequence of the target reference as the decoder input e ′ . This subsequence is, in fact, similar to our insertion data ẽins , as both are generated by randomly dropping tokens from the reference e. However, the subsequence e ′ is generated without specific constraints, and the percentage of dropped tokens is uniformly distributed between 0 and 1. Since we include dropping constraints in Section 7.2.1, the dropped token ratio for ẽins is thus concentrated in a certain range. Therefore, the insertion data may be simplified since we apply a probability p = 0.5 to use the e ′ when training our Edit-LevT model.

On the other hand, the ability to keep already parallel sentences unchanged in Edit-LevT is also preserved by the iterative refinement decoding, which will reach a convergence when it detects the initial input as a proper translation. Therefore, we would like to study if we can simplify the editing data required to train Edit-LevT models. To do so, we prepare a new editing training data in which the initial translation ẽ is composed only of ẽsub and ẽdel data. Simplifying the training data also allows Edit-LevT to see more substitution and deletion data during training, potentially improving the performance of these two editing types. In this section, we only experiment with monolingual Edit-LevT models.

Results reported in Table 8.5 suggest that training Edit-LevT with only ẽsub and ẽdel editing data does not hurt the performance in general and even brings gains over almost all situations for both directions. The insertion and extension tasks for Fr-En are not improved, but the performance remains almost the same. The improvement mostly comes from the compression task and deletion tasks. As expected, seeing more data helps to improve the ability to detect segments in the initial translations that should be removed. However, the performance gain for the substitution task is much smaller.

. Translating with Translation Memories

Similar to Section 7.5, we also adapt our Edit-LevT model to the MT with TMs task in this section to evaluate the ability of Edit-LevT to generalize to this practical scenario.

. Datasets and Experimental Settings

We reuse the same training and test datasets in Section 7.5.2 and consider the same FM and FM # baselines as in Section 7.5.3. As Edit-LevT also differs from the two baselines in both the task and the domains, we also consider fine-tuning the general Edit-LevT model. We use para + similar + related data to fine-tune the generic Edit-LevT model. Unlike the AR models, the similar translation or segments ẽ are not concatenated to the source for Edit-LevT. On the contrary, they are used to initialize the decoder during training (see Table 8.13). The initial translation ẽ for para data is set to empty. We follow the settings in Section 8.2 to use a probability of p = 0.5 to select initial translations ẽ or randomly noised reference subsequences e ′ .

We use the pre-trained Edit-LevT En-Fr sub+del model, as it yields better general results, and fine-tune it for 2 epochs with a learning rate of 9e -5

to keep approximately the same number of updates as Edit-MT since Edit-LevT uses a bigger batch size. The fine-tuned Edit-LevT model can perform translations with both similar sentences and related segments as the initial state within the same model. We evaluate both BLEU and TER scores using SacreBLEU.

. Results

We first perform standard translation on the multi-domain test sets without using the retrieved similar translations. As shown in Table 8.6, the generic Edit-LevT model performs worse than the generic AR Edit-MT model. This is not surprising, as AR models are always better than NAT models. By performing fine-tuning, Edit-LevT can greatly improve its performance. Even though it always cannot match the performance of Edit-MT, the gap is similar to that in Section 8.3.2, showing that Edit-LevT adapts to the new domains in a similar trend as Edit-MT, while its general ability lags behind Edit-MT. In the following of this section, we focus on translating with TMs. Results in Tables 8.7 and 8.8 indicates that the generic Edit-LevT performs similarly to the generic Edit-MT, but fine-tuning drastically improves the performance (+10.1 BLEU, -0.11 TER points on average). Nevertheless, the quality of fine-tuned Edit-LevT still lags behind Edit-MT. The related setting also benefits from fine-tuning, albeit by a smaller margin (+6.4 BLEU). The performance gap between Edit-LevT and Edit-MT on average is similar to that in Table 8.1, indicating that Edit-LevT also adapts to this task, with a similar trend as Edit-MT. Note that we compare the best AR Edit-MT configuration with prefixed tags, as the tags are not oracle and the effect of tags is less significant for Edit-MT in this setting. We also evaluate Edit-LevT on the two unseen domains (Office and ENV). Results in Table 8.9 show that in this setting, the fine-tuned Edit-LevT is also able to make good use of TMs to close the performance gap with both FM and FM # . It even surpasses the FM # model and achieves performance on par with FM in the ENV domain. This suggests that Edit-LevT has also adapted to the domain and the task, as it can effectively perform zero-shot TM-based translation on unseen domains.

. Analyses

To better understand Edit-LevT on TM-based translation, we further aggregate the 11 test sentences and break them down by the difference ∆ between ẽ and e (computed as edit distance) in Table 8.10, as in Section 7.5.5. The generic Edit-LevT model performs worse than the copy baseline in all cases. After fine-tuning, the performance gap between Edit-LevT and Edit-MT is smaller when minor edits are needed (∆ < 3). Fine-tuned Edit-LevT is better than FM and fine-tuned Edit-MT models at detecting the initial translations that do not need any edits (∆ = 0). As mentioned in Section 8.1, AR models need to regenerate the entire target sequence through a standard decoding process, even though they only need to copy the initial ẽ. Therefore, we cannot guarantee that they will always copy the initial ẽ exactly. On the contrary, Edit-LevT leaves the initial ẽ unchanged if it detects that ẽ is already parallel to the source. We also break down the aggregate test set by editing operations required to transform ẽ to e. Table 8.11 shows that the generic Edit-LevT performs worse than the copy baseline when substitutions are needed. A substitution in Edit-LevT requires first deleting a token and then generating another token. Once a token is deleted, information from this token is lost, and Edit-LevT has to condition on other existing tokens to predict a new token. Fine-tuning vastly improves the ability to substitute and delete from ẽ, separately or even in combination.

. More Studies of Non-autoregressive Translation with Translation Memories

In the previous section, we experiment with the Edit-LevT model to perform fine-tuning on the MT with TM task. In this section, we look more in-depth at the task of NAT with TM. To the best of our knowledge, this task has not been studied yet in the literature.

. Incorporating Translation Memories with TM-LevT

We adapt our Edit-LevT model to TM-LevT to better fit this task. The model architecture of TM-LevT does not change compared to Edit-LevT. The only difference is in the input source format (see Table 8.13). As LevTbased models only support deletions and insertions, reordering operations can only be obtained by first deleting a token, then reinserting it at another position. As these operations are performed independently, they may cause the erasure of valid words from a similar translation ẽ. To mitigate this risk, we make sure that ẽ is always fully available to the decoder by concatenating ẽ to the source f to make the input sequence, as done in (Bulte and Tezcan, 2019) and in Edit-MT in Chapter 7. Therefore, the initial translation ẽ in TM-LevT is present on both the source and target side as initialization.

Similar to Section 8.4, we concatenate parallel data and TM augmented data for training. In this section, we discard the related data and only use complete similar translations retrieved from the TM. The initial ẽ for parallel data is considered empty for both the encoder and decoder sides. We again use a probability of p = 0.5 to use ẽ or the noised subsequence e ′ as decoder initialization. For inference, we follow the same input format as training when performing NAT with TMs.

. Datasets

The dataset we use is the same as in Section 8.4. However, to extend the applicability of using TMs, we perform a different range of fuzzy matches, requiring a fuzzy match score FM ∈ [0.4, 1). For each domain, we prepare two test sets with 1,000 sentences each: one containing randomly selected sentences having a close match (FM ≥ 0.6) in the TM, the other containing sentences with an acceptable match (FM ∈ [0.4, 0.6)). The remaining data is used for training. Details about the retrieved TM ratios for both FM ranges are shown in Table 8.12. By releasing the fuzzy match score range to 0.4, the number of sentences with TM matches approximately doubles. We also use all retrieved (up to 3) TM matches for training and only the best match for testing. This time, the initial set of 4.4M parallel sentences is thus augmented with about 5M examples for which a TM match is available. The Byte Pair Encoding (BPE) model and vocabulary are the same as in Section 8.4. In this section, we only consider using the complete sequence of retrieved TM matches.

. Experimental Settings

We follow the same training configuration and procedure as Edit-LevT in Section 8.3.1 to train TM-LevT. We compare TM-LevT with the FM approach of [START_REF] Curran Associates | Neural fuzzy repair: Integrating fuzzy matches into neural machine translation[END_REF]. Note that the training data used in this section is different from Section 8.4, as it contains more TM matches with an FM score greater than 0.4. We also train a LevT-FM model with exactly the same training data as FM. That is to say, LevT-FM also has TMs concatenated on the source side, while the target side is a vanilla LevT decoder. Table 8.13 gives a summarization of all compared models. These baselines use the same training data as TM-LevT, and process examples with and without TM matches on the source side. Contrarily to TM-LevT, TM matches for these models only appear concatenated to the source sentence, and translation always starts from scratch for both baselines. We also report results of a "do-nothing" baseline which simply outputs the TM matches. Performance is measured by SacreBLEU [START_REF] Post | A call for clarity in reporting BLEU scores[END_REF] and COMET [START_REF] Rei | COMET: A neural framework for MT evaluation[END_REF].

. Main Results

We evaluate the performance of both standard MT and MT with TMs on the two test sets introduced in Section 8.5.2. We report aggregated results computed on the concatenation of all domains (11k sentences) in the main text (Table 8.14). The results broken down by domains when performing MT with TMs on the test sets with FM ≥ 0.6 are also shown in Tables 8.15 and 8.16. Detailed results with a breakdown by domains for other settings are shown in Appendix D. As shown in Table 8.14, the FM baseline using TMs yields higher BLEU and COMET scores than the standard MT without TMs in both test sets. LevT-FM can also make good use of TM matches, but its performance lags way behind the FM model. However, unlike [START_REF] Gu | Levenshtein transformer[END_REF], who claim that the original LevT can perform zero-shot automatic post-editing, our results (+tgt TM in Table 8.14) show that, given a trained LevT-FM model, when we initialize the inference with ẽ instead of an empty sequence, LevT-FM is not able to make proper edits and almost considers ẽ as the final translation, even though the TM match ẽ is always present on the source side. This difference may come from the fact that editing NMT outputs and TMs are quite different, even for human translators [START_REF] Sánchez-Gijón | Post-editing neural machine translation versus translation memory segments[END_REF], as mentioned in Section 3.3.

Surprisingly, TM-LevT performs remarkably well when translating from scratch, even surpassing the FM model on BLEU scores on the FM ≥ 0.6 set, which is arguably easier as it is more similar to the training data. Compared to LevT-FM, TM-LevT significantly improves the performance of standard translation. Note that these models are trained without using KD on the target side, which implies that initializing the TM-LevT decoder with TMs may be an orthogonal method to KD to improve standard translation quality. When translating using TM matches, TM-LevT almost closes the gap with the FM approach on BLEU, especially for the subset containing higher quality matches, where it even outperforms the FM approach in some domains, as shown in Table 8.15. The gap in COMET score between TM-LevT and the FM model (see Table 8.16) is much smaller than reported by [START_REF] Helcl | Nonautoregressive machine translation: It's not as fast as it seems[END_REF], indicating that TM-LevT actually outputs valid translations. Figure 8.2 illustrates the performance gains obtained by translating with TMs of different similarity ranges. The FM score ranges also indicate the similarities between the test sentences and the training data. Higher similarities imply that the test sets are closer to the training data, therefore, easier to translate from scratch. For both FM and TM-LevT, using more similar TMs yields larger improvements, while using less similar TMs can still obtain significant performance gains. However, the improvements obtained by TM-LevT are always smaller than FM in both cases. The benefit of using less similar TM matches is important as high-similarity TMs are often difficult to find in real-world applications.

The FM approaches are known to improperly copy "unrelated tokens" from TM matches into output translations, as we report in [START_REF] Xu | Boosting neural machine translation with similar translations[END_REF]. We define here unrelated tokens as those present in ẽ but not in e, which is different from [START_REF] Xu | Boosting neural machine translation with similar translations[END_REF]. More precisely, we compute the edit operations between ẽ and e based on edit distance and count tokens in ẽ that should be deleted or substituted as unrelated tokens. We then compute the edit operations between ẽ and the hypothesis ê produced by our models to evaluate if these tokens are correctly deleted or mistakenly kept in the output. We omit the use of a word alignment model to detect unrelated tokens aligned with the source f , as introduced in Section 7.5.1, as word alignment models are not perfect and yield additional errors that may lead to an imprecise measure of unrelated tokens. Table 8.17 reports the percentage of such unrelated tokens that are not removed from the hypotheses. TM- LevT is less prone than the FM model to recopy unrelated parts of the TM matches. LevT-FM does even better in this setting, but its lower BLEU scores suggest that this approach also discards related tokens. Table . : The average percentage of unrelated tokens from the retrieved TM matches appearing in the final translations evaluated on the multi-domain test sets.

. The Effect of Knowledge Distillation

KD is now the "by default" technique used in most NAT models, as it reduces the complexity and lexical diversity of target sentences, thereby helping NAT approaches mitigate the multimodality issue [START_REF] Zhou | Understanding knowledge distillation in non-autoregressive machine translation[END_REF]Xu et al., 2021b). Here we conduct experiments with KD to evaluate its effectiveness in NAT with TMs. We train a Transformer-base teacher model on the 4.4M parallel data and use it for distillation. The distillation process is done by translating the source sentences using the teacher model with beam search. As we use TMs, which are also in the target language, we thus consider two situations: applying KD only to the target reference and to both TMs used in training and the reference. Note that applying KD simply replaces the target reference and/or TMs with the corresponding distilled data generated by the AR teacher model, while all other settings are unchanged.

As expected, using KD on the reference side does improve the performance of TM-LevT on standard translation, as reported in Table 8.18. This further confirms the hypothesis in Section 8.5.4 that KD is complementary to initializing the TM-LevT decoder with TMs for standard translation. However, KD seems unnecessary when editing a similar initial translation (w/TM) and yields a significant drop in scores compared to using real data for training. One possible explanation is that there is a mismatch between real TM matches and distilled training references. However, this mismatch also exists in training as we perform inference under the same condition as training. Applying KD also to TM matches crosses out this mismatch both for training and inference. Nevertheless, it still hurts the performance on both tasks with and without TMs.

. Ablation Analysis

We further study the effectiveness of each component of our method. We experiment with several settings and train a new model for each contrast:

• Removing the initial translation from the target side (-tgt TM), which makes the model similar to the original LevT model, but with an additional init-del operation; • Removing the TM from the source side (-src TM); • Removing the final-del operation in training (-final-del); • Using the reference deletion operation for init-del instead of the selfprediction operation (-self-pred). As illustrated in Table 8.19, training without TM matches on the target side vastly degrades the scores in all test conditions, indicating again that standard MT can also benefit from training with TMs as initial targets. This also implies that only performing the init-del without initial translations is meaningless. On the contrary, training without TM matches on the source side improves standard MT, as also pointed out by [START_REF] Curran Associates | Neural fuzzy repair: Integrating fuzzy matches into neural machine translation[END_REF] and found in Section 7.4.3. However, it has a negative impact when translating with TMs. This highlights the benefits of remembering ẽ on the source side. We also compare alternative implementations of the deletion operation. Results in Table 8.19 (-final-del) show that removing the final deletion operation mainly impacts TM-LevT in the standard MT setting, where the detection of wrong predictions matters most (Huang et al., 2022). This further demonstrates that unrelated tokens from TMs and prediction errors made by the model are quite different, and training with both operations is necessary. Last, we experiment by using only reference deletion labels to train the insertion operation instead of using the union of both the reference and model predictions (see Section 8.2 and Figure 8.1). We observe (-self-pred) a slight performance drop with respect to the complete TM-LevT, suggesting that this component is less important in the model than others. Using TMs in MT is expected to not only improve the translation quality but also reduce the decoding load, as certain segments in TMs can be directly reused, and models only need to edit a similar translation. However, the decoding load is not really reduced in FM or LevT-FM, as they always perform a standard translation process with only an extended source input. The decoding procedure always starts from scratch. TM-LevT, on the contrary, uses an initial translation to speed up decoding. We measure the average number of iterations needed in inference in Table 8.20. As can be seen, translating with TMs reduces the decoding effort for TM-LevT by about 20% fewer iterations, while LevT-FM remains almost unchanged. We also find that TM-LevT needs fewer iterations to converge than LevT-FM in all conditions. The main advantage of using NAT models is the inference speedup when generating multiple tokens in a single decoding step. Table 8.21 compares the average decoding time per sentence on each domain for the AR FM approach and TM-LevT. TM-LevT is much faster than FM when performing MT both with and without TMs. Even though [START_REF] Helcl | Nonautoregressive machine translation: It's not as fast as it seems[END_REF] state that NAT models are not as fast as it seems since the evaluation condition matters, we use the same hardware condition and inference batch size for all cases, making our results comparable, at least in our scenario. The decoding time of FM greatly varies across domains, while it is more stable for TM-LevT. This is mainly due to the length of generated sequence. Generating longer sequences for FM takes more time, while NAT models can parallel the generation process of all tokens. We also notice that TM-LevT takes more time when translating with TMs than without TMs, even though it requires fewer decoding iterations. This is probably due to the data processing step when initializing the target sequence with a TM. Besides, encoding an ex-tended source input also takes more time, which is consistent for both FM and TM-LevT.

. Conclusion

In this chapter, we studied Bi-sync with non-autoregressive approaches to simulate the bilingual writing scenario. We demonstrated that nonautoregressive Edit-LevT could also perform well on the general Bi-sync tasks. Multilingual Edit-LevT even outperformed monolingual ones on the more realistic compression and extension tasks. Edit-LevT could also be quickly adapted to multi-domain MT with TMs tasks, where it could outperform strong AR baselines when tested on unseen domains. In the in-depth study of NAT with TMs, we found that by copying the TM matches both on the source side and on the target side as an initial target sequence, our TM-LevT model vastly outperformed the LevT-FM model and achieved BLEU scores that approached those of a strong AR model both when decoding from scratch and when editing a TM match. TM-LevT also generated translations that contained fewer unrelated tokens and reduced the decoding load with fewer iterations. Finally, we demonstrated that training with TMs helps to improve NAT performance on standard MT without resorting to KD.

In our future work, we would like to experiment with actual users in realistic interactions and explore questions related to human-machine interaction in a bilingual writing assistant system, e.g., the visualization of automatic edits or verifying that the user inputs are not erased by the system, etc. Besides, we would like to further explore more in NAT systems, which are computationally faster, and try to improve their performance to generate high-quality automatic edits within a low latency. For this, we intend to study a carefully designed training curriculum. We would also like to experiment on other language pairs for the NAT with TMs task, especially on more distant language pairs, to further validate the TM-LevT model.

In this dissertation, we have explored new methods to assist bilingual writing in two different simulated situations. The first simulation allows a single sentence to be composed in mixed-language (MXL), which is then translated into monolingual sentences in both component languages. The second scenario assumes texts in two languages are separate and makes it possible to write in either language, while the other language is automatically updated to keep both sides synchronized. In both situations, our ultimate purpose is to perform double authoring, i.e., to have two versions of the same text in the two languages of equally good quality.

For the first case, we have thoroughly studied the task of generating monolingual sentences from an MXL input. We proposed the dual decoder model, a single model capable of simultaneously generating texts in both languages. We also proposed a method to generate synthetic MXL data from parallel sentences and showed that the artificial data could effectively train our dual decoder model to translate both regular monolingual and MXL sentences. Our dual decoder model achieved results on par with baseline approaches, containing two separately trained translation models for each language. The proposed dual decoder model could almost perfectly disentangle tokens in one language from the other language and make necessary corrections when the input sentence contained errors. We have achieved state-of-the-art performance on the SemEval 2014 L2 Writing Assistant task for the Spanish-English direction. Besides, we further applied the dual decoder model to four other tasks and discovered that dual decoding is prone to exposure bias. We explored practical remedies to mitigate this issue and have achieved similar performance compared to simple multi-task approaches while constantly generating more consistent translation pairs. We also demonstrated that the dual decoder model could benefit from pre-trained regular translation models to achieve improved translation quality in various tasks, alleviating the common data scarcity problem in the four situations where generating synthetic data was not trivial.

In the second situation, we defined the framework of bilingual synchronization (Bi-sync), which aims at editing existing initial translations to restore parallelism with a source sentence. We proposed various methods to simulate initial translations that require different types of edits. The artificial editing data was effective in training both autoregressive and nonautoregressive models. Both our proposed autoregressive Edit-MT and the non-autoregressive Edit-LevT could make good use of initial translations to reach high translation quality, even in real-world scenarios. Experiments in multilingual settings demonstrated that one single model could perform Bi-sync in both directions without decreasing the overall performance, making it possible to treat both languages equivalently without distinguishing a source language. In addition, we studied more specific tasks with edit-based models. We demonstrated that models trained for the general Bi-sync task could be quickly adapted to parallel corpus cleaning and neural machine translation (NMT) with translation memories (TMs) tasks with only slight fine-tuning. The performance even outperformed dedicated systems trained on specific tasks. We paid particular attention to the non-autoregressive translation (NAT) with TMs task and explored the TM-LevT model, which greatly surpassed the original Levenshtein Transformer model and approached a strong autoregressive method with a lower appearance rate of unrelated words and a much faster decoding speed. Training with TMs also improved the performance of TM-LevT on standard translation without using knowledge distillation (KD). Our work is one of the first studies in this new subdomain of NAT and translation with TM. We hope it can motivate more research work to further explore this area.

Future Outlook

Along with our work, there still remain several promising research directions. We discuss below some future perspectives with open questions that emerged from our study in this thesis.

Our work in this thesis was based on simulated situations where we statically performed translations without actual interaction with humans. It will be a promising direction to design user interfaces, collect real user feedback to evaluate the differences between simulated conditions and real-world scenarios, study more precisely how expert and non-expert translators will use this framework, and improve bilingual writing systems to suit this use case better. It is also interesting to contrast our systems with post-editing NMT outputs or interactive translation systems in terms of edition time, especially for texts that need incremental updates.

As mentioned in Chapter 6, the exposure bias issue is amplified in the dual decoder model. We proposed to generate partial synthetic reference data to mitigate this problem but have not stepped further. Future work may concentrate on developing other methods to better alleviate this problem so that the dual decoder model can generate high-quality and consistent translations.

NAT is a newly emerged domain in the area of NMT. However, due to the conditional independence assumption, the performance of NAT models is mostly lagging behind autoregressive baselines. Besides, NAT models highly rely on KD from autoregressive teacher models. KD implicitly defines the upper bound performance of NAT models to be around the teacher model.

Even though there were studies analyzing why KD is effective for NAT, it is still a mystery why KD can improve NAT. More work and better analyses could be conducted to determine the key reason behind KD. We showed that training with TMs could improve NAT performance without using KD. We hope this provides a new direction and wish our work can inspire more studies on NAT models without relying on KD.

Tables B.1 and B.2 reports results for the multi-target translation experiments of Section 6.3 using the IWSLT tst2015 test set, a setting also used in (He et al., 2021a). D.4, D.5, and D.6. The variation in scores across domains is large, confirming that TM matches can be very beneficial for some technical domains (e.g., ECB, EME, GNO, KDE, JRC), for which we often find good matches that help to greatly increase the performance. On the other hand, News, Wiki, and TED yield fewer matches, which only helps when the similarity is high (FM ≥ 0.6).

E -Résumé Français

Dans un monde de plus en plus globalisé, il est de plus en plus courant d'avoir à s'exprimer dans une langue étrangère ou dans plusieurs langues. Cependant, pour de nombreuses personnes, parler ou écrire dans une langue étrangère n'est pas une tâche facile. Les outils de traduction automatique peuvent aider à générer des textes en plusieurs langues. Grâce aux progrès récents de la traduction automatique neuronale (NMT), les technologies de traduction fournissent en effet des traductions utilisables dans un nombre croissant de contextes. Pour autant, il n'est pas encore réaliste d'attendre des systèmes NMT qu'ils produisent des traductions sans erreur. En revanche, les utilisateurs ayant une bonne maîtrise d'une langue étrangère donnée peuvent trouver des aides auprès des technologies de traduction aidée par ordinateur.

Lorsqu'ils rencontrent des difficultés, les utilisateurs écrivant dans une langue étrangère peuvent accéder à des ressources externes telles que des dictionnaires, des terminologies ou des concordanciers bilingues. Cependant, la consultation de ces ressources provoque une interruption du processus de rédaction et déclenche une autre activité cognitive. Pour rendre le processus plus fluide, il est possible d'étendre les systèmes d'aide à la rédaction afin de prendre en charge la composition de textes bilingues. Cependant, les études existantes se sont principalement concentrées sur la génération de textes dans une langue étrangère. Nous suggérons que l'affichage de textes correspondants dans la langue maternelle de l'utilisateur peut également aider les utilisateurs à vérifier les textes composés à partir d'entrées bilingues. Dans cette thèse, nous étudions des techniques pour construire des systèmes d'aide à la rédaction bilingues qui permettent la composition libre dans les deux langues et affichent des textes monolingues synchronisés dans les deux langues. Nous présentons deux types de systèmes interactifs simulés.

La première solution permet aux utilisateurs de composer des textes dans un mélange de langues, qui sont ensuite traduits dans leurs équivalents monolingues. Nous étendons le modèle Transformer pour la traduction en ajoutant un décodeur duel: notre modèle comprend un encodeur partagé et deux décodeurs pour produire simultanément des textes en deux langues. Nous ajoutons une couche d'attention croisée du décodeur dans chaque bloc du décodeur pour construire l'interaction entre les deux décodeurs. En conséquence, les deux décodeurs sont complètement synchronisés et notre modèle de décodeur duel est capable de générer simultanément les traductions dans les deux langues. Nous explorons également le modèle de décodeur duel pour plusieurs autres tâches, telles que la traduction multi-cible, la traduction bidirectionnelle, la génération de variantes de traduction et le sous-titrage multilingue.

La deuxième contribution vise à étendre les systèmes de traduction commerciaux disponibles en ligne en permettant aux utilisateurs d'alterner librement entre les deux langues, en changeant la boîte de saisie du texte à leur volonté. Dans ce scénario, le défi technique consiste à maintenir la synchronisation des deux textes d'entrée tout en tenant compte des entrées des utilisateurs, toujours dans le but de créer deux versions également bonnes du texte. Pour cela, nous introduisons une tâche générale de synchronisation bilingue et nous implémentons et expérimentons des systèmes de synchronisation autorégressifs et non-autorégressifs. Nous proposons un nouveau modèle de traduction non-autorégressif basé sur le modèle Levenshtein Transformer qui peux effectivement éditer une phrase de cible pour rendre le parallélisme des deux langues. Nous étudions également l'utilisation de modèles de synchronisation bilingue pour d'autres tâches spécifiques, telles que le nettoyage de corpus parallèles et la NMT avec mémoire de traduction, afin de mieux évaluer la capacité de généralisation des modèles proposés. Notre travail est en particulier le premier à étudier en profondeur les modèles de traduction non-autorégressifs complétés par des mémoires de traduction.

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 An example of simultaneously generating monolingual translations from an MXL sentence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 A graphical view of the dual decoder model. . . . . . . . . . . . . . . . . . . . . . 5.3 A graphical view of the dual beam search process. . . . . . . . . . . . . . . . . . . 5.4 Examples of generated MXL sentences when taking English as the matrix language and varying the number r of replacements of embedded French segments (in boldface). The alignment units are shown in color. . . . . . . . . . . . . . . . . . . . . 5.5 Evolution of the BLEU score of source MXL data and their target translations for En-Fr. (a) Direction MXL→En. The solid curve takes Fr as the matrix language,

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Evolution of the BLEU score of source MXL data and their target translations for En-Es. (a) Direction MXL→En. (b) Direction MXL→Es. . . . . . . . . . . . . . . . 5.7 Translation of a noisy MXL sentence with French as both the matrix and target language. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Instances of dual decoding: multi-target translation ( § 6.3) translating into Chinese (Zh) and Japanese (Ja); bidirectional decoding ( § 6.4) generating sequences from left to right(L2R) and right to left (R2L); multi-style decoding ( § 6.5) producing polite and informal translations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 A graphical view of the asynchronous dual beam search process generating translation ê2 with a reference translation in e 1 as forced prefix. . . . . . . . . . . . . . . . . . 6.3 Examples of the dual model generating more consistent translations than the indep model for De→En/Fr. Both models are trained from scratch using partial synthetic reference data (ps). Differences between indep and dual are in bold. Consistent translations in dual between En and Fr are marked with the same color. . . . . . . 6.4 A graphical view of various captioning and subtitling strategies. T refers to transcripts. C and S denote captions and subtitles, respectively. . . . . . . . . . . . . . 6.5 Example of a triplet (transcript, caption, subtitle) from our tri-parallel data. Differences between transcript and caption are in bold. The segmentation tags [eob] and [eol] refer to change-of-screen and end-of-line, respectively. . . . . . . . . . . . . 6.6 Examples of dual decoding improving both captioning and subtitling. Major improvements are marked in bold. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6 TM entries with the corresponding LCS of words with the source sentence (left) and word alignments (right). Black dots and squares indicate matches of LCS and word alignments, respectively. Related target tokens are in bold. . . . . . . . . . . . . . . 8.1 A complete training step for Edit-LevT. . . . . . . . . . . . . . . . . . . . . . . . . 8.2 Comparison of performance gains when translating with TMs of different similarity ranges for FM and TM-LevT. FM ∈ [0, 0.4) is a held-out set of 1,000 sentences per domain with no match found. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  SemEval 2014 Task 5 for Fr-En. . . . . . . . . . . . . . . . . . . . . . . 5.7 Results of SemEval 2014 Task 5 for En-Es. . . . . . . . . . . . . . . . . . . . . . . 5.8 Results of BLEU scores on SemEval data. Performance is computed on full sentences. 6.1 The number of lines in the trilingual IWSLT data. English is used to identify trilingual sentences and is therefore not shown in this table. . . . . . . . . . . . . . . . . . . 6.2 Statistics of WMT bilingual data used in pre-training experiments for multi-target translation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  2round and Ref refer to 2-round decoding with respectively model predictions and references as forced prefixes in one direction. . . . . . . . . . . . . . . . . . . . . . 7.1 Statistics of training and test data for Bi-sync. . . . . . . . . . . . . . . . . . . . . 7.2 BLEU scores of Edit-MT performing standard translation on newstest2014 for both En-Fr and Fr-En. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 BLEU scores of Edit-MT on Bi-sync for En-Fr. Deletions are evaluated separately

  7.11 BLEU (B) scores and average edit distance (AD) broken down by the distance ∆ between ẽ and e. Each column represents a range of distances. N denotes the number of sentences in each group. . . . . . . . . . . . . . . . . . . . . . . . . . . 7.12 BLEU scores and average edit distance (AD) broken down by the editing operations required between ẽ and e. Each column represents a combination of edits. I, S, D refer to insertion, substitution, and deletion, respectively. . . . . . . . . . . . . . . . 7.13 Label conversion scheme between CLTE task and Edit-MT editing tags. . . . . . . 7.14 Accuracy scores on the SemEval CLTE tasks. FT denotes Edit-MT models fine-tuned for the CLTE classification tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.15 BLEU scores on MSLT task of models trained with different subsets of OpenSubtitles. 8.1 BLEU scores of Edit-LevT performing standard translation on newtest2014 for both En-Fr and Fr-En. The base-mono results are copied from Section 5.5. . . . . . . . 8.2 BLEU scores of Edit-LevT on Bi-sync for En-Fr. Deletions are evaluated separately for the two generation methods (Del 1 and Del 2). The best performance of NAT methods is in bold. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3 BLEU scores of Edit-LevT on Bi-sync for Fr-En. . . . . . . . . . . . . . . . . . . . 8.4 BLEU scores of multilingual Edit-LevT on Bi-sync for both En-Fr and Fr-En. The multilingual model denoted with multi is the same model used for both directions. The best performance in each block is in bold. . . . . . . . . . . . . . . . . . . . . 8.5 BLEU scores of Edit-LevT trained with different editing data on Bi-sync for both En-Fr and Fr-En. Models trained with only substitution ẽsub and deletion ẽdel editing data are denoted with sub+del. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6 BLEU scores of Edit-LevT by performing standard translation without using TMs on test sets from multiple domains. All is computed by concatenating test sets from all domains, with 11k sentences in total. Copy refers to copying the retrieved similar translations to the output. + FT refers to models fine-tuned on the multi-domain TM data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.7 BLEU scores of Edit-LevT on test sets from multiple domains when translating with TMs. +t refers to AR models decoding with prefixed tags. + R implies using the related segments instead of a full initial sentence for inference. + FT refers to models fine-tuned on the multi-domain TM data. The best performance in each block is in bold. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.8 TER scores of Edit-LevT on test sets from multiple domains when translating with TMs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9 BLEU scores of Edit-LevT on unseen domains when translating with TMs. + tag refers to decoding with the oracle tag as a forced prefix. . . . . . . . . . . . . . . . 8.10 BLEU (B) scores and average edit distance (AD) of Edit-LevT broken down by the distance ∆ between ẽ and e. Each column represents a range of distances. N denotes the number of sentences in each group. Best performance of each column is in bold. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.11 BLEU scores and averaged edit distance (AD) of Edit-LevT broken down by the editing operations required between ẽ and e. Each column represents a combination of edits. N denotes the number of sentences in each group. I, S, D refer to insertion, substitution, and deletion, respectively. . . . . . . . . . . . . . . . . . . . . . . . . 8.12 Dataset statistics for NAT with TMs experiments, with ratios of sentences with at least one TM match for various similarity ranges. . . . . . . . . . . . . . . . . . . . 8.13 Summarization of the architecture and data format of different models used in this chapter. Extended source implies concatenating the initial translation or similar TM match ẽ to the source sentence. Target initial refers to initializing the decoder with ẽ. LevT + init-del is our proposed model with an additional initial deletion operation. 8.14 BLEU and COMET scores on aggregated multi-domain test sets for various TM similarity ranges. w/o TM is standard MT, w/ TM adds a retrieved match on the source side and uses the match to initialize TM-LevT inference. Copy simply copies the retrieved match in the output. +tgt TM refers to using TM match as the initial target for LevT-FM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.15 BLEU scores for each domain when performing MT with TMs with FM ≥ 0.6. All is computed by concatenating all test sets (11k sentences in total). . . . . . . . . . 8.16 COMET scores for each domain when performing MT with TMs with FM ≥ 0.6. . 8.17 The average percentage of unrelated tokens from the retrieved TM matches appearing in the final translations evaluated on the multi-domain test sets. . . . . . . . . . 8.18 BLEU scores for KD effects on the aggregated multi-domain test sets. + KD applies KD to the training references, + KD TM applies KD to both references and TM matches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.19 BLEU scores for various configurations. -tgt TM (resp. -src TM) is the model trained without TM match on the target (resp. source) side. -final-del is trained without the final-del operation, -self-pred only applies reference deletions during training. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.20 Averaged decoding iterations per sentence evaluated on the aggregated multi-domain test sets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.21 Average decoding time (×10 -3 s) per sentence evaluated on the multi-domain test sets for both FM ≥ 0.6 and FM ∈ [0.4, 0.6). . . . . . . . . . . . . . . . . . . . . . B.1 BLEU and scores for multi-target models on tst2015. Pseudo (ps) refers to models trained from scratch with partial synthetic reference data. FT indicates models finetuned from the pre-trained multilingual (multi) models. FT+ps refers to models fine-tuned using partial synthetic reference data. . . . . . . . . . . . . . . . . . . . B.2 Similarity scores (SIM) for multi-target models on tst2015. The similarity scores are computed as the cross-lingual similarity between sentence embeddings of the two target translations computed with LASER. Pseudo (ps) refers to models trained from scratch with partial synthetic reference data. FT indicates models fine-tuned from the pre-trained multilingual (multi) models. FT+ps refers to models fine-tuned using partial synthetic reference data. . . . . . . . . . . . . . . . . . . . . . . . . . C.1 TER and BERTScore for captions (En) and subtitles (Fr). The base and pipeline settings are trained from scratch with original data. +FT indicates models fine-tuned from the pre-trained translation model. share refers to tying all decoder parameters.

( a )

 a System of Chen et al. ( ) (top) and commercial translation systems (bottom).

Figure . :

 . Figure . : Multiple designs of bilingual writing systems. Texts mixing two languages are distinguished using different colors and patterns.

( b )

 b Text infilling model of Xiao et al. ( a). The model tends to fill each [gap] in the initial target sequence and only produces translation segments. Segments are separated by a [eob] token.

Figure . :

 . Figure . : Two methods to incorporate human inputs for IMT.

  Combien de temps dure un rhume ? [sep] How much time does a cold take ?

Figure . :

 . Figure . : Single and dual-encoder models incorporating augmented sources.

Figure

  Figure . : A complete training step for LevT. We omit the begin-of-sentence and end-of-sentence tokens which are always present in each target sequence.

  Ding et al. (2021a) propose progressive multi-granularity training, which is based on the same general idea as GLAT, starting with easier training examples and then more difficult ones. They break down bilingual sentences into parallel words and phrases. The training curriculum is simply first training NAT models on words, then phrases, and finally sentences.

  Figure . : Masking procedures of Li et al. (). In the first step, the French word "Je" is aligned to "I" in English and to "我" in Chinese. Therefore, "Je" is replaced with "我", and "I" is masked. In the second step, tokens from both sentences are randomly masked.

  4.5 . Non-autoregressive Translation with Augmented Sources 4.5.1 . Lexical Constraints for Non-autoregressive Translation

Figure . :

 . Figure . : An example of simultaneously generating monolingual translations from an MXL sentence.

1t

  is conditioned not only on the source f and previous outputs e 1 <t but also on the previous outputs of the other target sequence e 2 <t . The probability of e 2 t is computed accordingly. Note that we are also able to simulate other asynchronous dependency patterns, where each symbol e 2 t is generated conditioned on e 1 <t+k and e 2 <t , thus reproducing the chained model of

  Figure . : A graphical view of the dual decoder model.

Figure

  Figure . : A graphical view of the dual beam search process.

  some examples of generated MXL sentences.MatrixIn Oregon , planners are experimenting with giving drivers different choices . r = 1 Dans Oregon , planners are experimenting with giving drivers different choices . r = 2 Dans Oregon , les planificateurs are experimenting with giving drivers different choices . r = 3 Dans Oregon , les planificateurs are experimenting en offrant aux drivers different choices . Embedded Dans l'Orégon , les planificateurs tentent l'expérience en offrant aux automobilistes différents choix .

Figure . :

 . Figure . : Examples of generated MXL sentences when taking English as the matrix language and varying the number r of replacements of embedded French segments (in boldface). The alignment units are shown in color.

  Figure . : Evolution of the BLEU score of source MXL data and their target translations for En-Fr. (a) Direction MXL→En.The solid curve takes Fr as the matrix language, where we progressively inject more En segments; for the dash-dot curve, En is the matrix language, with a growing number of Fr segments. (b) Direction MXL→Fr. The target BLEU is always much higher than the source BLEU, with about a -point difference. The gap between the dash-dot and solid curves is due to the basic sentence structure of the matrix language (see Section . . ). As dash-dot curves represent insertion in the reference target sentence, the corresponding BLEU score is always higher than the solid curve and actually reaches (in the absence of any embedded language).

  Figure . : Evolution of the BLEU score of source MXL data and their target translations for En-Es. (a) Direction MXL→En. (b) Direction MXL→Es.

Figure . :

 . Figure . : Translation of a noisy MXL sentence with French as both the matrix and target language.

Figure . :

 . Figure . : Instances of dual decoding: multi-target translation ( § . ) translating into Chinese (Zh) and Japanese (Ja); bidirectional decoding ( § . ) generating sequences from left to right(L R) and right to left (R L); multi-style decoding ( § . ) producing polite and informal translations.

  .4 (b)), where subtitles are

Figure

  Figure . : A graphical view of various captioning and subtitling strategies. T refers to transcripts. C and S denote captions and subtitles, respectively.

Figure . :

 . Figure . : Example of a triplet (transcript, caption, subtitle) from our triparallel data. Differences between transcript and caption are in bold. The segmentation tags [eob] and [eol] refer to change-of-screen and end-of-line, respectively.

  of various decoding methods. Models fine-tuned with the original data are denoted with w/real. All BLEU scores are averaged over the two outputs. 2-round and Ref refer to 2-round decoding with respectively model predictions and references as forced prefixes in one direction.

  Figure . : Confusion matrices for predicted tags of Edit-MT for both directions.

Figure . :

 . Figure . : Example attention matrix of Edit-MT when translating without forced oracle tag. The model generates a copy tag and then copies the initial translation as output.

Figure . :

 . Figure .: Example attention matrix of Edit-MT when translating with a forced substitution tag. The model properly replaces "que" with "auxquels" by paying more attention to "to" in the source and adds "accès" (access) to generate a good translation following the oracle tag.

  Figure . : TM entries with the corresponding LCS of words with the source sentence (left) and word alignments (right). Black dots and squares indicate matches of LCS and word alignments, respectively. Related target tokens are in bold.

Figure . :

 . Figure . : A complete training step for Edit-LevT.

Fuzzy

Figure . :

 . Figure . : Comparison of performance gains when translating with TMs of different similarity ranges for FM and TM-LevT. FM ∈ [0, 0.4) is a held-out set of 1,000 sentences per domain with no match found.

  Unrelated token rate ↓ FM ≥ 0.6 FM ∈ [0.4, 0

  1.1 Multiple designs of bilingual writing systems. Texts mixing two languages are distinguished using different colors and patterns. . . . . . . . . . . . . . . . . . . . . . . 3.1 Two methods to incorporate human inputs for IMT. . . . . . . . . . . . . . . . . . 3.2 Single and dual-encoder models incorporating augmented sources. . . . . . . . . . . 4.1 A complete training step for LevT. We omit the begin-of-sentence and end-ofsentence tokens which are always present in each target sequence. . . . . . . . . . . 4.2 Masking procedures of

  , Josep Crego, and François Yvon. 2022b. Bilingual synchronization: Restoring translational relationships with editing operations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8016-8030, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics • Jitao Xu, Josep Crego, and François Yvon. 2022c. Non-autoregressive machine translation with translation memories. CoRR, abs/2210.06020 • Jitao Xu, François Buet, Josep Crego, Elise Bertin-Lemée, and François Yvon. Minh Quang Pham, Jitao Xu, Josep Crego, François Yvon, and Jean Senellart. 2020. Priming neural machine translation. In Proceedings of the Fifth Conference on Machine Translation, pages 462-473, Online. Association for Computational Linguistics • Jitao Xu, Josep Crego, and Jean Senellart. 2020. Boosting neural machine translation with similar translations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1580-1590, Online. Association for Computational Linguistics • Jitao Xu, Minh Quang Pham, Sadaf Abdul Rauf, and François Yvon. 2021a. LISN @ WMT 2021. In Proceedings of the Sixth Conference on Machine Translation, pages 232-242, Online. Association for Computational Linguistics

2022a. Joint generation of captions and subtitles with dual decoding. In Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022), pages 74-82, Dublin, Ireland (inperson and online). Association for Computational Linguistics • Jitao Xu and François Yvon. 2021b. One source, two targets: Challenges and rewards of dual decoding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8533-8546, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics • Jitao Xu and François Yvon. 2021a. Can you traducir this? Machine translation for code-switched input. In Proceedings of the Fifth Workshop on Computational Approaches to Linguistic Code-Switching, pages 84-94, Online. Association for Computational Linguistics •

  performs far behind AR Transformer baselines. Translations are multi-modal, meaning that a source sentence can be accurately translated into multiple correct target sentences. Such multi-modal translations possibly exist in the training data, and NAT models are trained to cover all possible modes. During inference, without explicit indications, NAT models may generate tokens that belong to different translations due to the independence assumption. For example, a French sentence [Merci .] can be translated into English sentences [Thanks .] or [Thank you .]. However, NAT models may generate sequences like [Thanks you .] or [Thank .

  representations of dimension d is then passed through the standard output linear projection layer to obtain the n + 1 content-location logits of dimension |V |, where |V | is the target vocabulary size. Insertion Transformer models the joint content-location probability by flattening the n + 1 logits in a single vector of size (n + 1) × |V |, then performs a softmax operation to obtain the joint probability distribution over all possible tokens for all slots. It can also compute the content probability of each location by separately performing the softmax operation for each location to generate tokens for multiple locations simultaneously. However, Insertion Transformer cannot generate all tokens in one single step, as it can only produce at most one token for each insertion slot. It has to iteratively generate tokens to increase insertion slots for the next iteration. The inference time complexity of the Insertion Transformer is thus logarithmic.

  al.,

	Our implementation of MXL data generation is open-sourced at https://
	github.com/jitao-xu/code-switch-gen
	https://www.statmt.org/wmt14/translation-task.html
	https://www.statmt.org/wmt13/translation-task.html

https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176. bin

Table . :

 . Summarization of the model architecture and data of systems used in this chapter. Transformers implies using two separate models, one for each direction. Transformer refers to one model that can perform translation in two directions. indep decoders indicates two independent decoders.

	System	Architecture	MXL Data
	base-mono	Transformers
	bilingual	Transformer
	base	Transformers
	multi	Transformer
	indep	encoder + indep decoders
	dual	encoder + dual decoders

Table . :

 . Analyzing the copy rate of tokens evaluated on mxl-newstest2014

	Test set	mxl-newstest2014 mxl-newstest2013
	Direction	MXL-En MXL-Fr	MXL-En MXL-Es
	to copy	42,148	47,337	37,653	41,053
	copied	41,555	46,449	37,442	40,650
	copy rate (%)	98.6	98.1	99.4	99.0

Table . :

 . Percentage of sentences for which all target words have been precisely copied without and with order changes (Copy and Swap, respectively) and some target words in the MXL source have been changed

	Direction		En-Fr			En-Es
		Copy Swap Word-change Copy Swap Word-change
	MXL-En	.	.	.	.	.	.
	Matrix En	.	.	.	.	.	.
	Matrix Fr/Es	.	.	.	.	.	.
	MXL-Fr/Es	.	.	.	.	.	.
	Matrix Fr/Es	.	.	.	.	.	.
	Matrix En	.	.	.	.	.	.

Table . :

 . Results of BLEU scores on SemEval data. Performance is computed on full sentences.

			Accuracy Word Accuracy Recall
	UEdin-run		.	.	.
	UEdin-run		.	.	.
	UEdin-run		.	.	.
	CNRC-run		.	.	.
	multi
	free-dec		.	.	.
	token-cst		.	.	.
	presuf-cst	.	.	.
	indep
	free-dec		.	.	.
	dual
	free-dec		.	.	.
	Table . : Results of SemEval	Task for Fr-En.
			Accuracy Word Accuracy Recall
	UEdin-run		.	.	.
	UEdin-run		.	.	.
	UEdin-run		.	.	.
	multi
	free-dec		.	.	.
	token-cst		.	.	.
	presuf-cst	.	.	.
	indep
	free-dec		.	.	.
	dual
	free-dec		.	.	.
	Table . : Results of SemEval	Task for En-Es.
	Model Fr-En test En-Es test Fr-En dev En-Es dev
	multi	.	.	.	.
	indep	.	.	.	.
	dual	.	.	.	.

  al., 2019); • Bidirectional decoding, where we simultaneously decode in two directions, providing a new implementation of the idea of Watanabe and Sumita (2002) and Finch and Sumita (2009); • Multi-style decoding, where we generate coherent translation alternatives in different styles, an idea we use to compute polite and informal variants of the same input (Sennrich et al., 2016a); • Multilingual subtitling, where we produce subtitles in two different languages at the same time from automatic speech recognition (ASR) transcripts. Figures 6.1 and 6.5 illustrate examples of these applications. Considering multiple applications allows us to assess the challenges and rewards of dual decoding from various angles and to better evaluate the actual agreement between the two decoders' outputs.

	f	I could do that again if you want .
	Zh	只要 你 愿意 我 可以 重复 一遍 。
	Ja	もう 一 回 やり ましょ う か
	L R	Je peux le refaire si vous le voulez .
	R L	. voulez le vous si refaire le peux Je
	polite	Ich kann das noch mal machen , wenn Sie wollen .
	informal Ich kann das noch mal machen , wenn du willst .

Table . :

 . The number of lines in the trilingual IWSLT data. English is used to identify trilingual sentences and is therefore not shown in this table.

		Original De Original Fr Trilingual
	Train	209,522	236,653	205,397
	Dev	2,693	3,083	2,468
	tst2014	1,305	1,306	1,168
		Original Zh Original Ja Trilingual
	Train	235,078	226,834	213,090
	Dev	3,064	3,024	2,837
	tst2014	1,297	1,285	1,214

  En indep ps I had to look differently to be able to map the past in a map. Fr indep ps J'ai dû voir différemment pour pouvoir tracer le passé dans une carte. Examples of the dual model generating more consistent translations than the indep model for De→En/Fr. Both models are trained from scratch using partial synthetic reference data (ps). Differences between indep and dual are in bold. Consistent translations in dual between En and Fr are marked with the same color.

	De	Ich konnte mir das niemals vorstellen.
	En indep ps I've never imagined that.
	Fr indep ps Je ne pouvais jamais l'imaginer.
	En dual ps	I could never imagine that.
	Fr dual ps	Je ne pouvais jamais l'imaginer.
	De	Ich musste anders sehen, um die Vergangenheit in einer
		Karte nachzeichnen zu können.
	En dual ps	I had to see differently to be able to track the past in a map.
	Fr dual ps	J'ai dû voir différemment pour pouvoir tracer le passé dans
		une carte.
	Figure . : Model		De-En De-Fr En-De En-Fr En-Zh En-Ja
	dual		.	.	.	.	.	.
	dual 2-round	.	.	.	.	.	.
	dual ref		.	.	.	.	.	.
	dual FT		.	.	.	.	.	.
	dual FT l T 1 wait-3	.	.	.	.	.	.
	dual FT l T 2 wait-3	.	.	.	.	.	.
	dual FT 2-round	.	.	.	.	.	.
	dual FT ref	.	.	.	.	.	.
	Results using 2-round sequential decoding for one decoder are obtained in
	a second decoding pass using either automatic (2-round) or reference (ref)
	translations.

Table

.

: BLEU scores for asynchronous decoding: sequential decoding on the dual models trained from scratch (top), wait-3 models fine-tuned on the pre-trained models, and sequential decoding on the dual FT models (bottom).

Table . :

 . BLEU scores for bidirectional decoding models trained with actual data (top) and synthetic data (bottom). Pseudo (ps) refers to models trained with partial synthetic reference data. ps-dup systems are trained with twice as much data as ps settings.

	Model	En-De En-Fr En-Zh En-Ja Avg BLEU
	base	.	.	.	.	.
	indep	.	.	.	.	. (+ . )
	dual	.	.	.	.	. (-. )
	indep ps	.	.	.	.	. (+ . )
	dual ps	.	.	.	.	. (+ . )
	indep ps-dup	.	.	.	.	. (+ . )
	dual ps-dup	.	.	.	.	. (+ . )

Table . :

 . Results of politeness MT models. Tags are used for the pre-train model to generate the desired styles. Decoders of indep and dual compute two translations in one decoding step, while the results using sequential asyn-

	Model	Tag	neutral polite informal BLEU
		reference	-
	pre-train	none	.
		polite	.
		informal	.
		oracle	.
		Decoders neutral polite informal BLEU
	indep	polite	.
		informal	.
	dual	polite	.
		informal	.
	2-round informal polite	.
	2-round polite	informal	.

chronous decoding for one decoder are obtained with the 2-round procedure of Section . . .

Table . :

 . BLEU scores for captions (En) and subtitles (Fr), with structural and lexical consistency scores between the two hypotheses. The consistency scores are in percentage. The base and pipeline settings are trained from scratch with original data. +FT indicates models fine-tuned from the pretrained translation model. share refers to tying all decoder parameters.

			BLEU		Consistency ↑
	Model	En	Fr	Avg Structural Lexical
	base	.	.	.	.	.
	base + FT	.	.	.	.	.
	pipeline	.	.	.	.	.
	pipeline + FT	.	.	.	.	.
	dual + FT	.	.	.	.	.
	share + FT	.	.	.	.	.

Table . :

 . Transcript take time to write down your values your objectives and your key results do it today En pipeline +FT Take time to write down [eol] your values, your objectives, [eob] and your key results do it today. [eob] BLEU scores for multiple initialization settings when fine-tuning dual decoder models. -decoder refers to only initializing the subtitling decoder with pre-trained parameters.

	En share +FT	Take time to write down your values, [eol] your objec-
		tives, [eob] and your key results do it today. [eob]
	En ref	Take time to write down your values, [eob] your objec-
		tives and your key results. [eob] Do it today. [eob]
	Fr pipeline +FT Prenez le temps d'écrire vos valeurs, [eol] vos objec-
		tifs, [eob] et vos principaux résultats [eol] le font au-
		jourd'hui. [eob]
	Fr share +FT	Prenez le temps d'écrire vos valeurs, [eob] vos objectifs
		et vos résultats clés. [eob] Faites-le aujourd'hui. [eob]
	Fr ref	Prenez le temps d'écrire vos valeurs, [eob] vos objectifs
		et vos résultats clés. [eob] Faites-le aujourd'hui. [eob]
	Transcript	and as it turns out what are you willing to give up is ex-
		actly the right question to ask
	Fr share +FT	Et il s'avère que ce que vous voulez abandonner [eob]
		est exactement la bonne question à poser. [eob]
	Fr ref	Et il s'avère que [eob] « Qu'êtes-vous prêts à abandon-
		ner ? » [eob] est exactement la question à poser. [eob]
	Figure . : Examples of dual decoding improving both captioning and subti-
	tling. Major improvements are marked in bold.
		Model	En	Fr	Avg
		dual -decoder + FT	.	.	.
		dual + FT	.	.	.
		share + FT	.	.	.

En pipeline +FT And as it turns out, what are you willing [eol] to give up is exactly [eob] the right question to ask? [eob] En share +FT And as it turns out, what are you willing [eol] to give up [eob] is exactly the right question to ask? [eob] En ref And as it turns out, [eob] "What are you willing to give up?" [eob] is exactly the right question to ask. [eob] Fr pipeline +FT Et il s'avère que ce que vous voulez abandonner [eol] est exactement [eob] la bonne question à poser ? [eob]

Table . :

 . Table 7.1 summarizes the main statistics for training and test data. Statistics of training and test data for Bi-sync.

	Datasets		# Sentences
	Training		33.9M
	newstest2014		3,003
	Compression (Ive and Yvon,	)	526

Table . :

 . BLEU scores of Edit-MT on Bi-sync for En-Fr. Deletions are evaluated separately for the two generation methods (Del 1 and Del 2 ). + tag refers to decoding with the oracle tag as a forced prefix. The best performance is in bold.

	Model	Ins Sub Del 1 Del 2 Comp Ext
	copy	.	.	.	.	.	.
	Edit-MT	.	.	.	.	.	.
	+ tag	.	.	.	.	.	.
	Model	Ins Sub Del 1 Del 2 Comp Ext
	copy	.	.	.	.	.	.
	Edit-MT	.	.	.	.	.	.
	+ tag	.	.	.	.	.	.

Table . :

 . BLEU scores of Edit-MT on Bi-sync for Fr-En.

Table . :

 . BLEU scores of multilingual Edit-MT on Bi-sync for both En-Fr and Fr-En. The multilingual model denoted with multi is the same model used for both directions.

	En-Fr	Ins Sub Del 1 Del 2 Comp Ext
	copy	.	.	.	.	.	.
	Edit-MT (En-Fr)	.	.	.	.	.	.
	+ tag	.	.	.	.	.	.
	multi Edit-MT	.	.	.	.	.	.
	+ tag	.	.	.	.	.	.
	Fr-En	Ins Sub Del 1 Del 2 Comp Ext
	copy	.	.	.	.	.	.
	Edit-MT (Fr-En)	.	.	.	.	.	.
	+ tag	.	.	.	.	.	.
	multi Edit-MT	.	.	.	.	.	.
	+ tag	.	.	.	.	.	.

Table . :

 . Data used for experiments in Section . . FM ratio is the ratio of sentences with at least one matched similar translation. FM train is the actual number of examples with a similar translation used for training after setting aside 1,000 test sentences. Each train sentence is matched with up to 3 similar translations. GNO and Ubu refer to GNOME and Ubuntu, respectively.

	Domain	Raw FM ratio FM train
	ECB	195,956	51.73%	234,943
	EME	373,235	65.68%	624,109
	Epp	2,009,489	10.12%	465,228
	GNO	55,391	39.31%	42,697
	JRC	503,437	50.87%	587,859
	KDE	180,254	36.00%	136,456
	News	151,423	2.12%	4,048
	PHP	16,020	34.93%	10,350
	TED	159,248	11.90%	39,895
	Ubu	9,314	20.32%	1,738
	Wiki	803,704	19.87%	409,755
	Total	4,457,471	24.27% 2,557,078
	Office	49,845	43.76%	-
	ENV	13,632	6.81%	-

Table . :

 . BLEU scores by performing standard translation without using TMs on test sets from multiple domains. All is computed by concatenating test sets from all domains, with 11k sentences in total. Copy refers to copying the retrieved similar translations to the output. + FT refers to the model finetuned on the multi-domain TM data.

	7.5.4 . Results
	BLEU ↑ ECB EME Epp GNO JRC KDE News PHP TED Ubu Wiki All
	copy	.	.	.	.	.	.	.	.	.	.	.	.
	FM	.	.	.	.	.	.	.	.	.	.	.	.
	FM #	.	.	.	.	.	.	.	.	.	.	.	.
	Edit-MT	.	.	.	.	.	.	.	.	.	.	.	.
	+ FT	.	.	.	.	.	.	.	.	.	.	.	.

Table . :

 . BLEU scores on test sets from multiple domains when translating with TMs. + R implies using the related segments instead of a full initial sentence for inference. + FT refers to the model fine-tuned on the multi-domain TM data. The best performance in each block is in bold.

	BLEU ↑	ECB EME Epp GNO JRC KDE News PHP TED Ubu Wiki	All
	copy	.	.	.	.	.	.	.	.	.	.	.	.
	FM	.	.	.	.	.	.	.	.	.	.	.	.
	FM #	.	.	.	.	.	.	.	.	.	.	.	.
	Edit-MT	.	.	.	.	.	.	.	.	.	.	.	.
	+tag	.	.	.	.	.	.	.	.	.	.	.	.
	+R+tag	.	.	.	.	.	.	.	.	.	.	.	.
	+FT+tag	.	.	.	.	.	.	.	.	.	.	.	.
	+FT+R+tag	.	.	.	.	.	.	.	.	.	.	.	.
	TER ↓	ECB EME Epp GNO JRC KDE News PHP TED Ubu Wiki	All
	copy	.	.	.	.	.	.	.	.	.	.	.	.
	FM	.	.	.	.	.	.	.	.	.	.	.	.
	FM

Table . :

 . TER scores on test sets from multiple domains when translating with TMs.

Table . :

 . BLEU scores on unseen domains when translating with TMs.

						.	.
				FM #		.	.
				Edit-MT + tag	.	.
				+ FT + tag	.	.
	7.5.5 . Analyses
	B↑ ∆(ẽ, e)									-	>
	N
	copy	.	.	.	.	.	.	.	.	.	.
	FM	.	.	.	.	.	.	.	.	.	.
	Edit-MT+tag	.	.	.	.	.	.	.	.	.	.
	+FT+tag	.	.	.	.	.	.	.	.	.	.
	AD↓ ∆(ẽ, e)									-	>
	copy	.	.	.	.	.	.	.	.	.	.
	FM	.	.	.	.	.	.	.	.	.	.
	Edit-MT+tag	.	.	.	.	.	.	.	.	.	.
	+FT+tag	.	.	.	.	.	.	.	.	.	.

Table . :

 . BLEU (B) scores and average edit distance (AD) broken down by the distance ∆ between ẽ and e. Each column represents a range of distances. N denotes the number of sentences in each group.

Table . :

 . BLEU scores and average edit distance (AD) broken down by the editing operations required between ẽ and e. Each column represents a combination of edits. I, S, D refer to insertion, substitution, and deletion, respectively.

	BLEU↑	=	I	S	D	I+S	I+D	S+D	I+S+D	All
	N
	copy	.	.	.	.	.	.	.	.	.
	FM	.	.	.	.	.	.	.	.	.
	Edit-MT+tag	.	.	.	.	.	.	.	.	.
	+FT+tag	.	.	.	.	.	.	.	.	.
	AD↓	=	I	S	D	I+S	I+D	S+D	I+S+D	All
	copy	.	.	.	.	.	.	.	.	.
	FM	.	.	.	.	.	.	.	.	.
	Edit-MT+tag	.	.	.	.	.	.	.	.	.
	+FT+tag	.	.	.	.	.	.	.	.	.

Table . :

 . Label conversion scheme between CLTE task and Edit-MT editing tags.

	CLTE Fr-En	Edit-MT En-Fr Edit-MT Fr-En
	Bidirectional	Copy	Copy
	Forward	Deletion	Insertion
	Backward	Insertion	Deletion
	No Entailment Substitution	Substitution

Table . :

 . Accuracy scores on the SemEval CLTE tasks. FT denotes Edit-MT models fine-tuned for the CLTE classification tasks.

	Methods
	Best SemEval		.	.
	Carpuat et al. (	)	.	.
	Edit-MT En-Fr		.	.
	+ FT		.	.
	Edit-MT Fr-En		.	.
	+ FT		.	.

Table . :

 . BLEU scores on MSLT task of models trained with different subsets of OpenSubtitles.

	Cleaning Method BLEU Corpus size
	full	.	. M
	filtered	.	. M
	filtered + fixed	.	. M
	noisy	.	. M
	fixed	.	. M
	filtered ( . M)	.	. M

Table . :

 . BLEU scores of Edit-LevT performing standard translation on newtest2014 for both En-Fr and Fr-En. The base-mono results are copied from

	Model	En-Fr Fr-En
	base-mono	.	.
	Edit-MT	.	.
	LevT	.	.
	Edit-LevT	.	.
	Section . .

Table . :

 . BLEU scores of Edit-LevT on Bi-sync for En-Fr. Deletions are evaluated separately for the two generation methods (Del 1 and Del 2 ). The best performance of NAT methods is in bold.

	Model	Ins Sub Del 1 Del 2 Comp Ext
	copy	.	.	.	.	.	.
	Edit-MT	.	.	.	.	.	.
	LevT	.	.	.	.	.	.
	Edit-LevT	.	.	.	.	.	.
	Model	Ins Sub Del 1 Del 2 Comp Ext
	copy	.	.	.	.	.	.
	Edit-MT	.	.	.	.	.	.
	LevT	.	.	.	.	.	.
	Edit-LevT	.	.	.	.	.	.

Table . :

 . BLEU scores of Edit-LevT on Bi-sync for Fr-En.

Table . :

 . BLEU scores of multilingual Edit-LevT on Bi-sync for both En-Fr and Fr-En. The multilingual model denoted with multi is the same model used for both directions. The best performance in each block is in bold.

	En-Fr	Ins Sub Del 1 Del 2 Comp Ext
	copy	.	.	.	.	.	.
	Edit-MT (En-Fr)	.	.	.	.	.	.
	multi Edit-MT	.	.	.	.	.	.
	Edit-LevT (En-Fr)	.	.	.	.	.	.
	multi Edit-LevT	.	.	.	.	.	.
	Fr-En	Ins Sub Del 1 Del 2 Comp Ext
	copy	.	.	.	.	.	.
	Edit-MT (Fr-En)	.	.	.	.	.	.
	multi Edit-MT	.	.	.	.	.	.
	Edit-LevT (Fr-En)	.	.	.	.	.	.
	multi Edit-LevT	.	.	.	.	.	.

Table . :

 . BLEU scores of Edit-LevT trained with different editing data on Bisync for both En-Fr and Fr-En. Models trained with only substitution ẽsub and deletion ẽdel editing data are denoted with sub+del.

	En-Fr	Ins Sub Del 1 Del 2 Comp Ext
	Edit-LevT	.	.	.	.	.	.
	Edit-LevT sub+del	.	.	.	.	.	.
	Fr-En	Ins Sub Del 1 Del 2 Comp Ext
	Edit-LevT	.	.	.	.	.	.
	Edit-LevT sub+del	.	.	.	.	.	.

Table . :

 . BLEU scores of Edit-LevT on test sets from multiple domains when translating with TMs. +t refers to AR models decoding with prefixed tags. + R implies using the related segments instead of a full initial sentence for inference. + FT refers to models fine-tuned on the multi-domain TM data. The best performance in each block is in bold.

	BLEU ↑	ECB EME Epp GNO JRC KDE News PHP TED Ubu Wiki	All
	copy	.	.	.	.	.	.	.	.	.	.	.	.
	FM	.	.	.	.	.	.	.	.	.	.	.	.
	FM #	.	.	.	.	.	.	.	.	.	.	.	.
	Edit-MT+t	.	.	.	.	.	.	.	.	.	.	.	.
	+ FT + t	.	.	.	.	.	.	.	.	.	.	.	.
	Edit-LevT	.	.	.	.	.	.	.	.	.	.	.	.
	+ R	.	.	.	.	.	.	.	.	.	.	.	.
	+ FT	.	.	.	.	.	.	.	.	.	.	.	.
	+ FT + R	.	.	.	.	.	.	.	.	.	.	.	.
	TER ↓	ECB EME Epp GNO JRC KDE News PHP TED Ubu Wiki	All
	copy	.	.	.	.	.	.	.	.	.	.	.	.
	FM	.	.	.	.	.	.	.	.	.	.	.	.
	FM

Table . :

 . TER scores of Edit-LevT on test sets from multiple domains when translating with TMs.

	Model	Office ENV
	copy	.	.
	FM	.	.
	FM #	.	.
	Edit-MT + tag	.	.
	+ FT + tag	.	.
	Edit-LevT	.	.
	+ FT	.	.

Table . :

 . BLEU scores of Edit-LevT on unseen domains when translating with TMs. + tag refers to decoding with the oracle tag as a forced prefix.

Table . :

 . BLEU (B) scores and average edit distance (AD) of Edit-LevT broken down by the distance ∆ between ẽ and e. Each column represents a range of distances. N denotes the number of sentences in each group. Best performance of each column is in bold.

	B↑ ∆(ẽ, e)									-	>
	N
	copy	.	.	.	.	.	.	.	.	.	.
	FM	.	.	.	.	.	.	.	.	.	.
	Edit-MT+tag	.	.	.	.	.	.	.	.	.	.
	+ FT + tag	.	.	.	.	.	.	.	.	.	.
	Edit-LevT	.	.	.	.	.	.	.	.	.	.
	+ FT	.	.	.	.	.	.	.	.	.	.
	AD↓ ∆(ẽ, e)									-	>
	copy	.	.	.	.	.	.	.	.	.	.
	FM	.	.	.	.	.	.	.	.	.	.
	Edit-MT+tag	.	.	.	.	.	.	.	.	.	.
	+ FT + tag	.	.	.	.	.	.	.	.	.	.
	Edit-LevT	.	.	.	.	.	.	.	.	.	.
	+ FT	.	.	.	.	.	.	.	.	.	.

Table . :

 . BLEU scores and averaged edit distance (AD) of Edit-LevT broken down by the editing operations required between ẽ and e. Each column represents a combination of edits. N denotes the number of sentences in each group. I, S, D refer to insertion, substitution, and deletion, respectively.

	BLEU↑	=	I	S	D	I+S	I+D	S+D	I+S+D	All
	N
	copy	.	.	.	.	.	.	.	.	.
	FM	.	.	.	.	.	.	.	.	.
	Edit-MT+tag	.	.	.	.	.	.	.	.	.
	+ FT + tag	.	.	.	.	.	.	.	.	.
	Edit-LevT	.	.	.	.	.	.	.	.	.
	+ FT	.	.	.	.	.	.	.	.	.
	AD↓	=	I	S	D	I+S	I+D	S+D	I+S+D	All
	copy	.	.	.	.	.	.	.	.	.
	FM	.	.	.	.	.	.	.	.	.
	Edit-MT+tag	.	.	.	.	.	.	.	.	.
	+ FT + tag	.	.	.	.	.	.	.	.	.
	Edit-LevT	.	.	.	.	.	.	.	.	.
	+ FT	.	.	.	.	.	.	.	.	.

Table . :

 . Dataset statistics for NAT with TMs experiments, with ratios of sentences with at least one TM match for various similarity ranges.

	Domain	Raw FM ≥ 0.6 FM ∈ [0.4, 0.6)
	ECB	195,956	51.73%	14.06%
	EME	373,235	65.68%	12.65%
	Epp	2,009,489	10.12%	25.30%
	GNO	55,391	39.31%	11.06%
	JRC	503,437	50.87%	16.67%
	KDE	180,254	36.00%	10.81%
	News	151,423	2.12%	9.65%
	PHP	16,020	34.93%	12.38%
	TED	159,248	11.90%	26.64%
	Ubu	9,314	20.32%	8.26%
	Wiki	803,704	19.87%	17.32%
	Total	4,457,471	24.27%	20.00%

Table . :

 . Summarization of the architecture and data format of different models used in this chapter. Extended source implies concatenating the initial translation or similar TM match ẽ to the source sentence. Target initial refers to initializing the decoder with ẽ. LevT + init-del is our proposed model with an additional initial deletion operation.

	Name	Architecture	Extended Source	Target Initial
	FM	Transformer		-
	FM#	Transformer (related segments)	-
	Edit-MT	Transformer		-
	LevT	LevT
	Edit-LevT LevT + init-del
	LevT-FM	LevT
	TM-LevT LevT + init-del

Table . :

 . BLEU and COMET scores on aggregated multi-domain test sets for various TM similarity ranges. w/o TM is standard MT, w/ TM adds a retrieved match on the source side and uses the match to initialize TM-LevT inference.Copy simply copies the retrieved match in the output. +tgt TM refers to using TM match as the initial target for LevT-FM.

					FM ≥ 0.6		FM ∈ [0.4, 0.6)
		BLEU ↑		w/o TM w/ TM w/o TM w/ TM
		copy		-		.	-		.
		FM			.		.	.		.
		LevT-FM		.		.	.		.
			+tgt TM	-		.	-		.
		TM-LevT		.		.	.		.
		COMET ↑	w/o TM w/ TM w/o TM w/ TM
		copy		-	.		-	-.
		FM			.	.		.	.
		LevT-FM		.	.		.	.
			+tgt TM	-	.		-	-.
		TM-LevT		.	.		.	.
		ECB EME Epp GNO JRC KDE News PHP TED Ubu Wiki	All
	copy	.	.	.	.	.	.	.	.	.	.	.	.
	FM	.	.	.	.	.	.	.	.	.	.	.	.
	LevT-FM	.	.	.	.	.	.	.	.	.	.	.	.
	+tgt TM	.	.	.	.	.	.	.	.	.	.	.	.
	TM-LevT	.	.	.	.	.	.	.	.	.	.	.	.

BLEU ↑

Table

.

: BLEU scores for each domain when performing MT with TMs with FM ≥ 0.6. All is computed by concatenating all test sets (11k sentences in total).

Table . :

 . COMET scores for each domain when performing MT with TMs with FM ≥ 0.6.

	COMET ↑	copy	FM	LevT-FM LevT-FM + tgt TM TM-LevT
	ECB	.	.	.	.	.
	EME	.	.	.	.	.
	Epp	-.	.	.	-.	.
	GNO	.	.	.	.	.
	JRC	.	.	.	.	.
	KDE	.	.	.	.	.
	News	-.	.	.	-.	.
	PHP	-.	.	-.	-.	.
	TED	-.	.	.	-.	.
	Ubu	.	.	.	.	.
	Wiki	.	.	.	.	.
	All	.	.	.	.	.

Table . :

 . BLEU scores for KD effects on the aggregated multi-domain test sets. + KD applies KD to the training references, + KD TM applies KD to both references and TM matches.

		FM ≥ 0.6		FM ∈ [0.4, 0.6)
	BLEU	w/o TM w/ TM w/o TM w/ TM
	TM-LevT	.	.	.	.
	+ KD	.	.	.	.
	+ KD TM	.	.	.	.

Table . :

 . BLEU scores for various configurations. -tgt TM (resp. -src TM) is the model trained without TM match on the target (resp. source) side. -finaldel is trained without the final-del operation, -self-pred only applies reference deletions during training.

		FM ≥ 0.6		FM ∈ [0.4, 0.6)
	BLEU	w/o TM w/ TM w/o TM w/ TM
	TM-LevT	.	.	.	.
	-tgt TM	.	.	.	.
	-src TM	.	.	.	.
	-final-del	.	.	.	.
	-self-pred	.	.	.	.

Table . :

 . Averaged decoding iterations per sentence evaluated on the aggregated multi-domain test sets.

Table . :

 . Average decoding time (×10 -3 s) per sentence evaluated on the multi-domain test sets for both FM ≥ 0.6 and FM ∈ [0.4, 0.6).

	Decoding time	FM			TM-LevT
	(×10 -3 s)	w/o TM w/ TM w/o TM w/ TM
	ECB	.	.	.	.
	EME	.	.	.	.
	Epp	.	.	.	.
	GNO	.	.	.	.
	JRC	.	.	.	.
	KDE	.	.	.	.
	News	.	.	.	.
	PHP	.	.	.	.
	TED	.	.	.	.
	Ubu	.	.	.	.
	Wiki	.	.	.	.
	Average	.	.	.	.

Table B .

 B ModelDe-En De-Fr En-De En-Fr En-Zh En-Ja Avg BLEU : BLEU and scores for multi-target models on tst2015. Pseudo (ps) refers to models trained from scratch with partial synthetic reference data. FT indicates models fine-tuned from the pre-trained multilingual (multi) models. FT+ps refers to models fine-tuned using partial synthetic reference data.

	base	.	.	.	.	.	.	.
	multi	.	.	.	.	.	.	. (-. )
	indep	.	.	.	.	.	.	. (+ . )
	dual	.	.	.	.	.	.	. (-. )
	indep ps	.	.	.	.	.	.	. (+ . )
	dual ps	.	.	.	.	.	.	. (+ . )
	indep FT	.	.	.	.	.	.	. (+ . )
	dual FT	.	.	.	.	.	.	. (+ . )
	indep FT+ps	.	.	.	.	.	.	. (+ . )
	dual FT+ps	.	.	.	.	.	.	. (+ . )
	Model	De→En/Fr En→De/Fr En→Zh/Ja Avg SIM
	base	.		.	.		.
	multi	.		.	.		. (+ . )
	indep	.		.	.		. (+ . )
	dual	.		.	.		. (+ . )
	indep ps	.		.	.		. (+ . )
	dual ps	.		.	.		. (+ . )
	indep FT	.		.	.		. (+ . )
	dual FT	.		.	.		. (+ . )
	indep FT+ps	.		.	.		. (+ . )
	dual FT+ps	.		.	.		. (+ . )

Table B .

 B : Similarity scores (SIM) for multi-target models on tst2015. The similarity scores are computed as the cross-lingual similarity between sentence embeddings of the two target translations computed with LASER. Pseudo (ps) refers to models trained from scratch with partial synthetic reference data. FT indicates models fine-tuned from the pre-trained multilingual (multi) models. FT+ps refers to models fine-tuned using partial synthetic reference data.

			TER ↓			BERTScore-F ↑
	Model	En	Fr	Avg	En	Fr	Avg
	base	.	.	.	.	.	.
	base +FT	.	.	.	.	.	.
	pipeline	.	.	.	.	.	.
	pipeline +FT	.	.	.	.	.	.
	dual +FT	.	.	.	.	.	.
	share +FT	.	.	.	.	.	.

Table C .

 C : TER and BERTScore for captions (En) and subtitles (Fr). The base and pipeline settings are trained from scratch with original data. +FT indicates models fine-tuned from the pre-trained translation model. share refers to tying all decoder parameters. D -Detailed Results on Each Domain for Non-autoregressive Translation with Translation Memories (Chapter 8) BLEU and COMET scores for each domain are in Tables D.1, D.2, D.3,

Table D .

 D : BLEU scores for each domain, the task is standard MT with FM ≥ 0.6. All is computed by concatenating all test sets (11k sentences in total). Copy refers to copying the TM match into the output.

	BLEU	ECB EME Epp GNO JRC KDE News PHP TED Ubu Wiki	All
	copy	.	.	.	.	.	.	.	.	.	.	.	.
	FM	.	.	.	.	.	.	.	.	.	.	.	.
	LevT-FM	.	.	.	.	.	.	.	.	.	.	.	.
	TM-LevT	.	.	.	.	.	.	.	.	.	.	.	.
		COMET	copy	FM		LevT-FM TM-LevT
		ECB	.		.		.	.
		EME	.		.		.	.
		Epp	-.		.		.	.
		GNO	.		.		.	.
		JRC	.		.		.	.
		KDE	.		.		.	.
		News	-.		.		.	.
		PHP	-.		-.		-.	-.
		TED	-.		.		.	.
		Ubu	.		.		.	.
		Wiki	.		.		.	.
		All		.		.		.	.

Table D .

 D : COMET scores for each domain, the task is standard MT with FM ≥ 0.6.

	BLEU	ECB EME Epp GNO JRC KDE News PHP TED Ubu Wiki	All
	copy	.	.	.	.	.	.	.	.	.	.	.	.
	FM	.	.	.	.	.	.	.	.	.	.	.	.
	LevT-FM	.	.	.	.	.	.	.	.	.	.	.	.
	TM-LevT	.	.	.	.	.	.	.	.	.	.	.	.

Table D .

 D : BLEU scores for each domain, the task is standard MT with FM ∈ [0.4, 0.6).

	COMET	copy	FM	LevT-FM TM-LevT
	ECB	.	.	.	.
	EME	.	.	.	.
	Epp	-.	.	.	.
	GNO	.	.	.	.
	JRC	.	.	.	.
	KDE	-.	.	.	.
	News	-.	.	.	.
	PHP	-.	-.	-.	-.
	TED	-.	.	.	.
	Ubu	-.	.	.	.
	Wiki	-.	.	.	.
	All	-.	.	.	.

Table D .

 D : COMET scores for each domain, the task is standard MT with FM ∈ [0.4, 0.6).

	BLEU	ECB EME Epp GNO JRC KDE News PHP TED Ubu Wiki	All
	copy	.	.	.	.	.	.	.	.	.	.	.	.
	AR	.	.	.	.	.	.	.	.	.	.	.	.
	LevT	.	.	.	.	.	.	.	.	.	.	.	.
	+tgt TM	.	.	.	.	.	.	.	.	.	.	.	.
	TM-LevT	.	.	.	.	.	.	.	.	.	.	.	.

Table D .

 D : BLEU scores for each domain, the task is MT with TMs with FM ∈ [0.4, 0.6).

	COMET	copy	FM	LevT-FM LevT-FM + tgt TM TM-LevT
	ECB	.	.	.	.	.
	EME	.	.	.	.	.
	Epp	-.	.	.	-.	.
	GNO	.	.	.	.	.
	JRC	.	.	.	.	.
	KDE	-.	.	.	-.	.
	News	-.	.	.	-.	.
	PHP	-.	-.	-.	-.	-.
	TED	-.	.	.	-.	.
	Ubu	-.	.	.	-.	.
	Wiki	-.	.	.	-.	.
	All	-.	.	.	-.	.

Table D. : COMET scores for each domain, the task is MT with TMs with FM ∈ [0.4, 0.6).

https://github.com/facebookresearch/faiss

This is a joint work with François Buet.

We would like to thank Alina Karakanta for sharing her code to compute the consistency scores.

in both directions (En-Fr & https://www.statmt.org/wmt14

https://github.com/fyvo/ParallelCompression https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176. bin https://github.com/rsennrich/subword-nmt https://github.com/pytorch/fairseq

https://github.com/clab/fast_align

https://opus.nlpl.eu/ http://catalog.elda.org/en-us/repository/browse/ELRA-W0057/ https://github.com/SYSTRAN/fuzzy-match

# . . . . . . . . . . . . Edit-MT . . . . . . . . . . . . +tag . . . . . . . . . . . . +R+tag . . . . . . . . . . . . +FT+tag . . . . . . . . . . . . +FT+R+tag . . . . . . . . . . . .

We have not performed hyperparameter searching for fine-tuning, even though carefully fine-tuned models may achieve even better performance. We also acknowl-

. . . . . . . . . . . . FM # . . . . . . . . . . . . Edit-MT . . . . . . . . . . . . + FT . . . . . . . . . . . . Edit-LevT . . . . . . . . . . . . + FT . . . . . . . . . . . .

# . . . . . . . . . . . . Edit-MT+t . . . . . . . . . . . . + FT + t . . . . . . . . . . . . Edit-LevT . . . . . . . . . . . . + R . . . . . . . . . . . . + FT . . . . . . . . . . . . + FT + R . . . . . . . . . . . .

https://github.com/clab/fast_align https://github.com/Tiiiger/bert_score https://github.com/mjpost/sacrebleu Signature of BERTScore (En): microsoft/deberta-xlarge-mnli_L _no-idf_version= . . (hug_trans= . . )-rescaled_fast-tokenizer. Signature of BERTScore (Fr): bert-base-multilingual-cased_L _no-idf_version= . . (hug_trans= . . )-rescaled_fast-tokenizer.

Acknowledgments

Error Rate https://github.com/moses-smt/mosesdecoder https://github.com/rsennrich/subword-nmt. https://github.com/pytorch/fairseq Our implementation is open-sourced at https://github.com/jitao-xu/ dual-decoding. https://github.com/proycon/semeval2014task5 https://wit3.fbk.eu/2017-01-c https://www.statmt.org/wmt20/translation-task.html https://www.statmt.org/wmt14/translation-task.html For comparison with (He et al., a), we also report the results on tst2015 in Appendix B. https://github.com/facebookresearch/LASER http://data.statmt.org/rsennrich/politeness/ https://github.com/rsennrich/ParZu //github.com/pytorch/fairseq

trilingual test set. Table 6.1 summarizes the main statistics for trilingual training and test data used in this section.

For WMT bilingual data, we discard the ParaCrawl data and use all the rest for En-De, De-Fr, and En-Fr. We only use News Commentary, Wiki Titles, CCMT corpus, and WikiMatrix data for En-Zh. For En-Ja, we use all data except ParaCrawl and TED talks. The latter is actually the trilingual data that we do not use in the pre-training stage. We follow the preprocessing procedure as in Chapter 5 and discard sentence pairs with invalid language tags as computed by fasttext language identification model 5 [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF]. For all data, we use Moses tools [START_REF] Koehn | Moses: Open source toolkit for statistical machine translation[END_REF] to normalize punctuations and remove non-printing characters. We tokenize all English, German and French data using Moses tokenizer. 6 Chinese and Japanese sentences are segmented using jieba 7 and mecab, 8 respectively. For En→De/Fr and De→En/Fr, we use a shared source-target vocabulary built with 40K Byte Pair Encoding (BPE) units (Sennrich et al., 2016c) learned on WMT data with subword-nmt.

9 For En→Zh/Ja, we build a 32K BPE model for English and a joint 32K BPE for Chinese and Japanese, both learned on the WMT data. Detailed statistics for the WMT data that we actually use for each language pair are in Table 6.2. are marked in bold. Initial translations ẽins for insertion are generated by randomly removing segments in the reference sentence e. For ẽsub , e is first back-translated into an intermediate sentence f * using top-5 sampling, then translated back to ẽsub with LCD. The first method to generate ẽdel randomly inserts [gap] tokens into e and decodes with a GAP insertion model (Xiao et al., a). The ẽdel 1 is obtained by replacing [gap] with the predicted extra segments. The second method uses a model trained with WikiAtomicEdits data to translate e.

. Substitutions

To simulate substitutions, we apply round-trip translation with lexically constrained decoding (LCD) [START_REF] Post | Fast lexically constrained decoding with dynamic beam allocation for neural machine translation[END_REF] to generate initial translations for substitution ẽsub . Round-trip translation has already been applied for the APE task [START_REF] Junczys | Log-linear combinations of monolingual and bilingual neural machine translation models for automatic post-editing[END_REF]. The generation procedure requires two standard NMT models separately trained on parallel data, one for each direction. For each training example (f , e), we first (a) translate the original target e into an intermediate source sentence f * using top-5 sampling [START_REF] Edunov | Understanding back-translation at scale[END_REF]; 2 (b) generate an abbreviated version ẽ′ ins from e, using the method described above for insertions. We then translate f * using LCD, with ẽ′ ins as constraints, to obtain ẽsub . In this way, we ensure that at least half of e remains unchanged in ẽsub , while the other parts have been substituted during round-trip translation. In order to A -More Details for Extracting the Minimal Alignment Units (Chapter 5)

The minimal alignment units are extracted from bidirectional word-toword alignments generated by any alignment procedure, like fast_align [START_REF] Dyer | A simple, fast, and effective reparameterization of IBM model 2[END_REF]. Extracted units should follow the constraints [START_REF] Josep | Finite-statebased and phrase-based statistical machine translation[END_REF]: no word inside a unit is aligned to words outside this unit, and no smaller units can be extracted without violating the previous constraint. The extraction is decomposed into two steps:

• Firstly, in an iterative procedure, words on one side are grouped when they are linked to the same word or group on the other side. The procedure continues to group words on both sides until no new groups can be obtained. • The second step outputs the resulting groups (units), keeping the word order of target sentence words. Though, the sequence of units may modify the source sentence word order. The fast_align model proposed by Dyer et al. ( 2013) is a reparameterization of the IBM Model 2 [START_REF] Peter | The mathematics of statistical machine translation: Parameter estimation[END_REF] using lexical translation. The latter works as follows. Given a source sentence f of length S, it first predicts the length of target sentence T , then generates an alignment a = a 1 , a 2 , . . . , a T , which indicates which source word or null token each target word is translated from. Finally, it generates the T target words, where each target word e i depends only on the aligned source word f a i .

The alignment and target word prediction follow the distributions:

where θ(⋅ | f a i ) is the lexical translation probabilities, and δ(⋅ | i, T, S) is computed as:

exp λh(i, j ′ , T, S),

where p 0 is a null alignment probability, and λ ≥ 0 is a precision that controls the trend diagonal alignment. When λ is larger, the model is more likely to generate diagonal alignment. The probability that the word at position i of e is e i can be computed as:

P (e i , a i = j | f , T, S).

(A. )

The posterior probability over alignments is thus computed as:

The model assumes that all tokens in the target sentence are conditionally independent of each other. Therefore, the translation probability is measured as:

The fast_align model is optimized using the Expectation-Maximization algorithm. In the E-step, the posterior probabilities over alignments are computed as Equation (A. ). The lexical translation probabilities θ are updated in the M-step. The parameter λ is also updated during the M-step.

C -More Details for Multilingual Subtitling (Chapter 6)

C.1 . Data Processing Details

For the English to French language pair, MuST-Cinema (Karakanta et al., 2020b) contains 275k sentences for training and 1,079 and 544 lines for development and testing, respectively. The automatic speech recognition (ASR) system used by Karakanta et al. (2020a) to produce transcripts was based on the KALDI toolkit [START_REF] Povey | The kaldi speech recognition toolkit[END_REF] and had been trained on the clean portion of LibriSpeech [START_REF] Panayotov | Librispeech: An ASR corpus based on public domain audio books[END_REF] (∼460h) and a subset of . In order to emulate a real production scenario, we segment these transcripts as if they were from an ASR system performing segmentation based on prosody. As this kind of system tends to produce longer sequences compared to typical written text [START_REF] Cho | Segmentation and punctuation prediction in speech language translation using a monolingual translation system[END_REF], we randomly concatenate the English captions into longer sequences, to which we align the ASR transcripts using the conventional edit distance, thus adding a sub-segmentation aspect to the translation task. Edit distance computations are based on a Weighted Finite-State Transducer (WSFT), implemented with Pynini [START_REF] Gorman | Pynini: A Python library for weighted finite-state grammar compilation[END_REF], which represents editing operations (match, insertion, deletion, replacement) at the character level, with weights depending on the characters and the previous operation context. After composing the edit WFST with the transcript string and the caption string, the optimal operation sequence is computed using a shortest-distance algorithm [START_REF] Mohri | Semiring frameworks and algorithms for shortestdistance problems[END_REF]. The number of sentences to be concatenated is sampled normally, with an average of around 2. This process results in 133k, 499, and 255 lines for training, development, and testing, respectively. The approach to concatenate examples is proposed and implemented by François Buet.

For pre-training, we use all available WMT14 En-Fr data, 1 in which we discard sentence pairs with invalid language labels as computed by fasttext language identification model 2 [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF]. This pre-training data contains 33.9M sentence pairs.

C.2 . Consistency Score

We borrow the following example from [START_REF] Karakanta | Between flexibility and consistency: Joint generation of captions and subtitles[END_REF] to illustrate the lexical and structural consistency scores: https://statmt.org/wmt14 https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176. bin 0:00:50,820, 00:00:53,820 To put the assumptions very clearly: Enonçons clairement nos hypothèses : le capitalisme, 00:00:53,820, 00:00:57,820 capitalism, after 150 years, has become acceptable, après 150 ans, est devenu acceptable, au même titre 00:00:58,820, 00:01:00,820 and so has democracy. que la democratie.

As defined by [START_REF] Karakanta | Between flexibility and consistency: Joint generation of captions and subtitles[END_REF], for structural consistency, both captions (C) and subtitles (S) have the same number of 3 blocks. For lexical consistency, there are 6 tokens of the subtitles which are not aligned to captions in the same block: "le capitalisme ," , "au même titre". The Lex C→S is calculated as the percentage of aligned words normalized by the number of words in the caption. Therefore, Lex C→S = 20 22 = 90.9%; the computation is identical in the other direction, yielding Lex S→C = 17 23 = 73.9%, the average lexical consistency of this segment is thus Lex pair = Lex C→S +Lex S→C 2 = 82.4%. When computing the lexical consistency between captions and subtitles, we use the WMT14 En-Fr data as used in Chapter 5 to train an alignment model using fast_align 3 [START_REF] Dyer | A simple, fast, and effective reparameterization of IBM model 2[END_REF] in both directions and use it to predict word alignments for model outputs. The model is the same as used in Section 5.3. Zhang et al., 2020b). TER is computed using SacreBLEU.

C.3 . Additional Metrics

5 Note that for BERTScores, we remove segmentation tokens ([eob] and [eol]) from hypotheses and references, as special tokens are out-of-vocabulary for pre-trained BERT models.