Lorenzo Lupo
email: lorenzo.lupo@univ-grenoble-alpes.fr

Marco Dinarelli
email: marco.dinarelli@univ-grenoble-alpes.fr

Laurent Besacier
email: laurent.besacier@naverlabs.com

Divide and Rule: Training Context-Aware Multi-Encoder Translation Models with Little Resources

Multi-encoder models are a broad family of context-aware Neural Machine Translation (NMT) systems that aim to improve translation quality by encoding document-level contextual information alongside the current sentence. The context encoding is undertaken by contextual parameters, trained on documentlevel data. In this work, we show that training these parameters takes large amount of data, since the contextual training signal is sparse. We propose an efficient alternative, based on splitting sentence pairs, that allows to enrich the training signal of a set of parallel sentences by breaking intra-sentential syntactic links, and thus frequently pushing the model to search the context for disambiguating clues. We evaluate our approach with BLEU and contrastive test sets, showing that it allows multi-encoder models to achieve comparable performances to a setting where they are trained with ×10 document-level data. We also show that our approach is a viable option to context-aware NMT for language pairs with zero document-level parallel data.

Introduction

Neural machine translation (NMT) has seen substantial improvements in recent years, fostered by the advent of the Transformer model [START_REF] Vaswani | Attention is all you need[END_REF]. A remaining challenge for modern machine translation (MT) is the ability to contextualize translation of the current sentence with the other sentences in the document [START_REF] Läubli | Has machine translation achieved human parity? a case for document-level evaluation[END_REF]. For this reason, contextual NMT has recently triggered a lot of attention and many approaches have been proposed in the literature. A common taxonomy [START_REF] Kim | When and why is document-level context useful in neural machine translation[END_REF][START_REF] Li | Does multi-encoder help? a case study on contextaware neural machine translation[END_REF] divides them in two broad categories: single-encoder (concatenation) approaches [START_REF] Tiedemann | Neural machine translation with extended context[END_REF][START_REF] Ruchit Rajeshkumar Agrawal | Contextual Handling in Neural Machine Translation: Look Behind, Ahead and on Both Sides[END_REF][START_REF] Ma | A simple and effective unified encoder for documentlevel machine translation[END_REF][START_REF] Zhang | Long-short term masking transformer: A simple but effective baseline for document-level neural machine translation[END_REF] and multi-encoder approaches [START_REF] Jean | Does Neural Machine Translation Benefit from Larger Context[END_REF][START_REF] Tu | Context gates for neural machine translation[END_REF][START_REF] Bawden | Evaluating discourse phenomena in neural machine translation[END_REF][START_REF] Miculicich | Document-level neural machine translation with hierarchical attention networks[END_REF][START_REF] Voita | Context-aware neural machine translation learns anaphora resolution[END_REF]Maruf et al., 2019a;[START_REF] Zheng | [END_REF]. Multi-encoder models are more flexible and can be more efficient than concatenation approaches, but they have been criticized as being mere regularization methods [START_REF] Kim | When and why is document-level context useful in neural machine translation[END_REF][START_REF] Li | Does multi-encoder help? a case study on contextaware neural machine translation[END_REF]. In some cases, they have been even shown to perform worse than sentence-level systems on discourse-aware targeted test suites [START_REF] Lopes | Document-level neural MT: A systematic comparison[END_REF].

In this work, we address this criticism by showing that multi-encoder models learn effectively when given enough contextual training signal, i.e. the signal that pushes systems to translate in a context-aware fashion. We show that the simplest way to increase contextual training signal effectively is by increasing the amount of documentlevel training data. Large document-level parallel corpora are not always available, but some works have proposed data augmentation techniques to remedy this lack [START_REF] Sugiyama | Data augmentation using back-translation for contextaware neural machine translation[END_REF][START_REF] Stojanovski | ContraCAT: Contrastive coreference analytical templates for machine translation[END_REF][START_REF] Huo | Diving deep into context-aware neural machine translation[END_REF]. However, this solution is not always ideal because it requires more time and computation. We therefore introduce a simple and yet powerful technique to augment the contextual training signal that can be extracted from a small parallel training set: splitting parallel sentences in segments (see Figure 1). Feeding a context-aware model with a sequence of incomplete, consecutive segments, forces it to look for context (i.e., surrounding segments) more frequently, which makes the model learn faster to translate in a context-aware fashion. We train multiencoder models on split datasets and evaluate them in two ways: BLEU score, and a contrastive evaluation of anaphoric pronoun translation. Our main contributions are the following: (i) we show that context-aware multi-encoder models need to be trained carefully, because the contextual training signal is sparse; (ii) we propose a method to augment the contextual training signal by splitting sentences in two segments, with two variants; (iii) we S i,1 He said that it was a project of peace S i,2 and unity and that it brought people together . T i,1 Il disait que c' était un projet de paix T i,2 et d' unité et qu' il réunissait les gens . S j,1 I think single-cell organisms are S j,2 possible within two years . T j,1 Je pense que les organismes unicellulaires T j,2 sont possibles dans 2 ans .

Figure 1: Example of sentence pairs from En→Fr IWSLT17, after being tokenized and split in the middle. After the splitting, some syntactic relations span across two segments (underlined). Also, some source-side words are not parallel with their reference (in bold).

support this method with a statistical analysis of its impact on the distribution of discourse phenomena requiring context; (iv) we demonstrate that this method is both effective and efficient, as it allows multi-encoder models to learn better and faster than by simply increasing the training data; (v) we provide a proof of concept for the application of our approach to train context-aware models in a setting where document-level data are not available.

Background

Single-encoder approaches

The most straightforward approach to contextaware NMT consists in concatenating the contextual sentences to the current source sentence before feeding it to the standard encoder-decoder architecture [START_REF] Tiedemann | Neural machine translation with extended context[END_REF][START_REF] Ruchit Rajeshkumar Agrawal | Contextual Handling in Neural Machine Translation: Look Behind, Ahead and on Both Sides[END_REF][START_REF] Junczys-Dowmunt | Microsoft translator at WMT 2019: Towards large-scale document-level neural machine translation[END_REF][START_REF] Ma | A simple and effective unified encoder for documentlevel machine translation[END_REF][START_REF] Zhang | Long-short term masking transformer: A simple but effective baseline for document-level neural machine translation[END_REF]. A special token is introduced to make translation models distinguish the boundaries between sentences. Generation can then follow two strategies: the many-to-many strategy consists in decoding as many sentences in the target language as those input in the source language, and then discarding all the contextual sentences; the many-to-one strategy consists in decoding the current sentence only. The modeling capacity of concatenation methods is limited to few sentences because the complexity of attention scales quadratically with sentence length, although some recent works try to solve this constraint [START_REF] Tay | Efficient Transformers: A Survey[END_REF].

Multi-encoder approaches

Multi-encoder models couple a self-standing sentence-level NMT system, with parameters θ S , with additional parameters that encode and integrate the context of the current sentence, either on source side, target side, or both. We refer to this set of parameters as the contextual parameters θ C . The full context-aware architecture has parameters Θ = [θ S ; θ C ]. Most of the multi-encoder models can be described as instances of two architectural families [START_REF] Kim | When and why is document-level context useful in neural machine translation[END_REF], that only differ in the way in which the encoded representations of the context and the current sentence are integrated.

Outside integration. In this approach, depicted in Figure 2, the encoded representations are merged outside the decoder [START_REF] Maruf | Contextual neural model for translating bilingual multi-speaker conversations[END_REF][START_REF] Voita | Context-aware neural machine translation learns anaphora resolution[END_REF][START_REF] Zhang | Improving the transformer translation model with document-level context[END_REF][START_REF] Miculicich | Document-level neural machine translation with hierarchical attention networks[END_REF]Maruf et al., 2019a;[START_REF] Zheng | [END_REF]. This can happen in different ways, such as by simple concatenation of the encodings, or with a gated sum.

Inside integration. Here the decoder attends to the context representations directly, using its internal representation of the decoded history as query [START_REF] Tu | Learning to remember translation history with a continuous cache[END_REF][START_REF] Kuang | Modeling coherence for neural machine translation with dynamic and topic caches[END_REF][START_REF] Bawden | Evaluating discourse phenomena in neural machine translation[END_REF]Voita et al., 2019b;[START_REF] Tan | Hierarchical modeling of global context for document-level neural machine translation[END_REF].

Many of these works found useful to share the parameters (some or all of them) of the currentsentence encoder and the context encoder [START_REF] Voita | Context-aware neural machine translation learns anaphora resolution[END_REF][START_REF] Li | Does multi-encoder help? a case study on contextaware neural machine translation[END_REF]. In this way, the amount of contextual parameters to learn, |θ C |, is drastically reduced, as well as the computation required to encode a sentence. Shared representation can also be cached to be re-used and further processed by contextual parameters without the need of reencoding sentences from scratch, which represents an advantage with respect to single-encoder approaches. Both single and multi-encoder models can process one or multiple context sentences, from the past or future. Most of the approaches proposed in the literature focus on a few previous sentences, where most of the relevant context is concentrated.

Two-step training. Multi-encoder models are commonly trained following a two-step strategy [START_REF] Tu | Learning to remember translation history with a continuous cache[END_REF][START_REF] Zhang | Improving the transformer translation model with document-level context[END_REF][START_REF] Miculicich | Document-level neural machine translation with hierarchical attention networks[END_REF][START_REF] Li | Does multi-encoder help? a case study on contextaware neural machine translation[END_REF]. The first step consists in training θ S independently on a sentence-level parallel corpus C S . Secondarly, contextual parameters θ C are trained on a document-level parallel corpus C D , while fine-tuning or freezing θ S . Note that C S can also include sentences from C D .

Targeted evaluation

Most proponents of novel NMT systems evaluate them by computing BLEU [START_REF] Papineni | Bleu: a method for automatic evaluation of machine translation[END_REF] on the test data. However, BLEU is ill-equipped to capture the improvements achieved by contextaware MT [START_REF] Hardmeier | Discourse in Statistical Machine Translation. A Survey and a Case Study[END_REF], because contextualization can improve the translation of only a small fraction of the words in a document, while most of the words can be correctly translated without knowing the context. For instance, only a fraction of the anaphoric pronouns in a document has its nominal antecedent outside its own sentence. However, despite being sparse, these few cases strongly impact the quality of translation [START_REF] Läubli | Has machine translation achieved human parity? a case for document-level evaluation[END_REF][START_REF] Popescu-Belis | Context in Neural Machine Translation: A Review of Models and Evaluations[END_REF]. As a consequence, a number of discourse-targeted test sets and automatic metrics have been proposed to measure improvements in context-aware MT (Maruf et al., 2019b), the most widely adopted ones being contrastive test sets.

Contrastive test sets. Contrastive sets [START_REF] Bawden | Evaluating discourse phenomena in neural machine translation[END_REF][START_REF] Müller | A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation[END_REF]Voita et al., 2019a) consist of a number of source sentences, each paired with a correct translation and some incorrect ones. Models are assessed on their ability to rank the correct translation higher than the incorrect one. In many cases, the correct translation can only be identified by looking at context, which is provided for both source and target sides. Therefore, the ranking accuracy reflects the context-modeling ability of the translation system under evaluation.

A sparse training signal

Some works have criticized multi-encoder methods [START_REF] Kim | When and why is document-level context useful in neural machine translation[END_REF][START_REF] Li | Does multi-encoder help? a case study on contextaware neural machine translation[END_REF], arguing that they do not improve over sentence-level baselines in terms of BLEU when the baseline is well regularized. When there are improvements, it is argued that the context-encoder simply works as a noise-generator that makes training more robust, and the improvements are not to be attributed to better context-modeling. Along this path, [START_REF] Lopes | Document-level neural MT: A systematic comparison[END_REF] showed that multi-encoder architectures struggle to model contextual information, and even deteriorate the performance of a sentence-level baseline on contrastive test sets (see Section 6.4). Indeed, many proponents of multi-encoder models only show BLEU improvements, without providing any kind of targeted evaluation. This raises the suspicion that their context-modeling capability is negligible. We posit that training contextual parameters is non-trivial because of the sparsity of the training signal, i.e. the sparsity of words that need context to be correctly translated. As such, missing the right experimental setting can bring to unsuccessful training and unconvincing results.

More data. A straightforward way to offset the sparsity of training signal is to increase the volume of training data. In fact, existing works that report strong results with targeted evaluation train their contextual parameters with millions of documentlevel sentence pairs [START_REF] Bawden | Evaluating discourse phenomena in neural machine translation[END_REF][START_REF] Müller | A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation[END_REF]Voita et al., 2019b;[START_REF] Zheng | [END_REF][START_REF] Wong | Contextual neural machine translation improves translation of cataphoric pronouns[END_REF][START_REF] Kang | Dynamic context selection for document-level neural machine translation via reinforcement learning[END_REF]. In contrast, many works in the literature train models with the TED talks' subtitles released by the IWSLT shared tasks [START_REF] Cettolo | WIT3: Web inventory of transcribed and translated talks[END_REF], which only consist of a couple of hundred thousand parallel sentences (see Table 1). With a set of systematic experiments (see Section 6), we show that IWSLT's subtitles are not sufficient to effectively train multi-encoder models. It follows that one can not make fair comparisons between alternative architectures in such experimental settings. We also give an empirical confirmation to the intuition that increasing the volume of training data is a solution to this difficulty.

However, the need of a large training set is not ideal, since it slows down the speed of iteration, i.e. how long it takes to test a new model, which translates in slower progress. Moreover, large documentlevel training sets may not be available in many languages. In the following section, we propose an alternative solution that allows to drastically reduce the needed amount of training data.

Proposed Approach

One way to magnify the training signal for contextual parameters is to break intra-sentential syntactic links by splitting the sentence in two segments. In this way, a word that was disambiguated by the system by looking at its neighbours within the sentence, now requires contextual information from the other segment in order to be correctly translated. Within the framework of MT, if we split Algorithm 1: Split parallel corpus 1: input: Parallel corpus C, minimum source length l min , function wheresplit()

2: for i = 1, . . . , |C| do 3: if len(S i ) ≥ l min then 4: m S , m T = wheresplit(S i , T i , ...) 5: S i,1 = S i <m S and S i,2 = S i ≥m S 6: T i,1 = T i <m S and T i,2 = T i ≥m S 7:
end if 8: end for 9: return Split corpus C D the source sentence, its corresponding reference has to be split too. The proposed approach consists in splitting all the sentences of a parallel corpus C that have at least l min tokens, as described by Algorithm 1. Each source-side sentence S i , with index i = 1, ..., |C|, is split into S i,1 and S i,2 . Its corresponding reference T i is split into T i,1 and T i,2 . The resulting corpus is a document-level parallel corpus C D , such that, if the original corpus C was itself document-level, then C D keeps the same document boundaries than C. Figure 1 illustrates two examples of parallel sentences that are split in the middle. In both examples, a context-aware system needs to look at S 1 for translating S 2 correctly, i.e. to look at past context. In the first one, the English neuter pronoun "it" could be translated into "il" or "elle", according to the gender of its antecedent (there is no singular 3rd-person neuter in French). The antecedent "a project", which belongs to the previous segment, allows to disambiguate it into "il". In the second example, the adjective "possible" can be correctly translated into its plural version "possibles" by looking back at the noun it refers to: "organisms".

Splitting methods

In Algoritm 1, the wheresplit function returns the token indices m S and m T of S i and T i , respectively, where we need to split. In this work, we propose two possible definitions of this function.

Middle-split. The most simple strategy is to split both the source and the target sentence in the middle. In this case, we can take wheresplit = middlesplit(S i , T i ), which simply returns m S = len(S i )/2 and m T = len(T i )/2 . Aligned-splits. With the middle-split method it can happen that S i,j and T i,j , with j = 1, 2, are not parallel, as illustrated in the second example of Figure 1. The verb "are" belongs to S i,1 , but its translation "sont" does not belong to its corresponding reference segment T i,1 . This problem arises whenever the splitting separates a set of words from their reference, which ends up in the other segment. As a solution, we can calculate word alignments A i and use them to inform our splitting strategy by setting wheresplit = alignedsplit(S i , T i , A i ), where alignedsplit consists in splitting each sentence close to the middle, while trying to avoid to separate aligned words in different segments. Clearly, the proposed approach requires that the two languages do not have strong syntactic divergence, to avoid too large mismatches between S i,j and T i,j , with j = 1, 2. For more details, we refer to the appendix and to our implementation. 1

Impact of splitting on training signal

In order to give an explicit picture of how and why contextual training signal augments when splitting sentences, we processed the source training data of IWSLT17 with CoreNLP [START_REF] Manning | The Stanford CoreNLP natural language processing toolkit[END_REF] and we computed some statistics on coreference chains and dependency parse trees, before and after applying the middle-split method. Statistics show how splitting the sentences of a document augments the contextual training signal in two ways.

More cases. Splitting generates new cases that require context for disambiguation, making training signal more abundant. When syntactic dependencies are split in two segments, the model needs to access the context for reconstructing the syntactic structure of the source sentence and correctly translate it, as shown in Figure 1. In order to have an idea of the magnitude of this effect, we calculated the percentage of the sentences where the splitting method breaks the syntactic link between the main verb of the sentence (the root) and : (i) the subject or object (18.1% of the sentences); (ii) any complement (9.5%); (iii) any modifier (9.3%). If we consider all the dependencies with the root, except for those of punctuation, we find that 84.8% of the sentences contain a dependency that will be broken. These cases add up to the many other cases of broken dependencies, such as coreferences, which make the overall contextual training signal more abundant.

Denser cases. The splitting also has the effect of shortening the average length of text sequences, which eases the job of context-aware systems because they have to attend to fewer words while looking for context. In Figure 3, we show how many antecedents of an anaphoric pronoun are present in the data at a given distance d, expressed as number of sentences from the current one for original data, and number of segments for split data. d = 0 means that both the pronoun and its antecedent are in the same sentence (or segment); d = 1 means that the antecedent is in previous sentence (or segment), and so on. We show statistics up to d = 3, which is the maximum context distance that we experiment with. The absolute number of antecedents is normalized by the average length of a sentence or segment. The resulting bar plot shows that splitting sentences into segments makes pronomial anaphora more dense in the set of tokens that the model is attending, which fosters the learning of contextual parameters. The same effect applies to the other discourse phenomena that require contextual disambiguation.

Zero-resource context-aware learning

Another application of the split approach is that of training context-aware models for language pairs where document-level parallel data are not available. We call this setting zero-resource. If we only have access to a sentence-level corpus C, we can split it so that C D is a document-level corpus where each "document" D i is comprised of two parallel segments D i = {(S i,1 , T i,1 ), (S i,2 , T i,2 )}. In other words, the splitting turns each sentence in a standalone document of two segments, with which we can train a context-aware model that looks at one segment back (or ahead). Such model could be used to translate documents in a language where sentence-level MT was the only alternative. We provide a proof of concept of this in Section 6.2.1.

5 Experimental setup

Data

We conduct experiments for two language pairs: English→German and English→French. Following [START_REF] Kim | When and why is document-level context useful in neural machine translation[END_REF], we pre-train sentence-level baselines on large sentence-level parallel data to make them as robust as possible. In particular, we employ data from the WMT17 2 En→De news translation shared task (∼5.2M sentences), and WMT20143 En→Fr (∼35.8M sentences). Then, sentence-level baselines are fine-tuned on TED talks subtitles from IWSLT17 [START_REF] Cettolo | WIT3: Web inventory of transcribed and translated talks[END_REF], in both languages. We train context-aware models with two document-level data settings: Low resource setting -IWSLT: consisting of TED subtitles from IWSLT17, for both En→De/Fr;

High resource setting -NEI: consisting of three document-level corpora (i) News-Commentary-v12 for En→De and News-Commentary-v9 for En→Fr; (ii) Europarl-v7, for both En→De/Fr; (iii) IWSLT17's TED talks, for both En→De/Fr.

The resulting size of training sets after preprocessing is reported in Table 1. For all models, we use IWSLT17's test sets 2011-2014 as development set, and 2015 for evaluation. Pre-processing details can be found in the appendix.

Evaluation

Besides evaluating models on the translation of TED talks subtitles with case-sensitive detokenized BLEU [START_REF] Papineni | Bleu: a method for automatic evaluation of machine translation[END_REF], 4 we employ two outof-domain contrastive test suites for the evaluation of anaphoric pronoun translation.

En→De ContraPro [START_REF] Müller | A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation[END_REF]. A large-scale test set from OpenSubtitles2018 [START_REF] Lison | OpenSubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora[END_REF], that measures translation accuracy of the English anaphoric pronoun it into the corresponding German translations er, sie or es. Examples are balanced across the three pronoun classes (4,000 examples each). Each example requires identification of the pronominal antecedent, either in the source or target side, that can be found in the current sentence or any of the previous ones.

En→Fr ContraPro [START_REF] Lopes | Document-level neural MT: A systematic comparison[END_REF]. A large-scale test set from OpenSubtitles2018 [START_REF] Lison | OpenSubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora[END_REF], completely analogous to the previous one but focused on the translation of two English pronouns: it and they. The test set consists of 3,500 examples for each target pronoun type: il or elle for it, ils or elles for they.

Models

We experiment with three models: K0. A sentence-level Transformer baseline, pretrained on WMT data and then finetuned on IWSLT. The model architecture follows the Transformerbase introduced by [START_REF] Vaswani | Attention is all you need[END_REF].

K1. A context aware multi-encoder architecture with outside integration (see Section 2.2), that encodes a single past source sentence as context.

K3. A context aware multi-encoder architecture with outside integration, that encodes three past source sentences as context. 5For both K1 and K3, sentence-level parameters θ S follow the Transformer-base configuration (hidden size of 512, feed forward size of 2048, 6 layers, 8 attention heads, total of 60.7M parameters), while contextual parameters θ C follow hierarchical architecture with source-side encoder proposed by [START_REF] Miculicich | Document-level neural machine translation with hierarchical attention networks[END_REF] (hidden size of 512, feed forward size of 2048, 8 attention heads, total of 4.7M parameters). Context-aware models are trained following two-step strategy described in Section 2.2. Sentence-level parameters θ S of both K1 and K3 are initialized with K0 and freezed. This has the advantage of saving time and computation, since only a small fraction of parameters (θ S ) is trained (4.7M over a total of 65.2M). Details can be found in the appendix.

Results and Analysis

Training with more data

In this section we provide evidence about the inefficacy of training contextual parameters on small document-level data, while larger datasets do the job. In the first block of Table 2 we report performance of context-aware models trained on IWSLT (-orig models), and compare them with the same models trained on NEI (×10 training size). When trained on IWSLT alone, contextual models K1orig and K3-orig display good BLEU on the test set, generally without degradation with respect to the baseline K0, and with some improvements in the case of K1-orig. However, such marginal improvements in BLEU are difficult to interpret, as they do not necessarily correspond to better translation [START_REF] Freitag | BLEU might be guilty but references are not innocent[END_REF]. Similarly, accuracy on the contrastive test sets increases marginally over baseline, if at all. K1 even show a degradation of performance over the baseline for En→Fr. These results highlight the struggle of contextual parameters to learn an appropriate use of context, other than acting as mere regularizers, as it was suggested by [START_REF] Kim | When and why is document-level context useful in neural machine translation[END_REF] and [START_REF] Li | Does multi-encoder help? a case study on contextaware neural machine translation[END_REF]. However, K1 and K3 show a substantial improvement in their context-modeling capabilities when trained on larger data (-fromnei), gaining 3.5+ accuracy points each over the baseline (+11.38 for K1), in both language pairs. On the other hand, BLEU improves of a few decimal points only, showing its limits to measure improvements in context-aware translation. These results would suggest that multiencoder models need to be trained with more data before comparing them with alternative approaches. However, this is not ideal because it requires more time and computation for every experiment. Indeed, the last column of Table 2 shows that a model trained on NEI requires almost ×10 the time needed by a model trained on IWSLT.

Splitting data

Here we show that the proposed splitting approach is a more convenient alternative than using large training data. The second block of Table 2 displays performance of models that have been trained on the IWSLT data, after having split sentences in two segments following the middle-split method with l min = 7. For this training setting, we also split the validation set, while the test set is always kept in its original form. Beside the sentence-level pre-training on WMT, -split models are trained on IWSLTsplit alone prior to evaluation, whilefromsplit models are also finetuned on the original, non-split data, after training on IWSLTsplit. The middle-split method is very effective, as all models show strong improvements over the baseline in terms of accuracy on the test suites. The -fromsplit approach proves to be the most advantageous. This was expected, since providing the model with both original and split sentences (instead of split alone) increases the number of cases that require context. The result of -fromsplit models is striking, because they manage to outperform models trained on NEI with one-tenth of the data (the sole exception is K1-fromsplit on En→Fr). In addition, we see from last column of Table 2, that the proposed approach is efficient, as it cuts training time by a factor of 3 when training -fromsplit instead of -fromnei.

Proof of Concept: Zero-Resource

In this section we consider IWSLT as if it was a sentence-level parallel corpus, where each sentence is a stand-alone document. It follows that, after splitting, K1 can look at its past context only when translating the second segment of each split sentence, S i,2 , but not for S i,1 . The last row of Table 2 shows that a K1 model trained in this fashion can indeed learn to model context more effectively than both K0 and -orig baselines. However, it also displays a degradation in terms of BLEU. This is probably related to the substantial difference in document-length between the training and the evaluation setting, which could be offset, for example, by adding to the sentence-level split corpus some (longer) back-translated monolingual documents [START_REF] Sugiyama | Data augmentation using back-translation for contextaware neural machine translation[END_REF]. 6 According to the paired McNemar test [START_REF] Mcnemar | Note on the sampling error of the difference between correlated proportions or percentages[END_REF].

Impact of the splitting method

Following Section 4.1, we study whether a better alignment of source and reference segments is advantageous to our objective of learning contextual parameters. In our experiments, we use the opensource tool fast_align [START_REF] Dyer | A simple, fast, and effective reparameterization of IBM model 2[END_REF], which learns token alignments in an unsupervised fashion.

After applying the aligned-split method (always with l min = 7) to IWSLT, we train context-aware multi-encoder models on it and measure their accuracy on ContraPro. We report in Table 2 the improvements ∆acc achieved with this method compared to the middle-split one. Alignments prove to be useful in most cases, although we measure a degradation of performance for K3 over the En-De language pair. In general, the improvements over the simpler middle-split method are marginal, and thus may be worth the added complexity and the additional computational cost for alignment only for some specific language pairs and context-aware models. In fact, there are at least two factors to take into account while making this choice. First, the accuracy of alignment tools is language-dependent. E.g., fast_align is stronger on En-Fr than En-De (Ho and Yvon, 2019). Second, if we favor the alignment between source and target when deciding where to split, we might be doing it at the cost of breaking less syntactic links into separate segments, which is harmful with respect to the objective of the splitting method.

Ablation: shuffling context

As an ablation study, we want to verify that the proposed approach improves learning by making the context-aware model to rely on its modeling 2. A random context is incoherent with the current sentence in many cases, and thus misleading for a contextaware system. Indeed, -fromsplit models display a significant drop in performance when they are evaluated with inconsistent context, as well as -fromnei models, which confirms that they rely on context information to achieve the improvement in pronoun translations. Nonetheless, the same models prove to be robust against being shown a random context as they obtains a similar performance to K0. In other words, the splitting method does not produce models that are over reliant on context.

Comparison with the literature

In Table 3 we compare our results with those reported by [START_REF] Lopes | Document-level neural MT: A systematic comparison[END_REF], who evaluated various context-aware approaches on the same IWSLT setting as ours. They also adopt a large sentence-level pre-train, a two-step training strategy with frozen Transformer-base parameters for multi-encoder models, and the exact same evaluation setting as ours. We report their results for a baseline K0 like ours; a multi-encoder Zhang2018 with both inside integration and outside integration, that encodes three past source sentences [START_REF] Zhang | Improving the transformer translation model with document-level context[END_REF]; a multi-encoder Tu2018 with inside integration, that encodes all past context, both source-side and target-side; a many-to-one (2-to-1) single-encoder approach Concat21; a many-tomany (2-to-2) single-encoder approach Concat22. Lopes et al. ( 2020)'s multi-encoder models perform poorly or even lag behind K0, as it happens for -orig models when trained on IWSLT original data alone. Instead, concatenation approaches are very strong, because they do not have extra contextual parameters to train and simply finetune the same sentence-level Transformer architecture on the context-aware task. Our splitting strategy proves to be very effective, since -fromsplit models outperform Concat21 using the same amount of training data. Moreover, -fromsplit multi-encoder models beat the strong Concat22 on En→Fr, which has the non-negligible advantage of using also the target-side context, on top of the source-side. Finally, we try to train the contextual parameters of K1 and K3 on the much larger NEIsplit, where sentences are split according to the middle-split method, and then fine-tune on IWSLT for comparison. Results reported in Table 3 show further improvements on the anaphoric pronoun translation set, with no BLEU degradation with respect to the baseline. Nonetheless, considering the large increase in training data, these improvements seem to be marginal. Diminishing returns with the increase of training data could be related to the fact that context-aware models used for the experiments are limited to the source-side context, while target-side context is sometimes key for solving some examples in contrastive test suites.

Conclusions

Multi-encoder models are a broad family of context-aware NMT models. In this work we have shown that their contextual parameters are difficult to train because the training signal is sparse. We have proposed an approach based on splitting the training sentences, which augments the training signal by breaking syntactic links. After having analysed the implications of our approach on discourse phenomena, we have shown empirically that it allows to learn contextual parameters very effectively, and more efficiently than by simply increasing the amount of training data. Our approach also proves to be a viable option to zero-resource context-aware machine translation.

We leave to future work experiments with a syntactically motivated splitting method, which would use the syntactic information to be sure to break a syntactic link every time it is possible.

A Aligned-split method

In this section, we provide more details on the aligned-split method. As already mentioned, we use wheresplit = alignedsplit(S i , T i , A i ) , which takes as input the word alignments A i :

A i = {(j, k)|S i j and T i k are aligned},
where j = 1, ..., |S i | and k = 1, ..., |T i | are the indices of the words belonging to S i and T i , respectively. alignedsplit initially takes m S = len(S i )/2 and m T = max{k : (j, k) ∈ A i , j ≤ m S }. Then, it checks whether this choice is not breaking apart two aligned words. Formally, it checks that:

S i j ∈ S i,1 ∧ T i k ∈ T i,1 or S i j ∈ S i,2 ∧ T i k ∈ T i,2 . (1)
If this condition is not encountered, it tries to split the sentence pairs in the neighbouring distance, where condition (1) is met. If the condition can not be met, alignedsplit falls back to middlesplit. We refer to our open-source implementation for the full details.

B Experimental Setup

B.1 Data preprocessing

All datasets are tokenized with the Moses toolkit [START_REF] Koehn | Moses: Open source toolkit for statistical machine translation[END_REF], further cleaned by removing long sentences, and byte pair encoded [START_REF] Sennrich | Neural machine translation of rare words with subword units[END_REF] using 32k merge operations jointly for source and target languages. While IWSLT provides document boundaries for TED subtitles, the WMT releases of New-Commentary and Europarl do not provide them. Therefore, a small fraction of sentences in the NEI setting will be paired with wrong context. However, we found the models to be robust against occasional random context (see also [START_REF] Voita | Context-aware neural machine translation learns anaphora resolution[END_REF] and [START_REF] Müller | A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation[END_REF]). In order to make the models correctly learn how to translate headlines (the first line in a document), we need to have headlines in the training set. As such, we set artificial document boundaries in News-Commentary and Europarl, following the average document length of TED talks.

B.2 Training and evaluation

All models are implemented in fairseq [START_REF] Ott | fairseq: A fast, extensible toolkit for sequence modeling[END_REF]. After having pre-trained the baseline on 4 Tesla V100 for 200k steps, we train all models on a single Quadro RTX 6000, with a fixed batch size of approximately 16k tokens,7 as it has been shown that Transformers need a large batch size for achieving the best performance [START_REF] Popel | Training Tips for the Transformer Model[END_REF]. We stop training after 5 consecutive nonimproving validation steps. We train models with the optimizer configuration and learning rate schedule described in [START_REF] Vaswani | Attention is all you need[END_REF]. The maximum learning rate is 0.0005 for models trained on NEI, 0.003 for all the others. We use label smoothing with an epsilon value of 0.1 [START_REF] Pereyra | Regularizing Neural Networks by Penalizing Confident Output Distributions[END_REF]. Since the models are pre-trained on a large amount of parallel data (WMT), the dropout is set to 0.1, which gave the best results for the non-contextual baseline K0. At inference time, we use beam search with a beam of 4 for all models. The only hyperparameter that changes across training settings is length penalty (P len ). In particular, we adopt a length penalty of 0.6 for all models trained on En→De data (P len < 1 favors shorter sentences). For En→Fr models, the length penalty changes according to the training data employed: P len = 0.6 for models trained on original, non-split data, i.e., -orig and -fromnei models; P len = 1.3 for models trained on split-data alone, i.e., -split models (P len > 1 favors longer sentences); and P len = 1.0 for models that have been trained on both split and non-split data, i.e., -fromsplit models. This choice is intuitive considering that split corpora have a shorter average sentence length.

C Results Analysis

C.1 Accuracy by antecedent distance

In this section, we want to investigate more in details the performance of the proposed approach on the accuracy of ambiguous pronoun translation. We report in Table 5 the accuracy on En→De ContraPro, detailed by varying antecedent distance. We notice that all the improvements achieved by -fromsplit models are related to those pronouns whose antecedent is in the context (d ≥ 1), which is in line with the expectations of context-aware models exploiting context for disambiguation. K1fromsplit is very strong in translating pronouns with antecedent distance d = 1, surpassing K0 and -orig baselines by 22+ points of accuracy. Similarly, K3-fromsplit surpass baselines by a large margin on 0 ≤ d ≤ 3, beating all the other models on d = 2, 3, as expected. We notice however that K3-fromsplit lacks behind K1-fromsplit on d = 1. On one side, this could be explained by the fact that K1-fromsplit is more specialized at modeling a single past sentence. On the other side, we also notice that the hierarchical context-encoding architecture by [START_REF] Miculicich | Document-level neural machine translation with hierarchical attention networks[END_REF], at the core of K3, is not aware of the distance of the context sentences that are encoded. Hence, we believe that K3-fromsplit might perform worse on d = 1 than K1-fromsplit because it gives the same importance to further away context (d = 2, 3). Since pronouns with antecedent distance d = 1 are the most frequent in the test set, K1-fromsplit has the highest average result (reported in "Total").

C.2 Ablation: shuffling context

We have shown that well trained context-aware models display a significant drop in accuracy of ambiguous pronoun translation when they are evaluated with inconsistent context, which suggests that they are heavily relying on context for solving ambiguous translations. We complement these find-ings with the equivalent experiment on the IWSLT test set. We shuffle the test set in order to make context incoherent for most of the sentences in the documents, and then we evaluate the translation output of context-aware models with BLEU. Results reported in Table 6 display a small degradation in average quality performance as measured by BLEU. However, the changes are so small that are probably negligible, showing once again that BLEU is ill-equipped to measuring improvements in document-level translation.

Figure 2 :

 2 Figure 2: Multi-encoder approach integrating context outside the decoder.

Figure 3 :

 3 Figure 3: The number of antecedents of anaphoric pronouns at a given distance in terms of sentences/segments, normalized by the number of tokens that the model needs to attend for resolving the coreference.

Table 1 :

 1 Millions of En→De/Fr sentence pairs used for training context-aware models.

		News	Europarl	TED	Total
	IWSLT	x	x	0.20/0.23 0.20/0.23
	NEI	0.26/0.17 1.82/1.92 0.20/0.23 2.28/2.32

Table 2 :

 2 BLEU scores on the in-domain IWSLT test set and accuracies (in %) on the ContraPro test sets. The last column reports the training time spent on the document-level MT task by En→Fr models. All differences in accuracy between any model and the baselines (K0 and -orig) are statistically significant with p < 0.01. 6 ∆acc measures the change in accuracy when IWSLT is split with the aligned-split method instead of middle-split.

	En→De	En→Fr

Table 3 :

 3 Comparison between[START_REF] Lopes | Document-level neural MT: A systematic comparison[END_REF]'s evaluation of context-aware models (1st block), our models trained on split IWSLT (2nd), and split NEI (3rd). We report BLEU scores on IWSLT and accuracies (in %) on ContraPro. * : these models have the advantage of accessing target-side context, hence comparison is unfair.

		En→De	En→Fr
	Model	acc (∆acc)	acc (∆acc)

Table 4 :

 4 Accuracy results (and their changes) on contrastive examples when using inconsistent context.

	of the context. Table 4 shows performances on
	ContraPro after having shuffled the context of ev-
	ery contrastive example in the test set (c.f. Scher-
	rer et al. (2019)), and the delta w.r.t. the results
	with consistent context presented in Table

  the Most of Context in Neural Machine Translation. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3983-3989. International Joint Conferences on Artificial Intelligence Organization.

Table 6 :

 6 Changes in BLEU results when the test set is shuffled, making context incoherent.

Which will be soon open-sourced.

http://www.statmt.org/wmt17/translation-task.html

http://www.statmt.org/wmt14/translation-task.html

Moses' multi-bleu-detok[START_REF] Koehn | Moses: Open source toolkit for statistical machine translation[END_REF].

Although the splitting does not increase the number of inter-segment phenomena for d > 1, it strengthens the signal by making it more dense (see Section 4.2). Thus, K3 and any wider-context model can profit from the proposed approach.

The optimizer update is delayed to simulate the 16k tokens.

Acknowledgements

This work has been partially supported by the Multidisciplinary Institute in Artificial Intelligence MIAI@Grenoble Alpes (ANR-19-P3IA-0003).

Total