Jitao Xu

Josep Crego
email: josep.crego@systrangroup.com

François Yvon
email: francois.yvon@limsi.fr

Cela Cela

Bilingual Synchronization: Restoring Translational Relationships with Editing Operations

Keywords:

Machine Translation (MT) is usually viewed as a one-shot process that generates the target language equivalent of some source text from scratch. We consider here a more general setting which assumes an initial target sequence, that must be transformed into a valid translation of the source, thereby restoring parallelism between source and target. For this bilingual synchronization task, we consider several architectures (both autoregressive and non-autoregressive) and training regimes, and experiment with multiple practical settings such as simulated interactive MT, translating with Translation Memory (TM) and TM cleaning. Our results suggest that one single generic edit-based system, once fine-tuned, can compare with, or even outperform, dedicated systems specifically trained for these tasks. pas

Introduction

Neural Machine Translation (NMT) systems have made tangible progress in recent years [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF][START_REF] Vaswani | Attention is all you need[END_REF], as they started to produce usable translations in production environments. NMT is generally viewed as a one-shot activity process in autoregressive approaches, which generates the target translation based on the sole source side input. Recently, Non-autoregressive Machine Translation (NAT) models have proposed to perform iterative refinement decoding [START_REF] Lee | Deterministic non-autoregressive neural sequence modeling by iterative refinement[END_REF][START_REF] Ghazvininejad | Mask-predict: Parallel decoding of conditional masked language models[END_REF][START_REF] Gu | Levenshtein transformer[END_REF], where translations are generated through an iterative revision process, starting with a possibly empty initial hypothesis.

This paper focuses on the revision part of the machine translation (MT) process and consider bilingual synchronization (Bi-sync), which we define as follows: given a pair of a source (f ) and a target (ẽ) sentences, which may or may not be mutual translations, the task is to compute a revised version e of ẽ, such that e is an actual translation of f . This is necessary when the source side of an existing translation is edited, requiring to update the target and keep both sides synchronized. Bi-sync subsumes standard MT, where the synchronization starts with an empty target (ẽ = []). Other interesting cases occur when parts of the initial target can be reused, so that the synchronization only requires a few changes.

Bi-sync encompasses several tasks: synchronization is needed in interactive MT (IMT, [START_REF] Knowles | Neural interactive translation prediction[END_REF] and bilingual editing [START_REF] Bronner | Cosyne: Synchronizing multilingual wiki content[END_REF], with ẽ the translation of a previous version of f ; in MT with lexical constraints [START_REF] Hokamp | Lexically constrained decoding for sequence generation using grid beam search[END_REF], where ẽ contains target-side constraints [START_REF] Susanto | Lexically constrained neural machine translation with Levenshtein transformer[END_REF][START_REF] Xu | EDITOR: An Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints[END_REF]; in Translation Memory (TM) based approaches [START_REF] Bulte | Neural fuzzy repair: Integrating fuzzy matches into neural machine translation[END_REF], where ẽ is a TM match for a similar example; in automatic post-editing (APE) [START_REF] Félix Do Carmo | A review of the state-of-the-art in automatic postediting[END_REF], where ẽ is an MT output.

We consider here several implementations of sequence-to-sequence models dedicated to these situations, contrasting an autoregressive model with a non-autoregressive approach. The former is similar to [START_REF] Bulte | Neural fuzzy repair: Integrating fuzzy matches into neural machine translation[END_REF], where the source sentence and the initial translation are concatenated as one input sequence; the latter uses the Levenshtein Transformer (LevT) of [START_REF] Gu | Levenshtein transformer[END_REF]. We also study various ways to generate appropriate training samples (f , ẽ, e). Our experiments consider several tasks, including TM cleaning, which attempts to fix and synchronize noisy segments in a parallel corpus. This setting is more difficult than Bi-sync, as many initial translations are already correct and need to be left unchanged. Our results suggest that one single AR system, once fine-tuned, can favorably compare with dedicated systems for each of these tasks. To recap, our main contributions are (a) the generalization of several tasks subsumed by a generic synchronization objective, allowing us to develop a unified perspective about otherwise unrelated subdomains of MT; (b) the design of a Figure 1: Methods for generating synthetic initial translations ẽ for each edit type. Rectangle purple boxes refer to separate models used to generate the desired operations. Differences in artificial initial translations (in blue boxes) are marked in bold. Initial translations ẽins for insertion are generated by randomly removing segments in the reference sentence e. For ẽsub , e is first back-translated into an intermediate sentence f * using top-5 sampling, then translated back to ẽsub with LCD. The first method to generate ẽdel randomly inserts [gap] tokens into e and decodes with a GAP insertion model [START_REF] Xiao | BiTIIMT: A bilingual text-infilling method for interactive machine translation[END_REF]. ẽdel1 is obtained by replacing [gap] with the predicted segments. The second method automatically edits e with a model trained on WikiAtomicEdits data.

training procedure for a generic edit-based model; (c) an empirical validation on five settings and domains.

Methods

Generating Editing Data

We consider a general scenario where, given a pair of sentences f and ẽ, assumed to be related, but not necessarily parallel, we aim to generate a target sentence e that is parallel to f . We would also like ẽ and e to be close, as ẽ is often a valid translation of a sentence f that is close to f . Training such models requires triplets (f , ẽ, e). While large amounts of parallel bilingual data are available for many language pairs, they are hardly ever associated to related translations ẽ (except for APE). We therefore study ways to simulate synthetic ẽ from e, while preserving large portions of e in ẽ. String edits can be decomposed into a sequence of three basic edits (insertions, substitutions and deletions), we design our artificial samples so that edits from ẽ to e only involve one type of operation (Figure 1).

Insertions

We mainly follow [START_REF] Xiao | BiTIIMT: A bilingual text-infilling method for interactive machine translation[END_REF] to generate initial translations ẽins for insertion by randomly deleting segments from e. For each e, we first randomly sample an integer k∈[1, 5], then randomly remove k non-overlapping segments from e. The length of each removed segment is also randomly sampled with a maximum of 5 tokens. We also impose that the overall ratio of removed segments does not exceed 0.5 of e. Different from [START_REF] Xiao | BiTIIMT: A bilingual text-infilling method for interactive machine translation[END_REF], ẽins does not include any placeholders to locate the positions of removed segments. This makes ẽins a more realistic starting point as the insertion positions are rarely known in practical settings. Our preliminary experiments also show that identifying insertion positions makes the infilling task easier than when they are unknown.

Substitutions

To simulate substitutions, we apply round-trip translation with lexically constrained decoding (LCD, [START_REF] Post | Fast lexically constrained decoding with dynamic beam allocation for neural machine translation[END_REF] to generate initial translations for substitution ẽsub . Roundtrip translation is already used for the APE task in [START_REF] Junczys | Log-linear combinations of monolingual and bilingual neural machine translation models for automatic post-editing[END_REF]. This requires two standard NMT models separately trained on parallel data, one for each direction. For each training example (f , e), we first (a) translate e into an intermediate source sentence f * using top-5 sampling [START_REF] Edunov | Understanding back-translation at scale[END_REF];1 (b) generate an abbreviated version ẽ ins using the method described above for insertions. We then translate f * using LCD, with ẽ ins as constraints, to obtain ẽsub . In this way, we ensure that at least half of e remains unchanged in ẽsub , while the other parts have been substituted. To increase diversity, ẽ ins (used to create ẽsub ) is sampled with a different random seed than ẽins (used for the insertion task).

Deletions Simulating deletions requires the initial translation ẽdel to be an extension of e. We propose two strategies to generate ẽdel . The first uses a GAP insertion model as in [START_REF] Xiao | BiTIIMT: A bilingual text-infilling method for interactive machine translation[END_REF], in which word segments are randomly replaced with a placeholder [gap] to generate ẽgap . The task is then to predict the missing segments based on the concatenation of f and ẽgap as input. This differs from our own insertion task, as (a) insertion positions are identified as a [gap] symbol in ẽgap and (b) generation only computes the sequence of missing segments e seg , rather than a complete sentence.

We use GAP to generate extra segments for a pair of parallel sentences as follows. We randomly insert k∈ [1,5] [gap] tokens into e, concatenate it with f and use GAP to predict extra segments, yielding the synthetic target sentence ẽdel 1 . This method always extends parallel sentences with additional segments on the target side. However, these segments are arbitrary and may not contain any valid semantic information, nor be syntactically correct.

We thus consider a second strategy, based on actual edit operations collected in the WikiAtomicEdits dataset2 [START_REF] Faruqui | WikiAtomicEdits: A multilingual corpus of Wikipedia edits for modeling language and discourse[END_REF], which contains edits of an original segment x and the resulting segment x , with exactly one insertion or deletion operation for each example, collected from Wikipedia edit history. This notably ensures that both versions of each utterance are syntactically correct. We treat the deletion data of WikiAtomicEdits as "reversed" insertions, and use both of them to train a seq-toseq wiki model (x short →x long ), generating longer sentences from shorter ones. The wiki model is then used to expand e into an ẽdel 2 . Compared to ẽdel 1 , ẽdel 2 is syntactically more correct. However, it is also by design very close (one edit away) to e.

As both simulation methods have merits and flaws, we randomly select examples from ẽdel 1 and ẽdel 2 to build the final synthetic initial translation samples for the deletion operation ẽdel .

Copy and Translate Operations To handle parallel sentences that do not require any changes, we add a fourth copy operation, where the initial translation ẽcp is equal to the target sentence (ẽ cp =e). Hence, the data used to learn edit operations is built with triplets (f , ẽ, e) where ẽ is uniformly randomly selected from ẽins , ẽsub , ẽdel and ẽcp . Finally, to maintain the capacity to perform standard MT from scratch, we also consider samples where ẽ is empty. The implementation of standard MT varies slightly upon approaches, as we explain below.

Model Architectures

We implement Bi-sync with Transformer-based [START_REF] Vaswani | Attention is all you need[END_REF] autoregressive and nonautoregressive models. The former (Edit-MT) is a regular Transformer with a combined input made of the concatenation of f and ẽ; the latter (Edit-LevT) is the LevT of [START_REF] Gu | Levenshtein transformer[END_REF].

Edit-MT In this model, ẽ is simply concatenated to f , with a special token to separate the two sentences. This technique has been used e.g. in [START_REF] Dabre | Enabling multi-source neural machine translation by concatenating source sentences in multiple languages[END_REF] for multi-source MT or in [START_REF] Bulte | Neural fuzzy repair: Integrating fuzzy matches into neural machine translation[END_REF] for translating with a similar example. The input side of the editing training data is thus f [sep] ẽ, as shown in Figure 1 (top).

On the target side, we add a categorical prefix to indicate the type of edit(s) associated with a given training sample, as is commonly done for multidomain or multilingual MT. For each basic edit (insertion, substitution and deletion), we use a binary tag to indicate if the operation is required. For instance, an ẽins needing insertions would have tags [ins] [!sub] [!del] prepended to e. Copy corresponds to all three tags set to negative as

[!ins] [!sub] [!del].
The tagging scheme provides us with various ways to perform edit-based MT: (a) we can perform inference without knowing the required edit type of ẽ by generating tags then translations; (b) when the edits are known, we can generate translations with desired edits by using the corresponding tags as a forced prefix; (c) inference can also only output the edit tags and predict the relation between f and ẽ. The ability to perform standard MT is preserved by training with a balanced mixture of editing data and parallel data.

The latter corresponds to an empty ẽ. For these examples, the target side does not contain any tags.

Edit-LevT LevT is trained with a randomly noised version of the reference as initial target, and decodes with empty sentences. In Bi-sync, we instead initialize the target side with the given ẽ for training and inference. To perform standard MT with the same model, we train with a tunable mixture of these two strategies, where p controls the proportion of each type of samples. Taking p = 0 is equivalent to train a LevT model with only parallel data. We use p = 0.5 in our experiments, making it equivalent to mixing editing and parallel data for the Edit-MT model. The value of p can be carefully designed with a schedule or curriculum to optimize the behavior of Edit-LevT, which we leave for future work. For Edit-LevT, we do not use any tags, as Edit-LevT already includes an internal mechanism to predict the edit operation(s).

3 Bilingual Re-synchronization

Datasets and Experimental Settings

We first evaluate Edit-MT and Edit-LevT on a basic resynchronization task where ẽ is assumed to be the translation of a former version of f , and only a limited amount of edits is sufficient to restore parallelism. We conduct experiments on WMT14 English-French data3 in both directions (En-Fr & Fr-En) and evaluate on two test sets. The first is an artificial derivation of the standard newstest2014 set, and the second is the small parallel sentence compression dataset4 of [START_REF] Ive | Parallel sentence compression[END_REF].

For the original newstest2014, we generate an initial version ẽ for each test sentence and each edit operation according to the methods of Section 2.1. For deletion, we test the performance of both generation methods, resulting in four versions (Ins, Sub, Del 1 , Del 2 ) of newstest2014 with 3,003 sentences each. The sentence compression dataset contains a subset of documents also selected from newstest2014, where sentences are compressed by human annotators while remaining parallel in the two languages. We only retain utterances for which the compressed and original versions actually differ on both sides, resulting in 526 test sentences.

Both experiments use the same training data, where we discard examples with invalid language tag as computed by fasttext language identifica-tion model5 [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF], yielding a training corpus of 33.9M examples. We tokenize all data using Moses6 and build a shared sourcetarget vocabulary with 32K Byte Pair Encoding (BPE) units [START_REF] Sennrich | Neural machine translation of rare words with subword units[END_REF] learned with subword-nmt.7 Since we use both parallel and artificial editing data to train edit-based models, the total training data contains about 68M utterances.

We conduct experiments using fairseq8 [START_REF] Ott | fairseq: A fast, extensible toolkit for sequence modeling[END_REF]. Edit-MT relies on the Transformerbase model of [START_REF] Vaswani | Attention is all you need[END_REF]. Model and training configurations are in Appendix A. Performance is computed with SacreBLEU [START_REF] Post | A call for clarity in reporting BLEU scores[END_REF].

Results

We first separately evaluate the learnability of each edit operation on our synthetic newstest2014 sets. We also derive two tasks from the compression dataset: parallel sentence compression (comp) and extension (ext). For compression, the task consists of producing a compressed target sentence e comp given the compressed source f comp and the original target e. For extension, the model should produce e with f and the compressed target e comp . These two tasks are respectively similar to the deletion and insertion tasks. There are slight differences, though, as (a) ẽ for these settings is always syntactically correct, (b) segments that are removed or inserted are selected for their lower informativeness. Therefore, these tasks are more about restoring an adequate, rather than a fluid, translation.

As mentioned in Section 2.2, the generation of translations in Edit-MT models is conditioned on predicted or oracle editing tags that are prefixed to the output: these two situations are contrasted using forced-prefix decoding with the correct tags. For the compression and extension tasks, we use the deletion and insertion tags, respectively.

Tables 2 and3 report results for both directions, to be compared with a "do-nothing" baseline which simply copies ẽ as the output. Edit-MT is able to edit the given ẽ for all types of required edits much better than Edit-LevT. It obtains large gains over the copy baseline for insertion,9 substitution and deletion for both directions. When tested on the compression and extension tasks, which have different edit distributions to the artificial editing data, Edit-MT still improves ẽ by 1.2-4.7 BLEU for En-Fr and 2.4-7.9 BLEU for Fr-En. By prefixing Edit-MT with the oracle editing type tags, we can further boost the performance on almost every task for both directions. Edit-LevT can also improve ẽ in most test situations, even though the gains are lower than Edit-MT models. However, due to the non-autoregressive nature of Edit-LevT, it obtains a decoding speedup of 2.3-3× with respect to Edit-MT when tested with the same inference batch size on the same hardware, as recommended by [START_REF] Helcl | Non-autoregressive machine translation: It's not as fast as it seems[END_REF].

To better understand Edit-MT and Edit-LevT on the resynchronization task, we further analyze their performance with respect to the edit distance ∆ between ẽ and e. For the results in Table 1, we merge test sentences of all edit types (Ins, Sub, Del 1 , Del 2 ) into one test set, then break them down by the value of ∆. For both directions, prefixing the oracle editing tags for Edit-MT yields a stable improvement for almost all ∆. Edit-LevT performs similar to Edit-MT when no editions are needed, but only starts to improve the copy baseline when more edits are required (∆ ≥ 8).

Translating with Translation Memories

As explained above, Bi-sync encompasses examplebased MT, whereby an existing similar translation retrieved from a TM is turned into an adequate translation of the source. Edit-MT actually uses the same architecture as the retrieval-based models of [START_REF] Bulte | Neural fuzzy repair: Integrating fuzzy matches into neural machine translation[END_REF]; [START_REF] Xu | Boosting neural machine translation with similar translations[END_REF]. In this section, we study the performance of our synchronization models in this practical scenario.

Datasets

We use the same multi-domain corpus as [START_REF] Xu | Boosting neural machine translation with similar translations[END_REF], which contains 11 different domains for the En-Fr direction, collected from OPUS10 (Tiedemann, 2012). We search for similar translations using Fuzzy Match. 11 The similarity between two English source sentences is computed as:

sim(f , f ) = 1 - ED(f , f ) max(|f |, | f |) , (1)
where ED(f , f ) is the edit distance between f and f , and |f | is the length of f . The intuition is that the closer f and f is, the more suitable ẽ will be. To (2020) proposed an alternative where, for each similar translation, they masked out segments that were not aligned with the source. This means that the initial similar translation only contains segments that are directly related to the source input. We also reproduce this related setting, which is very similar to the insertion task.

As we are mostly interested in the edit behavior, we split the data by keeping 1,000 sentences with a sufficiently similar translation (sim > 0.6) in the TM as the test set for each domain. The remaining data is used for training. We use all found similar translations for training and only the best similar translation for testing. This results in 4.4M parallel sentences (para), in which 2.6M examples are also associated to a similar translation (similar) or just the related segments (related). Data preprocessing is similar as in Section 3.1.

Experimental Settings

Our baselines reproduce two settings for TM-based MT: the FM setting of [START_REF] Bulte | Neural fuzzy repair: Integrating fuzzy matches into neural machine translation[END_REF] and the FM # setting of [START_REF] Xu | Boosting neural machine translation with similar translations[END_REF]. The former is trained using para + similar data, and the latter uses para + related. These two baselines are trained with the same configuration as in Section 3.1. We also report scores obtained by simply copying the retrieved similar translations, as in Section 3.2.

Edit-MT and Edit-LevT from Section 3 differ from FM and FM # both in task, and also in training domains. Hence, we consider fine-tuning (FT) our models. For Edit-MT, we use para+similar+ related data and fine-tune for only 1 epoch with a learning rate of 8e -5 . As we do not have information about the edit operations required to change a similar translation into the reference, we set all editing tags as on for similar data, and prefix the output with [ins] [sub] [del]. For the related data, we conjecture that mostly insertions are needed as the irrelevant segments have already been removed. We thus only activate the insertion tag. For Edit-LevT, we only use similar+related and fine-tune for 1 epoch with a learning rate of 9e -5 . Our finetuned models can perform both translation with a similar sentence and with related segments.

Results and Analysis

We evaluate with BLEU, and also show TER scores in Appendix C. We reproduce in Table 4 the overall good performance of FM and FM # . Both significantly improve the initial similar translations. The generic Edit-MT performs much worse, and does not even match the copy results. When prefixed with the editing tag 13 (+tag), we observe small improvements (+0.8 BLEU on average), that are further increased in the related scenario (+R). FT yields a much larger boost in performance (+13.4 BLEU). This highlights the effect of the task and domain mismatches on our initial results with Edit-MT. The related setting also benefits from FT, albeit by a smaller margin (+9 BLEU). 13 We conjecture with a substitution tag for zero-shot inference as we have no information about the required edit type. Fine-tuned model uses the same tag as FT data.

Model ∆(ẽ, e) Our best results overall, using FT, are superior to FM # and close to that of FM. This has practical implications, since FM # and FM are specifically trained to transform a retrieved translation, whereas the generic Edit-MT is initially trained with artificial edits then only slightly fine-tuned on the in-domain data. To appreciate this difference, we evaluate our models on two unseen domains (Office and ENV), neither of which are used to train FM and FM # , nor to fine-tune Edit-MT. Results in Table 6 unambiguously show that in this setting, the fine-tuned Edit-MT outperforms FM, suggesting that our edit-based model has not only adapted to the domain, but also to the task, as it can effectively perform zero-shot TM-based translations. Results obtained with the Edit-LevT model on these test sets lag far behind: even with FT on in-domain data, Edit-LevT still struggles to improve over the copy baseline.

0
We also perform the analysis with a breakdown by the edit distance ∆ as in Section 3.2 on the multidomain test sets. For the results in Table 5, all 11k test sentences are merged into one test set, then broken down by values of ∆. The generic Edit-MT starts to improve the similar translation for ∆≥3. However, it is difficult for Edit-MT to detect very small changes (∆<3) without FT. Once fine-tuned, Edit-MT performs similar to FM for small changes, which further confirms that Edit-MT adapts to the TM-based translation task. Edit-LevT models, however, only slightly improve the copy baseline for similar translations requiring large changes (∆≥8). However, it is better than other models at detecting ẽs that do not need edits (∆=0). We also provide results broken down the merged test set by different edit operations in Appendix C.

Parallel Corpus Cleaning

Our model restores synchronization between a pair of sentences. This is also useful for parallel TM cleaning tasks. Given a source sentence f and a possibly incorrect translation ẽ, we want to detect non-parallelism and to perform appropriate fixes. We study how Edit-MT fares with this new problem on two publicly available datasets: first on the SemEval 2012&3 Task 8: Cross-lingual Textual Entailment (CLTE, [START_REF] Negri | Divide and conquer: Crowdsourcing the creation of cross-lingual textual entailment corpora[END_REF][START_REF] Negri | Semeval-2012 task 8: Cross-lingual textual entailment for content synchronization[END_REF][START_REF] Negri | Semeval-2013 task 8: Cross-lingual textual entailment for content synchronization[END_REF]),14 then with the OpenSubtitles corpus [START_REF] Lison | OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles[END_REF].

Cross-lingual Textual Entailment

The CLTE task aims to identify multi-directional entailment relationships between two sentences x 1 and x 2 written in different languages. We evaluate on the Fr-En direction, where x 1 is in French and x 2 is in English. The tagging mechanism of Edit-MT (see Section 2.2) can readily be used for this classification task. Data descriptions and slight adjustments of tagging scheme are in Appendix D.

We treat x 1 as f and x 2 as ẽ to match the input format of Edit-MT, and perform zero-shot inference reusing the same Edit-MT model as in Section 3. We concatenate x 1 and x 2 as input, and truncate the target sequence by only taking the first three edit tags as the predicted label for the corresponding input pair, treating Edit-MT as a mere classification (CLF) model. We also slightly finetune Edit-MT with the 500 examples of CLTE training data for 5 epochs with a learning rate of 8e-5. 7, together with the best scores reported in [START_REF] Negri | Semeval-2013 task 8: Cross-lingual textual entailment for content synchronization[END_REF] for years 2012 and 2013 and the scores reported in [START_REF] Carpuat | Detecting cross-lingual semantic divergence for neural machine translation[END_REF], which are the best reported performance we could find. Note that these scores may be quite weak, as pre-trained language models did not exist at that time. We have tried to apply a pre-trained XLM model [START_REF] Conneau | Crosslingual language model pretraining[END_REF] to report better baselines for CLTE. However, fine-tuning XLM with only 500 sentences for 5 epochs did not outperform the reported baselines, as the fine-tuned XLM model needs to train new parameters for the linear output layer. Our Edit-MT, on the contrary, does not require any additional parameters during fine-tuning. As can be seen in Table 7, out-of-the-box Edit-MT fails to clearly detect the entailment relationships. This is not surprising, as there is a significant difference between our editing data and the CLTE test sets. For instance, the insertion initial translation ẽins is always grammatically incorrect, while all sentences in CLTE are syntactically correct. However, after slight fine-tuning on the CLTE data, Edit-MT for both directions can quickly adapt to the task, achieving state-of-the-art performance. This again hints that Edit-MT actually learns to identify various cases of non-parallelism.

Results are in Table

Fixing OpenSubtitles Corpus

We further evaluate the ability of Edit-MT to detect parallel sentences and fix noisy data. We experiment with the OpenSubtitles15 data [START_REF] Lison | OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles[END_REF] for the En-Fr direction. In this corpus, the French side is translated from English, but with noisy segments. A standard approach is to filter out noisy sentences from the training data when building systems. We aim to study whether Edit-MT can automatically identify and edit, rather than discard, noisy sentence pairs, so that training can use the full set of parallel data. We measure performance on the 10,159 segments of the En-Fr Microsoft Spoken Language Translation (MSLT) task [START_REF] Federmann | Microsoft speech language translation (MSLT) corpus: The IWSLT 2016 release for English, French and German[END_REF], which simulates a real world MT scenario.

The OpenSubtitles data is processed similarly as in Section 3.1. We first use the fine-tuned CLF model in the CLTE task to predict the relation for all sentence pairs in OpenSubtitles data. About 60% of the data is classified as parallel, indicating that no edit operation is predicted for these segments. Models trained on the 60% clean data are denoted as filtered. For the other 40% presumably noisy data, we reuse the Edit-MT En-Fr model of Section 3 to fix the translations, using the predicted edit tag as a prefix on the target side (see Sections 3 and 4). Models trained on the edited data are noted as fixed. We train NMT models with either all data (full) or just the 40% noisy data as baselines. For comparison, we also train a model using the same data size (15.8M) as the noisy subset, randomly selected from the filtered subset.

As shown in Table 8, aggressively filtering the noisy data improves over using the full training corpus (+2 BLEU for filtered) more than revising it (+1 BLEU for filtered + fixed). The second set of results yields similar conclusions with smaller datasets: here, the effect of automatically fixing a set of initially noisy data improves the BLEU score by 7.2 points and closes half of the gap with a clean corpus of the same size. Note that these results are obtained without adaptation, simply reusing the pre-trained Edit-MT model of Section 3. This suggests that in situations where the training data is small and noisy, editing-based strategies may provide an effective alternative to filtering.

Related Work

The prediction of translations based on a source sentence and an initial translation is first explored in the context of IMT, using a left-to-right sentence completing framework proposed by [START_REF] Langlais | TransType: a computer-aided translation typing system[END_REF]. The proposals of [START_REF] Green | Human effort and machine learnability in computer aided translation[END_REF]; [START_REF] Knowles | Neural interactive translation prediction[END_REF]; [START_REF] Santy | INMT: Interactive neural machine translation prediction[END_REF] explore ways to generate translations based on given prefix hints. A more general setting, enabling arbitrary insertions thanks to LCD, is studied for online IMT systems in [START_REF] Huang | Transmart: A practical interactive machine translation system[END_REF]. Note that LCD was initially developed for other purposes, namely enforcing lexical or terminological constraints [START_REF] Hokamp | Lexically constrained decoding for sequence generation using grid beam search[END_REF][START_REF] Post | Fast lexically constrained decoding with dynamic beam allocation for neural machine translation[END_REF][START_REF] Hu | Improved lexically constrained decoding for translation and monolingual rewriting[END_REF]. As this approach induces large decoding overheads, recent works in this thread explore NAT techniques: [START_REF] Susanto | Lexically constrained neural machine translation with Levenshtein transformer[END_REF] propose to inject lexical constraints into an edit-based LevT [START_REF] Gu | Levenshtein transformer[END_REF], an approach improved by [START_REF] Xu | EDITOR: An Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints[END_REF] with an additional repositioning operator.

Recent attempts to revise initial translations are explored by [START_REF] Marie | Touchbased pre-post-editing of machine translation output[END_REF], who propose a touch-based scenario where users select usable translations segments, while the more questionable ones are automatically retranslated iteratively. This idea is revisited by [START_REF] Grangier | QuickEdit: Editing text & translations by crossing words out[END_REF], where undesired words in the initial translation are crossed-out. The authors use a dual source encoder to represent the initial translation along with the source sentence, which is also explored by [START_REF] Wang | Touch editing: A flexible one-time interaction approach for translation[END_REF] in a touch-editing case. The text infilling task is also considered by [START_REF] Xiao | BiTIIMT: A bilingual text-infilling method for interactive machine translation[END_REF], based on a single source encoder; see also [START_REF] Yang | Wets: A benchmark for translation suggestion[END_REF] and [START_REF] Lee | IntelliCAT: Intelligent machine translation post-editing with quality estimation and translation suggestion[END_REF] for related proposals. These studies consider a slightly different task than ours, as they only predict the missing part of the initial translation. Nevertheless, they can all be adapted to our generic Bi-sync scenario. Similar approaches have also been studied in APE. Multi-source architectures have been explored in e.g. [START_REF] Junczys | MS-UEdin submission to the WMT2018 APE shared task: Dual-source transformer for automatic post-editing[END_REF][START_REF] Tebbifakhr | Multi-source transformer with combined losses for automatic post editing[END_REF][START_REF] Shin | Multiencoder transformer network for automatic postediting[END_REF][START_REF] Pal | A transformer-based multisource automatic post-editing system[END_REF], whereas [START_REF] Hokamp | Ensembling factored neural machine translation models for automatic post-editing and quality estimation[END_REF][START_REF] António | Unbabel's submission to the WMT2019 APE shared task: BERT-based encoder-decoder for automatic post-editing[END_REF] jointly encode source and translation as one input. [START_REF] Wisniewski | Why predicting post-edition is so hard? failure analysis of LIMSI submission to the APE shared task[END_REF][START_REF] Libovický | CUNI system for WMT16 automatic post-editing and multimodal translation tasks[END_REF][START_REF] Bérard | LIG-CRIStAL submission for the WMT 2017 automatic post-editing task[END_REF] focus on learning edit operations. [START_REF] Junczys | Log-linear combinations of monolingual and bilingual neural machine translation models for automatic post-editing[END_REF] also propose to generate APE training data with round-trip translation.

Bi-sync also encompasses TM-based methods. [START_REF] Gu | Search engine guided neural machine translation[END_REF] use a second encoder to represent TM matches, an idea extended with a more compact representation of TM matches in [START_REF] Xia | Graph based translation memory for neural machine translation[END_REF]. As explained above, [START_REF] Bulte | Neural fuzzy repair: Integrating fuzzy matches into neural machine translation[END_REF] use a single encoder, concatenating TM seg-ments with the source. [START_REF] Xu | Boosting neural machine translation with similar translations[END_REF] further add a second embedding feature indicating related segments in TM matches, and [START_REF] Pham | Priming neural machine translation[END_REF] propose to simultaneously consider the source and target sides of retrieved TMs. Retrieval-based MT is also explored in [START_REF] He | Fast and accurate neural machine translation with translation memory[END_REF]; [START_REF] Khandelwal | Nearest neighbor machine translation[END_REF]; [START_REF] Cai | Neural machine translation with monolingual translation memory[END_REF], trying to make the performance gain less dependent on the quality of the retrieved TM matches, or to enforce a tighter coupling between TM matches and translations.

Conclusion

This work introduced Bi-sync, the task of generating translations of a source sentence by editing a related target sentence. We have proposed various ways to create artificial initial translations for different editing types, that are needed for training. We have explored both autoregressive and non-autoregressive architectures, observing experimentally that our autoregressive Edit-MT model trained with artificial triplets performs bilingual resynchronization tasks in several real world scenarios. Edit-MT can also be quickly adapted to retrieval-based MT tasks, where we compared favorably to dedicated models. Finally, Edit-MT can also fix TMs by detecting parallel sentences and correct imperfect translations without adaptation. Another application that we wish to explore is APE.

Our NAT approach Edit-LevT is lagging behind Edit-MT. In the future, we would like to explore more NAT systems, which are computationally faster, and improve their performance. We intend to consider training curriculums and to modify the LevT model to better fit the Bi-sync task. We would also like to study ways to reduce the load of fully re-decoding the input sequence, especially when small changes, that need to be reproduced in the target, are iteratively applied to the source sentence.

Limitations

The generation of editing data for each type of edit operations requires lots of efforts and resources. This requires two separately trained NMT models to generate the data for substitution via roundtrip translation with LCD. The computational cost of LCD is very high compared to regular beam search, therefore consuming many computational resources. The generation of editing data for deletion also requires one separately trained model for each method with a complete decoding of the entire training corpora. Even though our data generation procedure is effective, the generation process may not be environmentally friendly. Due to computational limits, we were not able to conduct experiments on other languages pairs and tasks such as APE, in which large public datasets are available for other language pairs other than En-Fr.

As we decomposed the edits from one initial translation to the reference by basic edits (insertion, substitution and deletion), the generic Edit-MT and Edit-LevT models can mostly perform one type of edits at a time. It might be worth studying to combine several edit types into one single generated example, in order to approach more realistic scenarios for the generic models.

We mainly measured our results with BLEU and some additional scores in TER. However, other metrics like COMET [START_REF] Rei | COMET: A neural framework for MT evaluation[END_REF] can also be interesting: as pointed out by [START_REF] Helcl | Non-autoregressive machine translation: It's not as fast as it seems[END_REF], BLEU may be less appropriate to measure valid translations than COMET for NAT models.

A Edit-based Model Configurations

We conduct experiments using fairseq16 [START_REF] Ott | fairseq: A fast, extensible toolkit for sequence modeling[END_REF]. Edit-MT relies on the Transformerbase model of [START_REF] Vaswani | Attention is all you need[END_REF]. We use a hidden size of 512 and a feedforward size of 2,048. We optimize with Adam with a maximum learning rate of 0.0007, an inverse square root decay schedule, and 4,000 warmup steps. We also tie all input and output embedding matrices [START_REF] Press | Using the output embedding to improve language models[END_REF][START_REF] Inan | Tying word vectors and word classifiers: A loss framework for language modeling[END_REF]. Edit-MT is trained with mixed precision and a batch size of 8,192 tokens on 4 V100 GPUs for 300k iterations. We save checkpoints for every 3,000 iterations and average the last 10 saved checkpoints for inference. For Edit-LevT, we follow [START_REF] Gu | Levenshtein transformer[END_REF], using a maximum learning rate of 0.0005 with 10,000 warmup steps and a larger batch size of 16,384. For inference, we set a maximum decoding round of 10. Our experiments use the same multi-domain corpus as [START_REF] Xu | Boosting neural machine translation with similar translations[END_REF]. This corpus contains 11 different domains for the En-Fr direction, collected from OPUS17 [START_REF] Tiedemann | Parallel data, tools and interfaces in OPUS[END_REF]: documents from the European Central Bank (ECB); from the European Medicines Agency (EMEA); Proceedings of the European Parliament (Epps); legislative texts of the European Union (JRC); News Commentaries (News); TED talk subtitles (TED); parallel sentences extracted from Wikipedia (Wiki); localization files (GNOME, KDE and Ubuntu) and manuals (PHP). All these data were deduplicated prior to training. To evaluate the ability of our models to actually make use of TMs instead of memorizing training examples, we also test on two unseen domains: OpenOffice from OPUS and the PANACEA environment corpus18 (ENV). We follow [START_REF] Xu | Boosting neural machine translation with similar translations[END_REF] and search for the top 3 similar translations based on Fuzzy Match with a similarity score greater than 0.6 as computed by Equation (1) on the source side and without an exact match. Note that the ratio of sentences with at least one similar translation greatly varies across domains, as shown in Table 9. When reproducing the related setting, [START_REF] Xu | Boosting neural machine translation with similar translations[END_REF] used a placeholder token to mark the positions where segments are deleted. As discussed in Section 2.1, our models do not include such information. Therefore, we do not use placeholders for the related data.

B Details for MT with TMs

C Additional Results

Table 10 reports results on multi-domain test set for the task of translating with TMs measured on TER using SacreBLEU [START_REF] Post | A call for clarity in reporting BLEU scores[END_REF]. TER results show that even the generic Edit-MT model actually identifies useful edits, as we see improvements with respect to the copy baseline.

Table 11 shows results on BLEU broken down the aggregate test set by edit operations. We observe that the generic Edit-MT struggles to perform well when substitutions are needed. FT vastly improves the ability to substitute and delete from ẽ, separately or even in combination. Fine-tuned Edit-MT even outperforms FM when only deletions are required.

D Details for Parallel Corpus Cleaning

In the CLTE task, the goal is to identify multidirectional entailment relationships between two sentences x 1 and x 2 , written in different languages. Each (x 1 , x 2 ) pair in the dataset is annotated with one of the following relations: Bidirectional (x 1 ⇔x 2 ): the two fragments entail each

Table 1 :

 1 BLEU scores for all edit types (Ins, Sub, Del 1 , Del 2 ) broken down by the edit distance ∆ between ẽ and e for both En-Fr and Fr-En. Each column represents a range of distances. N denotes the number of sentences in each group. All is computed by concatenating all test sentences.

	En-Fr	0	1	2	3	4	5	6	7	8-10 >10	All
	N		458 2076 1133 1152 1117 1033 880 776 1852 1535 12012
	copy	100.0 88.9 84.7 81.5 77.8 75.2 70.3 68.0 65.3 52.9	75.9
	Edit-MT	96.4 92.6 89.3 87.3 85.8 83.3 82.3 80.4 79.4 71.7	84.1
		+ tag	95.5 92.6 90.4 88.3 86.9 84.4 83.0 81.3 80.1 72.1	84.7
	Edit-LevT 95.9 88.3 84.3 80.6 77.2 74.2 69.7 67.9 65.3 57.8	74.8
	Fr-En	0	1	2	3	4	5	6	7	8-10 >10	All
	N		341 2050 1439 1178 1125 1054 945 832 1643 1405 12012
	copy	100.0 90.1 85.8 80.6 76.4 71.8 68.7 67.0 63.8 51.7	75.6
	Edit-MT	97.6 92.8 91.1 87.8 83.8 82.5 80.1 80.2 77.3 69.2	83.6
		+ tag	98.3 93.2 91.6 88.6 84.8 83.3 80.8 80.5 77.9 69.9	84.2
	Edit-LevT 97.9 89.0 84.4 79.6 75.0 71.0 67.3 66.2 64.8 54.6	73.6
	Model	Ins Sub Del 1 Del 2 Comp Ext
	copy	54.0 71.5 71.0 78.7	68.4 66.7
	Edit-MT	75.9 77.0 86.9 94.7	73.1 67.9
	+ tag	76.9 78.5 88.6 94.7	74.0 72.7
	Edit-LevT 65.3 73.9 72.5 78.7	67.8 67.7

Table 2 :

 2

	Model	Ins Sub Del 1 Del 2 Comp Ext
	copy	51.8 70.9 71.0 78.7	63.4 61.4
	Edit-MT	73.6 74.6 87.5 95.8	65.8 69.3
	+ tag	74.6 76.2 89.1 96.2	67.0 71.6
	Edit-LevT 66.5 72.4 72.3 78.4	62.5 64.3

BLEU scores for Edit-MT and Edit-LevT on resynchronization tasks for En-Fr. Deletions are evaluated separately for two generation methods (Del 1 and Del 2 ). + tag refers to decoding with the oracle tag as a forced-prefix. Best performance is in bold.

Table 3 :

 3

BLEU scores for Edit-MT and Edit-LevT models on resynchronization tasks for Fr-En.

Table 4 :

 4 BLEU scores for the multi-domain test sets. All is computed by concatenating test sets from all domains, with 11k sentences in total. Copy refers to copying the similar translation in the output. +R implies using the related segments instead of a full initial sentence for inference. Best performance in each block are in bold.study the ability of our models to actually make use of TMs instead of memorizing training examples, we also test on two unseen domains: OpenOffice from OPUS and the PANACEA environment corpus 12 (ENV). Detailed description and statistics about these corpora are in Appendix B.Xu et al.

	Model	ECB EMEA Epps GNOME JRC KDE News PHP TED Ubuntu Wiki All
	copy	59.8	64.5	34.4	70.3	67.6 55.3 12.0 38.6 30.8	51.6	47.4 52.6
	FM	72.1	72.3	58.3	80.6	83.2 66.9 28.0 47.2 62.9	69.3	68.8 67.3
	FM #	69.3	68.1	58.2	74.2	80.1 65.2 28.6 44.3 62.6	68.1	69.0 65.0
	Edit-MT	59.3	62.5	34.7	69.8	68.0 50.6 12.1 38.0 31.2	52.3	45.6 51.8
	+ tag	60.3	63.0	35.7	70.3	68.4 51.9 12.9 38.8 32.6	52.1	45.6 52.6
	+ R + tag	56.0	53.9	45.9	64.9	68.5 50.0 17.7 39.7 44.9	59.9	52.8 53.3
	+ FT + tag	70.6	71.5	57.8	78.2	82.0 66.2 28.0 45.1 61.1	67.7	66.8 66.0
	+ FT + R + tag 66.4	63.6	57.3	71.3	77.6 60.5 28.0 42.1 60.9	66.7	65.0 62.3
	Edit-LevT	59.5	62.6	34.8	69.0	66.9 54.2 12.2 37.8 31.0	49.0	44.8 52.0
	+ R	49.6	49.1	43.6	57.8	61.0 42.9 16.5 34.3 41.0	49.2	50.6 47.6
	+ FT	60.5	63.2	35.2	68.3	66.5 54.5 12.1 39.0 32.9	50.7	45.9 52.4

Table 5 :

 5 BLEU scores for the multi-domain test sets broken down by the edit distance ∆ between ẽ and e. Each column represents a range of distances. N denotes the number of sentences in each group.

			1	2	3	4	5	6	7	8-10 >10	All
	N	540 2096 1107 882 827 782 689 607 1193 2277 11000
	copy	100.0 82.3 74.2 67.2 62.1 51.7 50.5 48.2 40.7 33.8	52.6
	FM	91.6 93.3 86.5 82.3 79.2 70.5 69.0 68.0 60.9 49.5	67.3
	Edit-MT+ tag	95.3 80.8 72.9 68.0 62.7 52.4 50.7 49.6 41.5 34.3	52.6
	+ FT + tag	91.6 91.1 85.8 80.9 77.7 68.8 68.4 66.6 59.0 48.0	66.0
	Edit-LevT	94.5 79.5 72.1 66.3 61.0 49.9 49.6 47.5 40.5 33.7	52.0
	+ FT	96.6 78.9 72.0 65.8 61.1 50.4 50.4 48.0 40.8 34.3	52.4
	Model	Office ENV
	copy	54.7	59.6
	FM	66.8	75.4
	FM #	64.0	70.6
	Edit-MT+ tag 56.2	60.3
	+ FT + tag	68.6	78.6
	Edit-LevT	53.0	60.0
	+ FT	51.9	59.6

Table 6 :

 6 BLEU scores on unseen domains.

Table 7 :

 7 Accuracy scores on the SemEval CLTE tasks. FT denotes Edit-MT fine-tuned for classification.

	Methods	2012 2013
	Best SemEval13	0.570 0.458
	Carpuat et al. (2017) 0.604 0.436
	Edit-MT En-Fr	0.350 0.284
	+ FT 0.716 0.466
	Edit-MT Fr-En	0.376 0.288
	+ FT 0.710 0.530

Table 9 :

 9 B.1 Datasets and Processing Details Data used for experiments of Section 4.2. FM ratio is the ratio of sentences with at least one matched similar translations, FM train is the actual number of examples augmented with a similar translation used for training, after setting aside 1,000 test sentences for each domain. Each training sentence is matched with up to 3 similar translations.

	Domain	Train FM ratio	FM train
	ECB	195,956	51.73%	234,943
	EMEA	373,235	65.68%	624,109
	Epps	2,009,489	10.12%	465,228
	GNOME	55,391	39.31%	42,697
	JRC	503,437	50.87%	587,859
	KDE	180,254	36.00%	136,456
	News	151,423	2.12%	4,048
	PHP	16,020	34.93%	10,350
	TED	159,248	11.90%	39,895
	Ubuntu	9,314	20.32%	1,738
	Wiki	803,704	19.87%	409,755
	Total	4,457,471	24.27% 2,557,078
	Office	49,845	43.76%	-
	ENV	13,632	6.81%	-

Early experiments showed that using sampling instead of beam search increases the diversity of the generated ẽsub .

https://github.com/google-research-datasets/ wiki-atomic-edits

https://www.statmt.org/wmt14

https://github.com/fyvo/ParallelCompression

https://fasttext.cc/

https://github.com/moses-smt/mosesdecoder

https://github.com/rsennrich/subword-nmt

https://github.com/pytorch/fairseq

BLEU gains for the insertion operation are artificially high. This is because the baseline is hindered by a high brevity penalty.

https://opus.nlpl.eu/

https://github.com/SYSTRAN/fuzzy-match

http://catalog.elda.org/en-us/repository/ browse/ELRA-W0057/

https://ict.fbk.eu/clte-benchmark/

https://opus.nlpl.eu/OpenSubtitles-v2018.php

https://github.com/pytorch/fairseq

https://opus.nlpl.eu/

http://catalog.elda.org/en-us/repository/ browse/ELRA-W0057/

Acknowledgement

We would like to thank the anonymous reviewers for their valuable suggestions and comments. This work was granted access to the HPC resources of IDRIS under the allocation 2022-[AD011011580R2] made by GENCI. The first author is partly funded by SYSTRAN and by a grant (Transwrite) from Région Ile-de-France.

The tagging mechanism of Edit-MT described in Section 2.2 can readily be used for this classification task with slight adjustments represented in Table 12.

We have not performed hyperparameter searching for FT, even though carefully fine-tuned models may achieve even better performance.