F Habert

Mochel

Cécile Delorme

MD Isaac Adanyeguh

PhD David Bendetowicz

MD Isabelle Le Ber

PhD Amélie Ponchel

PhD Aurélie Kas

PhD Marie-Odile Habert

PhD Fanny Mochel
email: fanny.mochel@upmc.fr

Multimodal neurometabolic investigation of the effects of Zolpidem on leukoencephalopathy-related apathy

Keywords: 2, Supplementary material: 1 video Apathy, Zolpidem, Leukoencephalopathy, FDG-PET, GABA FDG: 18 F Fluorodeoxyglucose, MRS: magnetic resonance spectroscopy, ALSP: axonal spheroids and pigmented glia, Glx: glutamine+glutamate, tCr: total creatine

Multimodal neurometabolic investigation of the effects of zolpidem on leukoencephalopathy-related apathy

Introduction

Apathy is defined as a quantitative reduction of goal-directed activity [START_REF] Levy | Apathy and the Functional Anatomy of the Prefrontal Cortex-Basal Ganglia Circuits[END_REF][START_REF] Marin | Apathy: a neuropsychiatric syndrome[END_REF][START_REF] Robert | Is it time to revise the diagnostic criteria for apathy in brain disorders? The 2018 international consensus group[END_REF]. Each dimension of apathy, namely cognitive, affective-emotional and auto-activation/self-initiating impairment seems to be related to distinct lesions of the prefrontal cortex and basal ganglia: respectively, dorsolateral, ventral (orbitofrontal/inferior) and medial prefrontal cortex regions as well as their corresponding projections to the basal ganglia [START_REF] Levy | Apathy and the Functional Anatomy of the Prefrontal Cortex-Basal Ganglia Circuits[END_REF]. Apathy should rather be seen as a "network-related" cerebral dysfunction, involving white matter or multifocal lesions, than a focal disorder [START_REF] Hahn | Apathy and White Matter Integrity in Alzheimer's Disease: A Whole Brain Analysis with Tract-Based Spatial Statistics[END_REF][START_REF] Yang | Deficiency of brain structural subnetwork underlying post-ischaemic stroke apathy[END_REF][START_REF] Heron | The anatomy of apathy: A neurocognitive framework for amotivated behaviour[END_REF][START_REF] Sheelakumari | Neuroanatomical correlates of apathy and disinhibition in behavioural variant frontotemporal dementia[END_REF].

Zolpidem, a GABAergic non-benzodiazepine hypnotic, was found to enhance arousal and neuropsychiatric functions in some patients with stroke [START_REF] Autret | Transient improvement of poststroke apathy with zolpidem: a single-case, placebo-controlled double-blind study[END_REF] or post-anoxic brain injuries, [START_REF] Cohen | Transient improvement of aphasia with zolpidem[END_REF][START_REF] Bomalaski | Zolpidem for the Treatment of Neurologic Disorders: A Systematic Review[END_REF].

Zolpidem can also alleviate catatonic syndromes [START_REF] Mastain | Effet favorable du zolpidem sur un état catatonique[END_REF][START_REF] Thomas | Test for catatonia with zolpidem[END_REF][START_REF] Zaw | Replication of zolpidem test for catatonia in an adolescent[END_REF][START_REF] Thomas | Catatonia in French Psychiatry: Implications of the Zolpidem Challenge Test[END_REF][START_REF] Javelot | Zolpidem test and catatonia[END_REF]. Zolpidem effects have never investigated in white matter disorders.

Here, we studied a patient with a genetic white matter disease characterized by severe apathy, who experienced marked improvement after Zolpidem. We undertook a multimodal approach using 18 F Fluorodeoxyglucose (FDG)-PET metabolism and a dedicated magnetic resonance spectroscopy (MRS) sequence for GABA and glutamine/glutamate metabolism, pre and post-Zolpidem, to understand what may mediate this effect.

Methods

Patient description and pharmacological intervention

The patient was 33 when she developed attentional difficulties and apathy. Her condition rapidly worsened with marked walking impairment, severe cognitive disturbances and spasticity. Brain MRI showed patchy asymmetric white matter hyperintensities on fluid-attenuated inversion recovery sequence, and lesions with increased diffusivity on diffusion weighted imaging sequence (data not shown). She was diagnosed with multiple sclerosis, but diagnosis was reconsidered due to the very atypical and rapid evolution. Axonal spheroids and pigmented glia (ALSP) was suspected and confirmed by CSF1R gene sequencing, which identified a heterozygous c.2342C>T (p.A781V) pathogenic variant [START_REF] Konno | Clinical and genetic characterization of adult-onset leukoencephalopathy with axonal spheroids and pigmented glia associated with CSF1R mutation[END_REF]. At age 38, she could only walk a few steps with help due to frontal disturbances. She had marked pyramidal syndrome but no muscle deficit. She had dementia with evidence of frontal dysfunction.

The patient was enrolled in a study on frontotemporal lobar degeneration approved by Inserm ethical committee (Inserm RBM 02-059). FDG-PET and MRS were performed before (after a wash-out of more than 12 hours) and one hour after taking orally 10 milligrams of Zolpidem. FDG-PET was used for the therapeutic evaluation of Zolpidem, with a low total radiation dose (4.6 mSv), and data use was approved by the French authority for the protection of privacy and personal data in clinical research (CNIL, approval No. 2111722). Written informed consent was obtained from the patient and her caregiver for clinical investigation including repeat FDG-PET and MRS, video and publication.

FDG-PET

A 10 minutes acquisition was performed with a hybrid PET/CT system (Biograph mCT Flow -Siemens Healthcare) 30 min after the intravenous injection of FDG (2 MBq/Kg). PET volumes were co-registered, spatially normalized in the Montreal Neurological Institute (MNI) space using SPM12 (https://www.fil.ion.ucl.ac.uk/spm/software/spm12/), smoothed and normalized in intensity. Twosample T-tests were used to compare the patient's metabolism to normal controls, with a general liner model with age and gender as covariates. The primary cluster-defining threshold was p=0.001, with the cluster-level extent threshold set at p = 0.05 for false discovery rate correction.

A z-score mapping implemented in BrainVisa software (http://brainvisa.info) was used to extract areas with differences in metabolism before and after Zolpidem. Relative difference was computed into zscores as described [START_REF] Huberfeld | Ictal brain hyperperfusion contralateral to seizure onset: the SPECT mirror image[END_REF][START_REF] Navarro | Motor cortex and hippocampus are the two main cortical targets in LGI1-antibody encephalitis[END_REF]. Clusters < 100 voxels (8ml) and voxels with absolute values < 1.5 z-score were excluded before displaying z-score maps onto anatomical MRI.

Magnetic Resonance Spectroscopy

MRS was performed on a 3-Tesla Siemens Prisma fit scanner (Siemens Medical Solutions, Erlangen, Germany). Single volume MEGA-Point-Resolved Spectroscopy (MEGA-PRESS, TR = 2000 ms, TE = 68 ms, averages = 48) was used to acquire data in a 17 x 35 x 15 mm 3 and 15 x 20 x 20 mm 3 volumes of interest in the medial frontal cortex and pallidum respectively. Consecutive application of spectral editing pulses at 1.9 ppm (MEGA-ON) and 7.5 ppm (MEGA-OFF) was performed for enhanced detection of low-concentrated metabolites including GABA and Glx (glutamine + glutamate). The difference of the ON and OFF spectra revealed the GABA and Glx signals that usually overlap with higher concentrated metabolites. Quantification of the spectra was done using LCModel [START_REF] Provencher | Estimation of metabolite concentrations from localized in vivo proton NMR spectra[END_REF]. The ratio of GABA/total creatine (tCr) and Glx/tCr were reported.

Results

Clinical evaluation

Behavioral examination before Zolpidem revealed major apathy involving all three dimensions, with executive dysfunction, affective indifference, emotional flattening, anhedonia, and lack of initiative to undertake spontaneous actions or elaborate about future projects (Supplementary Video, Part 1). One hour after taking 10 mg of Zolpidem, the patient was more alert, responsive and able to communicate more fluently (Supplementary Video, Part 2). There was a marked increase in the emotional and social content of her conversation. She did elaborate on some future projects such as doing an around the world travel with her family. Neuropsychological scores otherwise remained unchanged and a severe executive deficit persisted. Patient's participation to clinical examination clearly improved, including her ability to walk. There was also a marked increase in spontaneous motor expression, with increased and more rapid facial and bodily movements. The effect of Zolpidem was reproductive over days and lasted for about four hours.

FDG-PET

Pre-Zolpidem cerebral FDG-PET showed hypometabolism in the bilateral cerebellum, the pons, orbitofrontal cortex, thalamus, cingulate, and dorsolateral prefrontal cortex compared to a cohort of healthy controls (Figure 1A).

Post-Zolpidem cerebral FDG-PET showed increased metabolism in the orbitofrontal cortex, the median prefrontal cortex, the caudate nucleus and the thalamus compared to pre-Zolpidem acquisitions (Figure 2). Post-Zolpidem metabolism in the orbitofrontal cortex was not significantly decreased compared to healthy controls (Figure 1B).

Magnetic resonance spectroscopy

Post-Zolpidem, we observed increased ratios of GABA/tCr and Glx/tCr in the medial frontal cortex and pallidum compared to pre-Zolpidem (Figure 2B).

Discussion

We report the marked clinical response to Zolpidem of a patient with severe apathy in the context of a white-matter disease. Our multimodal neurometabolic study revealed that Zolpidem increased metabolism in the orbitofrontal cortex and basal ganglia, and increased GABA and Glx levels in the medial prefrontal cortex and pallidum.

ALSP is one of the most common leukodystrophies and is a model of microglia dysfunction [START_REF] Konno | CSF1R -related leukoencephalopathy: A major player in primary microgliopathies[END_REF].

Neuropsychiatric manifestations of ALSP include cognitive decline, apathy, and other behavioral changes resembling behavioral-variant frontotemporal dementia [START_REF] Konno | Clinical and genetic characterization of adult-onset leukoencephalopathy with axonal spheroids and pigmented glia associated with CSF1R mutation[END_REF]. ALSP predominantly affects the white matter, in particular the frontal subcortical white matter and corpus callosum [START_REF] Konno | CSF1R -related leukoencephalopathy: A major player in primary microgliopathies[END_REF][START_REF] Freeman | Adult Onset Leukodystrophy with Neuroaxonal Spheroids: Clinical, Neuroimaging and Neuropathologic Observations[END_REF]. FDG-PET in ALSP patients showed diffuse hypometabolism predominantly affecting the frontal and parietal lobes [START_REF] Freeman | Adult Onset Leukodystrophy with Neuroaxonal Spheroids: Clinical, Neuroimaging and Neuropathologic Observations[END_REF][START_REF] Ueda | A novel A792D mutation in the CSF1R gene causes hereditary diffuse leukoencephalopathy with axonal spheroids characterized by slow progression[END_REF].

Pre-Zolpidem FDG-PET in our patient displayed significant hypometabolism in brain regions involved in motivation and goal-directed behaviors, namely the orbitofrontal cortex, medial prefrontal cortex and basal ganglia. Post-Zolpidem FDG-PET showed increased metabolism in these regions.

These results are consistent with the three-dimensional model of apathy (1) as, post-Zolpidem, our patient showed emotional-affective and auto-activation improvement contrasting with persisting severe executive disturbance. This response was paralleled by increased FDG-PET metabolism in the orbitofrontal cortex, median prefrontal cortex and basal ganglia but no change in the dorsolateral frontal regions. It has been hypothesized that, in some leukodystrophies, frontal neuropsychiatric symptoms might reflect a "disconnection syndrome" secondary to white matter alterations rather than a cortical dysfunction [START_REF] Hyde | Psychiatric Disturbances in Metachromatic Leukodystrophy: Insights Into the Neurobiology of Psychosis[END_REF][START_REF] Baumann | Motor and psycho-cognitive clinical types in adult metachromatic leukodystrophy: genotype/phenotype relationships?[END_REF]. Apathy is frequent in other white matter diseases, such as small vessel diseases [START_REF] Reyes | Apathy: A major symptom in CADASIL[END_REF] and multiple sclerosis [START_REF] Rosti-Otajärvi | Behavioural symptoms and impairments in multiple sclerosis: a systematic review and meta-analysis[END_REF].

Zolpidem increased GABA and Glx in our two MRS regions of interest, the medial frontal cortex and the pallidum. In healthy controls, decreased GABA/Cr ratio in response to Zolpidem has been reported previously in the thalamus but with no effect on Glx [START_REF] Licata | A therapeutic dose of zolpidem reduces thalamic GABA in healthy volunteers: a proton MRS study at 4 T[END_REF]. The binding of Zolpidem in healthy controls, in which the glutamatergic/GABAergic balance is not impaired, leads to cortical inhibition.

Conversely, increased GABA/Cr ratio has been reported in the thalamus and anterior cingulate of patients with major depressive disorder after Zolpidem administration [START_REF] Licata | Zolpidem increases GABA in depressed volunteers maintained on SSRIs[END_REF]. Hence, the effect of Zolpidem seems to be influenced by changes in receptor configurations and how the glutamatergic / GABAergic balance is affected by neuropsychiatric disorders.

Zolpidem can improve cognitive function in some patients with post-hypoxic brain injuries, which also affect the white matter [START_REF] Whyte | Incidence of Clinically Significant Responses to Zolpidem Among Patients with Disorders of Consciousness: A Preliminary Placebo Controlled Trial[END_REF]. One common hypothesis is that the cortical disconnection related to white matter alterations induces allosteric changes in GABA receptors, which may become hypersensitive in order to keep their function even with low GABAergic tone. In line with this hypothesis, a 18 F Flumazenil PET study showed increased GABA receptors activity in the frontoparietal regions in a patient with hypoxic brain injury [START_REF] Kim | Zolpidem-Induced Arousal by Paradoxical GABAergic Stimulation: A Case Report With F-18 Flumazenil Positron Emission Tomography and Single Photon Emission Computed Tomography Study[END_REF]. Zolpidem may enhance cortical function by reversing this functional diaschisis, as suggested by our MRS findings. Zolpidem is specific for GABA1-receptors containing the α1 subunit, which are predominantly expressed in the thalamus and the striatum [START_REF] Churchill | GABAA receptors containing alpha1 and alpha2 subunits are mainly localized on neurons in the ventral pallidum[END_REF][START_REF] Chen | Electrophysiological and behavioral effects of zolpidem in rat globus pallidus[END_REF][START_REF] Mortensen | GABA Potency at GABAA Receptors Found in Synaptic and Extrasynaptic Zones[END_REF][START_REF] Wisden | The Distribution of 13 GABA, Receptor Subunit mRNAs in the Rat Brain. I. Telencephalon[END_REF]. Zolpidem binding may inhibit the internal pallidum, which would in turn disinhibit thalamus and activate thalamo-cortical projections and result in a "deep brain stimulation-like" effect [START_REF] Schiff | Mesocircuit Mechanisms Underlying Recovery of Consciousness Following Severe Brain Injuries: Model and Predictions[END_REF][START_REF] Badillo | Zolpidem for the Treatment of Dystonia[END_REF].

This study has several limitations. It was not placebo-controlled and was performed on a single subject. The severe frontal cognitive impairment made it difficult to provide a more detailed clinical examination, for example social cognition evaluations that require a preserved working memory and attention. Although we cannot draw definite conclusions about the mode of action of Zolpidem on apathy, our multimodal approach suggests that its effects are related to increased metabolism in the orbitofrontal cortex and basal ganglia secondary to GABA modulation.

  Cognitive examination was limited due to severe distractibility and slowing of thoughts. She scored 17/30 on the Mimi Mental State Examination, 2/18 on the Frontal Assessment Battery and 10/30 on the Montreal Cognitive Assessment.

Figure 1 -

 1 Figure 1 -Pre-and post-Zolpidem FDG-PET In blue are shown regions with significant hypometabolism in the patient compared to a cohort of healthy controls (p<0.05 after cluster correction). A-Pre-Zolpidem. B-Post-Zolpidem.

Figure 2 -

 2 Figure 2 -Post-Zolpidem neurometabolic response A. FDG-PET. In red are shown regions with significant post-Zolpidem increased metabolism compared to the pre-Zolpidem acquisition. B. MR-Spectroscopy. GABA/Cr and Glx/Cr ratios in the medial frontal and pallidum before and after 10 mg of Zolpidem. Metabolite concentrations are in µmol/g. Cr: total creatine; Glx: glutamine+ glutamate.


Anh Khoa

Ngo Ho

Franc ¸ois Yvon
email: francois.yvon@limsi.fr

Generative latent neural models for automatic word alignment

Word alignments identify translational correspondences between words in a parallel sentence pair and are used, for instance, to learn bilingual dictionaries, to train statistical machine translation systems or to perform quality estimation. Variational autoencoders have been recently used in various of natural language processing to learn in an unsupervised way latent representations that are useful for language generation tasks. In this paper, we study these models for the task of word alignment and propose and assess several evolutions of a vanilla variational autoencoders. We demonstrate that these techniques can yield competitive results as compared to Giza++ and to a strong neural network alignment system for two language pairs. 1 1 As is custom, target sentences are completed with a "null" symbol, conventionally at index 0. 2 We omit the initial step, consisting in sampling the lengths I and J and the dependencies wrt. these variables.

Introduction

Word alignment is one of the basic tasks in multilingual Natural Language Processing (NLP) and is used to learn bilingual dictionaries, to train statistical machine translation (SMT) systems [START_REF] Koehn | Statistical Machine Translation[END_REF], to filter out noise from translation memories [START_REF] Pham | Fixing translation divergences in parallel corpora for neural MT[END_REF] or in quality estimation applications [START_REF] Specia | Quality estimation for Machine Translation[END_REF]. Word alignments can also be viewed as a form of possible explanation of the often opaque behavior of a Neural Machine Translation [START_REF] Stahlberg | An operation sequence model for explainable neural machine translation[END_REF]. Word alignment aims to identify translational equivalences at the level of individual lexical units [START_REF] Och | A systematic comparison of various statistical alignment models[END_REF][START_REF] Tiedemann | Bitext Alignment. Synthesis Lectures on Human Language Technologies[END_REF] in parallel sentences.

Successful alignment models either rely on bilingual association measures parameterizing a combinatorial problem (eg. an optimal matching in a bipartite graph); or on probabilistic models, as represented by the IBM Models of [START_REF] Brown | The mathematics of statistical machine translation: Parameter estimation[END_REF] and the HMM model of [START_REF] Vogel | HMM-based word alignment in statistical translation[END_REF]. All these models use unsupervised learning to estimate the likelihood of alignment links at the word level from large collections of parallel sentences.

Such approaches are typically challenged by low-frequency words, whose co-occurrences are poorly estimated; they also fail to take into account context information in alignment; finally, they make assumptions that are overly simplistic (eg. that all alignments are one-to-many or many-to-one), especially when the languages under focus belong to different linguistic families. Even though their overall performance seem fair for related languages (eg. French-English), there is still much room for improving. Indeed, the error rate of automatic alignments tools such as Giza++ [START_REF] Och | A systematic comparison of various statistical alignment models[END_REF] or Fastalign [START_REF] Dyer | A simple, fast, and effective reparameterization of IBM model 2[END_REF], even for high resource languages, is still well above 15-20%; and the situation is much worse in low-resource settings [START_REF] Martin | Word alignment for languages with scarce resources[END_REF][START_REF] Xiang | Diversify and combine: Improving word alignment for machine translation on low-resource languages[END_REF][START_REF] Mccoy | Phonologically informed edit distance algorithms for word alignment with low-resource languages[END_REF].

As for most NLP applications [START_REF] Collobert | Natural language processing (almost) from scratch[END_REF], and notably for machine translation [START_REF] Cho | On the properties of neural machine translation: Encoder-decoder approaches[END_REF][START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF], neural-based approaches offer new opportunities to reconsider some of these issues. Following up on the work of eg. [START_REF] Yang | Word alignment modeling with context dependent deep neural network[END_REF][START_REF] Alkhouli | Alignment-based neural machine translation[END_REF][START_REF] Wang | Neural hidden Markov model for machine translation[END_REF], we study ways to take advantage of the flexibility of neural networks to design effective variants of generative word alignment models.

Our main source of inspiration is the model of [START_REF] Rios | Deep generative model for joint alignment and word representation[END_REF], who consider variational autoencoders [START_REF] Kingma | Auto-encoding variational Bayes[END_REF][START_REF] Rezende | Stochastic backpropagation and variational inference in deep latent gaussian models[END_REF] to approach the unsupervised estimation of neural alignment models. We revisit here this model, trying to analyze the reasons for its unsatisfactory performance and we extend it in several ways, taking advantage of its fully generative nature. We first generalize the approach, initially devised for IBM model 1, to the HMM model; we then explore ways to effectively enforce symmetry constraints [START_REF] Liang | Alignment by agreement[END_REF]; we finally study how these models could benefit from monolingual data. Our experiments with the English-Romanian and English-French language pairs show that our best model with symmetry constraints is on par with a conventional neural HMM model; they also highlight the remaining deficiencies of these approaches and suggest directions for further developments.

Neural word alignment variational models

The standard approach to probabilistic alignment [START_REF] Och | A systematic comparison of various statistical alignment models[END_REF] is to consider asymmetric models associating each word in a source sentence f J 1 = f 1 . . . f J of J words with exactly one word from the target sentence e I 0 = e 0 . . . e I of I + 1 words. 1 This association is governed by unobserved alignment variables a J 1 = a 1 ...a J , yielding the following model:

p(f J 1 , a J 1 |e I 0 ) = J j p(a j |a j-1 1 , f j-1 1 , e I 0 )p(f j |a j 1 , f j-1 1 , e I 0 ) (1)
Two versions of this model are considered here: in the IBM model 1 [START_REF] Brown | The mathematics of statistical machine translation: Parameter estimation[END_REF], the alignment model p(a j |a j-1 1 , f j-1

1 , e I 0 ) is uniform; in the HMM model of [START_REF] Vogel | HMM-based word alignment in statistical translation[END_REF], Markovian dependencies between alignment variables are assumed and a j is independant from all the preceding alignment variables given a j-1 . In both models, f j is conditionally independent to any other variable given a j and e I 1 . Under these assumptions, both parameter estimation and optimal alignment can be performed efficiently with dynamic programming algorithms. In this approach, e I 1 is not modeled.

A fully generative model

We now present the fully generative approach introduced by [START_REF] Rios | Deep generative model for joint alignment and word representation[END_REF]. In this model, the association between a source word f j and a target word e i is mediated by a shared latent variable y i , assumed to represent the joint underlying semantics of mutual translations. In this model, the target sequence e I 1 is also modeled, yielding the following generative story: 2 1. Generate a sequence y I 0 of d-dimensional random embeddings by sampling independently from some prior distribution e.g. Gaussian

Generate e I

1 conditioned on the latent variable sequence y I 1 3. Generate a J 1 = a 1 ...a J denoting the alignment from f J 1 to y I 0 4. Generate f J 1 conditioned on y I 0 and a J This yields the following decomposition of the joint distribution of f J 1 and e I 1 , where we marginalize over latent variables y I 0 and a J 1 :

p(f J 1 , e I 1 ) = y I 0 p(y I 0 )p θ (e I 1 |y I 1 ) a J 1 p θ (a J 1 )p θ (f J 1 |y I 0 , a J 1 ) dy I 0 (2)
Directly maximizing the log-likelihood to estimate the parameters is in general intractable, especially when neural networks are used to model the generation of f J 1 and e I 1 . The standard approach in neural generative models [START_REF] Kingma | Auto-encoding variational Bayes[END_REF] is to introduce a variational distribution q φ for the latent variables and to optimize the so-called evidence lower-bound (ELBO). Following [START_REF] Rios | Deep generative model for joint alignment and word representation[END_REF] we consider tractable alignment models and use the variational distribution only for modeling y I 0 conditioned on e I 1 . This yields the following objective:

J(θ, φ) = -E q φ (y I 1 ) (log p θ (e I 1 |y I 1 )) -E q φ (y I 0 ) ([log a J 1 p θ (a J 1 )p θ (f J 1 |y I 0 , a J 1 )) + KL[q φ (y I 0 |e I 1 )||p(y I 0 )] (3)
where E p (f ) denotes the expectation of f with respect to p, and KL is the Kullback-Leibler divergence. Objective (3) is a sum of three terms that are referred respectively as the reconstruction cost, the alignment cost and KL divergence cost. The last term can be computed analytically when the prior and the variational distributions are Gaussian and we thus assume the following parameterization q φ (y I 1 |e I 1 ) = i N (y i |u i , s i ), where the mean u i and the diagonal covariance matrix diag(s i ) are deterministic functions of e I 1 . As is custom, the expectations in equation ( 3) are approximated by sampling values of y i as y i = u i + s i • i , where i is drawn from a white Gaussian noise. The reparameterization trick removes the sampling step from the generation path, and makes the whole objective differentiable [START_REF] Kingma | Auto-encoding variational Bayes[END_REF].

Introducing Markovian dependencies

The experiments in [START_REF] Rios | Deep generative model for joint alignment and word representation[END_REF] only consider basic assumptions regarding the alignment model p θ (a J 1 ), corresponding to IBM model 1. Our first variation of this model considers a richer transition model assuming Markovian dependencies, for which the exact marginalization of alignment variables implied by equation (3) remains tractable with the forward algorithm. The alignment cost is the expectation of the source given the latent variables:

E q φ (y J 0 ) ([log a J 1 J j=1 p θ (f j |y aj )p θ (a j |a j-1 )]) (4)
As is usual with HMM variants of alignment models, we parameterize the transition distribution p θ (a j |a j-1 ) on the distance (jump) between the values of a j and a j-1 [START_REF] Och | A systematic comparison of various statistical alignment models[END_REF]. This model is referred to below as HMM+VAE.

Towards symmetric models: a parameter sharing approach

A first benefit of having a fully generative model (in both alignment directions), which jointly models f J 1 and e I 1 , is that it becomes easy to encourage these models to share information and to improve their joint performance. Our alignment models involves two decoders, one for the source and one for the target (in each direction). These components are used to compute a distribution over vocabulary words given a d-dimensional variable, and are conceptually similar.

Our first step is thus to simultaneously train the alignment models in both directions, making sure that they use the same decoder respectively for f J 1 and e I 1 . This means that the same network computes p θ (e I 1 |y I 1 ) (when e I 1 is in the target) and p θ (e I 1 |y J 0 , a J 1 ) when e I 1 is the source.

There is only one encoder computing the variational parameters in each direction, and these remain distinct in this approach. Our joint objective function now comprises six terms including two reconstruction costs, two alignment costs and two KL divergence costs. From this, we see that a first benefit of this method is computational as it greatly reduces the number of parameters to train. We also expect that it will yield two additional benefits: (a) to help improve the alignment model, which is more difficult to train for lack of observing the "right" alignment variables; in comparison the reconstruction of the target sentence is almost obvious, as each e i is generated from the right y i ; (b) to make the alignments more symmetrical, thereby facilitating their interpretation and their recombination. This model is denoted +VAE+SP below.

Enforcing agreement in alignment

The idea of training two asymmetrical models opens new ways to control the level of agreement between alignments, an idea already considered eg. in [START_REF] Liang | Alignment by agreement[END_REF][START_REF] Grac ¸a | Learning tractable word alignment models with complex constraints[END_REF].

Following the former approach, we implement this idea by adding an extra cost that rewards agreement between asymmetric alignments. For non null alignment links, this cost is based on the alignment posterior distributions and is defined as:

i>0,j>0 |p(a j = i|f J 1 , e I 1 ) -p(b i = j|f J 1 , e I 1 )|, (5)
where b J 1 is the alignment variables introduced when e I 1 is the source of the alignment, and f J 1 is the target. Both for the IBM-1 and for the HMM variants, these posterior distributions can be computed effectively, in the latter case using the forward-backward algorithm.

In the case of the null links, the agreement term should reward configurations where one source word is aligned with the null symbol in one direction, and is not aligned to any target word in the other direction. This yields the following additional term (for the canonical source to target direction, the reverse term is analogous):

J j=1 |1 -p(a j = 0|f J 1 , e I 1 ) - I i=1 p(b i = j|f J 1 , e I 1 )| (6)
For this model (+VAE+SP+AC), the objective function comprises nine terms, each with its own dynamics, which makes optimization more difficult due to the heterogeneity between costs.

Training with monolingual data

Leaving the alignment module aside, the model can be used as a simple autoencoder which can be (pre)trained monolingually. We use supplementary monolingual sentences ėM 1 that just go through the encoding-decoding process, and add an extra monolingual reconstruction term J mono in the objective (3):

J mono (θ, φ) = -E q φ ( ẏM 1 ) (log p θ ( ėM 1 | ẏM 1 )) + KL[q φ ( ẏM 1 | ėM 1 )||p( ẏM 1 )] (7)
where ẏM 1 is the latent variable associated to ėM 1 . Alternatively, we consider training the alignment model monolingually. We implement this idea by adding a random noise to the target sentence, to make it more similar to a source sentence and amenable to alignment. In this case, the extra reconstruction term is:

J mono (θ, φ) = -E q φ (ÿ N 0 ) ([log äM 1 p θ (ä M 1 )p θ ( ėM 1 |ÿ N 0 , äM 1 )) + KL[q φ (ÿ N 0 |ë N 1 )||p(ÿ N 0 )] (8)
where ëN 1 is a noisy version of ėM 1 , ÿN 1 is the latent variable for ëN 1 , and äM 1 denotes the alignment variables betwen ėM 1 and ÿN 0 . In our experiments, we only use IBM Model 1 as our alignment model.

Experiments

Datasets

Our experiments use two standard benchmarks from the 2003 word alignment challenge [START_REF] Mihalcea | An evaluation exercise for word alignment[END_REF], respectively for aligning English with French and Romanian. We consider two different settings: for French, we use a large training corpus of parallel sentences from the Europarl corpus [START_REF] Koehn | Europarl: A Parallel Corpus for Statistical Machine Translation[END_REF]. In the case of Romanian, we use the SETIMES corpus used in WMT'16 evaluation, 3 These corpora are preprocessed, lowercased and tokenized with standard tools from the Moses toolkit.5 Following notably [START_REF] Garg | Jointly learning to align and translate with transformer models[END_REF], we perform the alignment between subword units generated by Byte-Pair-Encoding [START_REF] Sennrich | Neural machine translation of rare words with subword units[END_REF], implemented with the SentencePiece model [START_REF] Kudo | SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing[END_REF] and computed independently6 in each language with 32K merge operations. This makes the training less computationally demanding and greatly mitigates the rare-word problem, which is a major weakness of historical count-based model. Our results and analyses are however based on word-level alignments. Subword-level alignments are converted into word-level alignments as follows: a link between a source and a target word exists if there is at least one link alignment between their subwords.

Implementation

Our models are close in structure to the model proposed by [START_REF] Rios | Deep generative model for joint alignment and word representation[END_REF], and are made of three main components: an encoder to generate the latent variables y I 0 from e I 1 , and two decoders to respectively reconstruct e I 1 and f J 1 , with the help of the alignment model. The encoder is composed of a token embedding layer (128 units), two LSTM layers (each comprising 64 units), and dense output layers to independently generate the mean vectors (u 1 . . . u I ) vectors and the diagonal of the covariance matrices (s 1 . . . s I ). The latent variable y I 1 has 64 units. 7 Our encoder is formally defined as:

- → h i = RN N ( --→ h i-1 , E(e i )
)

h i = W h concat( - → h i , ← - h i ) s i = softplus(W s h i + b s ) u i = W u h i + b u y i = u i + s i • i
where E(e i ) ∈ R 128 is the embedding of word e i , is a noise variable ∼ N (0, 1) and softplus = log(1 + exp(x)) is an activation function returning a value positive. The vector y 0 is independently generated from a pseudo-sentence made of one dummy token; it is identical for all target sentences. Note that the decoder model does not try to reconstruct this token. The reconstruction decoder is given by:

p θ (e i |y i ) = [softmax(W v y i + b v )] ei ,
and the alignment model with emission and transition components is:

p θ (f j |e aj ) = [softmax(W v y aj )] fj p θ (a j -a j-1 ) = [softmax(W ∆ y aj-1 )] aj -aj-1
where

W v ∈ R 64×V , b v ∈ R V , with V the target vocabulary size. W ∆ ∈ R 64×301 with jump values in the interval [-150, +150].
For experiments with monolingual data, our noise model follows the technique in [START_REF] Lample | Unsupervised machine translation using monolingual corpora only[END_REF]. We randomly delete input words with probability p wd = 0.1. We then slightly shuffle the sentence, where the difference between the position before and after shuffling each word is smaller than 4.

In all cases, our optimizer is Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] with an initial learning rate of 0.001; the batch size set to 100 sentences. We use all training sentences of length lower than 50. All parameters of the Giza++ and Fastalign baselines are set to their default values. IBM-1+NN and HMM+NN correspond to basic neuralizations of the IBM models as in [START_REF] Rios | Deep generative model for joint alignment and word representation[END_REF][START_REF] Ngo-Ho | Neural Baselines for Word Alignments[END_REF] for both word-level and BPE-level. These models are trained by maximizing the likelihood with the expectation-maximization algorithm. We train all models for 10 iterations. Results with symmetric alignments use the grow-diag-final (GDF) heuristic proposed in [START_REF] Koehn | Edinburgh system description for the 2005 IWSLT speech translation evaluation[END_REF].

Evaluation protocol

We use the alignment error rate (AER) [START_REF] Och | Minimum error rate training in statistical machine translation[END_REF], accuracy, F-score, precision and recall as measures of performance. AER is based on a comparison of predicted alignment links (A) with a human reference including sure (S) and possible (P) links, and is defined as an average of the recall and precision taking into account the sets P and S. AER is defined as:

AER = 1 - |A ∩ S| + |A ∩ P | |A| + |S|
where A is the set of predicted alignments. Note that the Romanian-English reference data only contains sure links; in this case AER and F-measure are deterministically related.

Results

The top part of Table 2 reports the AER score of the IBM-1 baselines: the count-based model (IBM-1 Giza++) and the two neural variants, operating at the word (IBM1+NN) and subword (IBM-1+BPE) levels. We also report the performance of three VAE variants (IBM1+VAE+BPE, IBM1+VAE+BPE+SP, IBM-1+VAE+BPE+SP+AC). A first observation is neural baselines are better than Giza++, and that using BPE units brings an additional gain.

The basic model (IBM-1+VAE) falls short to match these results and proves way worse than the two neural version of the IBM-1 model. These results are in line with the findings of [START_REF] Rios | Deep generative model for joint alignment and word representation[END_REF], who report similar difference in performance. Sharing the parameters between directions greatly improves this baseline with a reduction in AER of about 8 points (En-Fr) and 6 points (En-Ro) for both directions, as well as for symmetrization. The reconstruction model, which is well trained in one direction, helps to improve the emission model in the reverse direction. We observe that the gain is more significant when the morphologically rich language is on the target side: this is were the emission model is the weakest and benefits most from parameter sharing. Adding an extra agreement cost fails to produce markedly better alignments for Fr-En; we however observe a gain of about 2 AER points for the symmetricized alignments in En-Ro. Overall, our best VAE model outperform the neural baseline in the large training condition (English-French); we do not see this for the other language pair, where the performance remains much below the neural baseline.

Model

English The effect of adding a transition component in these models is less clear, as shown in Table 3, where we report the performance of HMM-based variants. Both symmetrization strategies prove again very effective to improve the basic VAE model, and our best system (+AC) achieves AER scores that are close, yet slightly inferior, to the HMM+NN+BPE baseline. One possible issue that we do not fully solve via symmetrization is related to the null word, which, as explained above, is not part of the reconstruction model, and which does not improve with joint learning.

Model

English 4 Error Analysis

Balancing the terms in the VAE objective

One well-known issue of VAEs for text applications is posterior collapse [START_REF] Bowman | Generating sentences from a continuous space[END_REF][START_REF] Higgins | Beta-VAE: Learning basic visual concepts with a constrained variational framework[END_REF], where the variational distribution collapses towards the prior distribution. This is because the KL term can get arbitrarily small, with a moderate effect on the reconstruction cost, assuming a strong reconstruction model (a recurrent network in typical applications). We also encountered this problem in our setting, but the interpretation is a bit different: when the KL term goes to zero, all words in the dictionary become indistinguishable and the reconstruction costs reaches its maximum, corresponding to the entropy of the uniform distribution of the target vocabulary. The difference in dynamics between these scores is observed in Figure 1 (left), where we apply weights equal to α, β and γ respectively to the reconstruction cost, the alignment cost and the KL divergence term. This effect is mitigated if we proportionally decrease the weight of the KL term (middle). This second graph reveals the need to also better balance the importance of the other two terms. Using larger weights for the reconstruction term (α = 10) and even more for the alignment term (β = 50), we keep the KL divergence high and make sure that the optimization focuses on decreasing the two other terms8 .

Unaligned words

In asymmetrical models, the number of links that are generated is constant and equal to the total number of "source" words. A source word is deemed unaligned when it is linked to the special NULL token on the target side; a target word is unaligned when it emits no source word. We perform an in-depth analysis of these special links. Results for the alignment from French into English are in Table 4; we observe similar trends for the other direction and for the other language pair. We compute the alignment accuracy as the proportion of words (on both sides) for which the binary decision (aligned or non-aligned) is correct; we also report the precision and recall for unaligned words. Results in Table 4 show that the number of unaligned words varies in great proportion, with a minimum of about 3000 words (HMM+NN) and a maximum of nearly 6600 (IBM1+VAE+BPE and HMM+VAE+BPE). For this language pair, the reference contains 821 unaligned words. They also demonstrate the inability of all models to correctly predict null links, the best model achieving a precision of only 13.1%. Predicting so many unaligned words is extremely detrimental to the performance of the two basic VAE models for which we observe a very poor recall for non-null links, which is hardly compensated by the good precision scores. We see here clearly the effect of the symmetrization constraints (especially for the HMM model) where the reward associated with symmetric predictions reduces the tendency to align French words with the NULL English, and to leave too many English words unaligned. Even there (HMM+VAE+BPE+SP+AC), the number of predicted non-null links is about half as what we see for HMM+NN: as it predicts much more links than the others, this model also as an clear edge when it comes to post-hoc symmetrization, since the "grow-diag-final" heuristics heavily depends on the size of the intersection. Note that this problem has a much stronger overall effect in English-Romanian than in English-French. This is because the English-Romanian test set only contains sure links, which means that a low recall for aligned words directly impacts the AER. We do not see this for the French-English data, which contain many possible links that have no impact on recall [START_REF] Fraser | Measuring word alignment quality for statistical machine translation[END_REF].

Model

Incidentally, we also observe a null-word problem for HMM+NN+BPE; presumably splitting words in small units that are unrelated across languages can also make the model prefer the null alignment over links between actual words. These results clearly point out one deficiency of the current approach: for lack of having a proper model for the latent representation of the NULL token, the VAE-based approach tends to leave too many words unaligned.

Symmetrization and agreement

We now study the effects of sharing parameters across alignment directions. We consider the English-Romanian test, for which the relationship between precision, recall and AER is straightforward. Detailed scores for all IBM-1 models are in Table 5. We see the clear benefits of sharing parameters, which contributes a jump of both precision, recall and F-measure compared with the baseline VAE. Models SP and SP+AC generate more alignment links (about +500 links) than the baseline model. This enhancement helps to outperform Giza++ but is insufficient to surpass the conventional neural network models, especially when using BPE. Numbers in Table 5 show that the gain in recall is largest in the direction En-Ro: this is because the better reconstruction of English words boosts the translation model. We now measure more directly the level of agreement between the two alignment directions for English-French (Table 6). We note that the model integrating agreement costs (+SP+AC) leads to a higher number of agreements in comparison to the other VAE-based models, and also yields the best scores in terms of intersection AER.

Model

Training with monolingual data

A last extension concerns the use of monolingual data in the low-resource condition. To compute the performance of the reconstruction model (R-ACC), we compute the proportion of words for which the model's prediction actually corresponds to the correct word. Experiments are performed with English-Romanian.9 Results in Table 7

Related work

The majority of recent approaches to neural word alignment fall into two categories: heuristic and probabilistic. A representative heuristic approach is [START_REF] Legrand | Neural network-based word alignment through score aggregation[END_REF], which learns association scores between source and target word embeddings without any underlying probabilistic model. This simple approach is used to clean up translation memories in [START_REF] Pham | Fixing translation divergences in parallel corpora for neural MT[END_REF]. More recently [START_REF] Sabet | Simalign: High quality word alignments without parallel training data using static and contextualized embeddings[END_REF] directly takes pre-trained non-contextual and contextual multilingual representations [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] as their association scores, deriving individual word alignments by solving an optimal matching problem.

Early work on probabilistic neural alignment is [START_REF] Yang | Word alignment modeling with context dependent deep neural network[END_REF], where a feed-forward neural network is used to replace the count-based translation model of a HMM-based aligner. This approach is further developed in [START_REF] Tamura | Recurrent neural networks for word alignment model[END_REF] where a recurrent network helps to capture contextual dependencies between alignment links. This early work aims to improve the alignment quality for phrase-based MT. As discussed above, the work of [START_REF] Rios | Deep generative model for joint alignment and word representation[END_REF] also considers neural versions of IBM models, with the goal to improve word representations through cross-lingual transfer in low-resource contexts. Alignment is also the main focus of (Ngo-Ho and Yvon, 2019) which reviews a whole set of alternative parameterizations for neural IBM-1 and HMM models, varying the word embeddings (word and character based), the context-size in the translation model and the parameterization of the distortion model.

A much more active line of research tries to improve neural MT by exploiting the conceptual similarity between alignments and attention [START_REF] Koehn | Six challenges for neural machine translation[END_REF]. [START_REF] Cohn | Incorporating structural alignment biases into an attentional neural translation model[END_REF] modify the attention component to integrate some biases that are useful in alignements: a preference for monotonic alignements, for reduced fertility values, etc. They also propose, following [START_REF] Liang | Alignment by agreement[END_REF], to enforce symmetrization constraints, an idea also explored in [START_REF] Cheng | Agreement-based joint training for bidirectional attention-based neural machine translation[END_REF]; The same methodology is studied in [START_REF] Luong | Effective approaches to attention-based neural machine translation[END_REF][START_REF] Yang | Neural machine translation with recurrent attention modeling[END_REF], with the objective to introduce dependencies between successive attention vectors. The work of [START_REF] Peters | Sparse sequence-to-sequence models[END_REF] also aims to enhance the attention component of a sequence-to-sequence, by enforcing sparsity via the sparse-max operator.

The work reported in [START_REF] Alkhouli | Alignment-based neural machine translation[END_REF][START_REF] Wang | Hybrid neural network alignment and lexicon model in direct hmm for statistical machine translation[END_REF] explores ways to explicitely introduce alignments in NMT. They study various neuralizations of the standard generative alignment models, and also consider ways to exploit weak supervision from count-based models. This line of research is pursued by [START_REF] Kim | Structured attention networks[END_REF][START_REF] Deng | Latent alignment and variational attention[END_REF], where attention vectors are handled as structured latent variables in NMT; in this study, variational autoencoders are used represent the alignment structure. Finally, [START_REF] Garg | Jointly learning to align and translate with transformer models[END_REF] propose to jointly learn alignment and translation in a multi-task setting, thereby improving a Transformer-based model.

When compared to heuristic approaches, an obvious defect of IBM models is their directionality, which means that they deliver asymmetric alignments. Attempts to remedy this problem, while preserving the sound probabilistic underlying models have been many. [START_REF] Liang | Alignment by agreement[END_REF] propose to jointly train EM in both directions, enforcing directional link posteriors to agree as much as possible through an additional agreement term; this work is generalized in [START_REF] Liu | Generalized agreement for bidirectional word alignment[END_REF]. Grac ¸a et al. ( 2010) use a different technique and enforce symmetry via additional constraints on the posterior link distribution.

Since their introduction in [START_REF] Bowman | Generating sentences from a continuous space[END_REF], VAE models of text generation have been developed in multiple ways, and applied to many NLP tasks, in particular to Machine Translation [START_REF] Zhang | Variational neural machine translation[END_REF]. This approach generalizes the basic VAE approach by making the latent variable and the target sentence conditionally dependent from the observed source. One major difference with our work in that the model includes one latent variable per sentence, where we consider one for each target word.

Conclusion and outlook

In this paper, we have revisited the proposal of [START_REF] Rios | Deep generative model for joint alignment and word representation[END_REF] and explored variants of the variational autoencoder models for the unsupervised estimation of neural word alignment models. Our study has confirmed the previous findings and highlighted two promising aspects of this model. First, it is a full model of the joint distribution, which makes it easy and natural to introduce symmetrization constraints, as we have shown by proposing two such extensions. With these constraints, we were experimentally able to close the gap with strong baselines implementing neural variants of the conditional HMM models in the large data condition. Second, it opens new alleys to also incorporate monolingual data during training, which might especially prove useful in low-resource scenarios.

One remaining problem in this approach is the prediction of the null links, which is quite problematic in an encoder-decoder approach. We have shown in particular that the VAE model is strongly inclined to under-generate alignment links, which is detrimental to the overall AER performance. Symmetrization is a first answer to this problem, which however only partly fixes the issue. Another difficult problem with this model is controlling the optimization problem, a difficult task when the objective functions combines multiple terms with varying dynamics. More work is needed there to design better optimization strategies, with a better balance between the various sub-objectives.

Figure 1 :

 1 Figure 1: Visualizing the three terms of the ELBO for Romanian-English. The weights of the reconstruction cost, alignment cost and KL divergence are set to α, β, γ respectively.

Table 1 :

 1 which correspond to a more challenging scenario where the training data is limited in size. Additional experiments with monolingual data use the Romanian data from News Crawl 2019 (∼ 6M sentences) 4 . Basic statistics for these corpora are in Table1. Basic statistics for the data

	Corpus # sent. in train # sent. in test # tokens in test # non-null links
				Eng.	For.
	En-Fr	∼1.9M	447	7 020 7 761	17 438
	En-Ro	∼260K	246	5 455 5 315	5 988

Table 2 :

 2 AER scores for IBM-1 models. The best result in each column is in boldface.

			-French	English-Romanian
	IBM-1	En-Fr Fr-En GDF En-Ro Ro-En
	Giza++	40.0	33.9	25.1	56.0	53.5	51.1
	IBM1+NN	27.9	27.2	17.8	46.3	44.9	38.3
	IBM1+NN+BPE	25.7	24.0	14.6	43.4	40.4	34.4
	IBM1+VAE+BPE 33.4	34.3	24.9	56.3	55.6	51.3
	+SP	22.1	23.8	16.8	49.3	51.4	45.2
	+AC	22.8	23.6	17.8	49.1	49.2	43.3

Table 3 :

 3 AER scores for variants of the HMM model and for Fastalign.

			-French	English-Romanian
	HMM	En-Fr Fr-En GDF En-Ro Ro-En GDF
	Fastalign	15.1	16.2	14.2	33.3	32.9	30.4
	HMM Giza++	11.9	11.9	8.5	33.3	36.3	32.4
	HMM+NN	11.8	11.1	9.7	30.6	40.1	34.3
	HMM+NN+BPE	9.8	10.4	9.1	34.4	29.3	29.4
	HMM+VAE+BPE 18.9	12.9	13.9	50.2	38.6	42.7
	+SP	12.9	12.2	11.7	37.5	38.0	37.0
	+AC	11.4	10.8	9.6	35.5	38.8	35.1

Table 4 :

 4 Evaluation of null-alignment links when aligning French with English.

		# Unaligned Accuracy Precision Recall
	IBM1+NN	3836	74.0	8.7	49.7
	IBM1+NN+BPE	3633	75.8	10.1	54.4
	IBM1+VAE+BPE	6596	57.4	7.5	73.2
	+SP	5621	64.2	9.0	75.1
	+AC	5622	64.3	9.1	76.0
	HMM+NN	2994	80.4	13.1	58.1
	HMM+NN+BPE	4835	70.7	12.2	87.5
	HMM+NN+BPE+Joint	4843	70.3	11.6	83.5
	HMM+VAE+BPE	6591	58.6	8.7	84.9
	+SP	6581	59.0	9.1	89.3
	+AC	5579	65.5	10.4	86.0

Table 5 :

 5 Precision, recall and F-measure of IBM-1 models for English-Romanian

			Precision			Recall			F-measure
	IBM-1	En-Ro Ro-En GDF En-Ro Ro-En GDF En-Ro Ro-En GDF
	Giza++	58.8	49.9	73.8	35.1	43.5	36.5	43.9	46.4	48.8
	+NN	57.7	60.0	75.7	50.0	50.9	51.9	53.6	55.1	61.6
	+NN+BPE	63.9	64.1	80.4	50.6	55.6	55.3	56.5	59.5	65.5
	+VAE+BPE	56.6	53.9	79.5	35.4	37.6	35.0	43.6	44.3	48.6
	+SP	60.6	57.8	76.2	43.5	41.8	42.7	50.7	48.5	54.8
	+AC	61.3	58.9	76.9	43.5	44.6	44.8	50.8	50.8	56.6

Table 6 :

 6 show that +Mono helps improve the reconstruction model, which attains almost perfect reconstruction accuracy in both directions, Model HMM # Agree Ratio En-Fr Ratio Fr-En AER (inter) Agreement between alignments in two directions for English-French, in terms of the number of alignment links, its ratio to the total number of alignment links predicted by the model and the AER of the "intersection" heuristic.suggesting that the autoencoder is overfitting. The gain brought by monolingual data is found only for IBM-1, for the direction Ro-En. The extra-task of denoising the input (+Mono+Noise) further improves the AER compared to the parameter sharing approach.

	Giza++	4683	72.6		75.5	7.5
	+NN	4771	73.2		76.7	7.4
	+NN+BPE	4040	75.0		80.2	10.4
	+VAE+BPE	3160	69.1		76.0	18.7
	+SP	3586	86.2		86.5	13.0
	+AC	3989	83.6		84.8	10.1
	Model	English-Romanian Romanian-English
	+VAE+BPE+SP R-ACC	AER	R-ACC	AER
	IBM-1	84.6		49.3	93.0	51.4
	+Mono	98.1		49.1	98.1	47.8
	+Noise	98.4		48.8	97.9	47.6
	HMM	95.5		37.5	97.5	38.0
	+Mono	98.5		37.9	98.1	38.0
	+Noise	98.8		36.3	97.5	36.5

Table 7 :

 7 Training with a monolingual corpus (+Mono) and the noise model (+Noise) on English-Romanian data. R-Acc is the accuracy of the reconstruction model.

http://statmt.org/wmt16

See http://statmt.org/wmt19

https://github.com/moses-smt/mosesdecoder

We differ there from[START_REF] Garg | Jointly learning to align and translate with transformer models[END_REF] who use a joint BPE vocabulary.

In our BPE baseline experiments with En:Ro, we found that 64 hidden units were sufficient to obtain the best AER score after 10 iterations. As for the other meta-parameters, we decided to stick with these baseline values.

In our baseline experiment with English-Romanian (From http://www.statmt.org/wpt05/), using these weights resulted in an acceptable AER scores and seemed appropriate for our further experiment; a small exploration of the hyper-parameter space showed that these results were stable.

The Romanian corpus is from News Crawl 2019, the English corpus is from Europarl, and corresponds to the English side of the English-French data.

Acknowledgements

This work has been made possible thanks to the Saclay-IA computing platform.