Andrew Ng

Yoshua Bengio

Holger Schwenk

Loïc Barrault

Fethi Bougares

Paul Deleglise

Walid Aransa

Adrien Bardet

Anne Cécile

Nicolas Dugué

Grégor Dupuy

Yannick Estève

Mercedes García-Martínez

Mélanie Hardy

Malik Koné

Etienne I Micoulaut

Burak Arslan

Gözde Aytemur

˙ilke Bereketli

Emre Dogan

Özlem Durmaz

Fatma Gül Karagöz

Burcu Konakc

Günce Orman

Cem Özatalay

Atay Özgövde

Selin Pelek

Pınar Uluer Dogu Toksöz

Merve Ünlü

Galatasaray

Canan Akyüz

Sibel Aydın

˙ipek Daglı

Hande Erguner

Alp Kılıc

Ezgi Tunc ¸, C ¸igdem Vaizoglu Duygu Kuzuoglu

Emre I Yetkin

Alper Ac

Nihan Aksakallı

Tuna Altınel

Aslı Aydemir

Begüm Bas ¸das ¸

Can Candan

Yonca Demir

Nazım Dikbas

Hülya Dinc

Elif Ege

Aslı Odman

Özlem Özkan

Aslı Takanay

Ceren Uysal

Keywords: activation functions

Machine translation aims at automatically translating documents from one language to another without human intervention. With the advent of deep neural networks (DNN), neural approaches to machine translation started to dominate the eld, reaching stateof-the-art performance in many languages. Neural machine translation (NMT) also revived the interest in interlingual machine translation due to how it naturally ts the task into an encoder-decoder framework which produces a translation by decoding a latent source representation. Combined with the architectural exibility of DNNs, this framework paved the way for further research in multimodality with the objective of augmenting the latent representations with other modalities such as vision or speech, for example. is thesis focuses on a multimodal machine translation (MMT) framework that integrates a secondary visual modality to achieve be er and visually grounded language understanding. I speci cally worked with a dataset containing images and their translated descriptions, where visual context can be useful for word sense disambiguation, missing word imputation, or gender marking when translating from a language with gender-neutral nouns to one with grammatical gender system as is the case with English to French. I propose two main approaches to integrate the visual modality: (i) a multimodal a ention mechanism that learns to take into account both sentence and convolutional visual representations, (ii) a method that uses global visual feature vectors to prime the sentence encoders and the decoders. rough automatic and human evaluation conducted on multiple language pairs, the proposed approaches were demonstrated to be bene cial. Finally, I further show that by systematically removing certain linguistic information from the input sentences, the true strength of both methods emerges as they successfully impute missing nouns, colors and can even translate when parts of the source sentences are completely removed.

In 2016, more than 2000 academics -including myself -signed a petition for peace, calling the authorities to end the violence in eastern Turkey. Since then, more than 500 signatories have been discharged from their positions, and more than 700 of them have been put on trial for "terrorist propaganda".

As I complete the nal words of my thesis, I would like to dedicate my work to two academics for peace: Dr. Mehmet Fatih Tras ¸, who commi ed suicide a er being discharged from his university, and Prof. Füsun Üstel from Galatasaray University who is about to enter prison.

Résumé

La traduction automatique vise à traduire des documents d'une langue à une autre sans l'intervention humaine. Avec l'apparition des réseaux de neurones profonds (DNN), la traduction automatique neuronale (NMT) a commencé à dominer le domaine, a eignant l'état de l'art pour de nombreuses langues. NMT a également ravivé l'intérêt pour la traduction basée sur l'interlangue grâce à la manière dont elle place la tâche dans un cadre encodeur-décodeur en passant par des représentations latentes. Combiné avec la exibilité architecturale des DNN, ce cadre a aussi ouvert une piste de recherche sur la multimodalité, ayant pour but d'enrichir les représentations latentes avec d'autres modalités telles que la vision ou la parole, par exemple. Ce e thèse se concentre sur la traduction automatique multimodale (MMT) en intégrant la vision comme une modalité secondaire a n d'obtenir une meilleure compréhension du langage, ancrée de fac ¸on visuelle. J'ai travaillé spéci quement avec un ensemble de données contenant des images et leurs descriptions traduites, où le contexte visuel peut être utile pour désambiguïser le sens des mots polysémiques, imputer des mots manquants ou déterminer le genre lors de la traduction vers une langue ayant du genre grammatical comme avec l'anglais vers le franc ¸ais. Je propose deux approches principales pour intégrer la modalité visuelle: (i) un mécanisme d'a ention multimodal qui apprend à prendre en compte les représentations latentes des phrases sources ainsi que les caractéristiques visuelles convolutives, (ii) une méthode qui utilise des caractéristiques visuelles globales pour amorcer les encodeurs et les décodeurs récurrents. Grâce à une évaluation automatique et humaine réalisée sur plusieurs paires de langues, les approches proposées se sont montrées béné ques. En n, je montre qu'en supprimant certaines informations linguistiques à travers la dégradation systématique des phrases sources, la véritable force des deux méthodes émerge en imputant avec succès les noms et les couleurs manquants. Elles peuvent même traduire lorsque des morceaux de phrases sources sont entièrement supprimés. Language is the primary framework of communication that human beings use, when expressing their ideas and thoughts. e existence of thousands of languages in the world however, constitutes an obstacle to communication between the speakers of di erent languages. Although human translation is the gold standard for high quality translation across languages, nowadays we also require decent instantaneous translation facilities for di erent purposes such as quickly understanding a newly received document or making sense of a critical sign during a touristic trip. A computational solution to the instantaneous translation problem is not only important for the primary task of text translation but also is key to remove the communication barrier between speakers of di erent languages by means of a conversational tool that combines speech recognition, translation and speech synthesis for example. To that end, machine translation (MT) is speci cally interested in automatic language translation, through the use of statistical modeling tools of machine learning (ML). ese tools aim to capture the "complex relations" between two collections of sentences that are translations of each other. ese complex relations mostly refer to linguistic aspects such as syntax, semantics and pragmatics which are key to language understanding. An MT model should thus be able to understand a source language and then construct a uent and adequate translation in the target language. Until recently, the state-of-the-art approaches in MT heavily relied on multi-stage pipelines that divide the translation problem into smaller parts. ese parts are primarily responsible for modeling the phrase translation probabilities, learning the most likely target-to-source word alignments and ensuring the uency of the produced translations [START_REF] Koehn | Statistical Phrase-based Translation[END_REF]. Nowadays, deep neural networks based approaches [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF][START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF][START_REF] Vaswani | A ention is all you need[END_REF] are dominating the eld and considered to be the new state-of-the-art in MT. Unlike the multi-stage approach, neural MT models (NMT) are end-to-end and relatively easily trained with almost no feature engineering involved.

List of Notations

Regardless of the underlying statistical framework, MT requires large amount of parallel sentences to be able to learn a decent translation model. Luckily, we are in an era where massive amount of data is constantly produced and made publicly available through the Internet. e availability of such diverse data ranging from documents and images to videos, also gives rise to numerous new ideas to foster research on multimodal machine learning, a term coined to designate models that can leverage information coming from di erent modalities [START_REF] Baltrusaitis | Multimodal machine learning: A survey and taxonomy[END_REF]. is research area is inspired by the multimodal aspects of human learning i.e. the inherent ability of human beings to integrate simultaneous information from di erent sensory channels. In infant learning for example, lexical items produced through pointing gestures were shown to later migrate to the verbal lexicon of the children [START_REF] Jana | Gesture paves the way for language development[END_REF] whereas [START_REF] Abu-Zhaya | Building a multimodal lexicon: Lessons from infants' learning of body part words[END_REF] provide evidence that infants bene t more from tactile-speech than visual-speech interactions. e multisensory integration ability also allows us to achieve a be er understanding of the surrounding world [START_REF] Stein | e neural basis of multisensory integration in the midbrain: Its organization and maturation[END_REF][START_REF] Ernst | Humans integrate visual and haptic information in a statistically optimal fashion[END_REF] by reducing uncertainty, for example when we a empt to recognize speech in a noisy environment.

Similar uncertainties also arise in the case of MT where for example a word in a source sentence has multiple senses or when the gender information has to be inferred for translating from a gender-neutral language to another one that has grammatical gender. An example to the la er ambiguity is as follows: Translating "a basketball player" to French requires inferring the sex of the player in order to select between "un joueur" (male) and "une joueuse" (female). e primary objective of this thesis is thus to devise multimodal machine translation (MMT) systems which leverage contextual information from an auxiliary input modality. In order to do so, we explore a relatively new dataset called Multi30K [START_REF] Ellio | Multi30k: Multilingual english-german image descriptions[END_REF] which provides images, their natural language descriptions in English and the translations of these descriptions into three di erent languages. e choice of vision as the auxiliary modality here is motivated by the fact that the images are (almost) objective depictions of concrete concepts surrounding us, making them natural candidates to resolve the aforementioned linguistic ambiguities. Moreover, evidence from the literature also suggest their usefulness in terms of joint language and vision processing: [START_REF] Bergsma | Learning bilingual lexicons using the visual similarity of labeled web images[END_REF] and [START_REF] Kiela | Visual bilingual lexicon induction with transferred convnet features[END_REF] used images in a visual similarity based bilingual lexicon induction task i.e. the task of inferring the translation of a word without having access to data directly labeled for translation purposes; [START_REF] Vinyals | Show and tell: A neural image caption generator[END_REF] and Xu et al. (2015b) demonstrated the possibility to generate natural language descriptions for images using end-to-end deep neural networks.

To that end, I mainly propose two di erent interaction methods based on two di erent computational representations of images. Both types of features are obtained from state-of-the-art deep computer vision models [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF][START_REF] He | Deep residual learning for image recognition[END_REF] which are pre-trained to perform ImageNet large-scale image classi cation task [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF]. Before ge ing into the details of the proposed approaches, I rst provide an extensive background about ML, especially focusing on the ecosystem around deep neural networks (Chapter 2) and the underlying details of the state-of-the-art pretrained computer vision models (section 2.6. 3, p. 27). I then describe the conventional multi-stage MT and the state-of-the-art NMT approaches in chapter 3. In chapter 4, I explain the MMT task along with the Multi30K dataset and provide a detailed literature overview of the state-of-the-art in MMT.

e second part of the thesis consists of our contributions to MMT. is part begins with the introductory chapter 5 which gives a thorough description of the common experimental framework of the thesis, including details such as the pre-processing pipeline, the baseline NMT architecture and the underlying so ware used to train the models. Chapter 6 and chapter 7 introduce the two family of multimodal interactions.

e rst family of interactions incorporate global visual features which are high-level vectoral semantic representations, while the second family integrates more sophisticated convolutional features that preserve spatial information unlike the former. We conduct an extensive set of experiments followed by quantitative analyses for English→German and English→French translation tasks of Multi30K. Finally in chapter 8, I take a step back and provide several qualitative analyses to showcase the strengths and weaknesses of the explored MMT models, along with a novel probing framework to assess the visual awareness of the models. I conclude the thesis in chapter 9 where I discuss future perspectives about MMT and multimodal language understanding in general.

CHAPTER2

Deep Neural Networks

Machine learning (ML) is traditionally considered as a multi-stage framework which breaks down the task to be solved into two main stages. If we consider a supervised learning problem such as object recognition, the rst stage -referred to as feature engineering -would aim at extracting useful features from raw input images while the second stage would train a classi er to estimate the probability distribution over plausible object labels given the extracted input features. is feature engineering stage requires a substantial amount of human expertise and domain-knowledge. In addition, the quality of the obtained features heavily a ects the performance of the nal model.

Deep neural networks (DNN) on the other hand, propose to transform the explicit feature engineering stage into an intrinsic aspect of the model referred to as representation learning [START_REF] Goodfellow | Deep Learning[END_REF]. DNNs are able to jointly learn sophisticated feature extractors and an output logic -to perform classi cation or regression for example -by minimizing a task-relevant error signal through stochastic optimization. Unlike explicit feature engineering, this optimization framework enable DNNs to learn good feature extractors that even humans may not be able to come up with. In contrast to multi-stage ML, DNNs are also end-to-end: they require minimum to none pre/post processing allowing them to be easily trained and deployed. e idea behind DNNs dates back to 1950s. Initially, AI researchers were inspired by the massively interconnected network of neurons found in the biological brain. is biological evidence of intelligence guided the eld to come up with simple computational units such as the McCulloch-Pi s neuron [START_REF] Mcculloch | A logical calculus of the ideas immanent in nervous activity[END_REF] and later the perceptron algorithm [START_REF] Rosenbla | e perceptron: A probabilistic model for information storage and organization in the brain[END_REF]. Unfortunately, the lack of e cient training algorithms and the alleged inability of these models to learn the exclusive-OR (XOR) function had triggered the so-called AI winter where research on neural networks had lost traction [START_REF] Goodfellow | Deep Learning[END_REF]. Luckily, a group of researchers continued to work Figure 2.1: e graphical model of a neuron with four inputs: although the input neurons are not parameterized, they are generally depicted using nodes as well for notational purposes. Blue and green shades represent the input and output nodes respectively. in the eld resulting in the discovery of the missing piece of the equation, the backpropagation algorithm which is still a crucial element of DNN training [START_REF] Paul | Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences[END_REF][START_REF] Rumelhart | Learning representations by back-propagating errors[END_REF]. Today, DNNs are considered state-of-the-art in many elds including but not limited to object recognition, automatic speech recognition (ASR), language modeling (LM) and machine translation (MT) [START_REF] Lecun | Deep learning[END_REF].

In this chapter, I will rst introduce the fundamentals of DNNs with a focus on supervised learning. I will then proceed with recurrent neural networks (RNN), a type of DNN specialized for modeling sequential data such as natural languages. Finally, in order to lay the ground for joint language and vision processing, I will describe convolutional neural networks (CNN) which are state-of-the-art models in image and video processing.

Neurons and Fully-connected Networks

e basic computational unit in a DNN is a neuron. Parameterized with a set of weights {w i } n 1 and a bias term b, a neuron outputs the weighted sum of its inputs (Figure 2.1). e parameters of a modern neuron are real valued unlike the early McCulloch-Pi s neuron in the literature which used binary connections [START_REF] Mcculloch | A logical calculus of the ideas immanent in nervous activity[END_REF]. It is also possible to interpret the weighted sum as a dot product between the input vector x = [x 1 ; x 2 ; . . . ; x n ] ⊤ and the weight vector w = [w 1 ; w 2 ; . . . ; w n ] ⊤ as follows:

ŷ = i w i x i + b = x ⊤ w + b (2.1)
A neuron learns to produce a real valued response to some particular input pa ern where the response is proportional to the angular distance (i.e. closeness) between the input and the learned weight vector. is particular view of the neuron as a pa ern detector hints at the fact that, analogous to biological brain, complex reasoning ability may be achieved through an interconnected network of neuronsi.e. neural networks.

Before ge ing familiar with the concept of neural networks however, we need to de ne one more abstraction, namely, a layer, which is a logical computation unit grouping a set of neurons. e fundamental layer type in modern DNNs is the fully-connected layer (FC) which consists of h neurons, each connected to the incoming layer with dedicated weight vectors. e weight vector in equation 2.1 can be replaced with a matrix W where the i-th row corresponds to the weight vector of the i-th neuron in the layer. is way, the output of the layer becomes a vector ŷ given by a matrix-vector product:

ŷ = W x + b (W ∈ R h×n , x ∈ R n , b ∈ R h , ŷ ∈ R h ) (2.2)
We can now de ne a neural network (NN) as an interconnected topology made of input layers, output layers and hidden layers stacked in-between them. e la er layers are called hidden as their outcomes are not observable from the actual data generating processes i.e. they are considered to be variables expected to model latent structures discovered from the input. We will be using the term fully-connected neural networks (FCNN) to refer to networks that consist of FC layers. When naming FCNNs, the convention ignores the enumeration of the input and the output layers and only counts the number of hidden layers in-between. Figure 2.2 shows two FCNNs, a single-layer and a three-layer one, where the hidden layer neurons are shaded with gray. Let us express the computation performed by the rst FCNN where the output of the network is computed by successively feeding the output of each previous layer as input to the next one. ℓ i () denotes the function of the i-th layer where the parameters are W (i) and b (i) : 1) x + b (1) ) + b (2) (2.3)

ŷ = FCNN 1 (x) = ℓ 2 (ℓ 1 (x)) = W (2) ℓ 2 (W (
Figure 2.3: Commonly used non-linear activation functions.

Non-linear Neurons

So far, we have only covered linear layers where each neuron basically computes a different linear combination of the incoming connections. Although increasing the number of hidden layers seem to add computational capacity to the network, linear models are not able to capture non-linear input-output mappings, a traditional example being the XOR function. On the other hand, it has been shown that shallow FCNNs can act as "universal function approximators" once equipped with sigmoid non-linearities [START_REF] Cybenko | Approximation by superpositions of a sigmoidal function[END_REF]. For these reasons, modern DNNs are inherently designed with non-linear activation functions which themselves constitute an active area of research [START_REF] Glorot | Understanding the di culty of training deep feedforward neural networks[END_REF]Xu et al., 2015a;[START_REF] Clevert | Fast and accurate deep network learning by exponential linear units (ELUs)[END_REF][START_REF] Klambauer | Selfnormalizing neural networks[END_REF]. e three most commonly used non-linear activation functions are plo ed in Figure 2.3. Sigmoid activations are generally used to implement gating mechanisms in DNNs that regulate the information ow (section 2. 5.1, p. 20). Tanh and ReLU activations are more general purpose and o en used within RNNs (section 2.5, p. 18) and CNNs (section 2.6, p. 24) to induce complex pa ern recognition abilities. ese functions are mathematically de ned as follows:

sigmoid(x) = σ(x) = 1 1 + exp(-x) tanh(x) = 2σ(2x) -1 ReLU(x) =    0 x ≤ 0 x x > 0
e application of an activation function φ : R → R to a vector implies that it is applied to each component of that vector. e following depicts the three layer FCNN (Figure 2.2b) by assigning a non-linearity φ i () to each layer:

ŷ = FCNN 2 (x) = φ 4 (ℓ 4 (φ 3 (ℓ 3 (φ 2 (ℓ 2 (φ 1 (ℓ 1 (x))))))))

Multi-class Classi cation

Before diving into more sophisticated types of layers and networks, let us introduce the classical handwri en digit recognition to illustrate the steps involved in supervised training of a neural network. Figure 2.4 proposes a simple three-layer FCNN in order to estimate the probability distribution over a set of labels, given an input image. e model receives a a ened vector x ∈ R n 2 representing a grayscale square input image of shape n × n, feeds it through the subsequent hidden layers of size h each and produces a vector of predicted probabilities ŷ ∈ R k . e set of digit labels is de ned as K = {0, 1, . . . , 9} and the number of labels is given by the cardinality of the label set i.e. k = |K| = 10.

A well known dataset for handwri en digit recognition is the MNIST dataset (Le-Cun et al., 1998) which provides 60K training and 10K testing examples. We denote the training set by D = x (i) , y (i) : 1 ≤ i ≤ N where each element is an ordered pair of one a ened image vector x (i) and its target label y (i) ∈ K. Since the images provided by MNIST are of shape 28 × 28, the size of a a ened image vector is 28×28=784.

Both the digit recognition task and the various NMT models that will be explained in future sections, perform a multi-class classi cation i.e. predicts a discrete categorical distribution over a prede ned set of labels. A linear neuron produces an unbounded response which is obviously not what we expect from the output layer of such models. Instead, we would like that the output produces a valid probability distribution. We achieve this by using a special operator so max R k → [0, 1] k which normalizes its = Figure 2.5: High-level abstraction of the forward-pass step in a DNN: given an input image, the network assigns probabilities of 0.97 and 0.03 to labels 5 and 6 respectively. input vector so that the values lie between [0, 1] and sum to 1:

SOFTMAX (z) = exp(z 1 ) k i=1 exp(z i ) ; exp(z 2 ) k i=1 exp(z i ) ; . . . ; exp(z k ) k i=1 exp(z i )
Denoting the network by a function f : R n 2 → R k and se ing the output layer activation to so max, a forward-pass through the network (Figure 2.5) can now predict P (y | x) i.e. the conditional probability distribution over the labels given an image. For example, the last equation below fetches the probability of the input being a "5":

P (y | x) = ŷ = f (x) = SOFTMAX (ℓ 4 (φ 3 (ℓ 3 (φ 2 (ℓ 2 (φ 1 (ℓ 1 (x)))))))) P (y = "5" | x) = ŷ5 = 0.97 (2.4)

Maximum Likelihood Estimation

e training set D is just a sample from the true data generating distribution p data , which is what we actually want to understand in order to perform inference later on using unseen data. A common framework to achieve this is the maximum likelihood estimation (MLE) where the objective is to nd a set of parameters that maximize the likelihood of the training set, or to put it di erently maximize the probability of the ground-truth label assigned by the model. In order to cast this as an optimization problem, we rst need to pick a loss function suitable for multi-class classi cation. A common choice is negative log-likelihood (NLL) which is de ned below for a single example x (i) , y (i) : i) ; θ =log ŷy (i) Note that the explicit θ states that the model is parameterized by θ which is a attened parameter vector containing all the weights and the biases of the model. We can now de ne the training set NLL as the expected loss over all the examples:

NLL (i) = -log P y (i) | x (
L(θ) = 1 N N i=1 NLL (i) = - 1 N N i=1 log P y (i) | x (i) ; θ
As can be seen, NLL is a natural choice for classi cation since it approaches 0 when the output probability for the correct label approaches 1 and slowly goes to in nity otherwise. is way, we can cast MLE as minimizing the training NLL over the parameter space where the nal parameter estimate is denoted by θ * :

θ * = arg min θ L(θ) = arg min θ - 1 N N i=1 log P y (i) | x (i) ; θ

Training DNNs

One nice property of the network depicted so far is its compositional nature: each layer in the topology is a function of its inputs parameterized with the weights and biases of that layer. is means that the nal NLL loss is di erentiable with respect to all parameters involved in the network i.e. the parameter vector θ. When equipped with the necessary mathematical tools, the di erentiable nature of the network allows one to compute the gradient of the loss function with respect to θ denoted by ∇ θ L(θ). is gradient vector -composed of partial derivatives -quanti es how much the loss function changes in response to an in nitely small change in each parameter θ i . e following shows how the gradient vector is de ned for a network with D parameters i.e. θ ∈ R D :

∇ θ L(θ) =        ∂L(θ) ∂θ 1 . . . ∂L(θ) ∂θ D        (2.5)
Figure 2.6: e loss surface1 of a function with two parameters: gradient descent allows going downhill from the initial point (•) to a local minimum (blue ag).

Since the gradient vector points in the direction of greatest rate of increase and our objective is to minimize the loss, we can update θ by taking steps towards the negative of the gradient vector to decrease the loss (Figure 2.6). is iterative optimization method is called batch gradient descent (BGD) and it forms the basis of modern DNNs [START_REF] Rumelhart | Learning representations by back-propagating errors[END_REF][START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]. e described update rule is given as follows:

θ ← θ -α∇ θ L(θ) (2.6)
e scalar hyperparameter α is called the learning rate which tunes the size of the steps taken during the update rule. Correctly se ing the learning rate is of utmost importance since a too small learning rate can lead to slow convergence while a large one may provoke oscillations around local minima preventing convergence.

Minibatch Gradient Descent

Although the update rule for BGD (Equation 2.6) computes the gradient of the entire training set loss L(θ) with respect to the parameters, in reality we prefer to split the training set into smaller chunks called minibatches and use the gradient of the minibatch loss during training. is approach called minibatch gradient descent (MGD) has mainly two advantages over BGD: (i) it increases the number of parameter updates performed in a single sweep of the training set allowing a detailed exploration of the parameter space and (ii) it makes it possible to e ciently train a model over datasets of hundreds of thousands and even millions of training examples. e la er e ciency is due to the fact that CPUs and GPUs are highly tuned for batched linear algebra operations.

To sum up, let us denote the number of samples in a minibatch by B. It is trivial to see that by se ing B equal to the size of the training set, MGD reduces to BGD. On the other hand, se ing B=1 leads to the online BGD called stochastic gradient descent (SGD). SGD traverses the training set one example at a time and applies the update rule a er each such example. Although this is rarely used in practice because of its computational ine ciency, the term SGD o en appears in the literature to actually refer to MGD.

Adaptive Optimizers

Several adaptive extensions to gradient descent have been proposed in the last decade to integrate feature speci c learning rate scheduling [START_REF] Duchi | Adaptive subgradient methods for online learning and stochastic optimization[END_REF][START_REF] Ma Hew | Adadelta: an adaptive learning rate method[END_REF][START_REF] Kingma | Adam: A method for stochastic optimization[END_REF][START_REF] Sashank | On the convergence of Adam and beyond[END_REF]. e common idea behind these methods is to store the statistics of previous gradients (and possibly their magnitudes) and use their running averages to accelerate or slow down per-feature learning. Nowadays, these adaptive methods are generally the starting point for researchers and practitioners as they o er very good out-of-the-box performance, which is the reason I used an adaptive algorithm called ADAM [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] throughout the experiments in this thesis. Using ADAM, the new parameter vector θ t at timestep t is obtained as follows:

g t ← ∇ θ L(θ t-1 ) m t ← (1 -β 1 )g t + β 1 m t-1 v t ← (1 -β 2 )g t 2 + β 2 v t-1 θ t ← θ t-1 - α √ v t m t (2.7)
In the above, g t is a shorthand for the gradient vector while m t and v t are the exponential moving averages of the gradient and the squared gradient vectors (with decay rates β 1 and β 2 ). We can see from equation 2.7 that the base learning rate α is now scaled using √ v t and the actual gradient g t is replaced with an exponential moving average m t .

Parameter Initialization

e training starts by randomly sampling an initial θ vector through a procedure called parameter initialization, an active area of research itself [START_REF] Martens | Deep learning via hessian-free optimization[END_REF][START_REF] Glorot | Understanding the di culty of training deep feedforward neural networks[END_REF][START_REF] Andrew M Saxe | Exact solutions to the nonlinear dynamics of learning in deep linear neural networks[END_REF][START_REF] He | Delving deep into recti ers: Surpassing human-level performance on imagenet classi cation[END_REF][START_REF] Arpit | e bene ts of over-parameterization at initialization in deep relu networks[END_REF]. Failing to initialize the parameters correctly is likely to hinder the training process by causing slow convergence or even no convergence at all. e parameter initialization is even more important in DNNs with non-linear activation functions (Section 2.1) since incorrect initialization can cause neurons to saturate i.e. staying in a constant regime which propagates back a zero gradient that inhibits learning [START_REF] Goodfellow | Deep Learning[END_REF]. In the following, we make use of the initialization method proposed by [START_REF] He | Delving deep into recti ers: Surpassing human-level performance on imagenet classi cation[END_REF] where the variance of the sampled weights for a layer with H inputs is scaled by 2/H. is per-layer standard deviation makes sure that the variance of layer activations are preserved throughout the depth of the network. We speci cally sample the weights from the following gaussian distribution N (0; 2/H).

Regularization

So far, we have shown how to formulate the training problem from an optimization point of view. Although it may be intuitive to think that the overall aim of the minimization framework is to estimate a parameter vector θ * which obtains ∼0 loss, this is hardly what we would like to achieve. More precisely, such models perfectly memorizing (over ing) the training set will exhibit poor performance on a held-out test set i.e. they will not generalize well to unseen samples. Ideally, what we would like to end up with is a model which achieves a small training loss as well as a small gap between this training loss and the test set loss. e violation of these principles are referred to as under ing and over ing [START_REF] Goodfellow | Deep Learning[END_REF]. e over ing can be mitigated by carefully regularizing the capacity of the model to ensure the law of parsimony i.e. to encourage simpler solutions over very complex ones. On the other hand, under ing -when not caused by aggressive regularization -generally requires increasing the explicit capacity of the model de ned by the width, the depth and the types of layers in the case of a DNN.

In what follows, I describe three commonly used regularization techniques.

L 2 Regularization

One classical way of regularization is the so called L 2 penalty which is additively combined with the training loss to be minimized. Let us rede ne the loss function as a sum of the previously introduced training NLL and the L 2 penalty term and denote it by J :

J (θ) = L(θ) + λ ||θ|| 2 2 (2.8) = L(θ) + λ D i θ 2
i is penalty term scaled with λ imposes a constraint over the parameter space such that the L 2 norm of the parameter vector2 is minimal i.e. an arbitrary subset of weights is discouraged to become very large unless it is necessary [START_REF] Krogh | A simple weight decay can improve generalization[END_REF]. In other words, the penalty term encourages cooperation rather than relying on a set of neurons with large weights prone to capture features not necessarily useful towards generalization or even noise pa erns. L 2 regularization is generally used interchangeably with weight decay although the la er explicitly appears in the update rule (equation 2.6) while the former penalizes the loss as in equation 2.8 [START_REF] Loshchilov | Decoupled weight decay regularization[END_REF].

Dropout

Another regularization technique pervasively used throughout the literature is the socalled dropout [START_REF] Srivastava | Dropout: A simple way to prevent neural networks from overing[END_REF], which when applied to a layer, stochastically samples a subset of the activations with a prede ned probability and multiplies them by zero (Figure 2.7). is procedure which is repeated for each batch during training, has the e ect of training exponentially many partially-connected networks that are optimized through the same objective function. e stochastic removal of hidden units prevents upper layers from becoming "lazy" i.e. relying on the constant availability of some highly predictive incoming states. When the model has to be used in evaluation mode, the dropout functionality is removed and the activations to the post-dropout layer are correspondingly scaled to match the expected incoming magnitude. Although there are advanced dropout variants especially suited for recurrent neural networks [START_REF] Gal | A theoretically grounded application of dropout in recurrent neural networks[END_REF][START_REF] Semeniuta | Recurrent dropout without memory loss[END_REF], the simple approach is quite e ective inbetween non-recurrent layers such as fully-connected and convolutional ones.

Early-Stopping

e nal regularization technique that I would like to mention is early-stopping. e idea here is to periodically evaluate the performance of the model on a special validation set and save the parameters if the performance improves over the previous best model. If there is no improvement for a predetermined amount of time (patience), the training is stopped and the last saved parameters are considered as the nal ones. Early-stopping thus avoids over ed models by returning back in time to the model with the best generalization ability. When dealing with language related tasks, we will o en see the interplay of the empirical loss L that the training minimizes with a task-speci c performance metric that can for example quantify "how good a translated sentence is". Although we may be more curious about the la er, these metrics are generally not di erentiable with respect to the parameters; hence the reason why we choose to minimize the empirical loss instead. Early-stopping also gives us the ability to use such task-speci c metrics in order to assess how well a model is doing.

Backpropagation

We previously saw that given an arbitrary input, the loss is computed by what we call a forward-pass through the network i.e. a successive application of functions de ned in the topology. We also know that each parameter will be accordingly updated with respect to its partial derivative ∂J ∂θ i . e missing piece in the overall training algorithm is the middle step which will compute those partial derivatives. In the context of neural networks, this step is achieved by the backpropagation (BP) algorithm for which an e cient formulation was rst proposed by [START_REF] Paul | Applications of advances in nonlinear sensitivity analysis[END_REF] and later popularized by [START_REF] Rumelhart | Learning representations by back-propagating errors[END_REF][START_REF] Lecun | A theoretical framework for back-propagation[END_REF] according to [START_REF] Schmidhuber | Deep learning in neural networks: An overview[END_REF]. BP is essentially a special case of reverse-mode automatic di erentiation (RAD) that propagates the scalar loss signal backward in order to compute the partial derivatives [START_REF] Günes ¸baydin | Automatic di erentiation in machine learning: A survey[END_REF]. When doing so, it de nes the overall function that the network computes in terms of smaller building blocks such as variables and operators (multiplication, addition, trigonometric functions, etc.). Each such building block (node) has well de ned forward/backward semantics that de ne the forward computation and backward gradient propagation scheme. During the forward-pass, each node stores intermediate results and keeps track of its dependencies while the backward-pass reuses those intermediate results and neatly propagates back the gradients into the necessary nodes. When a scalar loss function is used -typically the case with many DNN models -the time complexity of the forward and the backward propagations are almost the same [START_REF] Günes ¸baydin | Automatic di erentiation in machine learning: A survey[END_REF]. To concretize BP, let us give a toy example that illustrates a linear regression model with quadratic error (Figure 2.8). We de ne a set of intermediate variables z i 's (le ) and write down their partial derivatives with respect to their inputs (right):

J =((w 1 x 1 + w 2 x 2 + b) -y) 2 = z 4 2 z 4 =z 3 -y z 3 =z 1 + z 2 + b z 2 =w 2 x 2 z 1 =w 1 x 1 ∂J/∂z 4 =2z 4 ∂z 4 /∂z 3 =1 ∂z 3 /∂z 1 = ∂z 3 /∂z 2 = ∂z 3 /∂b =1 ∂z 2 /∂w 2 =x 2 ∂z 1 /∂w 1 =x 1
Once we compute the gradient of the loss J with respect to the model parameters using the chain rule, we clearly see that they are compositionally made up of intermediate gradient expressions (blue). Each parameter then receives its gradient a er the error is propagated back towards the inner parts of the network: [START_REF] Pascanu | On the di culty of training recurrent neural networks[END_REF] to renormalize the magnitude of the gradient vector if its norm is higher than a predetermined threshold.

∂J ∂w 1 = ∂J ∂z 4 ∂z 4 ∂z 3 ∂z 3 ∂z 1 ∂z 1 ∂w 1 = 2z 4 x 1 ∂J ∂w 2 = ∂J
e vanishing gradient problem is more of an issue in very deep CNNs and RNNs that will be depicted in the following sections. In both cases residual connections [START_REF] He | Deep residual learning for image recognition[END_REF] from the bo om layers to the top of the network are generally helpful to create auxiliary pathways for the gradients to backpropagate. For RNNs, advanced units with gating mechanisms [START_REF] Hochreiter | Long Short-term Memory[END_REF]Cho et al., 2014b) are de facto preferred over the original recurrent units (section 2.5.1, p. 20).

e Complete Algorithm

Now that we have all the fundamental pieces covered, we can formalize the overall training process as a well de ned algorithm (Figure 2.9). Once we have a neural architecture decided, the training starts by randomly initializing the parameter vector θ and se ing some other hyperparameters such as the early-stopping patience. A full sweep over the training set is referred to as an epoch which is itself randomly divided into minibatches of examples. An iteration consists of performing the forward-pass, the backward-pass and the parameter update over a single minibatch. In order to do early-stopping, the generalization performance of the model is periodically assessed over a held-out validation set D valid . e performance criterion here does not necessarily have to be the NLL loss used as the training objective but can be some other task-relevant metrics such as translation quality or accuracy. e period of the evaluation is also a ma er of choice that depends on the task and the size of the training set: it can range from some thousands of minibatches to one epoch or two. Finally, the training is stopped if no performance improvement occurs over the previously saved model a er P consecutive evaluations.

Recurrent Neural Networks

In this section, I will describe the prominent DNN type in sequential modeling, namely, recurrent neural networks (RNN). RNNs are extensively used in language related tasks such as machine translation [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF][START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF][START_REF] Johnson | Google's multilingual neural machine translation system: Enabling zero-shot translation[END_REF], image captioning (Xu et al., 2015b) and speech recognition [START_REF] Chan | Listen, a end and spell: A neural network for large vocabulary conversational speech recognition[END_REF]. RNNs [START_REF] Rey L Elman | Finding structure in time[END_REF][START_REF] Hochreiter | Long Short-term Memory[END_REF]Cho et al., 2014b) sequentially update their hidden state as a function of the previous hidden state and the newly presented observation. e hidden state can be thought as a progressive memory that learns how to compress the input into an e cient latent representation. e stateful processing turns out to be important to handle natural language sentences where, driven by a set of well de ned syntactic rules, the order of the words ma ers for correct and unambiguous semantics. RNNs also naturally t into the framework of language processing since recurrent processing easily accomodates variable-length sentences.

Let us denote an input sequence by X=[x 1 , . . . , x t , . . . , x T ]3 such that each element is a vector x t ∈ R D X representing a word. In the following, r() denotes the parameterized function associated with the vanilla RNN where the parameters are the bias b ∈ R D H and the matrices {W ∈ R D H ×D H , U ∈ R D H ×D X }. r() computes the hidden state h t ∈ R D H as follows (Figure 2.10a):

h t = r(h t-1 , x t ) = φ(W h t-1 + U x t + b)
(2.9)

A common choice for the non-linearity φ is the "tanh" function (Section 2.1, p. 5). e initial state h 0 can be set to 0 or to an auxiliary feature vector that we would like the model to consider as an a priori information. e successive application of r() to the input sequence X can be serialized by repeating the computation graph of r() along the time axis. Since the same parameterized function r() is reused along the time axis, the number of parameters in an RNN does not depend on the sequence length. An example of the unfolded view is given in gure 2.10b with a short input sequence X=[x 1 , x 2 , x 3 ]. Unfolding the graph leads to the following equation for the nal hidden state:

h 3 = r(r(r(h 0 , x 1 ), x 2 ), x 3 )
More o en, we may want to access to all of the hidden states computed throughout the recurrence. Let us introduce a high-level computational block RNN() which, given the input and the initial hidden state, returns all of the hidden states. is sequence of hidden states H is usually referred to as encodings (or annotations) hence the function RNN() itself an encoder. Various sentence representations can be derived from H if one would like to "summarize" the semantics using a single vector:

H = [h 1 , . . . , h T ] = RNN (X, h 0 ) ENCODE H -1 = h T GET LAST STATE H MAX = MAXP (H) GET MAX-POOLED STATE H AVG = 1 T h t GET AVG-POOLED STATE
Illustrative Example. Let us assume that we are given a hypothetical task of partially translating a sentence from one language to another. In order to cast the problem as classi cation over a predetermined set of words in the target language, let us further consider that partial translation in this context refers to predicting only the rst word of the target sentence. We can now construct a simple architecture with an RNN encoder that compresses the input sentence into a vector which is then used for the classi cation:

H = RNN (X, h 0 ) ENCODE ŷ = SOFTMAX (V H -1 + b v ) CLASSIFY LAST STATE
e output layer here is parameterized with

{V ∈ R |K|×D H , b v ∈ R |K| }
where K denotes the set of possible target words that we consider for the classi cation. Backpropagation rough Time (BPTT). RNNs are trained using the previously described backpropagation algorithm as well (Section 2.4.5, p. 15) with the only di erence that the error will now backpropagate through the recurrent function r(): the parameters of the RNN will now accumulate gradients across time since they are successively involved in the computation of all recurrent hidden states (Figure 2.11).

Gated RNNs

Language o en involves distant dependencies in the form of anaphoras4 or co-references5 to same entities for example. Moreover, tasks such as question answering and dialog modeling further increase the span of the dependencies towards sentence and even paragraph boundaries. Although vanilla RNNs are capable of storing complex contextual informations about the input, they face di culties when modeling dependencies between an early input x t ′ and a late hidden state h t where t ′ << t. ese di culties are mostly a ributed to instabilities during BPTT that cause gradients to vanish (section 2.4.5, p. 16) [START_REF] Bengio | Learning long-term dependencies with gradient descent is di cult[END_REF][START_REF] Hochreiter | Recurrent neural net learning and vanishing gradient[END_REF]. Gated RNNs incorporate sigmoid-activated gate mechanisms that dynamically regulate the information ow from the input to the hidden states as well as between successive hidden states. By doing so, they can learn to explicitly forget part of the signal or to remember it for an appropriate amount of time. e additive integration of previous states into current ones (equations 2.10 and 2.12) allows the gradient to backpropagate through distant timesteps without vanishing [START_REF] Jozefowicz | An empirical exploration of recurrent network architectures[END_REF].

Long Short-Term Memory

Long Short-Term Memory (LSTM) [START_REF] Hochreiter | Long Short-term Memory[END_REF] is the most popular gated RNN which in turn gave rise to several further variants. LSTMs have three gates and maintain an internal cell state in addition to the existing hidden state. At timestep t, the following computations are performed to obtain the hidden state h t :

i t = σ(W i h t-1 + U i x t + b i ) INPUT GATE f t = σ(W f h t-1 + U f x t + b f ) FORGET GATE o t = σ(W o h t-1 + U o x t + b o ) OUTPUT GATE c t = φ(W c h t-1 + U c x t + b c ) CANDIDATE CELL STATE c t = c t ⊙ i t + c t-1 ⊙ f t CELL STATE
(2.10)

h t = φ(c t ) ⊙ o t HIDDEN STATE
(2.11)

⊙ denotes element-wise multiplication while σ and φ correspond to sigmoid and tanh non-linearities respectively. Note that the vanilla RNN is exactly recovered by se ing i t = o t = 1, f t = 0 and by removing the non-linearity from equation 2.11.

Gated Recurrent Unit

Gated Recurrent Unit (GRU) (Cho et al., 2014b) is an LSTM variant which removes the auxiliary cell state and fuses the three gates into two, namely, update and reset gates:

z t = σ(W z h t-1 + U z x t + b z ) UPDATE GATE r t = σ(W r h t-1 + U r x t + b r ) RESET GATE h t = φ(W h (h t-1 ⊙ r t ) + U h x t + b h ) CANDIDATE HIDDEN STATE h t = h t ⊙ z t + h t-1 ⊙ (1 -z t ) HIDDEN STATE
(2.12)

When compared to LSTMs, GRUs obtain very similar performances in many sequential modeling tasks but with slightly less parameters [START_REF] Chung | Empirical evaluation of gated recurrent neural networks on sequence modeling[END_REF][START_REF] Gre | Lstm: A search space odyssey[END_REF][START_REF] Jozefowicz | An empirical exploration of recurrent network architectures[END_REF]. First neural approaches to machine translation incorporated both LSTMs [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF] and GRUs [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF].

Continuous Word Representations

In section 2.5, we assumed vectorial word representations as inputs to RNNs but did not describe their precise nature. A naive way of representing words as vectors is the onehot encoding which assigns the canonical basis vector e i = [0, . . . , 1, . . . , 0] ∈ {0, 1} |K| to the i-th word in the vocabulary where the size of the vocabulary is |K|. Figure 2.12 (le ) shows a 3D space with three one-hot encoded words. It can be easily seen that this approach does not encode the notion of word similarity at all since all word vectors are orthogonal and the pairwise euclidean distance between any pair is always √ 2.

One-hot vectors are also sparse and ine cient as each newly added word is assigned a new dimension in isolation i.e. the dimension of the space increases with the vocabulary size. e prominent approach to representing words in DNNs is to use "continuous" (real valued) word vectors embedded in |D X |-dimensional space with much lower dimensionality than the vocabulary size i.e. |D X | << |K|. is is depicted on the right side of Figure 2.12 where 5 words are embedded inside a 3D space. In contrast to binary valued one-hot vectors, real valued continuous representations also allow words to cluster around meaning centroids. Several techniques allow structuring continuous word spaces speci cally through the distributional hypothesis which suggests that "words appearing in similar surrounding contexts carry similar semantics" [START_REF] Zellig | Distributional structure[END_REF][START_REF] Rupert | A synopsis of linguistic theory 1930-1955[END_REF]. word2vec [START_REF] Mikolov | E cient estimation of word representations in vector space[END_REF] and GloVe (Pennington et al., 2014) learn such spaces by making use of very large corpora readily found on the Internet. ese models also provide pre-trained word vectors that can be transferred to other language related tasks, similar to how pre-trained CNNs can be used to represent images (section 2.6. 3, p. 27). e main approach in NMT is also to use low-dimensional continuous word vectors but by learning them jointly during the training process instead of reusing pre-trained word vectors. From a computational point of view, this is easily achieved by using an embedding layer that performs a lookup into a weight matrix E ∈ R |K|×D X where each row is a D X -dimensional word embedding. By making E a parameter of the model, the word vectors receive gradient updates leading to a structured word space optimized towards translation performance. e following extends the partial translation example by an embedding layer:

S = [ A, WOMAN, IS, PROGRAMMING, A, COMPUTER ] X = EMBEDDING (S) = [ x A , x WOMAN , x IS , x PROGRAMMING , x A , x COMPUTER ] H = RNN (X, h 0 ) ŷ = SOFTMAX (V H -1 + b v )

Vocabulary Granularity

Given a training set, a vocabulary of unique tokens is rst constructed prior to training.

e size of a vocabulary can range from hundreds to thousands of tokens depending on the size of the training set and the granularity of the vocabulary. e la er de nes how aggressively a sentence is segmented into smaller units, such as characters, subwords or words. Although word-level vocabularies are simple to construct and intuitive at rst sight, they have limited coverage avoiding them to achieve open-vocabulary translation:

• Word-level models can not synthesize novel words: although the model can learn to infer when to output a plural noun based on contextual evidence, they can not achieve this if the plural noun is not available in the vocabulary.

• Whenever a source word unknown to the vocabulary is encountered at translation time, the model has no way to represent it in the learned word vector space. Although we reserve a special out-of-vocabulary (OOV) embedding, this embedding is never learned during training since every word is "known".

To overcome the coverage problem, subword level segmentation methods [START_REF] Sennrich | Neural machine translation of rare words with subword units[END_REF][START_REF] Kudo | Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing[END_REF] are o en preferred over word-level vocabularies. [START_REF] Sennrich | Neural machine translation of rare words with subword units[END_REF] proposes an algorithm based on byte pair encoding (BPE) which segments words in the training set sentences based on their corpus frequency: the more frequent a word is, the less likely it will be segmented into smaller subwords.

e threshold here is roughly set by a hyperparameter called the number of merge operations which can typically range from 10,000 to 30,000 depending on the size of the dataset. It should be noted that as the segmentation is purely statistical, these methods do not perform a linguistically motivated morphological segmentation. For example, the word "networks" can be spli ed as "net -works" although one would expect it to be "network -s". Subword models can synthesize novel surface forms (which are not necessarily valid words) and can represent unknown words by a combination of known subword units in the vocabulary.

Convolutional Neural Networks

Nowadays it would be surprising to see FCNNs deployed for computer vision tasks even for the previously given simple digit recognition network. e rst reason behind this is the relationship between the input size and the model complexity: each neuron in the rst hidden layer has as many weights as the number of pixels in the input. For a reasonable hidden layer size of h=512, the number of parameters jumps from ∼1M to ∼140M when going from grayscale digit images of size 28×28×1 to colored real-life images of size 300×300×3, showing why fully-connected input layers are prohibitive when working with images of variable size. Another drawback of FCNNs is their inability to model hierarchical nature of visual inputs: images are inherently composed of objects which are themselves made of simpler concepts such as edges and primitive geometric pa erns. If we would like to detect whether an image contains a "ball" for example, an ideal model should be translation invariant i.e. be able to answer independently from the position of the ball. Tightly connecting the neurons to each input pixel is very unlikely to generalize in this case unless the model is exposed to a multitude of training cases with the ball appearing at all possible positions. Convolutional Neural Networks (CNN), which are today used successfully in the literature to process di erent modalities including images, audio and wri en language [START_REF] Lecun | Deep learning[END_REF], propose a neat solution to these issues using convolution and pooling operations. Once we fully understand these notions, our previous digit recognition network can be easily extended to incorporate a CNN at the input layer that replaces the ine cient FC input layer.

Convolutional Layers

Let us denote a 2D input of shape7 M ×M with X and a 2D lter of shape K×K with W . e convolution of the input X with the lter K, denoted by X * W , produces a feature map F of shape M ′ ×M ′ where each element is de ned as follows 8 :

F [i, j] = K h=0 K w=0 X[i + h, j + w]W [h, w]
is is illustrated in Figure 2.13 by a small grid (2x2 lter) sliding over a larger grid (3x3 input) to compute four scalar values that ll an output grid (2x2 feature map). Specically, each output f k acts similar to the simple neuron (Equation 2.1, p.5) by computing a dot product between its weights and some part of the input grid. is view allows us to compare the convolution and the simple neuron:

1.

e convolution allows local-connectivity: the number of parameters in the lter does not have to match the spatial resolution of the input. A valid (albeit larger) feature map is still obtained even the input size is doubled. O en, lters much smaller than the input size are used, mitigating the aforementioned parameter explosion.

2.

e convolution allows parameter reuse: although each output is connected to a di erent input region, a single set of weights {w i } is shared across the dot products. On the other hand, the neurons in a FC layer do not share parameters.

3. If we set the lter size equal to the input size, a single dot product f 1 comes out of the convolution hence f 1 becomes a fully-connected neuron.

A 2D convolutional layer is a computational unit composed of at least one lter, where lters extend towards a third dimension which is the channel dimension C. An input and a lter are now respectively denoted by X ∈ R M ×M ×C and W i ∈ R K×K×C where channel dimensions for both should match. e convolution of X with a lter W i yields a 2D feature map F i ∈ R M ′ ×M ′ where each element is the dot product between the lter weights and the corresponding input volume. A layer with C ′ lters then produces C ′ feature maps {F i } C ′ i=1 which when stacked together, forms an output volume F ∈ R M ′ ×M ′ ×C ′ . Figure 2.14 illustrates a convolutional layer where a 6x6x3 image input is transformed with C ′ =4 di erent lters of size 3x3x3 each. A convolutional layer is o en followed by a non-linearity such as ReLU [START_REF] He | Delving deep into recti ers: Surpassing human-level performance on imagenet classi cation[END_REF]. is combination intrinsically behaves like a visual pa ern detector which res to speci c pa erns highlighted by the convolution.

Pooling Layers

One side-e ect of the previously described convolution operation is how it shrinks the spatial resolution of the input from M ×M to M ′ ×M ′ where M ′ < M . is is generally an unwanted e ect that hinders the design of deep architectures since stacking more convolutional layers will quickly shrink the input image to 1x1. e common practice is then to pad the input to the convolution layers with explicit zero pixels so that M ′ =M and delegate the shrinking to pooling layers when required [START_REF] Goodfellow | Deep Learning[END_REF]. ese special layers independently operate on top of each feature map to summarize/fuse the local activation neighborhoods. Speci cally, they again operate over small regions like the convolution except that the lter is no longer learned through backpropagation. For example, an average pooling of size 3×3 will convolve a lter pre-lled with

1 9 = 1
3×3 to compute an average activation over the region. Modern CNNs generally use max-pooling which instead of taking an average, selects the highest activation as the region summary. e translation invariance property of CNNs (see the "ball" example in section 2.6) is o en a ributed to max-pooling since a shi to the most activated neuron in the input region does not in uence to output of the pooling. A variant of average pooling called global average pooling (GAP) is o en used a er the last convolutional layer in the network in order to produce a global vector that will be further projected to the number of classes de ned for the task.

In summary, deep CNNs are able to learn a hierarchical decision function where the deeper layers detect complex pa erns which are themselves composed of simpler patterns (Figure 2.15). is is supported by early studies in neuroscience as well: [START_REF] David | Receptive elds, binocular interaction and functional architecture in the cat's visual cortex[END_REF] discovered that the visual cortex of the cat contains simple and complex cells which respond to visual stimuli in increasing levels of complexity ranging from light intensity changes to geometric pa erns. In fact, as one of the very rst pa ern detection networks in the literature, neocognitron [START_REF] Fukushima | Neocognitron: A self-organizing neural network model for a mechanism of pa ern recognition una ected by shi in position[END_REF] already integrated locallyconnected units and pooling layers bearing a strong resemblance to modern CNNs except that it lacked backpropagation.

LOW-LEVEL FEATURES MID-LEVEL FEATURES HIGH-LEVEL FEATURES SHALLOW LAYERS SIMPLE CONCEPTS DEEP LAYERS ABSTRACT CONCEPTS

Pre-trained CNNs as Feature Extractors

Being able to categorize images of real-life objects is a relatively hard task to solve for an AI system. e factors a ecting its di culty range from the level of detail and complexity present in the images, to the size of the visual vocabulary i.e. the number of possible labels that can be assigned. An in uential resource in this respect is the Im-ageNet dataset [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF] which comprises 1.2 millions training images handlabeled with 1000 object categories. Together with the periodically held "ImageNet Large Scale Visual Recognition Challenge" (ILSVRC) [START_REF] Russakovsky | ImageNet Large Scale Visual Recognition Challenge[END_REF], this dataset fostered research in computer vision especially in the context of image classi cation and object localization. For the rst time in 2012, a deep CNN architecture called AlexNet [START_REF] Krizhevsky | Imagenet classi cation with deep convolutional neural networks[END_REF] won the competition by increasing the Top-5 classi cation accuracy around 11% compared to previous non-neural approaches. e following 5 years of ILSVRC witnessed an unprecedented progress in classi cation performance thanks to deeper and more parameter e cient CNN architectures such as 19-layers VGGNet (Simonyan and Zisserman, 2014), 152-layers ResNet [START_REF] He | Deep residual learning for image recognition[END_REF], and DenseNet which goes beyond 200-layers (Huang et al., 2017b). e Top-5 classi cation accuracy achieved by these models are in the range of 92-96%. e 34-layers variant of the ResNet is depicted in Figure 2 • Global features f ∈ R C ′ on the other hand are more abstractive and optimized towards the original task since they are extracted from before the output layer. In the case of ResNet, these features are nothing more than global average poolings of nal convolutional feature maps (Figure 2.16). Despite their simplicity, [START_REF] Razavian | CNN features o -theshelf: An astounding baseline for recognition[END_REF] showed how a linear classi er on top of them results in superior performance compared to previous state-of-the-art in tasks such as scene classication and image retrieval. Early works in image captioning successfully made use of these features as well [START_REF] Kiros | Unifying visual-semantic embeddings with multimodal neural language models[END_REF][START_REF] Mao | Deep captioning with multimodal recurrent neural networks (m-rnn)[END_REF][START_REF] Vinyals | Show and tell: A neural image caption generator[END_REF].

With all the evidence hinting at the expressiveness of pre-trained visual features, we will be experimenting with both spatial and global features for multimodal translation.

Summary

In this chapter, I rst described the building blocks necessary to construct fully connected DNNs in supervised learning framework and the notions of objective function and stochastic parameter optimization. A er giving a complete recipe that uses backpropagation and the SGD algorithm to train a DNN, I proceeded with the detailed explanations of RNNs and CNNs that will be extensively used to represent the visual modality and linguistic inputs such as sentences. Based on the background provided, the following chapter will introduce the current state-of-the-art in neural machine translation.

CHAPTER3

Neural Machine Translation

A machine translation (MT) system is a computer system that automatically translates content from one language to another without any human intervention. Inspired by the previous successes in cryptography, American scientist Warren Weaver claimed about the possibility of such systems for the rst time in 1947: "When I look at an article in Russian, I say: ' is is really wri en in English, but it has been coded in some strange symbols. I will now proceed to decode. '" Today, MT systems are capable of producing decent translations that may require minimum to none post-editing e ort, thanks to the massive amounts of publicly available bilingual data and powerful so ware and hardware components.

Two radically di erent approaches currently dominate the eld: phrase-based machine translation (PBMT) [START_REF] Koehn | Statistical Phrase-based Translation[END_REF] and neural machine translation (NMT) based on DNNs (Cho et al., 2014b;[START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF][START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF][START_REF] Gehring | Convolutional sequence to sequence learning[END_REF][START_REF] Vaswani | A ention is all you need[END_REF]. Although PBMTs seem to have their own advantages over NMTs in low-resource conditions or in terms of out-of-domain translation performance [START_REF] Koehn | Six challenges for neural machine translation[END_REF], NMT is now considered the prominent approach in the eld both actively researched and also deployed in many online translation services such as Google Translate and Microso Translator. NMT is an encoding & decoding machinery largely compatible with what Weaver previously suggested: an encoder encodes a sentence into an intermediate latent representation which is further consumed by the decoder to generate an appropriate translation. We can draw parallels between this intermediate representation and the concept of interlingual MT that encodes to and decodes from a language-agnostic meaning representation called an interlingua [START_REF] Delavenay | An introduction to machine translation[END_REF]. Although the di culty of manually constructing a rule-based interlingua limited these early systems to simplistic ad-hoc translation problems [START_REF] Nyberg | e kant system: Fast, accurate, high-quality translation in practical domains[END_REF], the idea itself remains elegant and is more likely to be exploitable by the di erentiable nature of NMTs forcing the model to obtain useful translation-oriented latent representations. In fact, multilingual NMTs [START_REF] Ha | Toward multilingual neural machine translation with universal encoder and decoder[END_REF][START_REF] Johnson | Google's multilingual neural machine translation system: Enabling zero-shot translation[END_REF][START_REF] Firat | Multi-way, multilingual neural machine translation[END_REF] demonstrated that such universal representations can indeed be learned by a single NMT system when trained on a combination of input and output languages.

e end-to-end & di erentiable nature of neural systems provides an exceptional exibility for exploring novel architectures integrating multiple modalities as well. is thesis is not an exception to the ongoing neural trend as the MMT systems that we will be exploring in the next chapters are pure extensions to the existing NMT systems.

In this chapter, I will rst start by introducing neural language modeling (NLM) which provides a generalized framework for formulating NMTs as conditional language models. A er brie y describing PBMTs, I will focus on the basic sequence-to-sequence NMT architecture (Cho et al., 2014b;[START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF] followed by its a entive extension [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF]. Lastly, I will talk about how to decode translations from an NMT system and introduce automatic evaluation metrics commonly used for MT evaluation. In the following, X = [X 1 , . . . , X S ] and Y = [Y 1 , . . . , Y T ] denote the source and target sequences where each individual token X s and Y t belong to the source and target vocabularies S and T, respectively.

e Language Modeling Perspective

e purpose of a language model (LM) is to estimate the probability of a sequence where the de nition of a sequence can range from sentences to large documents. Once trained, an LM can be used to predict the next token given the previous ones or can answer the question of "how likely is it to encounter this sequence?" by assigning a score to it. Formally, the sequence probability P (Y) can be decomposed into T conditional probabilities where each term is conditioned on the full previous context denoted by Y <t :

P (Y) = T t=1 P (Y t |Y <t ) = T t=1 P (Y t |Y 1 , . . . , Y t-1 ) (3.1) = P (Y 1 )P (Y 2 |Y 1 )P (Y 3 |Y 1 , Y 2 ) . . . P (Y t |Y 1 , Y 2 , . . . , Y t-1 )
Traditional n-gram LMs approximate this probability by relaxing the full context to a xed-size context of n previous tokens where n is typically three or four:

P (Y) = T t=1 P (Y t |Y <t ) ≈ T t=1 P (Y t | Y t-1 , . . . , Y t-n+1 )
n-gram LMs are powerful non-parametric models purely estimated from count statistics of a given monolingual corpus. Although powerful and widely used, they lack the potential bene ts of full context and requires the integration of techniques such as backing-o and smoothing [START_REF] Kneser | Improved backing-o for M-gram language modeling[END_REF] to prevent the sparsity problem i.e. the underestimation of rare or never occurring n-grams. DNN-based LMs (NNLM) a empted to resolve this sparsity problem by representing the context with the concatenation of word embeddings associated to n previous tokens [START_REF] Bengio | A neural probabilistic language model[END_REF][START_REF] Schwenk | Continuous space language models for statistical machine translation[END_REF]. is context vector is then non-linearly transformed and projected to the size of the vocabulary for further probability estimation using the same multi-class classi cation framework introduced in the previous chapter. is way, an n-gram that never occurred in the training set can still be represented if its constituents are known to the model. However, as the number of parameters in the non-linear layer depends on the size of the context n, these NNLMs were practically limited to xed-size contexts as well. In 2010, Mikolov et al. proposed a recurrent LM (RNNLM) that encodes variable-length sequences, making it possible to use arbitrarily long contexts instead of a predetermined context size.

An RNNLM estimates the probability of a single sequence Y as follows:

Y = [A, WOMAN, PLAYS, TENNIS, <eos>] OUTPUT Y ′ = [<bos>, A, WOMAN, PLAYS, TENNIS] INPUT [h 1 , . . . , h T ] = RNN (EMB (Y ′ ) , h 0 ) ENCODE c t = f (h t ) CONTEXT i.e. Y <t (3.2) P (Y) = T t=1 P (Y t | Y <t ) = T t=1 P (Y t | c t ) SEQUENCE PROB. (3.3) -log(P (Y)) = - T t=1 log (P (Y t | c t )) SEQUENCE NLL
Note how we were able to de ne a pseudo-input sequence Y ′ which is actually a timeshi ed version of Y. is allows us to formulate the RNNLM as a mapping from an input sequence to an output sequence, consistent with the notational framework introduced in the rst chapter. Special tokens such as <bos> and <eos> are generally used to mark the "beginning" and the "end" of the sequences. Each new hidden state produced by the RNN conveys information about the sequence processed so far i.e. the full-context Y <t . is is why we represent the context as a function of the recurrent hidden state h t (equation 3.

2) where f is an arbitrarily complex output block that projects h t to the size of the vocabulary. At each timestep P (Y t | c t ) estimates the probability that corresponds to the true token Y t by applying so max normalization. We nally obtain the training set NLL by simply averaging the sequence NLLs.

Phrase-based MT

PBMTs formulate the translation problem as a probability distribution which is now conditioned on the source side i.e. P (Y|X) [START_REF] Koehn | Statistical Phrase-based Translation[END_REF]. is conditional probability is further factorized into a "translation model (TM)" P (X|Y) and a "target LM" P LM (Y) component. In practice however, the factorization is o en expressed as a weighted loglinear model with weights λ i assigned to feature functions f i :

P (Y|X) = P (X|Y) P LM (Y) log (P (Y|X)) = N i=1 λ i f i (X, Y) + λ LM f LM (Y)
A feature function is a subcomponent that scores a given source-candidate pair with respect to a speci c aspect of the translation problem such as how well the words are aligned to each other. e TM component is essentially a feature function as well that estimates the likelihood of a target phrase given a source one, using the phrase table it constructs from the parallel training corpora. e LM component on the other hand is generally learned on a large, separate monolingual corpus in the target language so that it can be used to score the translation candidates with respect to their uency. e weights λ i are optimized (Och, 2003) to maximize the translation quality on a held-out development set using evaluation metrics such as BLEU [START_REF] Papineni | Bleu: A method for automatic evaluation of machine translation[END_REF]. Finally, the best translation Y * is obtained by searching through the hypothesis space of the model to satisfy the following:

Y * = arg max Y log (P (Y|X)) (3.4)
PBMTs are complex systems that incorporate many feature functions carefully designed by experts throughout years of research [START_REF] Koehn | Statistical Machine Translation[END_REF]. NMTs instead, propose to replace the whole pipeline used to estimate P (Y|X) with an end-to-end DNN that implicitly replaces the LM component as well.

Early Neural Approaches

Prior to purely end-to-end NMT models, there has been several a empts to couple DNNs and traditional MT systems. [START_REF] Schwenk | Continuous space translation models for phrase-based statistical machine translation[END_REF] proposed a DNN similar to NNLM [START_REF] Bengio | A neural probabilistic language model[END_REF] that estimates the phrase translation probabilities of a PBMT: the model projects all words in a source phrase into a continuous vector from which a joint distribution of words in the target phrase is estimated. e author provided empirical evidence that the system was able to provide meaningful phrase translations even for unseen source phrases. Another type of coupling exploited the distributional power of DNNs to rescore an n-best list i.e. a set of candidate translations obtained from a traditional MT system. e top candidate translation a er reordering is considered as the nal translation. e neural network joint model (NNJM) [START_REF] Devlin | Fast and robust neural network joint models for statistical machine translation[END_REF] is one such method that extends the NNLM by augmenting the n-gram target context with an m-gram source context X t :

P (Y|X) ≈ T t=1 P Y t | Y t-1 , . . . , Y t-n+1 , X t (3.5)
e model uses external word alignments in order to select the best possible source context window X t at each timestep t. e nal context vector is exactly formed as in NNLM i.e. by concatenating all embeddings related to the source and target contexts. e authors showed signi cant improvements over a state-of-the-art MT in Arabic-English and Chinese-English translation tasks.

Sequence-to-Sequence NMT

We de ne a sequence-to-sequence (S2S) NMT any neural system that reads a source sequence and then translates it into a target sequence. e rst a empt to S2S NMT came from [START_REF] Kalchbrenner | Recurrent continuous translation models[END_REF] where the authors proposed two di erent models which both utilize a CNN encoder and an RNNLM decoder. Unlike the ngram relaxation in equation 3.5, the RNNLM decoder here has access to full-context Y <t .

e rst model encodes a source sentence with a CNN to obtain a constant source context vector from which the RNNLM decodes the translation whereas the second one replaces the constant context with a dynamic n-gram one represented as convolutional feature maps. e results mostly focused on rescoring performance and on the sensitivity of the models to the source word order. Later on, Cho et al. (2014a,b) proposed a very similar S2S architecture to [START_REF] Kalchbrenner | Recurrent continuous translation models[END_REF] by replacing the CNN and the RNNLM components with their novel GRU layer (section 2. 5.1, p. 20). is model along with the concurrent work by [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF] are considered the rst successful encoder-decoder NMTs in the literature. I will now proceed with a detailed description of encoder-decoder NMTs since the multimodal architectures designed throughout this thesis are derivations of them.

Recurrent Encoder

A recurrent encoder encodes a given sequence X to a sequence of hidden states H by using a recurrent layer e() such as a plain RNN or a gated variant GRU (Cho et al., 2014b) or LSTM [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF]. For simplicity, the source embedding layer is also made part of the encoder so that a sequence X of one-hot encoded tokens is implicitly mapped to continuous token representations before further processing (section 2.5.2, p. 21). e following example illustrates the sequence of operations performed by the encoder ENC:

X = [A, WOMAN, PLAYS, TENNIS, <eos>] H = ENC (X, h 0 ← 0) = e (EMB (X) , h 0 ← 0) = e ([x 1 , . . . , x S ], h 0 ← 0) H = [h 1 , . . . , h S ]
e end of source sequences is explicitly tagged with an <eos> token so that the encoder can learn how sentences come to an end, which probably is useful in estimating the target sentence length during translation generation. e initial hidden state h 0 is o en set to 0 unless otherwise stated. Each produced encoding h i conveys information about the phrase processed so far up to that position including the token X i itself. We assume that an encoder always provides the full set of encodings H and we delegate the choice of source context type to the decoder.

Bidirectional Encoding

e RNNs process an input sequence from le -to-right in a unidirectional fashion. is means that the last hidden state is fully aware of the past context while the earlier ones have more and more limited context. In the limit, the rst hidden state h 1 has no access to any contextual information making it a mere word encoding. Bidirectional RNNs [START_REF] Schuster | Bidirectional recurrent neural networks[END_REF] propose a simple extension to unidirectional RNNs by sparing a dedicated RNN for right-to-le encoding. At a given timestep, the encoding h i now doubles its size by concatenating the le -to-right and right-to-le hidden states obtained from these two RNNs. By denoting the original and the reversed sequence with #» X and # » X respectively, the encodings produced by a bidirectional GRU encoder are given as follows:

H =     # » h 1 # » h 1   , . . . ,   # » h S # » h S     =   # » GRU ( #» X, # » h 0 ) # » GRU ( # » X, # » h 0 )  
From now on, all recurrent encoders are assumed to be bidirectional although we do not explicitly precise this in the equations and the gures for the sake of simplicity.

Recurrent Decoder

A recurrent decoder generates the target sequence one token at a time given the previous tokens Y <t (equivalent to Y ′ ≤t ) and a representation of the source sentence. Both Cho et al. (2014b) and [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF] propose to compress the whole source sentence into a constant context i.e. a single high dimensional vector c that does not evolve across decoding timesteps. Speci cally, Cho et al. (2014b) de nes the source context as a function of the last encoding (H -1 = h S ) as follows:

c = tanh (W c H -1 + b c )
Besides conditioning the decoder through its initial hidden state h ′ 0 , Cho et al. (2014b) also rede nes the GRU logic in the decoder so that c is concatenated to the hidden state at each timestep t. is ensures that the impact of c does not vanish across the recurrence in the decoder.

e following illustrates the sequence of operations performed by the decoder DEC. e input to the decoder is a time-shi ed version Y ′ of the true target sequence Y. Y ′ begins with the <bos> token to explicitly trigger a sentence start. e recurrent layer d() represented here is can be again any RNN variant:

Y = [UNE, FEMME, JOUE, AU, TENNIS, <eos>] Y ′ = [<bos>, UNE, FEMME, JOUE, AU, TENNIS] H ′ = DEC (Y ′ , h ′ 0 ← c) (3.6) = d (EMB (Y ′ ) , h ′ 0 ← c) H ′ = [h ′ 1 , . . . , h ′ T ]
Note how the model is consistently trained with the embeddings of the true previous tokens (equation 3.6), a technique called teacher-forcing [START_REF] Goodfellow | Deep Learning[END_REF]. When decoding translations however, the model has to receive its previous predictions since the true distribution is unknown. It has been shown that gradually exposing the model to its own mistakes -a technique called scheduled sampling -alleviates this problem and improves the performance [START_REF] Bengio | Scheduled sampling for sequence prediction with recurrent neural networks[END_REF].

Output Logic

Once the hidden states H ′ are obtained from the decoder, an output layer is used to project them into the size of the target vocabulary. is can be realized with a simple FC layer or a complex one such as the deep output [START_REF] Pascanu | How to construct deep recurrent neural networks[END_REF] used by Cho et al. (2014b). Although the hidden state h ′ t is already conditioned on the source context and the previous embedding intrinsically, deep output creates a residual link to the encoder and to the target embedding layer to alleviate possible vanishing gradients:

o t = tanh (V h h ′ t + V y y t-1 + V c c) (3.7) P (Y t | Y <t , X) = SOFTMAX (W o o t )
e nal linear transformation W o which projects the output o t to the size of the target vocabulary is generally considered a secondary embedding matrix referred to as output embeddings. If the size of the output vector o t is set to be equal to the size of a target word embedding, the two embedding layers in the decoder can be shared so that a single embedding matrix is learned for both purposes. is is called tied embeddings [START_REF] Inan | Tying word vectors and word classi ers: A loss framework for language modeling[END_REF][START_REF] Wolf | Using the output embedding to improve language models[END_REF] and shrinks down the number of parameters in an NMT substantially if the size of the vocabulary is very large. Figure 3.1 shows the complete computation graph from the encoder to the probability distribution P (Y t | Y <t , X). e training NLL is then computed as follows:

L(θ) = - 1 N N i=1 log(P (Y (i) )) = - 1 N N i=1 T t=1 log P (Y (i) t | Y (i) <t , X (i) )

Attention Mechanism

e NMT model described so far is limited in a way that it encodes arbitrarily long sentences into a single vector c. is bo leneck makes it hard for the model to come up with an encoding scheme that can encode both a very short and a very long sequence in an equally expressive way. In fact, Cho et al. (2014a) showed how the performance of the encoder-decoder NMT sharply decreases as the sentence length increases, unlike PBMT systems that are almost invariant to the sentence length. e a ention mechanism [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF] provides a nice solution to the problem by replacing the single vector c with a dynamic and time-dependent one c t . is allows the decoder to look at di erent portions of the source sentence as the decoding progresses. e authors show that the addition of the a ention mechanism combined with a bidirectional encoder mitigates the performance collapse that occurs as the sentences get longer. Today, stateof-the-art NMTs are equipped with a ention mechanisms between the encoder and the decoder and even in other components of the network (section 3.4.6, p. 40).

Formally, at each timestep t of the decoding process, the a ention mechanism receives the hidden state h ′ t of the decoder as a "query" vector and computes a relevance score between each encoding h i ∈ H and the query. e time-dependent context c t is then computed as the weighted sum of encodings where the weights are the normalized relevance scores that sum to one:

z i = SCORE (h i , h ′ t ) (3.8) α = [α 1 , . . . , α S ] ⊤ = SOFTMAX [z 1 , . . . , z S ] ⊤ c t = Hα = S i α i h i
Two common methods exist for computing the relevance scores: [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF] propose a parameterized FF layer while Luong et al. (2015b) simply use the dot product (Figure 3.2). In the context of NMT, both methods have been shown to perform equally well [START_REF] Britz | Massive exploration of neural machine translation architectures[END_REF]. e following illustrates both approaches at decoding timestep t using a query vector h ′ t and a single encoding h i . e linear transformations W e and W q are used to project the encoding and the query to a common space: We can now wrap all the underlying a ention computations into a layer function a() and modify the equation 3.7 to additively integrate the context into the output logic:

z i = SCORE (h i , h ′ t ) → w ⊤ a tanh(W e h i + W q h ′ t ) Bahdanau et al. (2014) → (W e h i ) ⊤ (W q h ′ t ) Luong et al. (2015b)
c t = a (H, h ′ t ) o t = tanh (O h h ′ t + O y y t-1 + O c c t )
In models with more than one recurrent layers, the context c t is o en propagated to the subsequent layers as the input. One such example is the Conditional GRU model that will be explained in the next section.

Conditional GRU Decoder

e conditional GRU (CGRU) implements a slightly di erent decoder logic with two GRU layers encapsulating the a ention mechanism (Sennrich et al., 2017). e recurrent hidden states of the GRUs are "transitional" in the sense that the previous hidden state of the second GRU is determined by the rst GRU. e second GRU then computes the new hidden state that becomes the previous hidden state of the rst GRU in the next timestep (Figure 3.3). e input to the second GRU is the context c t computed by the a ention layer. e following summarizes the CGRU NMT for the initial decoding timestep t = 1:

H = ENC (X, h 0 ← 0) h ′ 0 = tanh (W c {H -1 or H AVG } + b c ) (3.9) h ′ 1 = d 1 (EMB (<bos>) , h ′ 0 ) (3.10) c 1 = a (H, h ′ 1 ) h ′′ 1 = d 2 (c 1 , h ′ 1 ) o 1 = tanh (O h h ′′ 1 + O y EMB (<bos>) + O c c 1 ) P (Y 1 | <bos> , X) = SOFTMAX (W o o 1 )
h ′ and h ′′ denote the hidden states of d 1 and d 2 , respectively. Common choices when se ing h ′ 0 in equation 3.9 are the last (H -1 ) or the average (H AVG ) encoding although with a ention, we observe li le to none performance drop even it is set to 0.

Deep Models

e depth of an NMT model can be quanti ed by how many layers are used to process the source and target sequences. e main model described (Cho et al., 2014b) is a shallow NMT as both the encoder and the decoder consist of a single GRU layer. On the other hand, the model proposed by [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF] is a deep NMT as the encoder and the decoder are each constructed by stacking four LSTM layers. When many recurrent layers are stacked this way, each layer receives as input the set of encodings H produced by the previous layer except the rst layer which receives the input sequence X. e depth especially becomes an important factor for large-scale state-of-the-art deployments [START_REF] Johnson | Google's multilingual neural machine translation system: Enabling zero-shot translation[END_REF][START_REF] Gehring | Convolutional sequence to sequence learning[END_REF][START_REF] Vaswani | A ention is all you need[END_REF]. 2017) employ very deep FCNN encoders and decoders with a variant of a ention called "self-a ention". When applied on top of a set of hidden states, "self-a ention" computes the relevance of each one of them to the set of hidden states themselves. Since MT is a translation-variant problem, the lack of recurrent processing is o en remedied by explicitly encoding word positions through the use of special "positional embeddings".

Non-recurrent Approaches

Multitask Learning for NMT

Multitask Learning (MTL) [START_REF] Caruana | Multitask learning[END_REF]) is a learning paradigm where related tasks are trained in a parallel fashion. An MTL architecture generally passes through a common representation which is shared across the tasks and which encodes domain/modality relevant knowledge useful to improve nal generalization performance. [START_REF] Dong | Multi-task learning for multiple language translation[END_REF] successfully used MTL to learn a one-to-many NMT with a shared recurrent encoder and multiple target language decoders with dedicated a ention mechanisms. At training time, they form minibatches containing sentences from one language pair only and this language pair is randomly sampled at each iteration. In the end, the parameters of the shared encoder are always updated during the backward-pass while the decoders are selectively updated depending on the language pair considered. Luong et al. (2015a) further extended [START_REF] Dong | Multi-task learning for multiple language translation[END_REF] to many-to-one and many-to-many setups with tasks ranging from translation to captioning and parsing.

Evaluation of MT Outputs

Classical machine learning metrics such as precision, accuracy or recall are not directly applicable to sequence transduction problems where the output is a sequence of tokens. Although the gold standard for MT evaluation is manual evaluation, we need a cheap and easy way to approximately assess the quality of the obtained translations in order to evaluate, compare and select MT models. is is achieved by automatic metrics that measure the similarity between the machine generated translations and the reference sentences translated by human annotators. e most commonly used automatic metric in MT is BLEU [START_REF] Papineni | Bleu: A method for automatic evaluation of machine translation[END_REF] which is a document-level metric that computes the geometric mean of n-gram matching precisions (up to 4-gram precision in the default se ing) between the reference sentences and the MT outputs. Another metric often used for evaluating image captioning and multimodal machine translation systems is METEOR [START_REF] Lavie | Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments[END_REF][START_REF] Denkowski | Meteor universal: Language speci c translation evaluation for any target language[END_REF] which combines unigram precision and recall with an internal alignment mechanism between the words in reference and hypothesis sentences. Unlike document-level BLEU which is unreliable when used to evaluate individual sentences [START_REF] Song | BLEU deconstructed: Designing a be er MT evaluation metric[END_REF], METEOR is a sentence level metric by design that can also account for paraphrasing and synonyms for a set of languages including English, French and German. e highest achievable score for both metrics is 100.

Translation is a one-to-many problem in the sense that a single source sentence may have in nitely many acceptable human translations. e automatic metrics are by no means capable of fully handling such variabilites in the outputs but this can be achieved to some extent by using multiple reference sentences. Although BLEU and METEOR support multi-reference evaluation, very few datasets provide them for their test sets.

Manual Human Evaluation

Manual evaluation through human annotators is the primary evaluation method considered in the news translation shared task yearly held under the conference of machine translation (WMT). e type of manual evaluation preferred since 2017 [START_REF] Bojar | Findings of the 2018 conference on machine translation (wmt18)[END_REF] is called "direct assessment" (DA) (Graham et al., 2017) where human annotators are presented with an MT output and its associated reference and asked to score the quality of the translation using a [0, 100] scale. e collected annotations are then standardized within each annotator and then across all annotators to obtain an overall score for each system. A clustering based on signi cance test is nally performed to rank the systems.

Translation Decoding

Once an NMT model is trained, the translations for new sentences are generally decoded using the greedy search or the beam search [START_REF] Graves | Generating sequences with recurrent neural networks[END_REF][START_REF] Boulanger-Lewandowski | Audio chord recognition with recurrent neural networks[END_REF][START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF] algorithms. ese search algorithms iteratively explore the search space to nd the most likely translation for a given input, based on the loglikelihood estimate of the model. In this thesis, I always use the beam search which, given an input, proceeds as follows: the decoding rst starts with an empty hypothesis. At timestep t = 1, we expand the empty hypothesis with every possible word in the target vocabulary T resulting in a list of |T| partial hypotheses. is list of partial hypotheses is called the beam. Before reiterating the same procedure for t > 1, beam search computes the log-likelihood of each hypothesis in the beam and prunes the beam to top k most likely hypotheses. e search stops when the <eos> token is generated for all k hypotheses in the beam. e size of the beam k is a predetermined hyperparameter usually ranging between 2 and 20. e greedy search is a special case of the beam search where only the most likely hypothesis is kept (k = 1) at each iteration.

Ensembling

Ensembling is a technique that allows averaging the predictions of an arbitrary number of models during the inference step. In the context of DNNs, training the same model multiple times with di erent random initializations and averaging their decisions o en leads to substantial performance improvements. A common way of ensembling in NMT is to run the beam search algorithm on a set of trained models in a synchronized way and sum their log-likelihoods at each decoding step t. [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF] demonstrated that this improves over their single best NMT by 2.7 and 4.2 BLEU scores for a two-model and a ve-model ensemble, respectively.

Summary

In this chapter, I introduced the task of machine translation along with the prominent approaches currently used in the eld such as PBMTs and NMTs. I speci cally focused on the la er as it is the fundamental framework that our multimodal translation approaches will be based on. A er explaining in detail each component of NMTs such as the encoder, the decoder and the a ention mechanism, I brie y described the beam-search algorithm and the commonly used translation evaluation methods. We now have the necessary background to start discussing multimodal machine translation.

CHAPTER4

Multimodal Machine Translation

Human beings interact with their surrounding world mostly through visual, auditory and tactile sensory modalities. Language is o en communicated over these sensory channels and perceived as a visual, auditory and tactile stimuli when looking at a word depicted in a tra c sign, listening to a conversation or reading a book wri en using the Braille system, respectively. Besides being able to handle each sensory modality in an isolated way, humans also develop a complex ability of integrating multiple modalities for e cient perception and decision making [START_REF] Stein | e neural basis of multisensory integration in the midbrain: Its organization and maturation[END_REF], including uncertainty reduction [START_REF] Ernst | Humans integrate visual and haptic information in a statistically optimal fashion[END_REF]. Computational language understanding also benets from multimodality in ways similar to human perception. Silberer and Lapata (2012) showed that for semantic tasks such as word association and similarity, the joint modeling of linguistic and perceptual information correlates with human judgments be er than late fusion of independent representations. Recent a empts at audio-visual speech recognition are forms of uncertainty reduction where noisy speech u erances are successfully transcribed by lip-reading from the video stream [START_REF] Joon | Lip reading sentences in the wild[END_REF].

It is not a surprise that language understanding is at the heart of MT which requires inferring the meaning of a sentence in one language and transferring that meaning to another language. State-of-the-art approaches in NMT successfully leverage the distributional hypothesis [START_REF] Rupert | A synopsis of linguistic theory 1930-1955[END_REF] through the use of word embeddings and achieve meaning induction abilities solely by being exposed to large amounts of parallel sentences. Rios [START_REF] Gonzales | Improving word sense disambiguation in neural machine translation with sense embeddings[END_REF] show that without any kind of explicit supervision, an outof-the-box NMT is able to reach an accuracy of 70% for a word sense disambiguation (WSD) task in two di erent languages. However, there are many situations where purely distributional evidence is not su cient to correctly translate a sentence. Consider the case where the translation of a sentence depends on the resolution of an anaphora with the antecedent being in the previous sentence or translating from a gender-neutral language to another one that has grammatical gender. Contextual (or large-context) MT is speci cally interested in solving the former problem by integrating cross-sentence information from neighboring sentences, paragraphs or even external linguistic resources [START_REF] Tiedemann | Neural machine translation with extended context[END_REF][START_REF] Voita | Context-aware neural machine translation learns anaphora resolution[END_REF][START_REF] Bawden | Going beyond the sentence : Contextual Machine Translation of Dialogue[END_REF]. e grammatical gender problem however, can be solved by neither a human nor an MT system1 without any additional context. What is worse for the MT system is how its word choices would be a ected by the intrinsic gender bias of the training set [START_REF] Marcelo | Assessing gender bias in machine translation: a case study with google translate[END_REF], a major concern for language understanding methods based on word embeddings [START_REF] Bolukbasi | Man is to computer programmer as woman is to homemaker? debiasing word embeddings[END_REF][START_REF] Caliskan | Semantics derived automatically from language corpora contain human-like biases[END_REF].

Multimodal machine translation (MMT) aims to provide a generic framework where the translation task is supported by auxiliary modalities such as vision and/or audio. Besides the aforementioned ambiguity issues, the successful integration of additional modalities can also be useful to improve the robustness of the system to noise, which can manifest itself as spelling mistakes or missing input words. e research e orts in MMT has so far been conducted on the Multi30K dataset [START_REF] Ellio | Multi30k: Multilingual english-german image descriptions[END_REF] which contains multilingual image descriptions and their translations. A yearly evaluation campaign has been held around the dataset to foster research on MMT [START_REF] Specia | A shared task on multimodal machine translation and crosslingual image description[END_REF]Ellio et al., 2017;[START_REF] Barrault | Findings of the third shared task on multimodal machine translation[END_REF]. In this chapter, I rst introduce the dataset and the evaluation campaign and then provide a review of the current state-of-the-art in MMT. I also brie y describe our contributions -which will be further detailed in next chapters -and conclude with a quantitative comparison of the described approaches in terms of MT evaluation metrics.

Multi30K Dataset

Multi30K [START_REF] Ellio | Multi30k: Multilingual english-german image descriptions[END_REF] is currently the prominent dataset used for MMT research.

e dataset is derived from the Flickr30K dataset [START_REF] Young | From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions[END_REF] of image descriptions where ve English descriptions were crowd-sourced for each of the 31014 images. In order to construct a parallel translation corpus with associated images, one of the ve descriptions was professionally translated to German by human translators [START_REF] Ellio | Multi30k: Multilingual english-german image descriptions[END_REF]. Although the translators were originally given the English sentence without the image, [START_REF] Frank | Assessing multilingual multimodal image description: Studies of native speaker preferences and translator choices[END_REF] later collected "image-aware" post-edits from another human translator for the development and test set references. e dataset is later extended to include French (Ellio et al., 2017) and Czech [START_REF] Barrault | Findings of the third shared task on multimodal machine translation[END_REF] translations, leading to 31014 English→German, English→French and English→Czech translation pairs with English sentences shared across all pairs. Unlike the original German translations, the French and Czech annotators were also given the described images as a visual cue.

Shared Task on MMT

Multi30K is the primary training resource provided by the shared task on MMT, which is an evaluation campaign held under the Conference of Machine Translation (WMT) between 2016 and 2018 [START_REF] Specia | A shared task on multimodal machine translation and crosslingual image description[END_REF]Ellio et al., 2017;[START_REF] Barrault | Findings of the third shared task on multimodal machine translation[END_REF]. Each year, a new language pair was added to the o cial evaluation campaign leading to three independent MMT subtasks in 2018, namely, English→German, English→French and English→Czech (Figure 4.1). A multimodal, multilingual subtask was also proposed in 2018 with the aim of designing a many-to-one MT system that considers the image and its English, French and German descriptions to perform translation into Czech. e use of additional resources such as MT and image captioning datasets is o en encouraged and the submissions that use them are tagged as "unconstrained". At the end of the submission period, all participating systems are evaluated with METEOR and BLEU (section 3.5, p. 41) with METEOR being the primary metric. In 2017 and 2018, a human evaluation (section 3.5, p. 41) was also conducted using the direct assessment approach extended with the described images. In this thesis, we are solely interested in "constrained" English→German and English→French tasks.

Test Sets

Each year, a new test set is published to evaluate the performance of participating systems on unseen data. A er the evaluation period, the references of the new test sets are disclosed so that researchers are able to evaluate their systems on them as well. One exception to that is the latest test2018 set which is kept undisclosed for continuous MMT evaluation through an online competition server2 . Apart from the yearly test sets, a test set called testcoco was published as a more challenging secondary test set in 2017 (Ellio et al., 2017). testcoco contains 461 carefully selected image-sentence pairs that potentially include ambiguous verbs having multiple senses. Speci cally, it contains one to three samples per sense per verb for 56 verbs in total. For example, the following two senses of the English verb "to pass" require di erent verbs when translating to French: "a vehicle passing (dépasser) another vehicle" and "a vehicle passing (traverser) over a bridge". A -visually grounded -verb sense disambiguation can be helpful when translating this test set.

Dataset Statistics

I provide several sentence level and corpus level statistics for the English, German and French sentences of Multi30K in Table 4.1. ese statistics are collected on tokenized and lowercased sentences, following the experimental framework of the thesis (section 5.2, p. 59). We notice that the sentences are quite short containing ∼14 words on average across all languages. e descriptive nature of the sentences turns out to be a limiting factor in terms of syntactic and semantic diversity: 16.7% and 7.2% of English training set sentences start with the bigram "a man" and "a woman", respectively. In overall, with only 29K sentences available for training, the dataset is smaller (and also simpler in terms of sentence structure) than commonly used MT datasets ranging from hundreds of thousands to hundreds of millions of sentences [START_REF] Bojar | Findings of the 2018 conference on machine translation (wmt18)[END_REF]. Table 4.2 provides OOV statistics for Multi30K test sets. With its compound words and rich morphology, it is not surprising that the German test sets are the most a ected ones with more than 1/3 of test2017 and testcoco sentences containing at least one OOV word.

State-of-the-art in MMT

In this section, I review the current state-of-the-art in MMT by categorizing the approaches with respect to the type of visual features they integrate. First, I start with the models that make use of the global visual features and then move on to MMTs that incorporate convolutional (spatial) features. Although the main focus will be on neural approaches, prominent non-neural works are also described to some extent. Regardless of the type of feature involved, the majority of neural MMTs are inspired by previous works in neural image captioning (NIC) [START_REF] Mao | Deep captioning with multimodal recurrent neural networks (m-rnn)[END_REF][START_REF] Vinyals | Show and tell: A neural image caption generator[END_REF]Xu et al., 2015b). Unless otherwise stated, all models use features extracted from CNNs trained on ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF] image classi cation task (section 2.6.3, p. 27).

Global Visual Features

Although global features are spatially unaware and highly optimized for the initial task that they were trained for, notable works in NIC [START_REF] Mao | Deep captioning with multimodal recurrent neural networks (m-rnn)[END_REF][START_REF] Vinyals | Show and tell: A neural image caption generator[END_REF] successfully leveraged these features to generate natural language descriptions for images (Figure 4.2). Consequently, this type of feature turned out to be a ractive in MMT research as well, where they have been shown to be bene cial to some extent.

Grounded Encoders & Decoders

e simplest way of leveraging the visual information in NMT consists of conditioning the encoders and/or the decoders with visual feature vectors: [START_REF] Calixto | DCU-UvA multimodal MT system report[END_REF] and [START_REF] Libovický | CUNI system for WMT16 automatic post-editing and multimodal translation tasks[END_REF] use 4096-dimensional FC 7 features extracted from a VGG CNN [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] to initialize the hidden state of the recurrent decoder. Many extensions and re nements have been further proposed by concurrent works in 2017: [START_REF] Ma | OSU multimodal machine translation system report[END_REF] initialize both the encoder and the decoder with 2048-dimensional "average pooled" feature vector of ResNet-50 [START_REF] He | Deep residual learning for image recognition[END_REF] while Madhyastha et al. ( 2017) draw a comparison between the "average pooled" feature vector and the 1000dimensional nal probability vector of ResNet-152 [START_REF] He | Deep residual learning for image recognition[END_REF] in similar encoderdecoder initialization scenarios. e authors also experiment with additive interaction between the feature vector and the source embeddings and nd out that the probability vectors perform slightly be er than the "average pooled" ones. [START_REF] Zheng | Ensemble sequence level training for multimodal MT: OSU-Baidu WMT18 multimodal machine translation system report[END_REF] revisit decoder initialization and apply reinforcement learning techniques to ne-tune the model parameters with the objective of directly maximizing the BLEU score. ey report that when combined with scheduled sampling (section 3.4.2, p.35), the ne-tuning yields BLEU improvements for NMT but the gains do not apply to MMT.

A slightly di erent grounding method is proposed by [START_REF] Huang | A entionbased multimodal neural machine translation[END_REF] which consider the global feature vector as a "visual token" that can be prepended (or appended) to the sequence of source word embeddings. is implicitly allows the language a ention mechanism to a end to visual information as they are made part of the source sequence. In essence, the proposed method is nothing more than a reiteration of [START_REF] Vinyals | Show and tell: A neural image caption generator[END_REF] (Figure 4.2) at encoder side. ey further extend their approach by feeding the full image to a pre-trained object detection CNN [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF] to get four region proposals (i.e. bounding boxes) that contain salient objects. In addition to the global feature vector extracted from the full image (using a VGG CNN), they extract four more feature vectors for the proposed regions. A total of ve visual vectors are then prepended to the source embedding sequence. In a similar vein, Calixto and Liu (2017) and Calixto et al. (2017a) simultaneously prepend and append the visual feature vector to the source sequence to ensure that the bidirectional encoder always processes the image as the rst element.

ey also combine this with encoder and/or decoder initialization. Finally, [START_REF] Grönroos | MeMAD submission to the WMT18 multimodal translation task[END_REF] experiment with RNN and Transformer [START_REF] Vaswani | A ention is all you need[END_REF] based NMTs by incorporating the visual feature in many ways such as prepending it, multiplying the embeddings with it (Caglayan et al., 2017a) or using it as a gate before the output layer to visually modulate the probability distribution over target words. More interestingly, they explore global visual features extracted from many di erent CNNs trained for scene recognition, action recognition and object detection. However, they obtain li le to none improvement from the visual modality and discover that when the models are given a mean feature vector for every sample, the translations do not deteriorate.

Multi-task Learning (MTL). A radically di erent encoder grounding technique is the Imagination [START_REF] Ellio | Imagination improves multimodal translation[END_REF] which is a one-to-many MTL architecture that shares the sentence encoder across a translation task T and a visual prediction task V .

e la er aims to reconstruct the global visual feature vector from the "average pooled" source sentence encoding H AVG using a non-linear FC layer [START_REF] Chrupała | Learning language through pictures[END_REF]. A margin-based loss is used for the visual task to minimize the cosine distance between the true feature vector f and its reconstruction f while pushing away the la er from the "contrastive" global features sampled from the rest of the minibatch. e MTL loss is de ned as the convex combination of the NMT loss and the visual loss:

J = λ L T + (1 -λ)L V L V = f ′ =f max 0, α -distance( f , f ) + distance( f , f ′ ) f = tanh(W v H AVG + b v )
e model is exible in the sense that each task can be independently pre-trained on external resources and plugged into the model a erwards. Moreover, the visual features are not needed at test time as they are only used during training for grounding the shared encoder. e authors report improvements over their baseline especially in the constrained setup but when the NMT is pre-trained with additional data, the improvements do not seem to hold. Later on, [START_REF] Helcl | CUNI system for the WMT18 multimodal translation task[END_REF] apply the same idea to a Transformer based NMT and show slight improvements over their baseline. Finally, [START_REF] Zhou | A visual a ention grounding neural model for multimodal machine translation[END_REF] incorporate an auxiliary a ention mechanism over the source sentence where the visual feature vector is used as the query to the a ention. e margin-based loss now minimizes the distance between the true feature vector f and the output of the newly added a ention layer instead of the reconstructed feature vector as in the original formulation.

Other Approaches

In this section, I brie y describe hybrid approaches based on reranking, retrieval and system combination. ese approaches are o en multi-stage in the sense that they consist of multiple submodels a ached together in di erent ways. One of the earliest reranking based approaches is [START_REF] Shah | SHEF-Multimodal: Grounding machine translation on images[END_REF] where the authors train a PBMT system and integrate the 1000-dimensional probability vector extracted from a CNN as additional scores for reranking 100-best list of translation hypotheses. Speci cally, they consider each probability in the vector as a feature function for which a coe cient is estimated during the tuning step. eir choice of probability features is motivated by the hypothesis that using likelihood of ImageNet objects appearing in the image may be more helpful than the penultimate layer features for MMT. is visual reranking yields very slight improvements over their PBMT baseline. In a more recent work, [START_REF] Lala | She eld submissions for WMT18 multimodal translation shared task[END_REF] show that the 20-best translation candidates obtained from an NMT system actually contain high quality translations that potentially allow 10% absolute METEOR improvement. In order to select these candidates, they design a novel multimodal WSD system based on ResNet-50 global visual features and rerank their n-best list of translation candidates with scores assigned by the WSD system. However, they conclude that the Multi30K dataset do not signi cantly bene t from the proposed approach.

As for the retrieval based approaches, [START_REF] Duselis | AFRL-OSU WMT17 multimodal translation system: An image processing approach[END_REF] and [START_REF] Gwinnup | e AFRL-Ohio State WMT18 multimodal system: Combining visual with traditional[END_REF] consider the image as the driving modality for MMT instead of the language input. To this end, they train an image captioning system to generate candidate captions in the target language for each image. ey utilize two encoders based on pre-trained FastText word embeddings [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF] to encode a source sentence and the candidate target captions obtained from the captioning system. A er learning a mapping function between the source sentence space and the target caption space, they retrieve the target caption closest to the source caption in the learned mapping space. Finally, [START_REF] Zhang | NICT-NAIST system for WMT17 multimodal translation task[END_REF] propose a combined way of using retrieval and reranking. For a given sentenceimage pair, they rst retrieve a set of similar images from the training set based on the euclidean distance between the global visual features. e target sentences associated with the retrieved images are considered as candidate translations. ey learn a visually guided word-to-word alignment function between source words and the candidate target words and use this function to select the most probable target word for each source word in the sentence. e 10K-best list of their PBMT is reranked with scores provided by a bidirectional NMT which receives the concatenation of the source words and the aligned target words. e authors report that pure reranking substantially improves the translation scores but the multimodal candidate word selection method shows no bene t.

Spatial Features

We now turn our a ention to the second line of work in MMT that aims to integrate convolutional features into NMT. Unlike global features which provide a single vectorial representation, the spatial axis of convolutional features has the potential to allow an evolving integration scheme that ts within the iterative nature of encoders and/or decoders. However, these features are relatively less explored than global ones for MMT probably because of the challenges behind the design of multimodal fusion strategies that can take into account their representational complexity in an e cient way.

Decoder-based Multimodality

Inspired from the success of visual a ention in image captioning (Xu et al., 2015b), the majority of the previous work in "spatial MMTs" consider extending the a entive NMT with an auxiliary visual a ention mechanism. A "black box" depiction is given in Figure 4.3 where the top and bo om parts correspond to visual and language a ention mechanisms, respectively. To this end, [START_REF] Calixto | DCU-UvA multimodal MT system report[END_REF] propose to extend the CGRU decoder (section 3.4.4, p. 38) with a visual a ention layer a V that receives the spatial features V. e nal context c 1 -which becomes the input to the second GRU -is then de ned as the concatenation of language and visual contexts. e following summarizes the multimodal CGRU for the rst decoding timestep:

c 1 = c L 1 c V 1 = a L (H, h ′ 1 ) a V (V, h ′ 1 ) (4.1)
Later on, Calixto et al. (2017b) apply the "gating scalar" from Xu et al. (2015b) with the purpose of scaling the visual context vector c V based on the hidden state of the decoder:

β = σ (W g h ′ 1 + b g ) c V 1 = β a V (V, h ′ 1 )
Although both models perform equally well compared to their baseline, the authors report that the la er model learns to activate the gate for visually depicted concrete nouns. [START_REF] Libovický | A ention strategies for multi-source sequence-to-sequence learning[END_REF] propose two multimodal a ention variants, namely, the " at" and the "hierarchical" a ention. e at a ention combines the textual and the visual encodings along the time and spatial axes to form a at multimodal sequence

M = H ∈ R S×c ; V ∈ R K×c where M ∈ R (S+K)×c .
Here, S denotes the number of words in the source sentence while K would be 64 for convolutional features with 8x8 spatial resolution. is new sequence replaces the text-only input to the a ention layer originally found in the CGRU decoder. e nal context is then given by c 1 = a (M, h ′ 1 ). On the other hand, the hierarchical a ention follows the dedicated a ention formulation of [START_REF] Calixto | DCU-UvA multimodal MT system report[END_REF] but instead of concatenating the individual contexts, it utilizes a new "hierarchical" a ention layer a H on top:

c L 1 = a L (H, h ′ 1 ) c V 1 = a V (V, h ′ 1 ) c 1 = a H c L 1 ; c V 1 , h ′ 1
e authors show that the hierarchical a ention performs be er than the at one although it can not surpass their baseline.

Finally, with Transformer NMTs (TFNMT) [START_REF] Vaswani | A ention is all you need[END_REF] becoming more and more popular, researchers started to explore the integration of spatial features into the TFNMT decoder as well. Arslan et al. ( 2018) extend the decoder with the separate a ention mechanism of [START_REF] Calixto | DCU-UvA multimodal MT system report[END_REF] and fuse the obtained modality contexts with addition instead of the concatenation (Equation 4.1) while [START_REF] Libovický | Input combination strategies for multi-source transformer decoder[END_REF] integrate their previous at and hierarchical a ention and propose two more variants, namely, the "parallel" and the "serial" a ention. e parallel a ention closely follows Arslan et al. ( 2018) while the serial one applies the language and the visual a ention in a stacked way where the former produces the query vectors for the la er. Arslan et al. ( 2018) substantially improve over their baseline TFNMT in terms of BLEU but strangely report a very poor METEOR score. For [START_REF] Libovický | Input combination strategies for multi-source transformer decoder[END_REF], the parallel a ention works best with moderate improvements over their baseline for all three translation pairs.

Encoder-based Multimodality

Besides the commonly explored decoder-based multimodal strategies, two encoder-based methods exist in the literature (Delbrouck and Dupont, 2017a). e rst one modulates the batch normalization (BN) layer [START_REF] Io | Batch normalization: Accelerating deep network training by reducing internal covariate shi[END_REF] of the ResNet CNN which is used to extract the visual features. e BN layers are o en placed a er the convolutional layers to standardize the previous activations to zero mean and unit variance. At the same time, the layer also learns to rescale and reshi the normalized activations. Delbrouck and Dupont (2017a) propose to intervene at this speci c step by injecting tiny variations to the learned mean and variance of a BN layer where the variations are driven by the mean source sentence encoding H AVG . is has the impact of modulating the feature maps extracted from the CNN in a learnable way where each feature map can be a enuated or ampli ed based on the source sentence. e reported results suggest that the method performs slightly inferior to the winning system from MMT17 (Caglayan et al., 2017a). Unfortunately, they do not compare the results with respect to their underlying baseline.

e second method couples the visual a ention with the sentence encoder where the visual context is computed using the bidirectional hidden states. e visual contexts are then fused with the bidirectional states to yield a set of multimodal encodings. e CGRU decoder then applies its original a ention layer on top of the new multimodal encodings.

e authors only provide results for the combination of the rst method above and this method where the performance slightly surpasses the same MMT17 system.

Multimodal Fusion

e models presented so far employ addition, concatenation or a hierarchical a ention in order to fuse the individual contexts into the nal multimodal one. Delbrouck and Dupont (2017b) take a di erent approach and apply multimodal compact bilinear pooling (MCBP) [START_REF] Fukui | Multimodal compact bilinear pooling for visual question answering and visual grounding[END_REF] which is an e cient realization of the computationally expensive outer product. Assuming that the individual context vectors have the same dimensionality c, the outer product of two vectors c L 1 and c V 1 is a c × c matrix composed of elementwise multiplication of every element of the rst vector with every element of the second. If one would like to project this matrix back into a c-dimensional space in order to feed it into the second GRU for example, the number of parameters in that layer (c 3 ) quickly reaches hundreds of millions. MCBP approximates this operation efciently, showing notable improvements for visual question answering (VQA). In the context of MMT however, Delbrouck and Dupont (2017b) show that although MCBP seems to improve over concatenation (Equation 4.1), it is inferior to a simple elementwise multiplication between the contexts i.e.

c 1 = c L 1 ⊙ c V 1 .

Our Contributions

I now brie y describe the contributions of this thesis by drawing parallels to the stateof-the-art. In Caglayan et al. (2016a), we simultaneously explore a reranking method and an end-to-end MMT approach. For reranking, we train a PBMT, a recurrent NMT and an NLM conditioned on global visual features [START_REF] Aransa | Improving continuous space language models using auxiliary features[END_REF]. e scores provided by the NMT and the visual NLM are used to rerank the 1000-best list of translation candidates obtained by the PBMT. e visual NLM produces a single LM score per candidate unlike the concurrent work of [START_REF] Shah | SHEF-Multimodal: Grounding machine translation on images[END_REF] where each element of the visual feature vector is considered as an independent feature function. With slight gains over the baseline PBMT, the proposed model ranked rst in MMT16 campaign [START_REF] Specia | A shared task on multimodal machine translation and crosslingual image description[END_REF]. We do not further detail this approach but present it as a baseline whenever we provide a quantitative comparison across the state-of-the-art models. For the end-to-end MMT approach, we experiment with spatial features and propose the "multimodal a ention" for the rst time, concurrently with [START_REF] Calixto | DCU-UvA multimodal MT system report[END_REF]. We speci cally explore a shared multimodal a ention in contrast to their dedicated version. Later on, we extend our multimodal a ention approach with di erent levels of sharing along with two multimodal fusion techniques, namely, the addition and the concatenation (Caglayan et al., 2016b). Finally, we propose several other re nements in Caglayan et al. (2018) where we mainly show that feature normalization is crucial for the visual a ention to work correctly.

Chapter 7 details the multimodal a ention experiments and provides quantitative and qualitative analyses using up-to-date models.

As for the global visual feature based MMTs, in Caglayan et al. (2017a) we explore several interaction methods within the framework of recurrent NMTs. Speci cally, we start by replicating the RNN initialization techniques (Calixto and Liu, 2017) and then propose novel interaction schemes primarily based on elementwise multiplication of the visual features with several intermediate language representations of the NMT system. Our English→German and English→French submissions to MMT17 evaluation campaign (Ellio et al., 2017) ranked rst with respect to automatic metrics. Moreover, our German system ranked rst in human evaluation by signi cantly surpassing other submissions. We extensively cover these methods in Chapter 6 and provide quantitative and qualitative analyses again with up-to-date retrained models.

Finally, following the source degradation protocols that we introduce in Caglayan et al. (2019a), we conduct several probing experiments in Chapter 8 to shed a light on the visual awareness of our MMTs, as well as on the need for visual grounding in the context of Multi30K.

Type

Feat English→German. e highlighted system is unconstrained. e systems marked with ( †) are re-evaluated with tokenized sentences. e descriptions refer to the techniques previously mentioned in this section.

antitative Comparison

I nalize this section with a quantitative overview of the current state-of-the-art in MMT for English→German test2016 set as this is by far the most commonly used setup to report automatic metrics in literature. ere exists an unfortunate discrepancy between the scores reported in MMT16 papers and the ndings report [START_REF] Specia | A shared task on multimodal machine translation and crosslingual image description[END_REF] as the o cial evaluation for was performed using detokenized sentences. To synchronize the results across systems, I downloaded the submissions for MMT16 systems, tokenized and re-evaluated them accordingly. is results in an increase of around 2.5 and 4.5 points in BLEU and METEOR, respectively. Table 4.3 reports the nal BLEU and METEOR scores for constrained systems in the literature along with the best unconstrained MMT18 submission [START_REF] Grönroos | MeMAD submission to the WMT18 multimodal translation task[END_REF] that may be considered as an upper bound. Although we leave the detailed analyses to the upcoming chapters, we can say that the results do not seem to suggest a distinctive boundary between the performance of global and spatial features.

Summary

In this chapter, I introduced the motivations behind MMT, described the closely associated Multi30K dataset, and provided an overview of the state-of-the-art. I broadly categorized the approaches into two groups based on the type of visual features they incorporate i.e. global visual features and spatial features. A er brie y describing our contributions to MMT -that will be detailed in chapters 6, 7 and 8 -I summarized the current state-of-the-art in terms of automatic metrics. e next chapter details the common hyperparameters, the pre-processing work ow and the baseline NMT that will be extensively used throughout the remaining chapters.

CHAPTER5

Experimental Framework roughout the course of MMT evaluation campaigns, we have progressively tuned our models each year to start with competitive baselines in the rst place. Besides that, there has also been many changes in the way we have pre-processed the textual data and the CNN that we have used to extract visual features. is evolution makes it quite impossible to fairly compare our models among themselves and also to the current stateof-the-art. For this reason, the following chapters will present both the results obtained from the yearly evaluation campaigns and up-to-date results from systems speci cally trained for this thesis. e la er systems use the same visual features, hyperparameters and pre-processing pipeline in order to ensure be er comparability. In this chapter, I describe the experimental framework in detail and introduce the baseline NMT model on top of which the next chapters will be based on.

So ware

As part of this thesis, I developed a high-level DNN Toolkit in Python called nmtpy with a focus on training language and vision related modalities and multimodal tasks. e rst version of the toolkit (Caglayan et al., 2017b) was derived from the popular dl4mt1 codebase and relied upon eano ( eano Development Team, 2016) as the backend framework. e current version2 which I extensively use in this thesis, is based on Py-Torch [START_REF] Paszke | Automatic di erentiation in PyTorch[END_REF] framework. Although the fundamental model in nmtpy is the a entive NMT with CGRU decoder (Sennrich et al., 2017) Currently, nmtpy provides support for handling text les, arbitrary feature vectors, raw images and speech features in Kaldi format. As for the model inventory, it provides reference implementations for all the simple (chapter 6) and a entive MMT models (chapter 7) as well as a state-of-the-art speech recognition model and its multimodal extension (Caglayan et al., 2019b).

Besides my own works in MMT, nmtpy has also been successfully used by other researchers primarily for machine translation [START_REF] Burlot | Word representations in factored neural machine translation[END_REF][START_REF] García-Martínez | Lium machine translation systems for WMT17 news translation task[END_REF][START_REF] Lala | She eld submissions for WMT18 multimodal translation shared task[END_REF] and also for multimodal summarization [START_REF] Libovickỳ | Multimodal abstractive summarization of opendomain videos[END_REF], phonemic transcriptions for text-to-speech [START_REF] Vythelingum | Acoustic-dependent phonemic transcription for text-to-speech synthesis[END_REF] and audio-visual dialog state tracking [START_REF] Sanabria | CMU Sinbad's submission for the DSTC7 AVSD challenge[END_REF]. e tool was also extensively used and developed by the "Grounded Sequence to Sequence Transduction" research group3 within the Fi h Frederick Jelinek Memorial Summer Workshop in 2018.

Pre-processing

Image Features

We use a pre-trained ResNet-50 CNN [START_REF] He | Deep residual learning for image recognition[END_REF] provided by torchvision4 to extract visual features. For pre-processing the images, we resize the shortest edge to 256 pixels and then take a center crop of size 256x256. We extract spatial features of size 8x8x2048 from the nal convolutional layer (res5c relu) of the CNN. ese spatial features are also the ones used to obtain the global 2048D avgpool features. In contrast, the shared task provides 14x14x1024 spatial features extracted from the second to last convolutional layer (res4f relu) using images of size 224x224.

In chapter 6, we directly use normalized global features i.e. the L 2 norm of each feature vector is normalized to 1. In chapter 7, we provide an analysis of the impact of L 2 normalization and detail the experimental procedure there.

Text Processing

We use Moses [START_REF] Koehn | Moses: Open source toolkit for statistical machine translation[END_REF] scripts to lowercase, normalize and tokenize the sentences with aggressive hyphen spli ing (-a parameter). For subword experiments, we use the BPE [START_REF] Sennrich | Neural machine translation of rare words with subword units[END_REF] algorithm (section 2.5.2, p.23) to create subword level vocabularies. For each language pair involved, we train a joint BPE model on the concatenation of the source and target training sentences. e number of merge operations is set to 10K for all language pairs.

Training & Evaluation

e set of common hyperparameters used throughout the thesis are given in Table 5.1. All models are trained for a maximum of 100 epochs. e model performance is evaluated at the end of each epoch based on METEOR score [START_REF] Denkowski | Meteor universal: Language speci c translation evaluation for any target language[END_REF] of the val set of Multi30K. If the METEOR score does not improve for ten epochs, the training is early-stopped (section 2.4.4, p. 14). In a similar way, the learning rate is halved if no improvement occurs for three consecutive epochs. We do not x the seed of the random number generator and train all models three times with di erent random initializations. Once the training is over, we decode test set translations from each run separately using the beam search algorithm with a beam size of 12. Prior to evaluation, we recover all segmentation artifacts including the hyphen spli ing and the BPE in order to ensure comparability across systems. We use the multeval tool [START_REF] Clark | Be er hypothesis testing for statistical machine translation: Controlling for optimizer instability[END_REF] tokenized BLEU and METEOR scores along with their means and standard deviations across three runs. We also rely on multeval to report statistical signi cance of the systems with respect to a designated baseline.

Baseline NMT

We conduct a preliminary experiment to select our baselines for English→German and English→French. Speci cally, we test four systems that use word and subword vocabularies and report average BLEU and METEOR scores over three runs in Table 5.2. We observe that WRD→BPE systems with word-level source tokens and subword-level target tokens outperform other systems. Based on this, we select this system as the baseline architecture for the upcoming MMT experiments.

CHAPTER6

Simple Multimodal Machine Translation is chapter describes our simple MMT (SMMT) architectures that extend S2S NMTs by incorporating global visual features. ese features are generally extracted from stateof-the-art CNNs (section 2.6.3, p. 27) primarily trained for large-scale vision tasks such as ImageNet image classi cation task [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF]. Global features can be thought as continuous bag of "latent concepts" where a linear layer applied on top, successfully classi es a given image into one of the thousand object categories. Although these vectors are highly tuned for the primary task that they were trained for, they were also showed to be e ective for language related tasks such as bilingual lexicon induction [START_REF] Kiela | Visual bilingual lexicon induction with transferred convnet features[END_REF] and image captioning [START_REF] Mao | Deep captioning with multimodal recurrent neural networks (m-rnn)[END_REF][START_REF] Vinyals | Show and tell: A neural image caption generator[END_REF]. erefore, the majority of state-of-the-art MMTs (section 4.2.1, p. 47) rely on global features that are compact and thus easy to integrate into existing NMTs.

Our proposed SMMTs can be broadly divided into two categories: (i) initializing the sentence encoders and/or decoders similar to Calixto and Liu (2017), (ii) interacting the visual features and the intermediate language representations in the network in novel ways. We train the models on Multi30K dataset, following a xed set of hyperparameters (Table 6.1) and the pre-processing pipeline previously described (section 5, p. 57). Finally, we conduct a quantitative analysis for English→German and English→French translation directions using corpus level and sentence level automatic evaluation and compare our systems to the current state-of-the-art in MMT. e chapter comprises the following work as well as unpublished extensions to it:

Methods

We rst introduce SMMTs based on RNN initialization and then continue with elementwise interactions. A visual summary of all the models is sketched in Figure 6.1.

RNN Initialization

We de ne three models that aim to provide visual context to the encoders and/or decoders by initializing their hidden states with global visual features. e initialization based MMTs were rst explored in [START_REF] Calixto | DCU-UvA multimodal MT system report[END_REF] and later extended with other variants in Calixto and Liu (2017) (section 4.2.1, p. 48). Our models closely relate to these works with slight di erences that will be detailed. Common to all three methods is the projection of the visual feature vector into the hidden space of the relevant RNN layer(s):

v = tanh (W f f + b f ) W f ∈ R h×2048 (6.1)

Encoder Initialization (EINIT)

Let us rst remind the bidirectional sentence encoding step where the hidden states of both the forward and the backward GRUs are initialized with 0:

H = [h 1 , . . . , h S ] = ENC (X, h 0 ← 0) =   # » GRU ( #» X, # » h 0 ← 0) # » GRU ( # » X, # » h 0 ← 0)  
We propose to initialize both forward and the backward GRUs with v (Figure 6.1, method 1) unlike Calixto and Liu (2017) where separate projections are preferred:

# » h 0 = # » h 0 = v
Although not explicitly stated in the equation above, since our encoder is composed of two stacked GRU layers, the initialization is also applied to the forward and backward GRUs of the second encoder layer. We believe that providing the same projection to all RNN layers for all directions may be a more consistent signal as the visual context should be invariant to encoding direction. Sharing the projection is also parameter e cient.

Decoder Initialization (DINIT)

e decoder initialization (Figure 6.1, method 2) is the most commonly explored way of visual grounding in MMT probably inherited from early NMTs (Cho et al., 2014b;[START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF] where the decoder is conditioned on a compressed source sentence representation through its initial state. Although this conditioning is no longer crucial with the introduction of the a ention mechanism, the decoder layer(s) in NMTs are still initialized with some kind of information coming from the encoder (section 3.4.4, p. 38). e visual grounding method proposed here initializes the rst GRU in the CGRU decoder with the projected visual features by se ing h ′ 0 = v. To allow for a fair comparison between the NMTs and the MMTs explored in this thesis, we kept the decoder uninitialized in our baseline NMTs. is way, the proposed method does not have to override a textually initialized decoder. An alternative is to initialize the decoder in a multimodal fashion as in Calixto and Liu (2017) where h ′ 0 is computed with an FF layer receiving v and H -1 .

Encoder & Decoder Initialization (EDINIT)

is method (Figure 6.1, method 1+2) constrains the network to learn a single representation that would satisfy all forward backward encoder layers as well as the rst GRU in the decoder by using a single projection layer (equation 6.1). is is made possible since all ve GRUs in our baseline have the same hidden state dimension h (Table 6.1):

# » h 0 = # » h 0 = h ′ 0 = v Visual Beginning-of-Sentence (VBOS)
We previously saw that the target sequences in NMT training are prepended with a beginning-of-sentence token <bos> (section 3.4.2, p. 35). Once the model is trained and the parameters are xed, this embedding -hence the initial input to the decoder -stays constant across di erent sentences that are translated. With this model, we propose to replace the static <bos> embedding with a dynamic one conditioned on the image information. e approach (Figure 6.1, method 5) is similar to [START_REF] Vinyals | Show and tell: A neural image caption generator[END_REF] in the sense that the decoder receives the feature vector as the rst input but we further remove the explicit <bos> token so that the decoding is truly initiated with the visual context. e following shows the CGRU decoder logic for the rst timestep and then modi es it accordingly to replace the <bos> with the linearly projected visual feature vector. Di erent from the RNN initialization methods (equation 6.1), we use a linear transformation here to comply with the dynamics of the word embeddings:

h ′ 1 = GRU 1 (EMB (<bos>) , h ′ 0 ← 0) NMT h ′ 1 = GRU 1 ((W f f + b f ) , h ′ 0 ← 0) W f ∈ R e×2048

VBOS

Although we have experimented with this model in the context of S2S multimodal speech recognition (Caglayan et al., 2019b), this is the rst time that we explore it for MMT.

Elementwise Interaction

In Caglayan et al. (2017a), we propose three novel interaction types concerning source side and target side sentence representations. All variants employ multiplicative interaction between the language related representations and the transformed visual feature v. e multiplicative interaction di ers from additive interaction in terms of cross-modal nature of its backward dynamics: the gradient of the loss with respect to the language related vectorial representation is scaled by v and vice-versa (∂ab/a = b) whereas the gradient with respect to the sum is directly passed along to the inputs of the sum. [START_REF] Fukui | Multimodal compact bilinear pooling for visual question answering and visual grounding[END_REF] show that the multiplicative interaction performs signi cantly be er than the additive counterpart in VQA.

Let us denote the source and target sequences of embeddings by X=[x 1 , . . . , x S ] and Y=[y 1 , . . . , y T ], respectively. Below, we de ne the interactions in a single set of equations although they are never combined together during the experiments:

v = tanh (W f f + b f ) Y = [y 1 ⊙ v, . . . , y T ⊙ v] TMUL X = [x 1 ⊙ v, . . . , x S ⊙ v] SMUL H = ENC (X, h 0 ← 0, v) = [h 1 ⊙ v, . . . , h S ⊙ v]
EMUL TMUL multiplies the target embeddings with the visual vector (Figure 6.1, method 6) while the SMUL applies the same trick to source embeddings (Figure 6.1, method 3). EMUL integrates the multiplicative interaction into the output of the bi-directional encoder to modulate the source representations -on top of which a ention will be applied in the decoder -with the visual vector v (Figure 6.1, method 4). e size of the projection matrix W f is R e×2048 for embedding interactions TMUL and SMUL, and R c×2048 for the encoding interaction EMUL.

Results & Analysis

We report BLEU and METEOR scores for English→German and English→French translation directions on both test sets in Table 6.2. For German, we observe that the RNN initialization based variants EINIT, DINIT, EDINIT and the multiplicative interaction model TMUL obtain signi cantly di erent scores than the baseline on test2017 (p-value ≤ 0.05 according to multeval [START_REF] Clark | Be er hypothesis testing for statistical machine translation: Controlling for optimizer instability[END_REF]). On average, TMUL seems the best performing model, reaching up to 0.7 point gains in both BLEU and METEOR. On test2016 however, the only signi cantly di erent systems are the EINIT and DINIT variants with up to 0.7 point gains in BLEU. e results are less promising for French where none of the systems achieve signi cantly di erent scores than the baseline. Still, we can say that the EDINIT, DINIT and TMUL systems -which are also ranked top three for German -are the ones that closely follow the baseline for test2017. e results suggest that the systems behave quite di erently for German when compared to French. We now conduct a breakdown analysis based on sentence level METEOR scores to possibly gain some insights about this di erence. 54.0 ± 0.7 (↓ 0.4) 70.6 ± 0.4 (↓ 0.5) SMUL 60.9 ± 0.8 (↓ 0.5) 76.0 ± 0.4 (↓ 0.4) 53.9 ± 0.5 (↓ 0.

Sentence Level Analysis

e protocol that we use for sentence level analysis is as follows: First, for each sentence of the test2017 set, we compute the METEOR scores obtained by the three independent runs of a given system. Second, we average these three scores to obtain a smoothed sentence level METEOR. Finally, for each MMT, we count the sentences which have a smoothed METEOR equal to, be er than, or worse than the one obtained by the baseline NMT. In other words, for a given multimodal-monomodal translation pair, we completely disregard the absolute METEOR di erence between them and discretize the evaluation into three bins of ties (=), wins (>) and losses (<).

Figure 6.2 shows the results for both language pairs. First, we notice that French systems behave very similarly to each other, with "losses %" higher than "wins %". Only the EDINIT system marginally di ers in this aspect with 34.3% "losses" and 35.1% "wins". On the other hand, all German systems except the EMUL have more "wins" than "losses": e Figure 6.2: Sentence level METEOR breakdown for MMT systems: the results are ordered by German (DE) "wins -losses" gap.

EDINIT system improves %46.2 of the translations while deteriorating on 38.8%, exhibiting the largest "wins -losses" gap of 7.4%. Second, we observe a systematic di erence between the languages in terms of the "ties": On average, 30.1% of multimodal French translations preserve their METEOR while for German this percentage drops to 16.1%.

e ties are consistently stable across SMMT variants with a standard deviation of 1% and 0.6% for French and German, respectively. is shows that independent from the model type, there is always a nonnegligible portion of the test set for which any given model performs equivalent to the baseline NMT. e fact that the French portion is almost the double of its German counterpart raises another question: Is this di erence related to the integration of visual modality or not? To understand this, we train a "control" NMT (still with three runs) and compute the same statistics for it by comparing its sentence level METEOR scores to the actual baseline. Surprisingly, we observe almost the same "ties" percentages for the "control" NMT: 29.9% for French and 16.8% for German. is strongly suggests that the French task is simpler than German since ∼1/3 of the test set consistently obtains equivalent sentence METEOR scores independent from the underlying conditions.

In overall, this ne-grained analysis corroborates the hypothesis that there is less room for improvement for the French task when compared to German. Although this is already obvious in terms of the corpus level results where French BLEU scores are ∼22 higher than German ones (Table 6.2), the breakdown analysis with discretized bins revealed interesting details about the characteristics of both NMTs and MMTs.

MMT17 Evaluation Campaign

In 2017, we participated to the shared task on MMT (Ellio et al., 2017) for both German and French translation directions. Back at that time, we mainly experimented with the DINIT, EDINIT, EMUL and TMUL variants presented in this chapter and submi ed a 5-run ensemble of TMUL for German and a 6-run ensemble of mixed SMMTs for French, respectively. Our systems ranked rst among 11 French and 16 German systems according to test2017 METEOR. Moreover, our multimodal German submission ranked rst according to human evaluation (section 3.5, p. 41). Figure 6.3 plots the standardized human judgment score against METEOR for the top ranked constrained and unconstrained systems. e system ( 1) is our aforementioned TMUL ensemble while the system ( 6) is our 5-run NMT ensemble. Our winning ensemble also surpassed three unconstrained systems, namely, an Imagination MMT (2), a pure NMT (4) and an a entive MMT [START_REF] Baltrusaitis | Multimodal machine learning: A survey and taxonomy[END_REF].

MMT17 vs. Retrained Systems

We now would like to compare the retrained systems for this thesis to the ones from MMT17 (Caglayan et al., 2017a). e new hyperparameters (section 5.4, p. 60) are substantially di erent than MMT17 as we now have a 2-layer encoder, 256D embeddings (instead of 128) and 320D recurrent layers (instead of 256). Also, we now use word→BPE vocabularies instead of the previous BPE→BPE ones. Table 6.3 provides average METEOR with standard deviation for the baseline NMT and TMUL systems. First of all, we can say that the new hyperparameters result in an average improvement of 0.9 points for German baseline. Second, the TMUL system brings up to 0.7 METEOR improvement both for MMT17 and the retrained systems. For French however, the 2 points gain (67.5→69.5) in MMT17 no longer holds for the retrained systems: e new baseline easily closes that gap and further reaches 71.1 with an overall improvement of ∼4 METEOR.

EN→DE EN→FR

NMT 51.6 ± 0.5 ⇒ 52.5 ± 0.7 67.5 ± 0.7 ⇒ 71.1 ± 0.2 TMUL 52.2 ± 0.4 ⇒ 53.2 ± 0.1 69.5 ± 0.7 ⇒ 71.1 ± 0.3 Table 6.3: test2017 METEOR comparison of MMT17 systems to this thesis: the arrow (⇒) between the scores shows the transition from MMT17 to retrained systems. A vertical comparison reveals the multimodal improvements within each year.

System

German Rank (score) French Rank (score)

SMMT (Caglayan et al., 2017a) 1 (0.67) 4 (0.22) Reranking [START_REF] Zhang | NICT-NAIST system for WMT17 multimodal translation task[END_REF] 3 (0.44) 1 (0.45) SMMT (Calixto et al., 2017a) 5 (0.31) 3 (0.30) NMT (Caglayan et al., 2017a) 6 (0.20) 8 (-0.08) Table 6.4: Standardized human judgment scores for German and French: we compare the subset of the German systems (Figure 6.3) that also participated to French evaluation.

When we further compare the human evaluation rankings of German and French MMT17 submissions (Table 6.4), we notice how our French systems lag behind the other ones that were otherwise surpassed for English→German. e systems ranked 3 rd and 5 th for German move to the 1 st and 3 rd positions for French by obtaining almost the same standardized human judgment scores whereas our submissions obtain substantially lower scores compared to our German systems. e shi between same architectures trained for di erent languages is rather unexpected and is probably due to the di erences between German and French hyperparameters back at that time, especially the ones related to dropout and L 2 regularization. Once again, is points out the importance of carefully selecting the underlying hyperparameters to avoid starting with a baseline that under ts or over ts to the training set.

Comparison to State-of-the-art

In this section, I compare our best SMMT systems to a selection of state-of-the art MMT systems including a competitive Transformer-based a entive MMT [START_REF] Libovický | Input combination strategies for multi-source transformer decoder[END_REF]. I evaluate the systems exactly the same way as the section 4.2.4 (p.55). According to the results in Table 6.5, our newly trained systems obtain the best BLEU and METEOR scores among the constrained systems, improving over our MMT17 systems as well. I report the relative gains (or drops) of each system with respect to the baseline MT reported in their works. For example, the di erence between the baseline NMT and the DINIT model of Calixto and Liu (2017) is 2.8 points (52.3→55.1). ese relative di erences reveal a clear pa ern among the current state-of-the-art in MMT: As researchers converge to be er baselines, the apparent improvements due to multimodality tend to disappear. Our ndings about the mismatch between our French MMT17 systems and the retrained ones also supports this view. We develop more insights about this aspect in chapter 8.

System BLEU METEOR Description

Calixto and Liu (2017) RNN 36.9 (↑ 3.2) 54. [START_REF] Vaswani | A ention is all you need[END_REF]. We do not report ensemble results to ensure a fair comparison. e relative di erences inside parentheses are with respect to the baseline MTs reported in those works. e results are sorted by METEOR.

Summary

In this chapter, I presented several SMMT systems which are MMTs that incorporate global visual features extracted from pre-trained CNNs. e chapter covers the systems proposed in Caglayan et al. (2017a) and adds two more SMMT systems to the inventory, namely, the SMUL and the VBOS variants. I provide quantitative results for the German and the French translation tasks of Multi30K dataset, using BLEU and METEOR scores on two di erent test sets. I further compare the systems retrained for this chapter to our winning submissions in MMT17. e main conclusions can be summarized as follows:

• We observe signi cant improvements in BLEU and METEOR for English→German -especially on test2017 -but the same does not hold for English→French.

• We conduct a sentence level analysis based on METEOR scores to break down the large baseline di erence between German and French. e results corroborate the hypothesis that there is less room for improvement for French as the percentage of the test set consistently obtaining same METEOR across di erent NMT and MMT systems is 30% for French compared to 16% for German. In other words, French systems seem more stable and conservative in terms of the variability of the produced translations.

• Based upon a comparison between our MMT17 systems (Caglayan et al., 2017a) and the ones retrained in this chapter, we conclude that the RNN initialization based SMMTs along with the multiplicative TMUL variant exhibit moderate improvements for German regardless of the baseline performance. However, the signi cant multimodal improvements for French disappear with the retrained systems suggesting that the visual modality may be helpful only if the architecture has di culty to fully exploit the textual information. We leave a quantitative and qualitative exploration of this aspect to chapter 8.

e next chapter explores a substantially di erent MMT paradigm equipped with a multimodal a ention mechanism which exploits spatially aware convolutional features instead of the global visual features.

CHAPTER7

A entive Multimodal Machine Translation e previous success of the a ention mechanism led to the further exploration of the idea for multi-input and/or multi-output networks mostly in the context of multilingual NMT. [START_REF] Dong | Multi-task learning for multiple language translation[END_REF] and [START_REF] Zoph | Multi-source neural translation[END_REF] experimented with dedicated a ention layers in one-to-many and many-to-one NMT systems respectively, whereas [START_REF] Firat | Multi-way, multilingual neural machine translation[END_REF] proposed a shared a ention across multiple language pairs in a manyto-many framework. In overall, all these approaches seemed bene cial to translation performance according to the experimental results provided by the authors. However, the curious case of shared vs dedicated a ention layers were not further explored in a comparative manner. Being a many-to-one framework with multiple input modalities involved to perform translation, MMT with visual a ention lies at the intersection of the above approaches as well. Moreover, the aforementioned case of sharing the a ention becomes much more interesting for MMT where the nature of the modalities are radically di erent: word representations and the corresponding encoder are jointly learned during the training while the visual representations are -generally -frozen and pretrained for an external visual recognition task.

is chapter describes our e orts towards the design of a entive MMT (AMMT) architectures capable of integrating the visual modality through an additional visual attention module (Xu et al., 2015b). We begin by exploring a shared multimodal a ention (Caglayan et al., 2016a) similar to [START_REF] Firat | Multi-way, multilingual neural machine translation[END_REF] and then manipulate it progressively to reach a completely dedicated variant along with di erent multimodal fusion strategies (Caglayan et al., 2016b). We compare all methods using a xed set of hyperparameters (Table 7.1) and the previously described pre-processing pipeline (section 5, p. 57). We nalize the chapter with a quantitative analysis on English→German and English→French translation tasks of Multi30K and also provide some qualitative insights about the characteristics of visual a ention.

Revisiting the CGRU Decoder

I will rst start by describing the decoder logic in the CGRU architecture in detail before extending it with the proposed multimodal a ention mechanism. In the following, L and V denote the textual and the visual set of encodings, respectively. L is an alias for the usual set of bidirectional encodings H (section 3.4.4, p.38) while V represents the spatial features extracted from a pre-trained ResNet-50 CNN [START_REF] He | Deep residual learning for image recognition[END_REF]. We superscript all layers and transformations with V(isual) or L(anguage) to distinguish modality-speci c constructs.

We start by naming the existing a ention mechanism in the CGRU decoder the "language a ention layer" and assign the symbol ATT L to it. is layer computes the a ention distribution over the language encodings L by using the hidden state h ′ t of the rst decoder GRU as the query vector. Note that unlike the previous formulation (section 3.4.4, p. 38), here we separate out the context computation for reasons that will be clear once we introduce the multimodal fusion. e following equations summarize how to obtain the a ention distribution α L t at decoding timestep t. We use a compact notation here to not explicitly de ne the a ention scores for each source word position. is avoids clu ering the symbols with source position indices:

(L ∈ R S×c , W L e ∈ R c×a , h ′ t ∈ R 1×d , W L q ∈ R d×a , w L a ∈ R a×1 ) α L t = ATT L (L, h ′ t ) = SOFTMAX tanh L W L e + h ′ t W L q w L a α L t ∈ R S×1
Once the a ention distribution is computed, the language context c L t is easily obtained with a matrix-vector product (equation 7.1). Finally, we linearly transform c L t to make its size compatible with the input size d of the second GRU in the decoder (equation 7.2).

is transformed context i t becomes the input to the second GRU:

c L t = L ⊤ α L t c L t ∈ R c (7.1)
i t = W L d c L t W L d ∈ R d×c (7.2) h ′′ t = GRU 2 (i t , h ′ t ) (7.
3) e rest of the computations follow the original CGRU formulations i.e. the probability of the next target token is computed with a deep output logic (section 3.4.4, p. 38). e language a ention layer ATT L is parameterized by the following transformations: {w L a , W L e , W L q }. We further separate these three transformations into two groups where W L q is referred to as the "decoder-state" projection and {w L a , W L e } are considered to be "modality-relevant" projections (Caglayan et al., 2016b).

Visual Attention

We denote the spatial features extracted from the pre-trained ImageNet CNN with the 3D tensor F. Since we do not experiment with ne-tuning the CNN during MMT training, we extract the spatial features once for all the images in the dataset and plug these into our architecture a erwards as standalone features:

F = RESNET50 (IMG 256×256 ) F ∈ R 8×8×2048 (7.4)
CNN ATT Inside the network, a convolutional layer with c 1x1 lters is applied to the spatial features to make the feature dimension compatible with the language encodings L. A

FLATTEN operation is performed to a en the spatial dimensions (8x8) of the feature tensor into 64 (feature vectors) on top of which a secondary a ention mechanism can be applied without any change:

F ′ = CONV 1×1×c (F) F ′ ∈ R 8×8×c V = FLATTEN (F ′ ) V ∈ R 64×c (7.5)
e type of visual a ention explored so far in MMT is "spatial" (Xu et al., 2015b) in the sense that a probability mass is assigned to each position in the 8x8 grid of convolutional features (Figure 7.1). is way, the model is able to select "where" to a end in the image at each decoding timestep t. is formulation is quite similar to the language a ention where a probability mass is assigned to each of the S hidden states produced by the encoder.

Let us now create a second a ention layer ATT V with another set of parameters {w V a , W V e , W V q } in order to implement the visual a ention. Note how the two a ention formulations are exactly the same except the number of feature vectors which is the number of words S and the spatial resolution K = 64 for the language and the visual a ention, respectively:

(V ∈ R 64×c , W V e ∈ R c×a , h ′ t ∈ R 1×d , W V q ∈ R d×a , w V a ∈ R a×1 ) α V t = ATT V (V, h ′ t ) = SOFTMAX tanh V W V e + h ′ t W V q w V a α V t ∈ R 64×1 c V t = V ⊤ α V t c V t ∈ R c Name Modality Decoder State SS Shared W L e = W V e , w L a = w V a Shared W L q = W V q SD Dedicated W L q = W V q DS Dedicated W L e = W V e , w L a = w V a Shared W L q = W V q DD Dedicated W L q = W V q Table 7
.2: Sharing strategies for multimodal a ention: e name consists of the initials of "Shared" and "Dedicated" for modality and decoder state projections, respectively.

Feature Normalization

e spatial features are extracted a er a ReLU convolutional layer that recti es its input into [0, ∞]. On the other hand, the non-linearities in GRUs and in our baseline NMT in general are based on the tanh activation which squeezes its input to [-1, 1]. Our initial a empts to a entive MMT models in 2016 (Caglayan et al., 2016a) and 2017 (Caglayan et al., 2017a) editions of the shared task, had signi cantly poor performance compared to our respective baselines. We hypothesize that the reason behind this may be the activation ranges of language and visual features in the network which hinders the learning dynamics. Speci cally, the unbounded visual features may easily saturate the tanh neurons in the network unless special care has been taken to adjust the random initialization scheme of the network weights. In Caglayan et al. (2018), we take a simpler normalization approach by following previous empirical evidence in VQA research [START_REF] Kazemi | Show, ask, a end, and answer: A strong baseline for visual question answering[END_REF][START_REF] Yu | Multi-modal factorized bilinear pooling with co-a ention learning for visual question answering[END_REF] showing the bene t of applying L 2 normalization over the channel dimension of spatial features. Speci cally, this ensures that the L 2 norm of each of the 64 (8x8) spatial feature vectors (∈ R 2048 ) is 1. e normalization step comes right a er the extraction of the spatial features F (equation 7.4).

Sharing Strategies

In order to understand the e ect of sharing the a ention across the modalities, we propose four di erent strategies (Caglayan et al., 2016b) that are summarized in Table 7.2. When a parameter is shared, it is reused in both a ention layers {ATT L , ATT V } enforcing the model to learn a shared representation -for the outcome of that transformationto minimize the training loss. When parameters are dedicated to modalities, the model would have more exibility to independently optimize the corresponding transformation parameters. Unlike [START_REF] Firat | Multi-way, multilingual neural machine translation[END_REF] where a single a ention is shared across multiple languages, we believe that a dedicated visual a ention may be more appropriate for MMT simply because of the radically di erent nature of the modalities i.e. jointly learned embeddings and encoder for the language and visual features transferred from an image classi cation task.

Multimodal Fusion

e multimodal a ention mechanism computes two modality speci c context vectors {c L t , c V t }, independent from the choice of sharing strategy. A fusion has to be performed to compress these contexts into a single vector which would then be used as the input to the GRU 2 layer (Figure 7.2). A linear transformation already exists at this step for the textual NMT architecture (equation 7.2) to project from c-dimensional context space to the d-dimensional decoder input i t (equation 7. 3). e multimodal fusion is thus an extension to that step that considers both contexts when performing the projection. e following de nes the SUM and CONCAT fusion methods proposed in Caglayan et al. (2016b):

i t = φ W d c L t + c V t = φ W d c L t + W d c V t SUM FUSION i t = φ( W L d ; W V d c L t c V t ) = φ W L d c L t + W V d c V t CONCAT FUSION
e di erence between CONCAT and SUM is that the former uses dedicated parameters e SS model with SUM fusion (SS-SUM) implements a "completely shared" multimodal a ention while the DD model with CONCAT fusion (DD-CAT) performs a "completely dedicated" multimodal a ention with the least amount of crossmodal interaction involved. Another popular fusion method is the hierarchical a ention [START_REF] Libovický | A ention strategies for multi-source sequence-to-sequence learning[END_REF] that employ a third a ention mechanism on top of the multimodal contexts.

{W L d , W V d }

Results & Analysis

We rst evaluate the performance of eight MMT variants that result from combining four sharing strategies with two fusion methods. We start by comparing BLEU and ME-TEOR scores of these systems with and without L 2 normalization on the test2016 set of English→German direction (Table 7.3). A quick look at the results reveal that without normalization, the results are far from being competitive, achieving 1.5 METEOR and 1.8 BLEU less than the baseline on average. With feature normalization however, the systems reach the baseline performance, with DS models even slightly improving over it (0.3 METEOR and 0.5 BLEU points for DS-SUM). In order to understand the qualitative impact of normalization, we visualize the language and visual a entions for one of the MMT variants (SS-SUM) in Figure 7.3. e example shows that with normalized features, the model produces a meaningful spatial a ention where it rst focuses on the "person" and then highlights the "mountain" in the background.

We now extend the results to English→French and report metrics on both test sets (Table 7.4). First of all, we observe that all German MMT models perform at least as good as the baseline on test2017 although the di erences are only signi cant for the DS systems (p-value ≤ 0.05 according to multeval [START_REF] Clark | Be er hypothesis testing for statistical machine translation: Controlling for optimizer instability[END_REF]). On average, the DS-CAT performs be er than the baseline with 0.6 and 0.4 BLEU and METEOR points, respectively. e results are less promising for French where none of the systems are signi cantly di erent on any test set. Interestingly, we observe the same saturating behavior as in the case of simple MMT (section 6.2, p. 65): French MMT systems are quite stable and barely move in terms of automatic metrics while all German MMTs perform at least as good as the baseline on test2017. Although it is not possible to draw a conclusion about the individual strengths of the models, we notice that the top ranked MMTs for both languages have modality dedicated a entions i.e. DS-CAT for German and DD-CAT for French. e choice of multimodal fusion does not seem to make a crucial di erence.

In the following, we brie y look at the results of sentence level analysis protocol introduced in the previous chapter (section 6.2. 54.2 ± 0.3 (↓ 0.2) 70.9 ± 0.4 (↓ 0.2) DS-SUM 60.7 ± 0.0 (↓ 0.7) 76.0 ± 0.2 (↓ 0.4) 54.2 ± 0.4 (↓ 0.2) 71.0 ± 0.1 (↓ 0.1) SS-SUM 61.2 ± 0.4 (↓ 0.2) 76.2 ± 0.2 (↓ 0.2) 54.0 ± 0.4 (↓ 0.4) 71.0 ± 0.1 (↓ 0.1) DD-SUM 61.6 ± 0.4 (↑ 0.2) 76.5 ± 0.4 (↑ 0.1) 54.2 ± 0.3 (↓ 0.2) 71.0 ± 0.2 (↓ 0.1) SD-SUM 61.3 ± 0.2 (↓ 0.1) 76.2 ± 0.0 (↓ 0.2) 54.3 ± 0.2 (↓ 0.1) 71.1 ± 0.2 DD-CAT 61.2 ± 0.6 (↓ 0.2) 76.3 ± 0.3 (↓ 0.1) 54.1 ± 0.7 (↓ 0.3) 71.2 ± 0.1 (↑ 0.1) Table 7.4: Combined results on test2016 and test2017: Highlighted scores are signicantly di erent than the NMT (p-value ≤ 0.05). Results ordered by test2017 METEOR.

Sentence Level Analysis

Figure 7.4 plots the percentages of ties, wins and losses on test2017 for both language pairs. e conclusions are pre y coherent with the SMMT analysis (section 6.2.1, p. 66): On average, 28.9% and 15.5% of French and German multimodal translations preserve their METEOR consistently across AMMT variants with a standard deviation of 1.2%. Let us remind that these averages are once again almost the same as the retrained "control" and thus not related at all to multimodality. e German DS-CAT improves %45.6 of the translations while deteriorating on 39.1% (6.5% "wins -losses" gap) whereas the same system for French is the worse in this aspect with -5.7% "wins -losses" gap. In overall, the fact that the sentence level breakdowns for SMMTs and AMMTs look pre y similar hints at the fact that the behavior of the models is mostly driven by the language signal as well as the test set characteristics rather than the type of multimodality introduced.

In other words, the models do not seem to be stimulated by the visual input.

Analysis of the Visual Attention

We previously saw in Figure 7.3 (bo om) that the language a ention is able to preserve its certainty (peakiness) throughout the translation decoding despite the fact that the SS-SUM model has a completely shared a ention across both modalities. To be er understand the behaviour of di erent sharing strategies as well as the type of multimodal fusion, we collect statistics during the decoding of test2016 sentences and compute normalized entropies for language and visual a ention mechanisms. e normalization is performed by dividing the entropy per sample by the uniform entropy, taking into account the number of source words for each decoded sentence in the case of language a ention. For example, a visual a ention that "always" assigns a probability of 1 8×8 to each position in the 8x8 convolutional feature maps, obtains a normalized entropy of 100%, indicating the highest uncertainty. e nal entropy is computed by simply taking the average of per sample entropies. Figure 7.5 plots the computed entropies across the explored AMMT variants. First of all, we can see that the uncertainty of the language a ention does not seem to be a ected by the multimodality and behaves similarly to the baseline NMT. In contrast, the uncertainty of the visual a ention consistently increases as the a ention becomes more and more shared across modalities. In fact, the visual a ention of SS-SUM turns out to be "almost" uniform.

Finally, we visualize the spatial a ention of the models on a speci c example of test2016 in Figure 7.6. Since the entropy of each model radically di ers from each other, it is impossible to visualize the heatmaps with a normalized scale i.e. the magnitudes of the a ention are not quite comparable across models. Nevertheless, the plot still gives an idea about the internal view of each model: Although quite uniform, the SS-SUM produces a plausible a ention where tiny di erences in the probability mass determine the focus.

On the other hand, the peakiness of the visual a ention increases as the multimodal a ention becomes more and more dedicated.

EN→DE (test2017) BLEU METEOR

NMT 32.1 ± 1.1 52.5 ± 0.7 DD 32.3 ± 0.8 (↑ 0.2) 52.6 ± 0.5 (↑ 0.1) SS 32.3 ± 0.2 (↑ 0.2) 52.6 ± 0.4 (↑ 0.1) SD 32.5 ± 0.2 (↑ 0.4) 52.8 ± 0.2 (↑ 0.3) UVA 33.0 ± 0.3 (↑ 0.9) 52.8 ± 0.1 (↑ 0.3) DS 32.7 ± 0.2 (↑ 0.6) 52.9 ± 0.5 (↑ 0.4)

Uniform Visual Attention

Although the entropy analysis suggests that dedicating the a ention mechanism decreases the uncertainty of the spatial focus, the automatic evaluation metrics do not reect any preference towards a speci c kind of a ention. Moreover, the almost uniform shared a ention variants seem to produce quite plausible a ention maps (Figure 7.6).

is raises an interesting question: How important is the spatial certainty of the visual a ention for the model performance? In order to answer this, we propose an ablation experiment which consists of replacing the learnable visual a ention layer ATT V with a dummy layer that explicitly assigns a uniform probability of 1 8×8 to each spatial position. e nal model that we call "uniform visual a ention (UVA)" still uses the spatial features to compute the visual context c V t but this context no longer depends on the hidden state h ′ t of the decoder i.e. it stays constant across the decoding steps. In fact, this amounts to using the global visual features f at each decoding step since f is the global average pooled version of the spatial features (section 2.6.2, p. 26).

Table 7.5 compares the BLEU and METEOR scores of AMMT systems to UVA with the multimodal fusion operation set to "concatenation" for each model. We see that the UVA model obtains the highest average BLEU score and is signi cantly di erent than the baseline according to multeval. It also outperforms the "almost uniform" SS variant on average metrics. ese results suggest that the model bene ts more from a constant and spatially unaware visual signal compared to a noisy version of it which evolves throughout the decoding steps. [START_REF] Vaswani | A ention is all you need[END_REF]. We do not report ensemble results to ensure a fair comparison. e relative di erences inside parentheses are with respect to the baseline MTs reported in those works. e results are sorted by METEOR.

Comparison to State-of-the-art

Table 7.6 compares our best a entive MMTs to a selection of state-of-the-art systems. I also include the SMMT systems from the previous chapter to provide a global view of all models presented in this thesis. e rst conclusions are pre y much the same as SMMTs (section 6.3, p. 69): be er baselines seem to bene t less from the visual modality. Di erent from SMMTs though, we observe that a entive systems struggle more to maintain the baseline performance. In fact, the only system that substantially improves over their baseline with respect to both metrics is Calixto et al. (2017b). In our case, this issue is now addressed with L 2 normalization which allows our models to at least perform as good as the baseline on average. In overall, both our SMMT and AMMT systems perform equally well and obtain state-of-the-art scores with respect to automatic evaluation metrics. e gains in BLEU are slightly higher than METEOR, however it should be noted that BLEU exhibits a higher variance -at least in the case of Multi30K -as shown in the detailed quantitative results.

Summary

In this chapter, I presented several a entive MMT systems with di erent sharing levels and multimodal fusion techniques. I rst showed how L 2 normalization of spatial features is crucial for these models to reach the baseline performance, then conducted a quantitative analysis on English→German and English→French translation tasks of Multi30K. Although some of the models were shown to be signi cantly be er than the baseline for German, we struggle to reach a global conclusion about the performance of the AMMT systems. A er gaining some insights from the entropies of a ention distributions, we conduct a contrastive experiment where the visual a ention is replaced with a dummy layer which constantly puts a uniform a ention over the image features. e fact that this model obtains competitive scores as well raises an obvious question about whether the quantitative gains can be solely a ributed to multimodality or not.

We also observe that the quantitative results for AMMTs are mostly coherent with the SMMT models in the sense that both approaches yield mild improvements for German while barely moving for French. is is interesting as it strongly points out that it is the linguistic traits of the underlying language pairs and the dataset which seem to dominate the nal performance trends of the models rather than the visual feature type or the interaction scheme explored. e next chapter a empts to tackle these concerns by providing a set of ablation experiments to probe the visual awareness of SMMT and AMMT models explored throughout the thesis.

CHAPTER8

Deeper Analysis of MMT Systems

In previous chapters, we explored several multimodal integration methods for NMT by rst using the global visual features (SMMT) and then moving on to more sophisticated a entive approaches (AMMT) which incorporate spatially aware features. Upon various quantitative analyses and manual inspection of the model dynamics, we nd it hard to reach a conclusion on the strengths and weaknesses of the proposed architectures in terms of their ability to integrate the visual modality.

Recent evidence from the literature also suggest that the bene ts of the current MMT approaches are li le to none on Multi30K. [START_REF] Lala | She eld submissions for WMT18 multimodal translation shared task[END_REF] show that when used to rerank a list of translation candidates, their multimodal WSD method is not any be er than the monomodal counterpart. [START_REF] Ellio | Adversarial evaluation of multimodal machine translation[END_REF] demonstrate that the performance of state-of-the-art MMTs is marginally in uenced when they are adversarially a acked by incongruent images i.e. when source sentences are paired with images not being the ones described by those sentences. A er experimenting with a plethora of visual features and integration methods, [START_REF] Grönroos | MeMAD submission to the WMT18 multimodal translation task[END_REF] also nd out that their English→French MMT is not negatively a ected at all by the adversarial evaluation, corroborating the ndings of [START_REF] Ellio | Adversarial evaluation of multimodal machine translation[END_REF]. Finally, the organizers of the shared task point out that "the integration of visual modality does not seem to help reliably" (Ellio et al., 2017) and there may be a need for a more challenging task & dataset for which the images would be indispensable [START_REF] Barrault | Findings of the third shared task on multimodal machine translation[END_REF]. We believe that the underlying reason behind these negative conclusions may be the simple, short and repetitive nature of the Multi30K dataset rendering the source sentences su cient for the translation task. In turn, this may prevent the visual modality from intervening in the learning process if the model sees no bene t from it when minimizing the loss.

To investigate our hypothesis, here we propose to systematically deprive the models from textual context primarily by masking out visually depictable words from the source sentences (Caglayan et al., 2019a). We then evaluate these new models using the adversarial protocol in order to assess their visual sensitivity. But before doing so, we revisit the state-of-the-art English→German models (section 4.2.4, p. 55) once again to discuss the nature of the previously demonstrated multimodal gains.

Figure 8.1 shows the METEOR gains relative to the baseline MTs reported in the corresponding papers. We make sure that neural MMT systems are compared to NMT baselines while multimodal PBMTs (PBMMT) are compared to PBMT baselines. e plot makes it clear that the improvements due to the visual modality are only prominent if the underlying baselines are not optimal i.e. they are not able to fully exploit the language signal for some reason. e dashed vertical line sets an hypothetical boundary after which all baselines obtain a METEOR ≥ 54 and all corresponding multimodal gains are ≤ 0.7 METEOR. Recall that the multimodal gains for our French MMT17 systems no longer hold for the retrained systems where the new baseline is signi cantly be er than the old one (section 6.2.2, p. 68). We thus posit that the retrained French systems crossed a similar hypothetical boundary a er which the bene t of the visual modality becomes li le to none. is brings us to the previously introduced question of "whether the quantitative gains can be solely a ributed to multimodality or not". In order to answer this question at least for our current SMMT and AMMT systems, we now describe the "adversarial evaluation" method [START_REF] Ellio | Adversarial evaluation of multimodal machine translation[END_REF]

Adversarial Evaluation

e protocol starts with decoding a given test set using incongruent visual features. e incongruence is achieved by shu ing the order of the visual features so that a source sentence X i is explicitly aligned to a "wrong" visual feature V j =i . Consequently, an MMT system capable of integrating the visual modality would likely deteriorate in terms of automatic evaluation metrics. For a given test set, [START_REF] Ellio | Adversarial evaluation of multimodal machine translation[END_REF] repeat the decoding process ve times by re-shu ing the order each time, in order to lter out noisy measurements that can be caused by a speci c shu e. Here we take a slightly di erent approach and we create a single incongruent test set with reversed feature order i.e. the source sentences {X 1 , X 2 , . . . , X N } are deliberately misaligned to visual features {V

N , V N -1 , . . . , V 1 }
where N denotes the size of the test set. Finally, we decode this incongruent test set for each of the three runs of a given model and compute the mean and the standard deviation of BLEU and METEOR using multeval tool [START_REF] Clark | Be er hypothesis testing for statistical machine translation: Controlling for optimizer instability[END_REF] as in the previous chapters. is way, we are able to leverage the statistical signi cance tests of multeval to evaluate an incongruently decoded model to its congruently decoded baseline.

Table 8.1 shows the results for English→German SMMT systems. First of all, we notice that the averaged metric shi s (∆) due to incongruent decoding are quite small, with the EMUL system deteriorating the most. EMUL is also the only system for which the incongruently decoded translations are signi cantly di erent than the congruent counterparts with respect to METEOR. Other than the EMUL and EDINIT variants, the rest are barely reacting to the misaligned visual features, some of them even showing slight improvements, an interesting e ect also observed by [START_REF] Grönroos | MeMAD submission to the WMT18 multimodal translation task[END_REF] We observe a similar behavior among the AMMT systems (Table 8.2) although they seem to deteriorate slightly more than the SMMT systems. e dedicated a ention variant DS-CAT signi cantly worsens by incongruent decoding with respect to BLEU. e completely shared variant SS-SUM exhibits nonnegligible average improvements when decoded incongruently, a phenomenon which strongly suggests that the visual modality behaves as a structured noise which substantially in uences the output probability distribution at translation decoding time -at least -for this model.

Globally, the adversarial evaluation results for both types of MMT suggest one clear thing: e visual signal is not a vital contributor to the multimodal reasoning ability as none of the models completely breaks apart when challenged with unrelated visual features. In other words, the modalities are far from being cooperative. In theory, this should not reject the (weak) possibility that the visual modality may be providing a complementary signal for the models that consistently su er from incongruence. However, a manual inspection of the translations for these systems reveal no systematic signs for that: For the incongruent DS-CAT system, a sentence that substantially deteriorates actually replaces the word "footballspieler" with its hyphenated version "football-spieler" whereas another one reaches 100% METEOR by adding the previously missing "in der stadt (in the city)" phrase to its translation. e rst example also shows how fragile the automatic evaluation is when performed with a single set of references.

[v] [v] [v] D 2 Prog. Masking (k=2) a lady [v] [v] [v] [v] [v] D 0 Prog. Masking (k=0) [v] [v] [v] [v] [v] [v] [v]

Degradation Methods

In this section we propose to explicitly degrade the source sentences in Multi30K training and test sets at di erent scales. e idea here is to understand whether the explored models can gain multimodal reasoning abilities by learning to refer to the images when the information no longer exists in the source sentence. In the following, we describe two approaches, namely, the progressive and entity masking (Table 8.3), and then proceed with quantitative and qualitative analyses.

Progressive Masking

A progressively masked variant D k replaces all but the rst k tokens of source sentences with [v] . ese tokens are further considered as OOVs during training and test time. Overall, we form 16 degraded variants D k (Table 8. 3) where k ∈ {0, 2, . . . , 30}. We stop at D 30 since 99.8% of the sentences in Multi30K are shorter than 30 words. D 0 is a special case where the only information that the models can extract from a source sentence is its length. is is interesting as an NMT model trained on D 0 will only be able to generate a single sentence per source sentence length, since all sentences with the same number of words look the same to the decoder. On the other hand, an MMT has the potential to remedy that problem as it also has access to an auxiliary source of information, namely, the image features. In turn, the MMT system will behave as an image captioning system which can also guess the number of target words to be generated.

Progressive masking does not guarantee systematic removal of visual context, but simulates an increasingly low-resource scenario where the models have only access to sentence pre xes. e NMT and MMT models trained on a progressively masked variant no longer perform machine translation but translation completion. Although this may sound unrealistic, the task is still interesting as an NMT model will purely re ect the intrinsic biases of the dataset while MMT models will potentially apply debiasing with the help of the visual modality.

Results

To evaluate the models for progressive masking, we pick the completely dedicated AMMT (DD-CAT) and the encoder-decoder initialization SMMT (EDINIT) as our target models. For each progressively masked dataset D k , we train the two MMTs along with the baseline NMT on English→French task. A er the training, we follow the usual pipeline for decoding and evaluating the models. We compute the gain in METEOR over the masked NMT by averaging the gains of each model across the three runs.

Figure 8.2 shows the evolution of the multimodal gain as the sentence pre xes get shorter and shorter. e dashed gray curve marks the percentage of non-masked words in the training set i.e. the amount of remaining information. We observe that the improvements become prominent (≥ 1 METEOR) when the context size shrinks to ∼9 words which is equivalent to ∼68% source information (∼32% information dropped).

is point more or less re ects the average number of words per sentence which is ∼13 for the training set and ∼11 for the test set. A er that point, the gap widens signi cantly, reaching ∼7 METEOR at D 0 . Finally, the SMMT consistently lags behind the AMMT by around 1 METEOR, showing for the rst time that the spatial feature based MMTs are able to leverage more visual context than the global feature based ones.

Table 8.4 provides qualitative examples from a couple of progressive masking experiments. We can see that the AMMT system is able to produce surprisingly good sentences that re ect more than one aspect of the image. We have also checked to what extent the correctly predicted phrases co-occur within a same context in the training set. For the second example, "dansent dans une rue" occurs only once in the training set and it is not followed by "en ville". For the third example, "maillot de bain rose" occurs in six sentences but none of them starts with "une femme". dress" in Table 8.3. In these cases, we only replace the head noun (dress in this case) and leave the other words (blue) intact. e entity masking method is an aggressive degradation as it masks 26.2% of the words in both the train and the test2016. In terms of sentence statistics, this results in a training and test set where almost all the sentences contain at least one OOV token, with the average OOV per sentence being 3.4. Only 11 training sentences are not a ected by this process. Unlike the progressive variant, entity masking guarantees systematic removal of visual information from source sentences since the originally annotated entities are concrete nouns.

Results

We conduct an extensive set of experiments and train all SMMT and AMMT variants for both English→German and English→French tasks. We then compute the congruent and incongruent METEOR scores across the three runs of each model. Finally, we compute the relative gains of each MMT over the masked NMT (∆ over NMT).

Figure 8.3 visualizes the results of German and French SMMT systems. We rst notice that the encoder side interactions bene t the most from the visual modality unlike the target side interaction methods TMUL, VBOS and DINIT which seem quite ine ective. We observe critical performance drops with incongruent decoding, suggesting that the visual modality is now much more important than previously demonstrated [START_REF] Ellio | Adversarial evaluation of multimodal machine translation[END_REF]. In fact, a large multimodal gain is always coupled with a large incongruent drop.

As for the AMMT experiments, we present the incongruent drops and the multimodal gains for the concatenative variants as they tend to obtain slightly be er ME-TEOR scores on average when compared to their additive versions (Figure 8.4). e results suggest that the uniform visual a ention (UVA) along with the shared a ention system SS-CAT, bene t less from the visual modality. For English→French, DS-CAT and SD-CAT systems perform the best on average while for German the completely dedicated a ention DD-CAT obtains the best improvement. However, it should be noted that the di erences among the AMMT variants are small: e best French system is signi cantly be er than two AMMTs (p-value ≤ 0.05) while for German, the best system is not signi cantly be er than the other variants. When compared to SMMT systems, the best AMMT for German and French signi cantly improves over the EDINIT systems with 0.7 and 0.8 METEOR, respectively. In overall, we can say that an AMMT with at least some level of modality speci c dedication, outperforms any other MMT variants including the SMMT ones. Finally, Table 8.5 provides qualitative examples (more examples are provided in Appendix A) for entity masking experiments where the selected AMMT is the DD-CAT system. Similar to progressive masking, we observe that the MMT system successfully lls in the blanks with the help of the visual modality. When incongruently decoded, the AMMT model mostly behaves like the masked NMT and loses its ability to produce visually coherent sentences.

A manual inspection of visual a ention maps produced by the default, non-masked MMTs (chapter 7) and entity masked MMTs reveals that the a ention is much more plausible and "active" in the la er. An interesting example is given in Figure 8.5 where the masked MMT a ends to the correct region of the image and successfully translates a masked word that was otherwise a spelling mistake in the source sentence ("son" wri en as "song"). However, the non-masked MMT a ention is stuck at the lower right portion of the image. All masked MMT models are able to correctly translate this sentence unlike the non-masked ones that blindly rely on the spelling mistake.

Summary

In this chapter, I presented an in-depth study on the potential contribution of images for MMT. Speci cally, I analysed the behavior of our SMMT and AMMT systems under two degradation schemes where information is systematically removed from source sentences. e results show that the proposed SMMT and AMMT models successfully exploit the visual modality when the linguistic context is scarce, but tend to be less sensitive to the images when exposed to complete sentences during training. In the la er case, the language signal turns out to be su cient to accomplish the task and dominates the visual modality. We think that this dominance is expected since NMT is quite good at performing sequence-to-sequence transduction, leaving no space to the externally injected visual signal. Interestingly, this behavior corroborates the seminal work of [START_REF] Francis | Human sensory dominance[END_REF] in Psychophysics where it has been demonstrated that visual stimuli dominate over the auditory stimuli when humans are asked to perform a simple audiovisual discrimination task. In the light of these, it is likely that the majority of the current state-of-the-art models are a ected by this dominance since the adversarial evaluation did not reveal any signs of complete collapse in the literature [START_REF] Ellio | Adversarial evaluation of multimodal machine translation[END_REF][START_REF] Grönroos | MeMAD submission to the WMT18 multimodal translation task[END_REF] and also for our SMMT and AMMT models. We thus suspect that -at least for the Multi30K dataset -the consequences of integrating the visual modality are secondary, reminding previous work about the in uence of random perturbations to DNN training: [START_REF] Gulcehre | Noisy activation functions[END_REF] deliberately inject random noise into non-linear activation functions which in turn improves the dynamics of the gradients while [START_REF] Neelakantan | Adding gradient noise improves learning for very deep networks[END_REF] shows evidence of improved generalization when adding gaussian noise to the gradients.

Finally, the degradation experiments also reveal that the a entive models which integrate spatial features, perform signi cantly be er than the simple models that use global visual features. Our investigation also suggests that visual grounding can increase the robustness of MT systems by mitigating input noise such as errors in the source text.

CHAPTER9

Conclusion & Discussion

In this study, we concentrated on designing novel NMT systems that can leverage contextual information from auxiliary modalities. For this purpose, we speci cally worked with the Multi30K dataset [START_REF] Ellio | Multi30k: Multilingual english-german image descriptions[END_REF] which contains images and their translated descriptions. e visual context can be bene cial to this dataset as it can encourage MT systems to apply visually guided word sense disambiguation, missing word imputation, or gender marking between gender-neutral and gendered languages. Besides being an interesting task on its own, a successful MMT system is also important to foster research on multimodal language understanding in general.

We mainly explored two di erent multimodal approaches, which further determine the type of visual features used to represent the images. First, we extracted global visual features from state-of-the-art pre-trained CNN models and experimented with grounding the intermediate components of an NMT with vectorial image representations. For the second type of models, we again took advantage of pre-trained CNNs, but this time we extracted convolutional features that preserve spatial information unlike the previous global representations. ese richer features are then integrated into a novel "multimodal a ention" mechanism in the NMT, with the purpose of guiding the decoder to look at the image when translating a given sentence.

Upon extensive analyses based on automatic evaluation metrics, we observed moderate to signi cant improvements for English→German but the same did not hold for English→French. For the multimodal a ention based models, we quantitatively and qualitatively showed that L 2 normalization of the features is crucial for the visual attention to be e ective. To be er understand to what extent the visual modality is taken into account by the models, we conducted the adversarial evaluation protocol [START_REF] Ellio | Adversarial evaluation of multimodal machine translation[END_REF] and noticed that most of the models barely respond to incongruent decoding, some of them even mildly improving similar to what has been observed by [START_REF] Grönroos | MeMAD submission to the WMT18 multimodal translation task[END_REF]. is brought up the question of whether the proposed MMT systems are even architecturally capable of leveraging the visual modality or not. To that end, we arti cially created scenarios where the visual modality is "required" to perform well on the task, such as systematically removing su xes or masking out visually depictable nouns from the source sentences. is nal set of experiments clearly showed that the images are indeed taken into account by both global feature and spatial feature based MMT with the la er performing signi cantly be er than the former. We have also found evidence that the visual grounding can improve the robustness of MT systems by mitigating input noise such as spelling errors.

We now brie y discuss several perspectives and insights about the next steps in multimodal language learning.

Better MMT Approaches

Borrowing from the insights of this work, I think that it would be interesting to design MMT systems which integrate a sort of message passing mechanism across modalities: e modality a entions can then be guided by entropy-based gating mechanisms for example, so that at each timestep the more con dent modality can take over the other one when computing the a ended context. I also believe that there remains a lot to explore in terms of handling OOV words at inference time. Although subword segmentation mitigates the problem in theory, the issue is still there: Consider the source sentence "a path leads to a pagoda" where the OOV token "pagoda" gets segmented into "p@@ ag@@ o@@ da" using the BPE algorithm. Although the token is no longer an OOV, it is practically impossible for the current NMT and MMT models to generate a sequence of subwords that would form the French translation "pagode". In fact, BPE even encourages hallucination here as the model would be forced to translate the sequence of source embeddings [p@@, ag@@, o@@, da] into something1 . On the other hand, if we were to keep the token as an OOV, the model could detect it and a empt to refer to some kind of multimodal knowledge base in order to fetch most probable candidate words that would be integrated into the decoding logic. It may be possible to construct this knowledge base using state-of-theart pre-trained word embeddings and visual features in ways similar to visual bilingual lexicon induction methods [START_REF] Kiela | Visual bilingual lexicon induction with transferred convnet features[END_REF].

Better MMT Evaluation

During the extensive quantitative and qualitative analyses, we were o en faced with the question of how should the e ectiveness of an MMT be evaluated. Our sentence level analyses in the previous chapters clearly showed one thing: Even retraining a model with di erent random initialization yields substantially di erent translations for 70% to 85% of the test set. ese abrupt shi s from one run to another can be a confounding factor that may hide small but valuable improvements due to the multimodality. is is also not completely mitigated by human evaluation as humans will also show di erent levels of appreciations for this intrinsic translation variance: For example, we previously saw how the incongruent decoding replaced "footballspieler" with the wrong "footballspieler" version, an e ect unlikely to be related to multimodality. In light of these, I believe that the MMT systems should be evaluated with custom ne-grained protocols instead of corpus level metrics. An example of this was proposed for MT evaluation through a "challenge set" which probes the abilities of state-of-the-art MT systems in terms of several global and language-speci c linguistic phenomena [START_REF] Isabelle | A challenge set approach to evaluating machine translation[END_REF].

New Datasets & Tasks

Although a more challenging test set with ambiguous verb uses was published (Ellio et al., 2017) for Multi30K, there has not been any exciting results showing substantial improvements over NMT baselines. I believe that this makes sense as we do not know to what extent the training set of Multi30K is a ected by contextual ambiguities i.e. if the models are never challenged with multiple senses of a verb during training, it is quite unrealistic that they will be grounded to resolve such ambiguities at test time. e fact that the state-of-the-art baselines converged to extremely high BLEU and METEOR scores also suggest that we may need more challenging datasets for which the auxiliary modalities are vital. To that end, we proposed a new multimodal dataset called How2 which consists of more than 70K instructional videos with English subtitles and their crowd-sourced Portuguese translations (Sanabria et al., 2018). e unique combination of video, speech and bilingual subtitles allow the exploration of many tasks such as automatic speech recognition (ASR), speech translation and machine translation. For each one of them, a multimodal variant exists where the auxiliary modality can be visual and/or auditory.

Multimodal Speech Recognition

In the context of automatic speech recognition (ASR), the presence of a synchronized video stream of the narrator enables lipreading [START_REF] Joon | Lip reading sentences in the wild[END_REF], a technique to reduce the e ect of ambient noise. is approach can be de ned as a local grounding since the grounding happens between phonemes and their visual counterparts visemes. On the other hand, global grounding can always happen when the video consistently provides object, action and scene level cues correlated with the speech content as may be the case with the instructional videos of How2 dataset. Here, visual cues from the recording environment (indoor vs outdoor) or the interaction between salient objects (people, instruments, vehicles, tools and equipments) can be exploited by the recognizer in various ways to learn a be er acoustic and/or language model. In Caglayan et al. (2019b), we experimented with our EINIT, DINIT, EDINIT and VBOS grounding methods (Chapter 6), with the global visual features being extracted using the middle frame of video segments. We obtained moderate improvements of up to 1% reduction in word error rate using the EDINIT approach with other approaches performing mildly worse than it, similar to our MMT results in this work.

Simultaneous Contextual MT

In chapter 8, we showed the e ectiveness of the visual modality when sentence sufxes are systematically removed from the language input. is is an interesting insight which encourages us towards extending the currently available simultaneous NMT systems [START_REF] Cho | Can neural machine translation do simultaneous translation[END_REF][START_REF] Gu | Learning to translate in real-time with neural machine translation[END_REF][START_REF] Dalvi | Incremental decoding and training methods for simultaneous translation in neural machine translation[END_REF] with the visual modality. We believe this is a nice way of leveraging the multimodality which would potentially decrease the source context delay in simultaneous MT. une femme âgée en t-shirt bleu est assise sur un banc dans un parc an older woman in a blue t-shirt is si ing on a bank in the park AMMT: une femme âgée en maillot de bain rose est assise sur un rocher au bord de l'eau an older woman in a pink swimsuit is si ing on a rock at the seaside D 6 SRC: an older woman in a bikini [v]… NMT: une femme âgée en bikini est assise sur un banc dans un parc an older woman in bikini is si ing on a bank in the park AMMT: une femme âgée en bikini est assise sur un rocher au bord de l'eau an older woman in a swimsuit is si ing on a rock at the seaside D 8 SRC: an older woman in a bikini is sunbathing [v]… NMT: une femme âgée en bikini fait un bain de soleil sur un tro oir en ville an older woman in bikini is sunbathing on a sidewalk in the city AMMT: une femme âgée en bikini fait un salto arrière sur la plage an older woman in bikini performs a back loop in the beach D 10 SRC: an older woman in a bikini is sunbathing on a [v]… NMT: une femme âgée en bikini est en train de nager sur un banc dans un parc an older woman in bikini is swimming on a bank in the park AMMT: une femme âgée en bikini fait du soleil sur un rocher au bord de l'eau an older woman in bikini is sunbathing on a rock at the seaside D 12 SRC: an older woman in a bikini is sunbathing on a rock by [v]… NMT: une femme âgée en bikini nage sur un rocher au bord de l'eau an older woman in bikini swims on a rock at the seaside AMMT: une femme âgée en bikini fait du soleil sur un rocher au bord de l'eau an older woman in bikini is sunbathing on a rock at the seaside D SRC: an older woman in a bikini is sunbathing on a rock by the ocean NMT: une femme âgée en bikini fait du soleil sur un rocher au bord de l'océan AMMT: une femme âgée en bikini fait du soleil sur un rocher au bord de l'océan an older woman in bikini is sunbathing on a rock at the seaside Combined with the architectural flexibility of DNNs, this framework paved the way for further research in multimodality with the objective of augmenting the latent representations with other modalities such as vision or speech, for example. This thesis focuses on a multimodal machine translation (MMT) framework that integrates a secondary visual modality to achieve better and visually grounded language understanding. I specifically worked with a dataset containing images and their trans-lated descriptions, where visual context can be useful for word sense disambiguation, missing word imputation, or gender marking when translating from a language with gender-neutral nouns to one with grammatical gender system as is the case with English to French. I propose two main approaches to integrate the visual modality : (i) a multimodal attention mechanism that learns to take into account both sentence and convolutional visual representations, (ii) a method that uses global visual feature vectors to prime the sentence encoders and the decoders. Through automatic and human evaluation conducted on multiple language pairs, the proposed approaches were demonstrated to be beneficial. Finally, I further show that by systematically removing certain linguistic information from the input sentences, the true strength of both methods emerges as they successfully impute missing nouns, colors and can even translate when parts of the source sentences are completely removed.

  Column i of matrix A A i,j Element i, j of matrix A A A tensor A i,j,k Element (i, j, k) of a 3-D tensor A A :,:,i2-D slice of a 3-D tensor I n Identity matrix with n rows and n columns e (i) Standard basis vector [0, . . . , 0, 1, 0, . . . , 0] with a 1 at position i ∂y ∂x Partial derivative of y with respect to x ∇ x y Gradient of y with respect to x xii P

  Figure 2.2: Fully-connected networks with one and three hidden layers: e naming convention only re ects the number of hidden layers as in "single layer" and "3-layer".

Figure 2

 2 Figure 2.4: A simple FCNN for handwri en digit recognition: the dashed arrows on the le part indicate that the actual input is a a ened version of the 2D input image.

Figure 2 . 7 :

 27 Figure 2.7: A fully-connected layer with dropout regularization: bo om layer drops out half of its activations which is equivalent to multiplying them by zero.

Figure 2 . 8 :

 28 Figure 2.8: Computation graph of a simple linear regression model: y is the ground-truth value for this speci c input (x 1 , x 2 ) while the parameters are {w 1 , w 2 , b}.

  e basic RNN unfolded for three timesteps.

Figure 2 .

 2 Figure 2.10: A vanilla RNN and its unfolded view: in the right gure, the hidden states are shown with dashed lines. mmul signi es matrix multiplication.

  Figure 2.11: Backpropagation rough Time: the error backpropagates to each timestep (red). If x t 's are parameterized, the gradients will also ow towards them (bright red).

Figure 2 .

 2 Figure2.12: One-hot (le ) vs distributional (right) word representations 6 .

Figure 2 .

 2 Figure 2.13: e convolution of a 3x3 image with a 2x2 lter yields a 2x2 feature map.

Figure 2 .

 2 Figure 2.14: A ReLU convolutional layer with 4 lters of size 3x3x3: each (colored) lter applies a convolution with outputs indicated (•) in the corresponding feature maps.

Figure 2 .

 2 Figure 2.15: e feature compositionality of deep CNN models: high-level abstract concepts are represented using simpler ones. Figure adapted from Zeiler and Fergus (2014).

  Figure 2.16: 34-layer ResNet CNN with residual connections (He et al., 2016). Once trained, features are generally extracted from before or a er the nal GAP layer.

Figure 3 . 1 :

 31 Figure 3.1: NMT with constant source context: the (e)ncoder and the (d)ecoder are unfolded along the time axis. e dashed connections show the additional context inputs to GRU(Cho et al., 2014b). e orange o layer is the output logic.

Figure 3

 3 Figure 3.2: A decoding timestep (t = 1) with "dot-a ention" (Luong et al., 2015b): e transformations to the query and the encodings are omi ed for simplicity.

Figure 3 . 3 :

 33 Figure 3.3: Conditional GRU decoder: e hidden state of the rst GRU (d 1 ) becomes the query for the a ention. e context c t produced by the a ention is fed to the second GRU (d 2 ) as the input. e dashed connections refer to hidden state transitions.

  e sequential nature of RNNs prevents them from being parallelized across multiple devices during training. e parallelization is especially important when training largescale deep NMTs on massive amounts of parallel data, o en in the order of millions of sentences. ere has been many a empts to replace RNNs with deep CNNs and FC-NNs: Gehring et al. (2017) replace them by convolutional layers while Vaswani et al. (

Figure 4 . 1 :

 41 Figure 4.1: Bilingual subtasks of the shared task on MMT: An English→TRG model receives the image and the English sentence to be translated into TRG.

  Figure 4.2: Show and tell captioning system[START_REF] Vinyals | Show and tell: A neural image caption generator[END_REF]: e decoder is an LSTM which receives a visual feature vector as its very rst input.

Figure 4 . 3 :

 43 Figure 4.3: Multimodal decoder with visual a ention: At decoding timestep t = 2, the hypothetical decoder correctly generates "joueuse (female player)" instead of "joueur" by integrating the image information.

Figure 5

 5 Figure 5.1: e training work ow of nmtpy.

Figure 5 .

 5 Figure 5.1 summarizes the modular design and the training work ow of the toolkit: an experiment is fully de ned by a con guration le which sets the training options, the paths to the relevant training and test set les, the speci c model to be trained and its hyperparameters. Each input/output le is independently handled by the relevant iterator and a multimodal data loader coordinates these iterators to prepare minibatches of multimodal data. A model basically has to de ne a small set of methods to create the layers based on the received options and to realize the forward-pass. Finally, the training loop manages the whole training process where it also periodically evaluates the model using prede ned metrics from the metric inventory.Currently, nmtpy provides support for handling text les, arbitrary feature vectors, raw images and speech features in Kaldi format. As for the model inventory, it provides reference implementations for all the simple (chapter 6) and a entive MMT models (chapter 7) as well as a state-of-the-art speech recognition model and its multimodal extension(Caglayan et al., 2019b).Besides my own works in MMT, nmtpy has also been successfully used by other researchers primarily for machine translation[START_REF] Burlot | Word representations in factored neural machine translation[END_REF][START_REF] García-Martínez | Lium machine translation systems for WMT17 news translation task[END_REF][START_REF] Lala | She eld submissions for WMT18 multimodal translation shared task[END_REF] and also for multimodal summarization[START_REF] Libovickỳ | Multimodal abstractive summarization of opendomain videos[END_REF], phonemic transcriptions for text-to-speech[START_REF] Vythelingum | Acoustic-dependent phonemic transcription for text-to-speech synthesis[END_REF] and audio-visual dialog state tracking[START_REF] Sanabria | CMU Sinbad's submission for the DSTC7 AVSD challenge[END_REF]. e tool was also extensively used and developed by the "Grounded Sequence to Sequence Transduction" research group 3 within the Fi h Frederick Jelinek Memorial Summer Workshop in 2018.

Figure 6 . 1 :

 61 Figure 6.1: Visual summary of SMMT methods: f ∈ R 2048 is the feature vector extracted from ResNet-50 (He et al., 2016). Each model is characterized by one or more numbered paths as de ned in the right side index.

Figure 6 . 3 :

 63 Figure 6.3: Human judgment score vs METEOR for German MMT17 participants: Colors of the circles represent signi cantly di erent clusters based on Wilcoxon signed-rank test (p-value ≤ 0.05). Systems within a cluster are tied. e highlighted systems 2, 4 and 7 on the right are unconstrained. Figure adapted from Ellio et al. (2017).

Figure 7 . 1 :

 71 Figure 7.1: Spatial a ention mechanism a ends on the convolutional feature maps extracted from a raw image (Xu et al., 2015b).

Figure 7

 7 Figure 7.2: NMT with multimodal a ention mechanism: modality speci c contexts reach the fusion module which aims to compress the representations into a single vector.

Figure 7 . 3 :

 73 Figure 7.3: e qualitative impact of L 2 normalization for multimodal a ention: (Top, unnormalized) visual a ention does not make sense (bo om, normalized) the a ention shi s focus from "person" to the "mountain". e model is SS-SUM.

Figure 7 . 4 :

 74 Figure 7.4: Sentence level METEOR breakdown for a entive MMT systems: e results are ordered by German (DE) "wins -losses" gap.

Figure 7 . 5 :Figure 7 . 6 :

 7576 Figure 7.5: e normalized entropies of a ention distributions: the language a ention has consistently lower entropy than the visual a ention which converges to uniform distribution when the a ention is completely shared.

Figure 8 . 1 :

 81 Figure 8.1: State-of-the-art multimodal gains over corresponding baselines: e x-axis shows the baseline METEOR scores on English→German test2016 for a set of state-ofthe-art systems. e systems with shaded scores are the best SMMT and AMMT systems from this thesis.

Figure 8 . 2 :

 82 Figure 8.2: Multimodal gain in METEOR for progressive masking: e dashed gray curve indicates the percentage of non-masked words in the training set. e large dot marks the point a er which the gap surpasses 1 METEOR.

Figure 8 . 3 :

 83 Figure 8.3: Entity masking results for German and French SMMTs: e boundary between the colored bars represents the METEOR score of the given MMT system.

Figure 8 . 4 :

 84 Figure 8.4: Entity masking results for German and French AMMTs: e boundary between the colored bars represents the METEOR score of the given MMT system.

Figure 8 . 5 :

 85 Figure 8.5: e impact of source degradation to visual a ention: (top) Non-masked MMT translates the misspelled "son" (song → chanson) while the masked MMT (bo om) performs a correct translation ([v]→ enfant) by exploiting the visual modality.

  SRC: a jockey riding his [v][v] NMT: un jockey sur son vélo (a jockey on his bike) AMMT: un jockey sur son cheval REF: un jockey sur son cheval (a jockey on his horse) SRC: a shing net on the deck of a [v][v] NMT: un let de pêche sur la terrasse d'un bâtiment (a shing net on the terrace of a building) AMMT: un let de pêche sur le pont d'un bateau (a shing net on the deck of a boat) REF: un let de pêche sur le pont d'un bateau rouge (a shing net on the deck of a red boat) SRC: girls are playing a [v][v][v] NMT: des lles jouent à un jeu de cartes (girls are playing a card game) AMMT: des lles jouent un match de football REF: des lles jouent un match de football (girls are playing a football match) SRC: a child [v][v][v][v][v][v] NMT: un enfant avec des lune es de soleil en train de jouer au tennis (a child with sunglasses playing tennis) AMMT: un enfant est debout dans un champ de eurs (a child is standing in eld of owers) REF: un enfant dans un champ de tulipes (a child in a eld of tulips) Table A.2: Additional progressive masking examples: underlined and bold words highlight bad and good lexical choices, respectively. English translations are provided in parentheses. un homme vêtu d'un t-shirt bleu et d'un jean est assis sur un banc a man wearing a blue t-shirt and a jean is si ing on a bank AMMT: une femme en maillot de bain rouge est assise sur un rocher au bord de l'eau a woman in a red swimsuit is si ing on a rock at the seaside D 2 SRC: an older [v]… NMT: un vieil homme vêtu d'un t-shirt blanc et d'un jean est assis sur un banc an older man wearing a blue t-shirt and a jean is si ing on a bank AMMT: une femme âgée vêtue d'un maillot de bain rouge est assis sur un rocher au bord de l'eau an older woman wearing a red swimsuit is si ing on a rock at the seaside D 4 SRC: an older woman in [v]… NMT:

Figure A. 1 :

 1 Figure A.1: Additional visual a ention example for entity masking where terrier, grass and fence are dropped from the source sentence: (a) Non-masked MMT is not able to shi a ention from the salient dog to the grass and fence, (b) the a ention produced by the masked MMT rst shi s to the background area while translating "on lush green [v]" then focuses on the fence.

3

  Neural Machine Translation 3.1 e Language Modeling Perspective . . . . . . . . . . . . . . . . . . . . . 3.2 Phrase-based MT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Early Neural Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Sequence-to-Sequence NMT . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 Recurrent Encoder . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Recurrent Decoder . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.3 A ention Mechanism . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.4 Conditional GRU Decoder . . . . . . . . . . . . . . . . . . . . . . 3.4.5 Deep Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.6 Non-recurrent Approaches . . . . . . . . . . . . . . . . . . . . . . 3.4.7 Multitask Learning for NMT . . . . . . . . . . . . . . . . . . . . . 3.5 Evaluation of MT Outputs . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Translation Decoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Table 4 .

 4 2: OOV statistics for Multi30K test sets. Sentence percentages re ect the percentage of sentences containing at least one OOV word.

		English	German	French
	Split	Words Avg. Len Words Avg. Len Words Avg. Len	Sents
	train	380K	13.1	364K	12.6	416K	14.4	29000
	val	13.4K	13.2	13.1K	12.9	14.6K	14.4	1014
	test2016 13.0K	13.1	12.2K	12.2	14.3K	14.2	1000
	test2017 11.4K	11.4	10.9K	10.9	12.8K	12.8	1000
	testcoco	5.2K	11.4	5.2K	11.2	5.8K	12.5	461
	Total	423K	13.0	405K	12.5	464K	14.3	32475
	Table 4.1: Tokenized word and sentence statistics for Multi30K.
			English		German		French
		Sents (%) Words (%) Sents (%) Words (%) Sents (%) Words (%)
	test2016	11.8	1.0	23.8		2.5	12.3	1.0
	test2017	15.5	1.7	31.7		3.6	13.8	1.3
	testcoco	11.1	1.1	34.5		3.6	16.1	1.5

Table 5 .

 5 1: e common set of hyperparameters used in the thesis: the decoder embeddings are tied[START_REF] Wolf | Using the output embedding to improve language models[END_REF].

	to compute

Table 5 .

 5

2: NMT performance on test2016 with di erent segmentation schemes.

  Table 6.1: Hyperparameters and intermediate dimensions for SMMTs: the dimension v of the transformed visual features depends on the type of interaction.

	Name	Symbol & Value
	Source sentence length	S
	Target sentence length	T
	Embedding dim.	e = 200
	RNNs hidden dim.	h = 320
	Single textual encoding dim.	c = 2h = 640
	All textual encodings Global visual feature Transformed visual feature	H ∈ R S×640 f ∈ R 2048 v ∈ R v
	• Ozan Caglayan, Walid Aransa, Adrien Bardet, Mercedes García-Martínez, Fethi
	Bougares, Loïc Barrault, Marc Masana, Luis Herranz, and Joost van de Weijer.
	2017a. LIUM-CVC submissions for WMT17 multimodal translation task. In Pro-
	ceedings of the Second Conference on Machine Translation, Volume 2: Shared Task
	Papers. Association for Computational Linguistics, Copenhagen, Denmark, pages
	432-439.

Table 6

 6

	5) 70.8 ± 0.3 (↓ 0.3)

.2: Combined SMMT results on test2016 and test2017: Highlighted scores are signi cantly di erent than the NMT (p-value ≤ 0.05). Ordered by test2017 METEOR.

Table 6 .

 6 5: Comparison of state-of-the-art SMMTs on German test2016: TF stands for Transformer

	3 (↑ 2.0) Encoder Prep. & App.

Table 7 .

 7 1: Hyperparameters and intermediate dimensions for a entive MMTs. is chapter comprises the following published works: • Ozan Caglayan, Walid Aransa, Yaxing Wang, Marc Masana, Mercedes García-Martínez, Fethi Bougares, Loïc Barrault, and Joost van de Weijer. 2016a. Does multimodality help human and machine for translation and image captioning? In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics, Berlin, Germany, pages 627-633.

• Ozan Caglayan, Loïc

Barrault, and Fethi Bougares. 2016b

. Multimodal a ention for neural machine translation. Computing Research Repository arXiv:1609.03976. • Ozan Caglayan, Adrien Bardet, Fethi Bougares, Loïc Barrault, Kai Wang, Marc Masana, Luis Herranz, and Joost van de Weijer. 2018. LIUM-CVC submissions for WMT18 multimodal translation task. In Proceedings of the ird Conference on Machine Translation. Association for Computational Linguistics, Belgium, Brussels, pages 603-608.

  while the la er shares a single W d for the context transformations. In fact,

	EN→DE		METEOR	BLEU
	System	SUM	CONCAT	SUM	CONCAT
	NMT		58.4 ± 0.3	38.9 ± 0.8
	DD + L 2	56.3 ± 0.4 (↓ 2.1) 56.9 ± 0.3 (↓ 1.5) 58.2 ± 0.3 (↓ 0.2) 58.0 ± 0.2 (↓ 0.4)	36.5 ± 0.6 (↓ 2.4) 37.4 ± 0.6 (↓ 1.5) 39.3 ± 0.4 (↑ 0.4) 38.6 ± 0.3 (↓ 0.3)
	SS + L 2	56.4 ± 0.4 (↓ 2.0) 57.4 ± 0.1 (↓ 1.0) 58.2 ± 0.3 (↓ 0.2) 58.4 ± 0.2	37.0 ± 0.4 (↓ 1.9) 37.1 ± 0.4 (↓ 1.8) 38.7 ± 0.2 (↓ 0.2) 38.9 ± 0.2
	SD + L 2	56.9 ± 0.1 (↓ 1.5) 57.0 ± 0.3 (↓ 1.4) 58.4 ± 0.7 58.1 ± 0.2 (↓ 0.3)	37.4 ± 0.6 (↓ 1.5) 37.3 ± 0.3 (↓ 1.6) 39.0 ± 0.6 (↑ 0.1) 38.6 ± 0.7 (↓ 0.3)
	DS + L 2	56.8 ± 0.4 (↓ 1.6) 57.2 ± 0.2 (↓ 1.2) 58.7 ± 0.1 (↑ 0.3) 58.5 ± 0.1 (↑ 0.1)	37.3 ± 0.6 (↓ 1.6) 36.8 ± 0.4 (↓ 2.1) 39.4 ± 0.1 (↑ 0.5) 39.2 ± 0.4 (↑ 0.3)

Table 7 .

 7 3: e impact of L 2 normalization on MMT performance on test2016: All di erences are against the baseline NMT.

  1, p. 66).

		test2016		test2017
	BLEU	METEOR	BLEU	METEOR
		English→German
	NMT 38.9 ± 0.8 SD-SUM 39.0 ± 0.6 (↑ 0.1) 58.4 ± 0.7 58.4 ± 0.3 DD-SUM 39.3 ± 0.4 (↑ 0.4) 58.2 ± 0.3 (↓ 0.2) DD-CAT 38.6 ± 0.3 (↓ 0.3) 58.0 ± 0.2 (↓ 0.4) SS-CAT 38.9 ± 0.2 58.4 ± 0.2 SS-SUM 38.7 ± 0.2 (↓ 0.2) 58.2 ± 0.3 (↓ 0.2) SD-CAT 38.6 ± 0.7 (↓ 0.3) 58.1 ± 0.2 (↓ 0.3) DS-SUM 39.4 ± 0.1 (↑ 0.5) 58.7 ± 0.1 (↑ 0.3) DS-CAT 39.2 ± 0.4 (↑ 0.3) 58.5 ± 0.1 (↑ 0.1)	32.1 ± 1.1 32.4 ± 0.6 (↑ 0.3) 52.5 ± 0.2 52.5 ± 0.7 32.5 ± 1.1 (↑ 0.4) 52.6 ± 0.4 (↑ 0.1) 32.3 ± 0.8 (↑ 0.2) 52.6 ± 0.5 (↑ 0.1) 32.3 ± 0.2 (↑ 0.2) 52.6 ± 0.4 (↑ 0.1) 32.7 ± 0.4 (↑ 0.6) 52.7 ± 0.2 (↑ 0.2) 32.5 ± 0.2 (↑ 0.4) 52.8 ± 0.2 (↑ 0.3) 32.6 ± 0.4 (↑ 0.5) 52.9 ± 0.3 (↑ 0.4) 32.7 ± 0.2 (↑ 0.6) 52.9 ± 0.5 (↑ 0.4)
		English→French
	NMT 61.4 ± 0.3 SD-CAT 61.0 ± 0.7 (↓ 0.4) 76.2 ± 0.4 (↓ 0.2) 76.4 ± 0.2 DS-CAT 61.5 ± 0.1 (↑ 0.1) 76.3 ± 0.1 (↓ 0.1) SS-CAT 61.3 ± 0.4 (↓ 0.1) 76.2 ± 0.3 (↓ 0.2)	54.4 ± 0.3 53.8 ± 0.5 (↓ 0.6) 70.8 ± 0.4 (↓ 0.3) 71.1 ± 0.2 54.0 ± 0.1 (↓ 0.4) 70.8 ± 0.1 (↓ 0.3)

Table 7 .

 7 5: Uniform visual a ention (UVA) on German test2017: UVA obtains the best average BLEU as well as a competitive METEOR. Highlighted scores are signi cantly di erent than the NMT (p-value ≤ 0.05). All systems are CONCAT variants.

Table 7 .

 7 6: Comparison of state-of-the-art AMMTs on German test2016: TF stands for Transformer

		System BLEU	METEOR Description
	Caglayan et al. (2016a)	RNN 29.3 (↓ 4.6) 48.5 (↓ 4.3) Shared A ention
	Helcl and Libovický (2017)	RNN 31.9 (↓ 2.7) 49.4 (↓ 2.3) Hierarchical A ention
	Calixto et al. (2016)	RNN 28.8	49.6	Separate A ention
	Arslan et al. (2018)	TF 41.0 (↑ 2.4) 53.5 (↓ 1.5) Parallel A ention
	Calixto et al. (2017b)	RNN 36.5 (↑ 2.8) 55.0 (↑ 2.7) β-gated A ention
	Caglayan et al. (2017a)	RNN 37.0 (↓ 1.1) 57.0 (↓ 0.3) Separate A ention
	Libovický et al. (2018)	TF 38.6 (↑ 0.3) 57.4 (↑ 0.7) Parallel A ention
	Caglayan et al. (2016a) PBMT 36.2 ( 0.0) 57.5 (↑ 0.1) Reranking (Visual NLM)
	Delbrouck and Dupont (2017a)	RNN 40.5	57.9	BNM + Enc. A ention
	SMMT (chapter 6)	RNN 39.0 (↑ 0.1) 58.5 (↑ 0.1) EDINIT
		39.5 (↑ 0.6) 58.6 (↑ 0.2) DINIT
	AMMT ( is chapter)	RNN 39.4 (↑ 0.5) 58.7 (↑ 0.3) DS-SUM + L 2

Table 8 .

 8 . 1: Adversarial evaluation of SMMT systems on English→German test2016: e incongruently decoded EMUL system is signi cantly di erent (p-value ≤ 0.05) than its congruent counterpart with respect to METEOR. e ∆'s are computed by subtracting the congruent mean from the incongruent mean.

		BLEU			METEOR
	Congruent Incongruent	∆	Congruent Incongruent	∆
	NMT 38.9 ± 0.8 VBOS 38.9 ± 0.1 TMUL 38.8 ± 0.1 SMUL 39.0 ± 0.6 EINIT 39.6 ± 0.4 DINIT 39.5 ± 0.1 EDINIT 39.0 ± 0.4 EMUL 38.6 ± 0.4	39.0 ± 0.1 38.9 ± 0.1 39.0 ± 0.6 39.5 ± 0.7 39.4 ± 0.1 38.8 ± 0.4 38.2 ± 0.1	↑ 0.1 ↑ 0.1 0.0 ↓ 0.1 ↓ 0.1 ↓ 0.2 ↓ 0.4	58.4 ± 0.3 58.3 ± 0.2 58.3 ± 0.2 58.2 ± 0.4 58.4 ± 0.2 58.6 ± 0.3 58.5 ± 0.3 58.1 ± 0.3	58.4 ± 0.1 58.3 ± 0.1 58.2 ± 0.3 58.5 ± 0.3 58.6 ± 0.3 58.4 ± 0.4 57.8 ± 0.4	↑ 0.1 0.0 0.0 ↑ 0.1 0.0 ↓ 0.1 ↓ 0.3

Table 8 .

 8 . 2: Adversarial evaluation of AMMT systems on English→German test2016: e incongruently decoded DS-CAT system is signi cantly di erent (p-value ≤ 0.05) than its congruent counterpart with respect to BLEU.

		BLEU			METEOR
	Congruent Incongruent	∆	Congruent Incongruent	∆
	NMT 38.9 ± 0.8 SS-SUM 38.7 ± 0.2 DD-CAT 38.6 ± 0.3 UVA 39.3 ± 0.4 SD-SUM 39.0 ± 0.6 SD-CAT 38.6 ± 0.7 DD-SUM 39.3 ± 0.4 DS-SUM 39.4 ± 0.1 SS-CAT 38.9 ± 0.2 DS-CAT 39.2 ± 0.4	39.3 ± 0.3 38.6 ± 0.2 39.2 ± 0.6 38.9 ± 0.8 38.5 ± 0.6 39.0 ± 0.7 39.1 ± 0.2 38.6 ± 0.4 38.8 ± 0.4	↑ 0.6 0.0 ↓ 0.1 ↓ 0.1 ↓ 0.1 ↓ 0.3 ↓ 0.3 ↓ 0.3 ↓ 0.4	58.4 ± 0.3 58.2 ± 0.3 58.0 ± 0.2 58.2 ± 0.3 58.4 ± 0.7 58.1 ± 0.2 58.2 ± 0.3 58.7 ± 0.1 58.4 ± 0.2 58.5 ± 0.1	58.5 ± 0.1 58.0 ± 0.3 58.3 ± 0.4 58.4 ± 0.6 58.1 ± 0.3 58.3 ± 0.2 58.6 ± 0.2 58.2 ± 0.3 58.2 ± 0.1	↑ 0.3 0.0 ↑ 0.1 0.0 0.0 ↑ 0.1 ↓ 0.1 ↓ 0.2 ↓ 0.3

Table 8 .

 8 3: A depiction of the proposed text degradations: D is the original test set.

Table 8 . 5

 85

	a														une	mère	et	sa
	mother
	and
	her
	young
	song													jeune	chanson	profitant	d'	une
	enjoying
	a
	beautiful
	day
	outside													belle	journée	dehors	.	<eos>
	.
	<eos>
	une	mère	et	sa	jeune	chanson	profitant	d'	une	belle	journée	dehors	.	<eos>
	a														une	mère	et	son
	[v]
	and
	her
	young
	[v]													jeune	enfant	profitant	d'	une
	enjoying
	a
	beautiful
	[v]
	outside													belle	journée	dehors	.	<eos>
	.
	<eos>
	une	mère	et	son	jeune	enfant	profitant	d'	une	belle	journée	dehors	.	<eos>

: Entity masking examples from English→French models: underlined and bold words highlight bad and good lexical choices, respectively. English translations are provided in parentheses. e red INC lines are incongruent AMMT outputs.

Table A .

 A 3: Successive outputs from progressively masked NMT and AMMT.

	a boston																un	terrier	de	boston
	terrier
	is
	running
	on lush															court	sur	l'	herbe	verte
	green
	grass
	in
	front of															luxuriante devant	une	clôture	blanche
	a
	white
	fence
	.															.	<eos>
	<eos>
	un	terrier	de	boston	court	sur	l'	herbe	verte	luxuriante	devant	une	clôture	blanche	.	<eos>
																(a) Non-masked MMT
	a boston																un	berger	allemand	court
	[v]
	is
	running
	on lush															sur	l'	herbe	verte	devant
	green
	[v]
	in
	front
	of a															une	clôture	blanche	.	<eos>
	white
	[v]
	.
	<eos>
	un	berger	allemand	court	sur	l'	herbe	verte	devant	une	clôture	blanche	.	<eos>
																(b) Entity-masked MMT

Illustration adapted from Huang et al. (2017a) with permission.

In general, L 2 penalty term is not applied to biases[START_REF] Goodfellow | Deep Learning[END_REF].

Note that the subscripts are in bold here compared to the notation x t previously used to denote the t-th element of a vector.

e music was so loud that it could not be enjoyed.

I like this book a lot because it provides an introduction to some concepts that my thesis will be based on, " she replied.

Figure adapted from Holger Schwenk's slides for his talk entitled Neural Machine Translation and Universal Multilingual Representations.

We limit ourselves to square inputs and lters here since we will be working with square images.

e bias terms are omi ed for simplicity.

Google Translate palliated this problem by suggesting alternative translations to the user.

https://competitions.codalab.org/competitions/19917

https://github.com/nyu-dl/dl4mt-tutorial

https://github.com/lium-lst/nmtpytorch

www.clsp.jhu.edu/workshops/18-workshop/grounded-sequence-sequence-transduction

https://pytorch.org/docs/stable/torchvision/models.html

We discovered French hallucinations such as "pylessive" and "limetière" when NMT and MMT systems translate the word "pagoda".

Acknowledgements

SRC: trees are in front [v][v][v][v][v] REF: des arbres sont devant une grande montagne (trees are in front of a big mountain) NMT: des vélos sont devant un bâtiment en plein air (bicycles are in front of an outdoor building) AMMT: des arbres sont devant la montagne (trees are in front of the mountain) INC: des taxis sont devant la fenêtre d'une voiture (taxis are in front of the window of a car)

des lles agitent des drapeaux violets tandis qu'elles dé lent dans la rue (girls wave purple ags as they parade down the street) NMT: des lles en t-shirts violets sont assises sur des chaises dans une salle de classe (girls in purple t-shirts are si ing on chairs in a classroom) AMMT: des lles en costumes violets dansent dans une rue en ville (girls in purple costumes dance on a city street) INC: des lles en maillots rouges faisant du vélo dans une rue en ville (girls in red shirts riding a bicycle in a city street)

une femme âgée en bikini bronze sur un rocher au bord de l'océan (an older woman in bikini is tanning on a rock at the edge of the ocean) NMT: une femme âgée avec un t-shirt blanc et des lune es de soleil est assise sur un banc (an older woman with a white t-shirt and sunglasses is si ing on a bank) AMMT: une femme âgée en maillot de bain rose est assise sur un rocher au bord de l'eau (an older woman with a pink swimsuit is si ing on a rock at the seaside) INC: une femme âgée en t-shirt blanc est debout à côté d'un grand arbre (an older woman in white t-shirt is standing next to a large tree) Finally, if we look at the incongruently decoded AMMT outputs, we can see that the models start to hallucinate, con rming that the e ect of visual features is not random.

In overall, we conclude that the models are able to guide the decoder to produce both uent and visually adequate sentences and when doing so they are not merely retrieving sentences out of the training set. More examples are provided in appendix A. Table A.3 is especially interesting as it compares the successive outputs of the NMT and the MMT for a set of masked datasets D k .

Entity Masking

Here we take advantage from an extension of Flickr30K dataset which provides coreference chains to annotate visually depictable entities in the image descriptions [START_REF] Bryan A Plummer | Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models[END_REF]. Since Multi30K is derived from Flickr30K, we can exploit these annotations for the source-side train, val and test2016 sentences. Speci cally, we replace every annotated noun with a special token [v] as in the case of progressive masking. e annotations are not limited to single nouns but can extend to noun phrases such as "a blue

APPENDIXA

Additional Masking Examples