Rachel Bawden Limsi
email: rachel.bawden@limsi.fr

Machine Translation of Speech-Like Texts: Strategies for the Inclusion of Context

Keywords: traduction automatique, contexte, parole, genre, pronoms, tag questions machine translation, context, speech-like texts, gender, pronouns, tag questions

Whilst the focus of Machine Translation (MT) has for a long time been the translation of planned, written texts, more and more research is being dedicated to translating speech-like texts (informal or spontaneous discourse or dialogue). To achieve high quality and natural translation of speechlike texts, the integration of context is needed, whether it is extra-linguistic (speaker identity, the interaction between speaker and interlocutor) or linguistic (coreference and stylistic phenomena linked to the spontaneous and informal nature of the texts). However, the integration of contextual information in MT systems remains limited in most current systems. In this paper, we present and critique three experiments for the integration of context into a MT system, each focusing on a different type of context and exploiting a different method: adaptation to speaker gender, cross-lingual pronoun prediction and the generation of tag questions from French into English.

RÉSUMÉ

Traduction automatique de l'« oral-écrit » : Stratégies pour l'intégration du contexte Bien que la Traduction Automatique (TA) se soit concentrée jusqu'à présent sur la traduction de textes écrits et édités, de plus en plus de travaux sont consacrés à la traduction de textes informels et spontanés (discours et dialogues). Pour traduire de tels textes relevant de l'« oral-écrit », il devient indispensable de prendre en compte des informations contextuelles, qu'elles soient de nature extra-linguistique (identité du locuteur, interaction entre le locuteur et l'interlocuteur) ou linguistique (coréférence et phénomènes stylistiques propres à la parole). Or l'intégration d'informations contextuelles dans les systèmes de TA reste limitée dans la plupart des systèmes actuels. Dans cet article, nous présentons et analysons trois expériences d'intégration du contexte dans un système de TA mettant en jeu des formes de contexte et donc des méthodologies différentes: l'adaptation au genre du locuteur, la traduction de pronoms et la génération de « tag questions » anglaises à partir du français.

Introduction

Speech-like texts (social networking, speech transcriptions, subtitles and other informal written exchanges) 1 pose new challenges for Natural Language Processing (NLP) and in particular Machine Translation (MT), when compared to the genres of parliamentary discourse and journalistic texts, which have been the main focus of text processing applications to date. As the need for high quality MT of this more informal genre increases, more and more MT research is being dedicated to exploring the difficulties speech-like texts present and how we might hope to overcome them [START_REF] Hardmeier | Discourse in statistical machine translation. a survey and a case study[END_REF][START_REF] Guillou | Incorporating Pronoun Function into Statistical Machine Translation[END_REF].

Speech-like texts are fundamentally different from planned written texts for a number of different reasons. One of these is their highly contextualised nature. Whilst it is true that all texts are related to a certain genre and context of production, the extra-linguistic context of a speech-like production is very often essential to understanding its linguistic content and therefore to producing accurate and coherent translations. Unlike journalistic texts, which are addressed to a heterogenous audience, absent at the moment of the text's production, in discourse, the speaker and the interaction with the listener can greatly influence the style (in terms of formality and politeness), vocabulary choices and even the grammatical agreement (in languages that have grammatical gender agreement for example) of the texts. Spontaneity of production allows speakers to continuously adapt to each other's reactions, making the aforementioned contextual aspects even more important. There is also the added possibility that speakers align to each other's linguistic choices, make reference to entities mentioned in the other speaker's speech turn and rely more on implicit common ground to root their speech, potentially resulting in more ambiguity for automatic processing.

Traditional MT methods suffer from a considerable flaw when it comes to dealing with context, this being that sentences are processed independently of each other, and translation choices are often made based on very local context, which is especially true of phrase-based statistical MT (SMT) techniques. Whilst attentional Neural MT (NMT) approaches can partially alleviate this problem, the degree to which we can control which context is used in translation is limited. As for extra-linguistic information, it must first be identified and then integrated into the translation process, through pre-processing, during decoding or as a post-edition step.

We illustrate the different strategies that can be used to integrate contextual information into the MT process by presenting a discussion orientated around three separate experiments for the language pair English-French, each illustrating the integration of a different type of contextual information. The scope and nature of context are very different in each case, presenting different challenges to MT. The experiments described illustrate the nature of the different problems faced, as well as provided a basis for the critique of the methods used, highlighting the fact that the problem is far from being solved.

The first experiment is an illustration of the use of domain adaptation, a very simple technique used to take into account a very coarse-grained and static type of information, that of speaker gender, which applies on the sentence-level (Section 2). The second experiment deals with a finer notion of context, necessitating a more sophisticated modelling of an utterance's linguistic context, in a task to predict the French translation of the English subject pronouns it and they, which, unlike their equivalent French pronouns, are unmarked for gender (Section 3). In this case, a very specific contextual element (the antecedent) determines the translation of a very specific textual element (the pronoun). Finally, in Section 4, we present our third experiment, a post-editing task to improve the translation of a particular stylistic phenomenon common in spoken English, the English tag question, whose usage is largely determined by speaker attitude and the interaction between speaker and listener. Whilst the phenomenon affected by context (the presence and form of the English tag question) is clearly identified, determining which context can be useful is much more complex. never produced orally, as in the case of social network posts and discussions.

Speaker identity (as defined by demographic factors such as age, gender and social background, as well as other aspects such as personality and mood) greatly influences the way in which speakers communicate, in terms of lexical choices, syntactic structure, politeness strategies, etc. This variation has long been studied in the sociolinguistics literature, in particular with respect to the impact of gender on communication style [START_REF] Lakoff R | Language and woman's place: text and commentaries[END_REF][START_REF] Coates | Women, Men and Language: A Sociolinguistic Account of Gender Differences in Language[END_REF]. Whilst speaker adaptation has long been a part of speech recognition systems [START_REF] Kuhn R | Eigenvoices for speaker adaptation[END_REF], only recently has research in NLP turned towards adapting systems to take into account this extra-linguistic information (cf. works by [START_REF] Volkova | Exploring demographic language variations to improve multilingual sentiment analysis in social media[END_REF] for sentiment analysis and [START_REF] Hovy | Demographic Factors Improve Classification Performance[END_REF] for three separate NLP classification tasks). Speaker identity is just as important in MT; van der [START_REF] Van Der | Measuring the Effect of Conversational Aspects on Machine Translation Quality[END_REF] show how MT performance fluctuates with respect to individual speaker identity, gender and register in five different language pairs for film and TV subtitles, and [START_REF] Mirkin | Motivating Personality-aware Machine Translation[END_REF] find that MT translated output does not conserve speaker personality and gender traits as well as human translations do.

One of the simplest methods for integrating coarse-grained, sentence-level information such as aspects of speaker identity is to perform domain adaptation. Domain in this context refers to a broader concept than the genre of text and can include any sentence-level aspect representing the class of sentence, such as sentence type, speaker gender and formality. Domain adaptation is a well-known approach in MT and is used almost systematically. In its simplest form, it implies simply choosing an in-domain development set to tune the system (cf. [START_REF] Pecina | Simple and Effective Parameter Tuning for Domain Adaptation of Statistical Machine Translation[END_REF] for a study of the effectiveness of in-domain tuning for SMT systems when dealing with a low-resource domain). But its implementation can be more complex, for example through similarity-based data selection techniques for both training and tuning data [START_REF] Axelrod | Domain Adaptation via Pseudo In-Domain Data Selection[END_REF] or by weighting different components of the MT system that have been trained on class-partitioned data [START_REF] Kuhn R | Mixture-Model Adaptation for SMT[END_REF][START_REF] Finch | Class-Dependent Modeling for Dialog Translation[END_REF].

Experiments and results

Our preliminary experiments into the use of domain adaptation to take into account the gender of the speaker (Bawden et al., 2016) give us the opportunity to critique this method, in particular in terms of the possibility of extending the technique to multiple aspects of speaker identity. Similar experiments on gender adaptation were performed concurrently by van der Wees et al. (2016) and [START_REF] Wang | Automatic construction of discourse corpora for dialogue translation[END_REF]. Speaker gender is a good starting point for integrating information concerning speaker identity, since it is relatively easy to identify (through knowledge of the person's name if the text is written) and is generally binary. As well as influencing word choice and style, speaker gender is also explicitly visible in languages which display grammatical gender agreement with the speaker. For example, the English sentence "I am happy" would be translated into French "Je suis heureuse" for a female speaker and "Je suis heureux" for a male speaker. When translating from English into French, standard MT systems are limited to selecting the most probable form given the linguistic context, which in most cases is the masculine form "heureux".

We performed a series of simple domain adaptation experiments, consisting of adapting a standard SMT system trained using Moses (Koehn et al., 2007), by modifying both the data used to train the components of the system and the data used for tuning. Our aim was to evaluate the effect on translation performance of training gender-specific language models and gender-specific translation models and of tuning to a gender-specific dataset. The idea of domain adaptation by data partitioning is to annotate the sentences of the data according to the different aspects being studied (the different domains in the broader sense of the term) and to use the data selectively in the training and tuning of the MT system based on these annotations. In our case, the domain was the gender of the speaker, so each dataset was divided into two, corresponding to the sentences spoken by female characters and those spoken by male characters. 2 We used subtitles annotated for speaker gender, taken from the TVD Big Bang Theory reproducible corpus [START_REF] Guinaudeau | TVD: A Reproducible and Multiply Aligned TV Series Dataset[END_REF], resulting in approximately 10,000 gender-annotated utterances. The results of the automatic evaluation are summarised in Table 1, using two automatic metrics, BLEU [START_REF] Papineni | BLEU: A Method for Automatic Evaluation of Machine Translation[END_REF] and METEOR [START_REF] Denkowski | The Meteor metric for automatic evaluation of machine translation[END_REF]. 3 The baseline system is shown in the first row of the table -no specific adaptation is performed and all development data (containing both male and female speakers' utterances) is used to tune the model. The results show that small gains in translation performance can be obtained from domain adaptation, although these improvements are not significative. Providing a gender-specific tuning set gives gains of +0.1 BLEU and +0.004 METEOR for the male speakers' utterances, and gains of +0.56 BLEU for the female speakers' utterances. Adding a language model that has been trained exclusively on female or male utterances also provides some very slight improvements for both genders, as does the addition of a gender-specific translation model. However a manual evaluation revealed that the slight improvements seen were not due to improvement in grammatical gender agreement at all. Most improvements were thanks to an improved lexical choice when compared to the baseline translation, followed by lexical additions or deletions. It appears that domain adaptation did indeed adapt slightly to the differently gendered datasets, but not necessarily for the reasons we might think; each dataset has its own distinct lexical properties, and it is possible that the improvements were simply due to minor lexical specificities of the two datasets and not necessarily due to a real gender bias. Data sparsity, in particular for female speakers is a problem with this method, which could in part explain why the male-adapted model performs highly on the female-specific test set.

Analysis and perspectives

The main problems found with this method is that data partitioning leads to smaller datasets being used for training and tuning. MT is a domain in which having large amounts of data is a must to produce a robust and high-performing model, so the decision to reduce the size of the data used is not one that should be taken lightly.

It is difficult to imagine this method being extended to integrate other coarse-grained, sentence-level features. In theory, the method is simple, requiring simply annotating data according to the set of features (e.g. sentence type, formality, topic). However, in practice, partitioning of the data depending on the set of class labels, whose number is multiplied each time a new feature is added, would result in smaller and smaller datasets on which to train and tune models, which would inevitably lead to degradation in translation performance. A solution to this problem proposed by [START_REF] Saluja | Context-aware Language Modeling for Conversational Speech Translation[END_REF], which involves weighting different sentences according to a similarity measure between the test sentence and each training sentence (based on their set of class labels), showed some improvements, but in a limited domain. The use of factored translation models (Koehn & Hoang, 2007) allows more linguistic input to be provided for each of the words, but has had limited impact on translation quality and suffers from computational problems when scaling up to large datasets.

Recently, a new method of performing domain adaptation has been introduced, in the context of neural machine translation (NMT). [START_REF] Sennrich | Linguistic Input Features Improve Neural Machine Translation[END_REF] show how it is possible to control for politeness on the sentence-level by introducing an additional feature (side constraint) as an arbitrary feature at the beginning of each sentence. The flexibility of this approach appears promising for integrating a range of different sentence-level aspects, including extra-linguistic information related to speaker identity.

Linguistically motivated cross-lingual pronoun prediction

Other forms of context necessarily require a more sophisticated representation of the linguistic properties of discourse. One of the most studied contextual aspects of discourse is coreference and the translation of pronouns, particularly between languages that do not have the same system of grammatical gender [START_REF] Le | Aiding pronoun translation with co-reference resolution[END_REF][START_REF] Guillou | Incorporating Pronoun Function into Statistical Machine Translation[END_REF]. Take for example the language pair English-French. French common nouns and pronouns are marked for grammatical gender, whereas the English pronouns it and they are not. In order to correct translate the pronoun it in the English sentence "it was blue", it is necessary to know the grammatical gender of the French coreferential antecedent of the pronoun. For example, if it refers to a box (une boîte), the correct translation is the feminine pronoun elle, whereas if it refers to a toe (un orteil), the correction translation is the masculine pronoun il. There is also a third option (which can be used in certain contexts) to translate it using the gender-neutral demonstrative ce.

The problem of ensuring accurate pronominal translation has received much interest in the discourse in MT community and a shared task has been organised for the past few years, dedicated to providing solutions to the problem (e.g. [START_REF] Guillou | Findings of the 2016 WMT Shared Task on Cross-lingual Pronoun Prediction[END_REF]. The task is a cross-lingual pronoun prediction task, for which participating classification systems aim to correctly predict target pronoun forms in the target sentence, based on contextual information in the source and target sentences. We focus on one of the language directions proposed by the 2016 task: English into French, for which the aim was to predict the correct French translation of the English subject pronouns it and they, from the set of possible classes il, ils, elle, elles, on, ce, cela and OTHER. An example of the data provided for the task is shown in Figure 3. Target sentences were provided as lemma-tag pairs, rather than as surface forms, to avoid systems relying solely on the grammatical gender of the immediately surrounded words to determine gender, which is an unrealistic scenario in a real translation setting.4

Pronoun class Lemma+tag English source sentence French target sentence Word alignment ils il|PRON They 're extremely costly . REPLACE_0 être|VER très|ADV coûteux|ADJ .|. 0-0 1-1 2-2 3-3 4-4 Figure 1: An example of pronoun prediction task data at WMT 2016. The first column represents the class label to be predicted and the second column the lemma and tag of the pronoun. The position of the pronoun to be predicted is marked by the placeholder REPLACE_0 in this example.

A variety of different strategies were used by task participants, who relied on different amounts of contextual, linguistic information. The winning system [START_REF] Luotolahti | Cross-lingual pronoun prediction with deep recurrent neural networks[END_REF] was a stacked recurrent neural network (RNN) system, which did not rely on any other contextual information than the information provided (as shown in Figure 3) and did not look beyond sentence boundaries. A second neural network approach also scored highly in the task: Dabre et al. (2016) used a simple RNN architecture with an attention mechanism and trained only using IWSLT data (whereas the winning system was trained on all data provided). Many of the other participants, including the current authors, chose to concentrate on the integration of more linguistic information into simpler (mainly linear) classification systems. Many participants used part-of-speech (PoS)-tagging, parsing and coreference chains. Stymne's ( 2016) system, a linear classifier using multiple linguistic annotations, was ranked second out of nine systems, showing that simple linear classifiers can perform just as highly as more sophisticated architectures such as neural network systems if good quality linguistic information is used.

Experiments and results

Our system (Bawden, 2016), followed this second strategy of including as much relevant linguistic knowledge into the system as possible. One of the main aims of our submission was to evaluate the capacity of linguistic tools and resources to provide accurate linguistic knowledge that, if perfect, should be sufficient to predict the correct pronoun. We used a variety of linguistic annotations and heuristics to provide features to a random forest classifier, implemented in Scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine Learning in Python[END_REF], relying heavily on linguistic tools and external sources. As source-side information, we used automatic PoS-tagging, dependency parsing and coreference resolution annotations, provided by the Stanford toolkit [START_REF] Manning | The Stanford CoreNLP Natural Language Processing Toolkit[END_REF]. On the target-side, we used dependency parsing using the Mate Parser [START_REF] Bohnet | A Transition-Based System for Joint Part-of-Speech Tagging and Labeled Non-Projective Dependency Parsing[END_REF]) and a parse model trained on lemmas and morphological information found in the morphological and syntactic lexicon, the Lefff [START_REF] Sagot | The Lefff, a Freely Available and Large-coverage Morphological and Syntactic Lexicon for French[END_REF]. We also integrated the prediction provided by the language model baseline [START_REF] Tiedemann | A Linear Baseline Classifier for Cross-Lingual Pronoun Prediction[END_REF] and identified local windows of syntactic and morphological patterns that were particularly linked to certain classes. 5We chose to explicitly perform coreference resolution, using the Stanford toolkit. Since the tool is not available for French, we performed coreference resolution on the English source sentence and Table 2: A decomposition of results on the test set for our submission to the WMT 2016 cross-lingual pronoun prediction task. The results are slightly higher than those reported in the official scoreboard due to the resolution of a minor bug found after the submission deadline.

automatically transferred the information through the alignments to the French target sentence. As shown in Figure 2, (i) we used the automatic alignments to find the pronoun to which the placeholder is aligned, (ii) we then use the English coreference chains to identify the antecedent of the English pronoun, and (iii) we again use the automatic alignment to identify the French antecedent of the pronoun. The gender of the pronoun is given by the French antecedent (found in the lexicon) and number is provided by the number of the English aligned pronoun. Both of these values were used alongside the other features in our classifier. The breakdown of the results of our system is shown in Table 2. The official metric for the task was macro-averaged recall, which means that more importance is given to rarer classes, such as elle, elles and on. The system scores ranged from 36.35 to 65.70, with the baseline system at 50.85. The overall score of our system 59.32 put us at sixth position out of nine systems (excluding the baseline system).

Our system scored +8.47 points higher than the baseline, showing that exploiting contextual linguistic information is useful for the task.

Analysis and perspectives

As with all systems submitted to the task, the most difficult classes to predict were the rarest classes: elle, elles and on. In terms of pronouns determined by coreference, where the neural network systems were limited by sentence boundaries, the linguistically-motivated systems were limited by the performance of the tools providing the linguistic annotation. Our analysis of the performance of the Stanford Coreference tool showed that a correct antecedent was provided in only 52.5% of cases and 32% of pronouns were linked only to other pronouns rather than to full noun phrases, even when searching for the noun phrase by transitivity. The tool also fails to identify impersonal uses of pronouns (supplying coreference chains for 18 impersonal pronouns out of 25). Whilst resources and linguistics tools can provide useful information for integration such contextual information, it is important not underestimate their lack of robustness in certain contexts. Neural classification methods, with the use of richer representations of both source and target sentences through embeddings, could provide a good setting for integrating this contextual information more effectively, without having to rely as much on the accuracy of the annotation tools.

Stylistic choices: the case of tag questions

The final focus of this paper is on an even more complicated notion of context, which is more difficult to characterise than the two previous types of contexts discussed in the two previous sections. Unlike speaker identity or grammatical gender in coreference chains, which are deterministic and easily discernible, we focus here on one particular aspect of stylistic choice, the English tag question. Common in spoken English (particularly British English), the tag question, which is best known in its canonical form, formed of an auxiliary verb and a pronoun of the form isn't it?, can we? or would you?, etc., is a much studied phenomenon in the field of linguistics [START_REF] Mcgregor | The English 'tag question': A new analysis, is(n't) it? On Subject and theme: A discourse functional perspective[END_REF][START_REF] Kimps | Declarative constant polarity tag questions: A data-driven analysis of their form, meaning and attitudinal uses[END_REF].

According to the Longman Grammar [START_REF] Biber | Longman Grammar of Spoken and Written English[END_REF], approximately 20% of the questions found in the conversational part of the corpus they analysed were tag questions.

Question tags are peripheral interrogative elements, typically appended to a declarative sentence, with the effect of modifying the sentence, to express a range of different attitudes, including doubt, surprise, contempt, etc. or simply to facilitate the flow of conversation. There are two main types of tag question, grammatical tag questions, also known as canonical tag questions, which are grammatically bound to the main anchor clause, for example:

(1) You do believe in happy endings, don't you?

(2) He can't do that, can he?

However, there also exist lexical (or invariant) tag questions, which are not grammatical bound in the same way, and are in fact the most frequent form of tag question, also commonly appearing in languages other than English. For example:

(7) He's a proper bad man, innit?

(8) There's got to be a cure, right?

The function of tag questions is complex and multi-facetted. They have often been referred to as dialogue facilitators, as ways of "keeping the conversation going" and "inviting listeners to communicate" [START_REF] Soars | New headway English course. Pre-intermediate student's book[END_REF]. However they have also been analysed as being modalisers to the main proposition of the utterance, modifying it to express the speaker's attitude to the proposition [START_REF] Mcgregor | The English 'tag question': A new analysis, is(n't) it? On Subject and theme: A discourse functional perspective[END_REF]. This bias based on the speaker's belief in the truth of the initial proposition is also linked to the understanding and common knowledge between speaker and interlocutor. The choice of question tag can have an important impact on the meaning of an utterance, in particular concerning speaker certainty, politeness and conduciveness. For example, a "they're a bit strange, aren't they?" suggests that the speaker thinks them to be strange, but demands confirmation from the listener in an inviting way, whereas "they're a bit strange, are they?" suggests that the listener has stated they are a bit strange and the speaker is questioning this, potentially in a slightly aggressive manner.

Tag questions are notably difficult to translate automatically because there are no direct equivalents to the English system of tag questions in other languages, with respect to canonical (grammatical) tag questions of the form isn't it?, won't you?. 6 The form of tag questions may be considered relatively easy to predict, once an utterance is known to be a tag question, because the grammatical structure of a tag question in most cases echoes the auxiliary verb of the main clause. However, there is also the option to use a lexical tag question (e.g. right?, see?), which complicates this decision. The most difficult decision is to know when to produce a tag question when translating from a language that uses different strategies for the same communicative purpose. SMT systems, which only use a very notion of local, immediate context, often struggle to provide a grammatical and coherent tag question.

NMT systems theoretically far better, but the task of generating a tag question where one does not appear in the source sentence is still a considerable challenge.

Predicting the use of English tag questions: presence and form

To our knowledge, ours is the first work on the MT of tag questions in English, despite the vast linguistic literature on the topic. For the language direction French to English, we aim to improve the MT output of sentences whose English reference translations correspond to English tag questions, without degrading translation performance overall. We formalise the problem as a post-edition, classification problem on top of a strong SMT baseline. Given a high-performing phrase-based system (Koehn et al., 2007), trained on parliamentary data, conference talks and subtitles, we modify the sentences classified as containing a tag question by appending the tag question form predicted by the classifier.

Tag question identification We first automatically annotate a subset of the OpenSubtitle parallel corpus [START_REF] Lison | OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles[END_REF] for the presence or absence of tag questions on the English side of the corpus and for the form of the question tag if the subtitle is indeed a tag question. We use a sequence of heuristic, lexical-based rules based on our knowledge of English syntax and frequent patterns. A manual evaluation on a 500-subtitle subset of our annotations shows that precision and recall are approximately 98% and 100% respectively for the identification of sentencefinal grammatical tag questions whose anchor clause is in the same subtitle. We divide the corpus consecutively into four datasets (train, dev1, dev2 and test), whose sizes are shown in Table 3. The number of different tag questions are also provided (corresponding to approximately 1% of sentences in each dataset). The distribution between the different question tag forms is very unequal (with the form "right" making up 20% of all tag question forms. This makes both learning and evaluation difficult. Experimental setup We perform classification in two separate steps, first by predicting whether or not a tag question should be used in the target translation and secondly by predicting the true form of the question tag (e.g. isn't it, don't you think) for those sentences predicted to be tag questions by the first classifier. Both classifiers are linear classifiers trained using Vowpal Wabbit [START_REF] Langford | Sparse Online Learning via Truncated Gradient[END_REF].

The first classifier, whose task is to predict the presence or absence of a tag question, is trained on features extracted only from source data from the train set. The second classifier uses information from the current and following source sentences, the current and following target sentence and the score from the first classifier. Only those sentences to which the first classifier assigns a score above a certain threshold are sent to the second classifier, all other sentences being considered as non-tag questions. The value of this threshold at training time (which sentences are selected for inclusion into the second classifier's training set) and at testing time (which sentences will be eligible for being assigned a tag question by the second classifier) constitute the two hyperparameters of our system. The first classifier is trained on the training set, and the second classifier is trained on the subpart of the dev1 set selected by the first hyperparameter, both hyperparameters being optimised on the dev2 set. Features include significant bag of word features (unigram, bigram and trigram) filtered using a G 2 test, the n-grams of the sentence, the presence of a lexical tag in French and the presence of an affirmative or negative reply in the following subtitle.

Step Table 4: Post-edition results on the test set (f-score, precision and BLEU). Scores marked "TQ" apply only to sentences which contain a tag question in the reference translation. Scores marked "ALL" apply to all sentences. The baseline score is the machine translated output and the topline is the machine translated output, post-edited with the gold question tags.

Classification Results

Results are shown in Table 4 for both classification steps. For the second step, we provide a total classification precision on all 214 labels (including the label none when no tag question is present). 7 However we also provide separate labelling precision for gold tag questions and for gold non-tag questions to give more insight into the performance of the classifier. 8 Finally, translation performance is evaluated using the automatic evaluation metric BLEU [START_REF] Papineni | BLEU: A Method for Automatic Evaluation of Machine Translation[END_REF]. In these preliminary experiments, we see that the overall labelling precision increases with respect to the baseline, but this is chiefly due to an increased precision in predicting when a subtitle is not a tag question. Our system predicts far more none labels than the baseline system, resulting in a lower precision for gold TQs. First observations show that our system has higher precision on those subtitles predicted as tag questions than the baseline, but this is at the expense of a lower recall (as reflected in the precision on gold TQs). We see slight increases in BLEU score for the entire corpus and on gold TQs. The lack of correlation between TQ precision and TQ BLEU should be investigated, and is possibly due to the fact that wrong question tag labels are penalised more than an absent tag label. Further experiments and a detailed manual evaluation would have to confirm this.

The main problem with evaluating such a subjective, stylistic aspect is that the automatic evaluation metrics used here do not take into account the fact that several correct tag questions could be possible for a given sentence.

Analysis and perspectives

In future work, we aim to build on these preliminary experiments to improve the classification process and consequently the translation of tag questions. Possible perspectives include combining statistical classification for the identification step with robust rules for the prediction of the grammatical tag forms, which are most often very predictable given a main sentence.

Although tag question prediction may appear a very minor aspect of translation, appearing in only 1% of sentences, correctly predicting the tag question form could greatly improve the fluidity, coherence and naturalness of translated dialogues. In this experiment, we identified tag questions in the English subtitles using carefully constructed rules, but a future step could be to automatically identify such phenomena, which are difficult to translate, using the knowledge gained through a linguistic analysis of our corpus and an in-depth comparison of reference and baseline translations.

Conclusion

Context, in its many different forms, is crucial when translating speech-like texts. We have shown, through three different experiments on three aspects of contextual information, how integrating context into the translation process can help translation performance, whether during decoding as with our first experiment on domain adaptation or as a post-edition step as with the last two experiments. However, these methods remain limited. Domain adaptation by data partitioning restricts the amount of data used, pronoun predicting using external resources is, at present, limited by the lacunas in automatic coreference resolution, and we are far from being able to model the many different aspects of style, which, like tag questions, can greatly impact translation quality. The arrival of neural learning methods, which provide a promising architecture for taking into account richer linguistic information (Sennrich & Haddow, 2016), pave the way for integrating rich, contextual information into MT systems, and importantly, could provide a unified approach for the integration of the different types of context.

8 TQ precision is the percentage of reference tag questions that are correctly labelled with the exact form of the tag question. Non-TQ precision is the percentage of reference non-tag questions that are correctly labelled.

Figure 2 :

 2 Figure 2: Use of coreference chains to determine gender and number of anaphoric (or, as shown here, cataphoric) pronouns.

Table 3 :

 3 Distribution of English TQs in the four datasets.

Our focus is on the speech-like nature of texts, rather than the medium of communication (written versus oral). The processing of oral discourse is an important research topic, but is not the study of this paper. We therefore choose to refer to the genre of informal, spontaneous productions as being speech-like, whether they are transcriptions of oral discourse or were

In the 2015 version of the task in which tokens were used on the target side, not a single system beat the baseline system, which was a simple n-gram language-model[START_REF] Hardmeier | Pronoun-focused MT and cross-lingual pronoun prediction: Findings of the 2015 DiscoMT shared task on pronoun translation[END_REF]. The use of just lemmas in the 2016 task encouraged participants to better model the context and use richer information than just surface forms.

These final features were particularly useful for predicting impersonal il pronouns, as well as anomalies in the training data, where the English pronoun was not translated by a French pronoun, and the gold class was therefore the class OTHER.

Grammatical tag questions do appear in some other languages, but this is rare[START_REF] Axelsson | A cross-linguistic study of grammatically-dependent question tags[END_REF].

The second classifier is trained on the predicted outputs of the first classifier and so it is necessary to include the "none" label in the second step to allow the system to go back on the decision of the first classifier.


Rachel Bawden Limsi
email: rachel.bawden@limsi.fr

Machine Translation, it's a question of style, innit? The case of English tag questions

In this paper, we address the problem of generating English tag questions (TQs) (e.g. it is, isn't it?) in Machine Translation (MT). We propose a post-edition solution, formulating the problem as a multiclass classification task. We present (i) the automatic annotation of English TQs in a parallel corpus of subtitles and (ii) an approach using a series of classifiers to predict TQ forms, which we use to post-edit state-of-the-art MT outputs. Our method provides significant improvements in English TQ translation when translating from Czech, French and German, in turn improving the fluidity, naturalness, grammatical correctness and pragmatic coherence of MT output.

Introduction

When it comes to the machine translation (MT) of discourse, revisiting the question of what constitutes a high quality translation is essential; which aspects of language should be tackled and how to evaluate them. A first step is to identify the many stylistic aspects of speech that pose a problem for current MT techniques and to study how they could be taken into account and evaluated.

We take a step in this direction by addressing a new aspect of discourse in MT, related to speaker attitude and style: the English tag question (hereafter TQ), i.e. utterances such as catchy, ain't it? and it wasn't him, was it?. When translating into English, TQs present two main challenges. The first is knowing when to generate one. Similar to the translation of discourse connectives, TQ use is a question of style and speaker attitude, and importantly, there is not often a direct, lexical correspondence across languages. TQs are common in English and far less so in other languages, which means that other contextual cues are necessary to determine whether a TQ should appear in the English translation. The second, in particular for canonical TQs (e.g. was it?, isn't it?), is that the overall grammaticality of the utterance is determined by the correct choice of tag, which, in a similar way to anaphor translation, is grammatically dependent on the rest of the MT output.

Our aim in this paper is to improve the generation of English TQs in an MT setting with English as the target language. We formulate this as a multi-class classification task, using features from both source sentences and machine translated outputs. The prediction of the appropriate question tag to use in the English translation (if any) is then used to post-edit MT outputs. Our results, when translating from Czech (CS), French (FR) and German (DE) into English (EN), display significant improvements, as shown by automatic and manual evaluations (Sec. 5).

English tag questions (TQs)

TQs are interrogative constructions, common in spoken English, formed of a main clause (typically declarative), followed by a peripheral interrogative element, the question tag:

(1) You do believe in happy endings, don't you?

(2) He can't do that, can he?

In its canonical form, the English question tag (in bold) is formed of an auxiliary verb, which can be negated, followed by a pronoun. It parallels the verb and subject (underlined) of the preceding host clause (in italics). The grammatical structure of TQs and agreement between the host and the tag gives them the name of grammatical TQs.1

There exists a second type of TQ, lexical TQs, formed of a word or phrase that is invariant to the subject and verb of the host clause. For example:

(7) He's a proper bad man, innit? (8) There's got to be a cure, right?

TQs' functions are complex and communicate information about speaker attitude, tone, the relationship between dialogue participants, common ground and dialogue flow [START_REF] Mcgregor | The English 'tag question': A new analysis, is(n't) it? On Subject and theme: A discourse functional perspective[END_REF]. Yet few languages have such a systematic use of TQs, in particular of grammatical TQs, as English [START_REF] Axelsson | A cross-linguistic study of grammatically-dependent question tags[END_REF]. When translating into English, the first difficulty is appropriately generating English TQs from a source sentence that does not have a TQ; the complex and often ambiguous functions of TQs (e.g. expressing doubt or surprise) can be expressed differently, and subtly, in the source language. The second difficulty is ensuring grammatical coherence of canonical TQs. Consider the German sentence Sie lebt noch, nicht wahr? and its English translation She's alive, isn't she?. The choice of the question tag isn't she? is dependent on the subject and verb of the anchor clause. Had the translation been She still lives, the correct question tag would have been doesn't she?.

Related work

Discourse is a growing field in MT [START_REF] Le | Aiding pronoun translation with co-reference resolution[END_REF][START_REF] Hardmeier | Discourse in statistical machine translation[END_REF][START_REF] Hardmeier | Discourse in Statistical Machine Translation[END_REF]. To our knowledge, there has been no previous work on TQs in MT, but the two main challenges described above are similar to those associated with two previously studied discursive aspects: the translation of discourse connectives and of anaphoric pronouns. As with TQs, their frequency is relatively low, but their mistranslation has a high impact on coherence, naturalness and therefore human understanding of translations [START_REF] Meyer | Machine Translation of Labeled Discourse Connectives[END_REF].

Discourse connectives (e.g. since, because) indicate the relation between discursive units and are linked to the overall coherence of a text in a similar way to TQs. They often have no direct mapping when translated [START_REF] Meyer | Implicitation of Discourse Connectives in (Machine) Translation[END_REF] and it is often necessary to generate a discourse connective or a TQ where one is not present in the source sentence. Previous work by [START_REF] Meyer | Implicitation of Discourse Connectives in (Machine) Translation[END_REF] consists in the disambiguation of discourse connectives in source sentences prior to translation, using automatic sense classification, which guides the MT system's choice of how a discourse connective should be translated (if at all). However they do not handle the case of generating discourse relations from a source sentences in which they do not appear lexically, as is our aim for TQs.

The difficulty of anaphoric pronoun translation is ensuring grammatical agreement between a pronoun and its coreferent, when the information relevant to grammatical agreement in the target language is not present in the source language. For example, the French translation of the pronoun it in I hear an owl but I can't see it is translated as le or la, depending on whether owl is translated as masc. hibou or fem. chouette. The position of the coreferent, as with the subject and verb for TQs, is not pre-determined, and identifying which words the translated pronoun or TQ must agree with is not always easy. The majority of works perform classification of pronominal forms in view to postediting MT output [START_REF] Guillou | Incorporating Pronoun Function into Statistical Machine Translation[END_REF], encouraged by the shared task on cross-lingual pronoun prediction at DiscoMT15 [START_REF] Hardmeier | Pronoun-focused MT and cross-lingual pronoun prediction: Findings of the 2015 DiscoMT shared task on pronoun translation[END_REF] and WMT16 [START_REF] Guillou | Findings of the 2016 WMT Shared Task on Cross-lingual Pronoun Prediction[END_REF]. We use the same strategy here, since the coherent use of TQs, like anaphors, is dependent on the MT translation.

Our post-edition approach

Our method to improve the generation of English TQs in MT is the automatic post-edition of stateof-the-art MT outputs. We formulate the problem as a supervised, multi-class classification task, exploiting lexical features from source sentences and their machine translations (Sec. 4.2). Possible labels are the different question tag forms (e.g. isn't it, ok). Predicted tags are either used to replace the question tag already present in the MT output or appended to it otherwise. We test our method for three source languages (CS, DE and FR) into EN.

Corpus annotation for English TQs

The first step is to produce annotated data. The corpora used cover three language pairs: CS-EN, DE-EN and FR-EN, and are large subsets of the most recent films of the OpenSubtitles2 parallel corpus [START_REF] Lison | OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles[END_REF]. The subtitles were automatically cleaned using heuristics and processed with the MElt tokeniser [START_REF] Denis | Coupling an annotated corpus and a lexicon for state-of-the-art POS tagging[END_REF] and the Moses truecaser [START_REF] Koehn | Moses: Open Source Toolkit for Statistical Machine Translation[END_REF]. We then developed robust, manually defined lexical rules to identify English TQs3 . We identify both the presence of TQs and the question tags themselves (e.g. is it?, right?).

A manual evaluation on a random subset of 500 grammatical TQs and 500 lexical TQs shows that annotations are near perfect (accuracy of ≈98% and recall of 100% on sentence-final grammatical TQs whose host clause is in the same subtitle). 4Each corpus was divided into three sets: TRAIN ( 23 ), DEV ( 1 6 ) and TEST ( 1 6 ). The distribution of TQs is shown in Tab. 1. TQs make up approximately 1% of subtitles, but are common among questions (≈20%). There are between 238 and 285 distinct English question tags, depending on the language pair (including a label none for non-TQs). The most frequent question tag is right? (≈20%), followed by ok? (≈16%). The majority of labels are grammatical question tags, but the most frequent (isn't it?) represents only ≈3% of all TQs, revealing a huge class imbalance.

Question tag classification

Given the class imbalance (Sec. 4.1) and the different nature of lexical and grammatical TQs, we hypothesise that first predicting the presence of an English TQ before selecting tag forms is preferable to directly predicting tags in a single, direct pass. We compare a single statistical classifier (hereafter CL-ONE), which directly predicts the question tag (including the label none), to a more complex system using a sequence of classifiers (hereafter CL-SEQ, see Fig. 1). In is a rule-based system, a choice that is better adapted to the sparse labels of grammatical TQs. Where there is no rule available, this system too can predict the label none.

Experimental setup All statistical classifiers are linear classifiers trained using Vowpal Wabbit [START_REF] Langford | Sparse Online Learning via Truncated Gradient[END_REF]. 5,6 To account for class imbalance, examples are weighted according to their relative frequency in the TRAIN set, and the degree of weighting is optimised on the DEV set. 7Features used for all statistical classifiers are described just below. The rule-based CL-SEQ gram system relies on the MT output alone.

Rule-based grammatical TQ prediction Our rule-based approach is designed to predict which grammatical tag should be appended to a given translation. The rules consist in the identification of certain lexical cues from the translation. For instance, utterance-initial words can be a good indicator of the use of a particular question tag: imperatives such as let's ... (indicative of the tag shall we), and claims about the interlocutor's perception such as you think... or you know... (indicative of the tag don't you). When there is a single auxiliary and subject, these are directly used to construct a question tag, using, as a simplification, the opposite polarity to that of the anchor clause, which is the most common polarity pattern in TQs. We include several rules to account for complex clauses and perform grammatical checking between the subject and auxiliary of the question tag. Table 2: Precision (P), Recall (R), F-score (F) and fine-grained labelling precision (P*) for the TEST set on each language pair. Results are given for each coarse-grained TQ class (gram, lex and non-TQ).

Labelling precision is calculated on the subtitles with the corresponding predicted coarse-grained label. Marked in grey are the cells containing the best F-scores for coarse-grained label groupings and the overall labelling precision (for fine-grained classes).

Features We use automatically and manually defined lexical feature templates. Unless indicated the features apply to both the source sentence and the MT output. The first set of features are automatically identified bag-of-word features, which represent the 500 uni-, bi-and tri-grams most associated with a TQ, as measured by a G 2 test.

The second set of features are manually defined, based on language-specific question-response patterns and recognisable lexical clues. They include (i) the presence of a question tag (and its form), (ii) the presence of a final question mark, (iii) (CS and DE only) whether the following subtitle contains a verb that appears in the current subtitle (and if so, we include as a feature the verb type and the preceding word in both the current and following subtitles) 8 , (iv) the following subtitle contains a specific response (from a predefined list of replies such as OK, yes, no, etc.), and (v) the first words of the MT output (1-4 gram), the last auxiliary, the last pronoun and the last pronoun-auxiliary pair.

"Baseline" MT outputs For Czech and German, we use the top systems at WMT16, both attentional encoder-decoder NMT models [START_REF] Sennrich | Edinburgh Neural Machine Translation Systems for WMT 16[END_REF]. For French, we trained a phrasebased model with Moses [START_REF] Koehn | Moses: Open Source Toolkit for Statistical Machine Translation[END_REF]. 9 Baseline predictions are automatically extracted from the MT outputs using our English TQ identification rules (Sec. 4.1).

8 In German and Czech, it is common for a reply to a yes/no question to repeat the verb of the question, e.g. Poslala jsi mu to? 'Did you send it to him?' -Poslala jsem 'Yes, I did' (lit. 'send (I) did') [START_REF] Gruet-Skrabalova | Verbs and particles in minimal answers to yes-no questions in Czech[END_REF]. 9 We use a combination of three phrase tables and three 4gram KenLM language models [START_REF] Heafield | Scalable Modified Kneser-Ney Language Model Estimation[END_REF], trained on Europarl, Ted Talks and 3M-sentence subtitles, tuned using kbmira on a disjoint 2.5K-sentence subset.

Results and analysis

As mentioned by [START_REF] Hardmeier | Discourse in statistical machine translation[END_REF], evaluating coherence-related MT phenomena is problematic. A question tag can be the correct choice without matching the question tag form in the reference translation (Sec. 2), making traditional metrics involving lexical comparison (including all standard MT evaluation metrics) ill-adapted to the task.

Despite this, we provide in Tab. 2 results using traditional metrics. To get a better view of the scores, we group final predicted question tags into their coarse-grained classes (gram, lex, none) and calculate precision (P), recall, F-score (F) for these three classes. Within each coarsegrained class, we also provide labelling precision (P*), corresponding to the number of question tags within that coarse-grained class assigned the correct question tag form according to the gold label. Labelling precision is also given overall, for all test sentences (in the final column).

Overall labelling precision is significantly improved for all language pairs with both classification systems, but in particular for CL-SEQ. This is partly due to a better prediction of non-TQs, represented by the high corresponding F-scores for CL-SEQ for all three language pairs. However it is also linked to a better labelling of grammatical and lexical TQs, which can be seen by the high labelling precision (P*) in the context of high recall (R). The higher scores of CL-SEQ over CL-ONE, particularly in terms of recall, which are most likely a result of question tag label sparsity, show that our two-tier strategy of predicting grammatical and lexical tags separately is better adapted than a single classifier.

There is a notable drop in recall between Tab. 4 shows predicted tags against gold tags for the DE→EN test set. For ease of illustration, the question tags are again regrouped into their three coarse-grained classes (gram, lex and none). The matrix reveals that for lexical tags and non-TQs, the majority were correctly classed into these coarse-grained classes. However, grammatical tags proved more difficult to predict, the majority being classed as non-TQs, most likely a result of the fact that no such tag question was present on the German source side. The most common errors on this test set were predicting a non-TQ when a lexical tag was expected. For example, CL-SEQ predicted none 1166 times when the gold tag right? was expected. However the number of correct predictions of right exceeded this number (1352 cases).

Manual analysis Given the drawbacks of automatic metrics, we manually evaluated a set of final translations post-edited with the predictions produced by the CL-SEQ system for DE→EN. We randomly selected 100 examples for which the baseline translation was modified and, comparing the baseline and CL-SEQ prediction, we labelled the example as improved (baseline incorrect and prediction correct) or degraded (baseline correct and prediction incorrect) or equal (both baseline and prediction (in)correct). Examples of these choices are provided in Tab. 3. We found that postedition made an improved choice of tag in 59/100 examples (compared to 13 worse choices). Only 25 of the 59 improved examples exactly matched the reference tag, confirming the problem of relying solely on traditional metrics. We notice in particular an improvement in the choice of grammatical TQs (33 out of the 59 improvements).

Discussion and perspectives

Improvements in the generation of English TQs in MT outputs, as seen by our manual analysis, result in improved grammatical coherence, particularly for grammatical TQs. However, TQ translation is far from solved. As a stylistic aspect, its prediction and evaluation are complex and should be further explored. Possible improvements include improving the choice of linguistic information used.

Interesting future work would be to compare the opposite approach to the task; augmenting source sentences with disambiguating information prior to translation, particularly within an NMT framework, which has good potential for handling nonlocal context and integrating extra features.

Table 1 :

 1 TQ distribution for each language pair.

				#English TQs		#labels
		#sents	all	gram.	lex.	gram.	lex.
	CS-EN	15.1M	146,782	44,572	102,210	269	15
	DE-EN	6.2M	57,435	18,396	39,039	221	16
	FR-EN	15.1M	149,847	44,651	105,196	254	16

  CL-SEQ, a first classifier (CL-SEQ coarse ) predicts a coarse-grained label gram (grammatical TQ), lex (lexical TQ) or none (non-TQ), which determines which classifier (CL-SEQ lex or CL-SEQ gram ) is used to predict the tag form. Both CL-SEQ coarse and CL-SEQ lex are statistical classifiers. CL-SEQ lex is trained on the TRAIN examples assigned the label lex by CL-SEQ coarse , which explains why it provides a second (i) CL-SEQcoarse predicts classes gram, lex and none (ii) CL-SEQlex predicts lex tags

		lex
		gram
		(iii) CL-SEQgram predicts gram tags
	none
	none	isn't it, don't you, etc.	innit, right, etc.
	Figure 1: CL-SEQ classification: (i) into coarse-
	grained classes, (ii-iii) prediction of forms
	chance to predict grammatical or non-TQs. CL-
	SEQ gram

Table 3 :

 3 Some examples from the manual comparison of the baseline translation and CL-SEQ predictions. An example is given for each of the possibilities: the prediction is better, worse or the translation and the prediction are equally good or poor with respect to tag question prediction.

	SEQ and CL-SEQ when it comes to grammatical
	TQ prediction, which is not as marked for lex-
	ical TQ prediction. This is most likely due to
	the huge class imbalance in the different question
	tags (221 grammatical tags vs. 16 lexical tags),
	which causes the purely statistical one-pass sys-
	tem to favour the more frequent lexical tags and
	struggle to predict the wide range of rarer gram-
	matical tags.
			predicted
		gram	lex	none
	gold	gram 1242 lex 387 3370 351 none 2050 4896 1023586 1341 2719

Table 4 :

 4 Confusion matrix of predicted versus gold tags (when question tags are grouped into their three coarse-grained classes) for DE→EN.

Although in theory their form is relatively systematic, their attested usage is more complex.

www.opensubtitles.org

All scripts and annotations will be made freely available in order to be able to recreate the corpus.

Recall for lexical TQs cannot be accurately measured, as identification relies on a closed list of forms found in the literature and observed in the data.

http://hunch.net/ ˜vw/

We use "OAA", FTRL-proximal optimisation, L2 regularisation (λ = 1e -6) and quadratic features.

 7 We vary weights from equal for all examples to weights that fully counterbalance the class distribution.