Rachel Bawden
email: rachel.bawden@inria.fr

Benoît Sagot
email: benoit.sagot@inria.fr

RoCS-MT: Robustness Challenge Set for Machine Translation

RoCS-MT, a Robust Challenge Set for Machine Translation (MT), is designed to test MT systems' ability to translate user-generated content (UGC) that displays non-standard characteristics, such as spelling errors, devowelling, acronymisation, etc. RoCS-MT is composed of English comments from Reddit, selected for their non-standard nature, which have been manually normalised and professionally translated into five languages: French, German, Czech, Ukrainian and Russian. In the context of the WMT23 test suite shared task, we analyse the models submitted to the general MT task for all from-English language pairs, offering some insights into the types of problems faced by state-of-the-art MT models when dealing with non-standard UGC texts. We compare automatic metrics for MT quality, including quality estimation to see if the same conclusions can be drawn without references. In terms of robustness, we find that many of the systems struggle with non-standard variants of words (e.g. due to phonetically inspired spellings, contraction, truncations, etc.), but that this depends on the system and the amount of training data, with the best overall systems performing better across all phenomena. GPT4 is the clear frontrunner. However we caution against drawing conclusions about generalisation capacity as it and other systems could be trained on the source side of RoCS and also on similar data.

Introduction

As the quality of state-of-the-art machine translation (MT) systems is becoming indistinguishable in certain scenarios and domains from that of human translators [START_REF] Kocmi | Findings of the 2022 conference on machine translation (WMT22)[END_REF], the task of tackling the translation non-standard texts is becoming an increasingly realisable aim. A considerable proportion of texts produced today are done so online in informal, unedited settings, e.g. on forums such as Twitter and Reddit, and MT is frequently to make posts accessible to a global audience. However, it has been shown that MT still struggles with usergenerated content (UGC) [START_REF] Gupta | On the real-world performance of machine translation: Exploring social media post-authors' perspectives[END_REF], as the type of language can differ considerably from the edited texts that have traditionally been used to train and evaluate MT models.

The RoCS-MT challenge set (Robust Challenge Set for Machine Translation) is designed to provide a test bed for the automatic translation of nonstandard UGC phenomena. It contains approximately 2k sentences from the online forum Reddit that have been manually normalised and professionally translated into five languages: French, German, Czech, Ukrainian and Russian. The sentences were selected specifically for the presence of non-standard phenomena, of which we provide manual annotations (e.g. spelling errors, devowelling, capitalisations, acronymisms, etc.). Inspired by other datasets such as the French Social Media Bank [START_REF] Seddah | The French Social Media Bank: a treebank of noisy user generated content[END_REF] and its parallel component [START_REF] Carlos | Comparison between NMT and PBSMT performance for translating noisy usergenerated content[END_REF], our aim is to provide an evaluation set that is more challenging than certain previous efforts, such as the commonly used MTNT dataset [START_REF] Michel | MTNT: A testbed for machine translation of noisy text[END_REF]. We also make different choices from most previous efforts concerning the guidelines for normalisation and translation of the source sentences. We choose to first normalise the source sentences before translation in order to optimise the quality of the translation and to reduce the arbitrariness that may be introduced when transferring non-standard variation to the target language (e.g. on which characters to apply spelling errors, how many characters to duplicate when elongating words). For normalisation, we aim to strike a balance between normalisation as much as possible while making sure that the normalised text remains natural. 1In this paper, we describe the creation of the challenge set, and in the context of the WMT23 test suite shared task, we analyse the models submitted to the general MT shared task for the from-English shared task language pairs: English→{Czech, German, Hebrew, Japanese, Russian, Ukrainian, Chi-nese} (en→{cs, de, he, ja, ru, uk, zh}). Through automatic and manual analysis of system outputs, we find that many of the phenomena remain challenging for most systems (in particular those that create potential out-of-vocabulary or rare words such as phonetically inspired spellings, contractions, devowelling and truncation). However, the difficulty varies depending on the phenomenon, the particular instance (notably how frequent the nonstandard word is) and the system, especially with respect to the quantity of training data. The highest performing systems overall generally do better across the board on all phenomena, whereas the weaker systems struggle in particular with certain phenomena. GPT4 has a clear lead over other systems, correctly translating even some of the most challenging examples and sometimes (although inconsistently) reproducing non-standardness in its outputs. However conclusions are limited given that the training data is unknown (as is the case of other unconstrained systems).

We make the challenge set, system outputs, evaluation code and guidelines (for the normalisation, annotation and translation) openly available for research purposes.2

Related Work

Several parallel UGC datasets exist across different language pairs. While some are extracted automatically from crawled data [START_REF] Ling | Microblogs as parallel corpora[END_REF][START_REF] Iñaki San | TweetMT: A parallel microblog corpus[END_REF][START_REF] Mubarak | Constructing a bilingual corpus of parallel tweets[END_REF], a majority are based on monolingual sentences that are then translated into the target language [START_REF] Sluyter-Gäthje | FooTweets: A bilingual parallel corpus of world cup tweets[END_REF][START_REF] Michel | MTNT: A testbed for machine translation of noisy text[END_REF][START_REF] Carlos | Comparison between NMT and PBSMT performance for translating noisy usergenerated content[END_REF][START_REF] Fujii | PheMT: A phenomenon-wise dataset for machine translation robustness on user-generated contents[END_REF][START_REF] Mcnamee | The multilingual microblog translation corpus: Improving and evaluating translation of user-generated text[END_REF]. The closest to our RoCS-MT dataset are [START_REF] Michel | MTNT: A testbed for machine translation of noisy text[END_REF] and [START_REF] Carlos | Comparison between NMT and PBSMT performance for translating noisy usergenerated content[END_REF], which were designed to contain challenging non-standard phenomena, whereas many of the existing datasets do not apply any such filter. Like RoCS-MT, the MTNT dataset [START_REF] Michel | MTNT: A testbed for machine translation of noisy text[END_REF] contains texts from Reddit. To target non-standard language, they select sentences that have a low probability using a language model trained on standard data. In practice, and as shown by Rosales [START_REF] Carlos | Comparison between NMT and PBSMT performance for translating noisy usergenerated content[END_REF], the amount of non-standard language remains limited with this method. Rosales [START_REF] Carlos | Comparison between NMT and PBSMT performance for translating noisy usergenerated content[END_REF] base their parallel dataset on the French Social Media Bank dataset [START_REF] Seddah | The French Social Media Bank: a treebank of noisy user generated content[END_REF], which targets nonstandard language by searching for specific nonstandard keywords. They show that this leads to a higher level of non-standard language, although the method is by nature more biased towards the keywords and phenomena used for data selection. An error analysis of the dataset was conducted in [START_REF] Carlos | Understanding the impact of UGC specificities on translation quality[END_REF], showing MT quality (using BLEU) for different UGC phenomena.

Despite significant effort to describe and classify UGC phenomena [START_REF] Michel | MTNT: A testbed for machine translation of noisy text[END_REF][START_REF] Sanguinetti | Treebanking user-generated content: A proposal for a unified representation in Universal Dependencies[END_REF], there is no consensus as to how texts should be normalised (and indeed translated). One extreme is to normalise all phenomena to standard forms, as is often done in lexical normalisation tasks [START_REF] Han | Lexical normalisation of short text messages: Makn sens a #twitter[END_REF][START_REF] Van Der Goot | Mul-tiLexNorm: A shared task on multilingual lexical normalization[END_REF], but which in several cases would lead to unnatural outputs (e.g. if lol and lmao, were systematically normalised to laughing out loud and laughing my ass off ). This makes translation difficult too, as the translations would also be unnatural. At the other end of the spectrum is the choice to not normalise source texts and in addition to attempt to translate the phenomena into the target language, with the disadvantage that some phenomena are language-specific3 and others would result in arbitrary decisions being made such as to which characters to apply spelling errors. The current datasets targeting particularly non-standard phenomena choose to at least in part transfer some phenomena to the target language, whereas we adopt a higher degree of normalisation (see Section 3.1.1 for more details), producing standard but naturalsounding translations.

Challenge Set Creation

Data Sourcing and Selection

The source sentences are taken from English posts on discussion platform Reddit4 using the API. 5We do not target a particular variety of English (e.g. British, American, etc.) and even include some non-native English,6 although we do not include code-switched texts. We get an initial pool of posts by searching for specific keywords from a manually drawn-up list as in [START_REF] Sanguinetti | Treebanking user-generated content: A proposal for a unified representation in Universal Dependencies[END_REF], e.g. ttyl, ppl, gr8, alot. The full list is given in Appendix A. For each keyword, we crawled both Reddit-wide and 3 specific subreddits (Casu-alUK, MadeMeSmile and entertainment) to ensure a diversity of informal topics7 at 6-month intervals between 2017 and 2022.

Once we had the initial pool of examples, in order to reduce the number of posts to manually review, we applied a very coarse-grained in-house 'non-standardness' classifier that we had trained on a small set of manually annotated tweets according to 4 labels (standard, mildly non-standard, moderately non-standard and very non-standard), and look at posts whose title or text was marked as anything other than 'standard'. From those posts, we manually select titles and passages from the text that contain interesting non-standard phenomena, including sentences not containing the initial keyword associated with the post. This means that although our initial search process is biased to our word list, the effect is diminished by taking additional non-standard phenomena. We automatically filter out any 18+ content (using the Reddit metainformation), and manually filter out any content that is sexually inappropriate, insulting or deals with sensitive (potentially triggering) topics such as suicide or drug addiction.

Sentence Splitting and Normalisation

We start by manually splitting the texts into sentences. In many cases, this corresponds to splitting on final punctuation (e.g. full stop, exclamation marks, etc.). However, the non-standard nature of the texts increases the number of cases where texts are split in places that are not marked by punctuation or where punctuation or newlines are added unexpectedly in the middle of what would ordinarily be considered a sentence.

For instance, the sequence I went grocery shopping I'm down to my last dollars soon (...) was split into the first sentence I went grocery shopping and the second sentence beginning with I'm down to my last dollars soon, despite the lack of a final punctuation between shopping and I'm.

The first author (a native English speaker) manually normalised each of the sentences produced by our manual sentence splitting, seeking help from people knowledgeable in the topics (e.g. video gaming) where necessary. The complete normalisation guidelines with examples can be found in the dedicated Github repository. 8 As with any guidelines for dealing with complex and evolving nonstandard phenomena, the decisions made are certainly not bulletproof and are likely to evolve in future work. Our aim was to reach a compromise between (i) normalising as much as possible of the text while (ii) rendering the output natural and realistic and (iii) not over-normalising such as to remove the style of the original text. We therefore normalise words such that the normalised variant could be spontaneously and naturally used.

Translation

Translation of the English sentences was carried out by paid professional translators. They had access to the original posts and both the raw and normalised versions of each sentence. Translation was carried out at the sentence level (following the manual segmentation and using as the source the normalised translation), although the translators had access to surrounding linguistic context, as well as additional context and translation notes provided by the first author during the normalisation step. There were also several exchanges between the first author and the translators in order to provide additional context and to answer questions. In order to preserve author anonymity, translators did not have access to meta-information about the authors (e.g. their gender). A single translation was produced for each sentence (we left the choice of speaker gender to the translators) with the exception of Ukrainian, for which two translations were produced for sentences where the speaker gender has an impact.

The target languages were chosen to cover four of those in the WMT2023 general translation task (Czech, German, Ukrainian and Russian), as well as French, which is an important language for our own research, although we do not analyse the French portion of the data in this article.

Translation Guidelines Translators were provided with guidelines (see Appendix B). They were instructed to translate the normalised versions of each sentence into the target language, using standard language but best matching the intention, naturalness and familiarity level of the sentence, similar to the guidelines set out in [START_REF] Mcnamee | The multilingual microblog translation corpus: Improving and evaluating translation of user-generated text[END_REF]. The decision to use standard language was to avoid the arbitrariness associated with attempting to reproduce non-standard phenomena in translation, which would make comparisons, particularly automatic ones, more difficult (e.g. which characters to alter to reproduce a spelling error, how many characters to repeat in the case of expressive repetition, etc.). They were also instructed to respect the manual segmentation provided,9 to respect punctuation choices made in the source where appropriate (e.g. conserving full stops) and to preserve English words in meta-linguistic discussions (i.e. where authors are writing specifically about English words). As in the normalisation guidelines, abbreviations, acronyms and simplifications were to be expanded unless the result would not make a natural sentence that could realistically be found. However, abbreviations linked to the names of places and institutions were to be kept as they were if used as such in the target language (e.g. French OTAN for English NATO). They were requested not to use MT systems to help them translate in order not to bias the translations produced.

Challenge Set Subsets

We create four subsets of the challenge set to test the impact of sentence segmentation (manual or automatic using spaCy) and of normalisation (manual or none, i.e. the original raw text):

• manseg-raw: Manual segmentation with original (raw) text

• manseg-norm: Manual segmentation with manual normalisation

• spacyseg-raw: spaCy segmentation with original (raw) text

• spacyseg-norm: spaCy segmentation with manual normalisation10

As shown in Section 3.3, the two different segmentation methods result in different numbers of individual sentences, and automatic segmentation with spaCy differs depending on whether the text has been normalised or not. In practice, in this article, we focus only on the manseg-raw and manseg-norm subsets, although we also release the system outputs for the spacyset-subset. We leave research on these other subsets (i.e. looking at the impact of sentence segmentation) to future work.

Dataset Characteristics

Some basic quantitative characteristics of the data are given Table 1.

Impact of sentence splitting While the number of sentences is fixed for the manual segmentation, spaCy segmentation is highly dependent on whether the text has been normalised or not, likely due to the tool being less well adapted to nonstandard text; when applied to raw text, the resulting number of sentences is far lower than manual segmentation (1660 vs. 1922), whereas the resulting number of sentences is more similar to manual segmentation when applied to the normalised text.

Tokenisation Normalisation impacts the number of tokens in the texts, as well as the number of unique tokens. When comparing the two normalised subsets on the one hand and the two raw subsets on the other (i.e. differing only in the sentence splitting), the number of tokens differs due to the fact that automatic segmentation tends to oversplit sentences on punctuation that in the manual segmentation would remain part of a token in the preceding sentence. The number of unique tokens inevitably drops after normalising, due to the homogenisation of non-standard forms (7175 vs. 6612) for manual segmentation.

Normalisation Types We manually annotated the texts for non-standard phenomena (e.g. spelling errors, acronyms, devowelling, capitalisation, pronoun drop, etc.), with the possibility of there being several types for a single span of text. Our annotations are at the word-level, with some phenomena spanning several words (e.g. capitalisation). Table 2 provides some statistics for the annotations occurring in at least 10 sentences, and some examples are given in Examples 1-4.

(

Translation Systems

In this article, we evaluate the systems submitted to the general translation task at WMT2023. There are both constrained and unconstrained systems, the two settings presenting significant differences in training data that should be taken into account when comparing systems.

Constrained systems Constrained systems followed similar strategies, with many systems doing data filtering/cleaning and data augmentation, using either bilingual or multilingual models and reranking. The constrained systems submitted were AIRC [START_REF] Rikters | AIST AIRC Submissions to the WMT23 Shared Task[END_REF], ANVITA, CUNI-Transformer and CUNI-DocTransformer [START_REF] Popel | CUNI English-Czech and English-Polish systems in WMT20: Robust document-level training[END_REF] (we refer to these system as CUNI-Trans and CUNI-DocTrans to save space in the results tables), CUNI-GA [START_REF] Jon | CUNI-GA submission at WMT23[END_REF], HW-TSC (Wu et al., 2023b), IOL_Research [START_REF] Zhang | IOL Research Machine Translation Systems for WMT23 General Machine Translation Shared Task[END_REF], NAIST-NICT [START_REF] Deguchi | NAIST-NICT WMT'23 General MT Task Submission[END_REF], Sam-sung_Research_Philippines (Cruz, 2023) (hereafter Samsung_RP), SKIM [START_REF] Kudo | SKIM at WMT 2023 General Translation Task[END_REF] and UvA-LTL (Wu et al., 2023a).

Unconstrained systems As in previous years of the shared task, translations were produced from anonymised online systems, corresponding in this addition to ONLINE-{A,B,G,M,W,Y} submissions. This year, translations from GPT4 were also produced using 5 few-shot examples (GPT4-5shot).11 Note that caution should be taken when comparing results from GPT4, given that it is very possible that source sentences from RoCS-MT are included in GPT4's training data. Two systems based on NLLB [START_REF] Nllb Team | No language left behind: Scaling humancentered machine translation[END_REF] were also submitted in the context of the metrics shared task: NLLB_Greedy and NLLB_MBR_BLEU (hereafter NLLB_MBR), which both rely on the same model but differ by the decoding strategy, either standard (greedy) or based on the Minimum Bayes Risk strategy [START_REF] Freitag | High quality rather than high model probability: Minimum Bayes risk decoding with neural metrics[END_REF]. A number of unconstrained systems were also submitted by participants, namely Lan-BridgeMT [START_REF] Wu | Exploring Prompt Engineering with GPT Language Models for Document-Level Machine Translation: Insights and Findings[END_REF], KYB , GTCOM (Zong, 2023), [START_REF] Li | KYB General Machine Translation Systems for WMT23[END_REF], PROMT [START_REF] Molchanov | PROMT Systems for WMT23 Shared General Translation Task[END_REF], Yishu [START_REF] Min | Yishu: Yishu At WMT2023 Translation Task[END_REF] and ZengHuiMT [START_REF] Zeng | Achieving State-of-the-Art Multilingual Translation Model with Minimal Data and Parameters[END_REF].

Evaluation and Analysis

Evaluation of UGC translation is more challenging than standard text; a correct translation can either be standard or non-standard in the target language, and there may be multiple ways of being non-standard that may not all be covered by available references. In our case, we chose to produce standard reference translations (See Section 3.1.2).

Any system that produces non-standard language may therefore be underestimated using referencebased metrics.

We test three different metrics (BLEU, COMET and COMET-QE) to evaluate the systems' translations of RoCS-MT, looking at how coherent they are between each other, and whether it is possible to use quality estimation to evaluate MT robustness in order to remove the need for reference translations (Section 5.1). We also look at the MT quality of each system per phenomenon by calculating COMET scores over subsets of the data. Finally, we perform a qualitative analysis, manually looking at how the different systems handle UGC phenomena, and confirming some of the trends using some simple automatic analyses (Section 5.2).

Automatic evaluation

BLEU [START_REF] Papineni | Bleu: a method for automatic evaluation of machine translation[END_REF], as a surface-level metric, is intuitively not robust to variation. It is therefore likely to be particularly ill-adapted to MT robustness evaluation, since MT systems' outputs can display standard or non-standard characteristics. We choose nevertheless to test this here, calculating BLEU scores using the sacreBLEU toolkit [START_REF] Post | A call for clarity in reporting BLEU scores[END_REF]. 12 We compare BLEU to referencebased COMET [START_REF] Rei | COMET: A neural framework for MT evaluation[END_REF] 13 for those language pairs for which we have a reference, and to COMET's reference-less (quality estimation) version, which we refer to as COMET-QE [START_REF] Rei | CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task[END_REF]. 14 We notably aim to test whether it is possible to use COMET-QE for evaluation rather than reference-based COMET, which would remove the dependency on reference translations and make evaluation possible for a wider range of languages.

Ukrainian has two reference translations for sentences for which the speaker's gender results in different translations is ambiguous between male 12 case:mixed|eff:no|tok:13a|smooth:exp|v:2.2.1 13 We use the default wmt22-comet-da model. 14 We use the default wmt22-cometkiwi-da. and female. While BLEU is designed to handle multiple references, this is not inbuilt into COMET. For these sentences, we choose to take the best COMET score of the two references. For COMET and COMET-QE, which also use the source sentence, we choose to evaluate system outputs against both the manseg-norm and manseg-raw source sentences, regardless of which set was translated by the system and take the highest score of all the source-reference combinations. This covers the case where non-standard (i.e. raw) sentences are normalised during the translation process.

We provide full results for COMET and COMET-QE in Table 3 and 4 respectively, and we include results for BLEU in Table 7 in Appendix D.

How coherent are the metrics? The trends of the three metrics are similar but not at all systematic (in terms of rankings) when evaluating translations of the normalised data (manseg-norm), with the same systems getting the highest scores across language pairs (amongst the best systems being ONLINE-W, ONLINE-B, GPT4). However, there are some clear inconsistencies between BLEU and the two COMET metrics when evaluating nonstandard data (manseg-raw). For example GPT4 is ranked above other systems by COMET and COMET-QE, whereas the BLEU scores of other systems (and in particular ONLINE-W and sometimes ONLINE-B) are higher. This indicates that GPT4 outputs are more surfacically different from the reference translations, which could be a result of paraphrasing or non-standard translations rather than a reflection of MT quality, especially given the high scores by COMET. This confirms that BLEU is poorly adapted to evaluating MT robustness and could even lead to misleading conclusions, confirming previous conclusions drawn by Rosales Núñez et al. ( 2021) about the inadequacy of BLEU for the evaluation of UGC MT. On the other hand, COMET-QE scores show more similar trends to COMET, suggesting that it could be possible to use it to evaluate without having to produce reference translations. We nevertheless add that COMET remains an automatic metric that does not produce perfect correlation with human judgments, more research would be necessary to stress-test the metric for MT robustness evaluation, particularly in terms of evaluating which of COMET and COMET-QE is better correlated with human judgments.

Which systems come out on top? The highest performing systems are the unconstrained online systems, with GPT4 getting significantly higher COMET and COMET-QE scores than other systems when translating non-standard (raw) text for all languages tested. Other systems that tend to produce high scores are ONLINE-W and to a lesser extent ONLINE-B. Apart from these online models, both NLLB models are the best-scoring ones, which might come from the fact that they are highly multilingual and therefore could be more robust to language variation. The constrained systems, whilst not the highest performing systems, appear to get comparable scores to at least some of the online systems.

Which systems are most robust? This question is linked to the previous question about MT quality on non-standard data. To take into account the base performance of the systems, we look at the difference in score between each system's translation of the non-standard sentences and their normalised versions (also in the previously mentioned Tables 3 and4. While there is a general trend that the higher performing systems also also have a smaller difference in quality (i.e. they are also more robust), there are some stand-out systems. GPT4 is the system with the lowest quality difference between original and normalised sentences for all language pairs tested. The NLLB models also have a low delta between the two subset, lower than or comparable to some of the more robust online systems. Similarly to the previous question, constrained systems are not the most robust in terms of their score difference. Notably for en-cs and en-de, the score differences are amongst the highest. However, some of the systems do show peformance in the same ballpark as some of the online systems.

Automatic analysis by UGC phenomenon

In order to analyse how systems handle different non-standard phenomena, we evaluate sentences by annotation types, by calculating COMET and COMET-QE scores for sentences containing at least one occurrence of a particular normalisation annotation. COMET results are given in Table 5 and we include a fuller analysis for COMET-QE results in Table 8 in Appendix E. Note that we only include annotation types that appear in at least 50 sentences, and that the 'all' column refers to the scores over all sentences and not just the ones annotated for UGC phenomena.

Scores are not directly comparable across annotation type. Performance by annotation type is consistent with previous conclusions, with GPT4 getting the highest scores across the board, and online systems and NLLB also doing well. It is striking that the systems that have higher scores in general tend to do better across the board on all annotation types, whereas the lower-scoring systems struggle with certain non-standard phenomena. They correspond in particular to phonetic distance, where a word is spelt differently according to how it is pronounced (e.g. HEERE'Z A QWESHCHUN FER YA 'Here's a question for you'), contractions (e.g. wud 'would'), devowelling (e.g. nvr 'never'), truncation (e.g. intro 'introductory') and spelling errors. These are notably phenomena that could well result in out-of-vocabulary words.

Are certain language pairs more difficult than others? It is tricky to compare across language pairs, since scores are not comparable. However, there are some indications that the en-cs set is more challenging, given the low scores across multiple annotation types for all systems other than GPT4. The fact that GPT4 has high scores for all annotation types listed shows that the lower scores of other models are not due to quality issues in the reference translations, and provides an upper bound against which other systems can be compared, thereby indicating that the systems struggled more.

Qualitative analysis

Non-standard variants of words Many of the non-standard phenomena that characterise the texts (e.g. acronyms, truncations, contractions, devowelling) represent a similar difficulty to unknown or rare tokens in MT. The treatment of these words differs according to the system used, and inevitably largely on the training data of the model. Many of the constrained systems struggle to translate such words, either copying the words into the translation or omitting them entirely. The degree to which the systems succeed in correctly translating these words appears to depend on how common it is. For example, tho, phonetically-inspired spelling of though, was translated successfully by multiple systems, although the devowelled word tmro 'tomorrow' proved more difficult.

Markers of expressivity

It is common for UGC texts to have markers of expressivity such as capitalisation or repetition of letters. We removed these markers in our normalised versions and reference

- - - - - - - - - - - - - - - - - - CUNI-GA 0.803 0.677 0.126 - - - - - - - - - - - - - - - - - - HW-TSC - - - - - - - - - - - - - - - - - - 0.793 0.740 0.054 IOL_Research - - - - - - - - - - - - - - - - - - 0.770 0.696 0.074 NAIST-NICT - - - - - - - - - 0.830 0.764 0.066 - - - - - - - - - Samsung_RP - - - - - - 0.797 0.732 0.065 - - - - - - - - - - - - SKIM - - - - - - - - - 0.837 0.785 0.052 - - - - - - - - - UvA-LTL - - - - - - 0.799 0.731 0.068 - - - - - - - - - - - -
Table 4: COMET-QE scores of systems on the manseg-norm and manseg-raw subsets.

translations for consistency. However, there are no guidelines as to how the different systems should translate them: either preserving the markers in the translation or normalising as we do in the reference. What we observe is variable behaviour depending on both the system and on the specific textual instances. Table 6 shows one of the more extreme examples of character repetition. Most of the systems fail to translate the words with character repetition and instead copy them (the case of moooorrreeee 'more', poollliitte 'polite' and Discouuurrrse 'Discourse'). There is greater robustness for the word neeeeed 'need', which is translated cor-rectly as brauchen by a majority of systems, but not by AIRC, ONLINE-M and ONLINE-W, which is interesting since ONLINE-W obtains very high BLEU scores for en-de. When systems do translate the words, it tends to be the standard form that is generated (i.e. without repetition). The only example here is GPT4, which translates moooorrreeee as viiiieeeel 'viel', conserving the expressivity of the source sentence. What is interesting is that this behaviour is far from consistent for GPT4, with the other non-standard words in the same example not undergoing the same treatment. This could suggest that while the system succeeded in deciphering the non-standard English texts, there is no systematic notion of generating non-standard translations; it is possible that an expressive version of viel was seen far more often in the training data, thus being a probable translation in this case.

Number of copied words

Bearing in mind the fact that unknown source words are often either copied or removed, we apply a simple automatic analysis to judge how often systems copy source words to diagnose a potential problem with outof-vocabulary words. We calculate how often a sentence's translation contains words that appear in both the source and system's translations (counting occurrences for each sentences and taking the average). We calculate the difference in the average number of copied words for themanseg-raw and manseg-norm subsets (i.e. how much more are words copied when systems are translating nonstandard data). Results are shown in Table 9 of Appendix F. Note that a certain number of 'copied' words is normal, as certain words are expected to be identical in source and target texts (e.g. some named entities, numbers, etc.). There are similar trends across languages: the same systems tend to have relatively fewer/more copied words in raw than norm. GPT4 has the smallest difference in number of copied words between norm and raw translations. Unsurprisingly, most other systems have more copying in raw than in norm, indicating a problem handling unknown vocabulary and confirming our previous qualitative analysis.

On a similar note, we also analysed the sourcetranslation length ratios for both raw and norm subsets. While this could be linked to copying or omit- ting unknown words (as we have seen), it is more likely to be linked to overgeneration problems, linked to systems encountering text that is out-ofdomain, which we occasionally observed in the system outputs. We observed that for all systems, the length ratio between manseg-raw translations and their source sentences was greater than those for manseg-norm. The effect was even greater for the texts when automatic sentence segmentation was applied (i.e. for spacyseg-subsets).

Conclusion

We have presented a new resource, RoCS-MT, a robustness challenge set for MT, designed to test MT systems on non-standard UGC. Our automatic and manual analysis show that non-standard texts are still a problem for many of the systems, including the unconstrained ones, and that certain phenomena such as phonetically inspired spellings pose a problem in particular. The comparison of COMET and COMET-QE metrics suggest that it may be possible to draw similar conclusions from automatic scoring without using references, although future work could go into more depth into analysing what is captured by the different metrics.

Limitations

The current test set is available for five from-English directions and it would be interesting to study other language directions, including those not involving English. The current version of the challenge set only contains variants for speaker gender for one of the language pairs, and we plan to add these for the other target languages in a future version.

Finally, a major limitation is one that is becoming widespread nowadays, which is that many of the systems trained and even used in research are trained on an unknown quantity of data for which the sources are unknown. Without being able to verify the fact, GPT4 and potentially some of the other systems are likely to be trained on some of the source sentences in the challenge set, and future models may even be trained on the reference translations we provide, despite it being indicated as a test set. This is a blocking factor for scientific comparison and one that goes beyond this particular resource.

A Non-standard keywords for sourcing of posts

The full list of keywords we searched for using the Reddit API is as follows: yyy, iii, eee, ppl, btw, imo, wtf, shes, hes, ima, shud, wud, cud, afaik, bcuz, hahaaa, dat, wen, wot, woz, bout, bro, gonna, lmao, ppl, smh, yall, omg, barley, fyi, beleive, seperate, lol, ttyl, muaha, mwah, air, afaik, fr, fyi, idk, ikr, irl, jk, nvm, plz, pls, cu, tbh, ur, wth, kk, 2mo, 2moro, tmrw, fwiw, nvm, thx, b4, ruok, m8, l8r, 2nite, gr8, lk, wt, w/, peeps, sooo, verry, innit, wasnt, ain't, definately, yous, nae, awfull, freind, untill, wierd, aweful, wether and alot.

The keywords were chosen as they illustrate well known non-standard phenomena, including:

• spelling errors (e.g. wierd 'weird', wether 'whether', alot)

• acronymisation (nvm 'never mind', fyi 'for your information)

• repetition of characters (e.g. hahaaa, eee, sooo)

• contractions (e.g. cud, gonna, shud))

• dialectisms (e.g. ain't, yous, nae, innit, yall)

• devowelling (e.g. tmrw 'tomorrow', pls 'please', jk 'joke')

• truncations, including abbrevations (e.g. peeps 'people', w/ "with') • etc.

Although the choice of keywords does create a certain bias in the types of language retrieved (especially given that several variants of some keywords are included), these keywords are used to identify posts that likely to contain other non-standard phenomena, so the final selected sentences are not restricted to those containing these keywords.

B Translation Guidelines

These guidelines are included because there are some specific constraints as to how the translations are to be carried out, and some particularities of the dataset to explain. The sentences to be translated are found in the excel spreadsheet in the column "Normalised segment". However, we also provide additional information that can help translation (see below for more information).

Origin of the text The texts to be translated are from the Reddit online forum (extracted using the API), taken from a range of different subreddits (so of different genres of text, e.g. relationship advice, advice about pets, video gaming strategy, etc.). They were selected due to their non-standard nature (spelling mistakes, abbreviations, lack of punctuation etc.).

Preprocessing of the text The texts have been manually pseudo-anonymised (usernames and names other than those representing celebrities and other well-known public figures are replaced with new names), split into "sentences" and normalised. It is the normalised versions of the sentences that are to be translated.

The sentences have been filtered to remove offensive or sensitive content (hate speech, taking drugs, suicide, etc.). However, profanities were kept as they were taken to be illustrative of the sociolect of online language. If however, you do not feel comfortable with translating something, please leave it blank and write a comment indicating that you have not translated it.

Additional context provided to help translation

The text is split into short documents with one or several sentences per document. In the excel document, a sentence's document is indicated by the value in the column "Post number", and the cells are also coloured such that it is visually easier to see which sentences belong to the same document (alternating grey and white). A Reddit post is associated with a title and a text with the main content of the post. The documents can contain either the title or a subset of the text or even both. The type of text associated with each sentence is indicated in the column "Text type". Titles are marked in bold to make them visually easier to see. Although the normalised text may be sufficient to carry out the translation, we also give access to the additional information just in case:

• the title of the post • the entire body of text associated with the post

• the raw version of the sentence (after pseudoanonymisation and segmentation into sentences)

• some translation notes have been added to provide some context about the posts (e.g. to give an idea of what is the subject of conversation, the meaning of some expressions and abbreviations, etc. in order to make translation easier). Very occasionally there are indications about how to translate (for instance for meta-linguistic questions where people discuss particular words, it is best to keep the English words, e.g. One word I simply can't say properly is water. . . → water should be kept in English in the translation.

Constraints (important) The dataset will be used to evaluate machine translation systems on their ability to handle non-standard texts. This crucially means that: the sentence boundaries that have been defined must not be modified. It is possible to translate a sentence using several sentences if that is what is natural. However, it is not possible to merge several source sentences to produce a single translation of both (i.e. one translation per row). translators should not use machine translation systems or other computational systems to aid translation as this could bias the translations to look like translations produced by Google Translate, DeepL, ChatGPT, etc.

More specific guidelines There are multiple posts that use slang terms (e.g. gaming or general online slang such as lol) and it possible that the correct translation will be an English borrowing. It is fine to use an English borrowing in this case, if this is what is generally used online. The punctuation choices should be kept as much as possible, as appropriate for the target language of translation (e.g. conserving full stops, exclamation marks, quotes, etc.). As described above, there are some instances of people talking about English words, and in this case, the English words should be kept as is. Another example: One says "Let's eat granny" making it seem like someone's going to eat their nan. However, the other example says "Lets eat, granny", implying a different meaning to the sentence. The phrases "Let's eat granny" and "Let's eat, granny" should be kept in English. These are indicated in the translation notes. Use of "non-standard" language:

• Any spelling mistakes that were in the raw sentence should not be reproduced in the translation (i.e. the normalised version should be used as the source sentence to translate).

• Formatting, including things like capitalisation, should (for the same reasons) follow the conventions of the normalised translation.

• Abbreviations, acronyms and simplifications (e.g. in English wdym = what do you mean, bc = because, rly = really, etc.) should be expanded, unless the result would not make a natural sentence that could realistically be found. An example of a non-natural expansion would be lol = laughing out loud, since this is not practically used.

• However, abbreviations linked to the names of places (e.g. USA, UK, UCL (=University College London) should be kept as they are if the acronym is also commonly used in the target language. In other cases, the most frequent equivalent translation should be used. (e.g. English UN = French ONU, English NATO = French OTAN).

The overall idea is that the translations should be natural and not contain the types of non-standard language that were normalised in the English versions, although they should match as best possible the style and familiarity.

Additional questions If you have any doubts or questions about the meaning of the sentences, please contact me at rachel.bawden@inria.fr to discuss things further.

C Prompt used for GPT4-5-shot

The prompt used for the GPT4-shot is the one from [START_REF] Hendy | How good are GPT models at machine translation? A comprehensive evaluation[END_REF]

D BLEU scores

We provide BLEU scores for language pairs with reference translations in Table 7. The results are provided (as with the COMET scores in the main part of the paper) for the original raw subset (manseg-raw) and for its normalised version ((manseg-norm)) as well as the difference between the two scores (δ).

E COMET-QE scores by annotation type

We provide in Table 8 COMET-QE scores per annotation type for all from-English language pairs of the shared task.

F Copying analysis

Table 9 shows results for our automatic analysis of the number of source words that are found in the output translations. We calculate the number of such words, averaged over the number of sentences for each of the subsets manseg-raw and manseg-norm and we calculate the difference between the two. Positive numbers indicate that more copied words are found when systems translate the non-standard output and negative numbers indicate that more copied words are found when systems translated the normalised sentences.

Table 1 :

 1 Basic statistics of the four subsets of the test suite. Tokens are defined as whitespace delimited character sequences. Sentences can either come from post titles or the body of the post.

	1) btw	I wud	prefer them rlly	quick.
	By the way, I would	prefer them really quick.
	acronym	contraction	devow.
	capitalisation
	punct_diff

Table 2 :

 2

					I shud	be
	So any ideas on what		I should	be ?
	spacing phon._dst. contraction punct.
	(3) Dhat	kwik	beizh	fawks	jmmpd
	That	quick	beige	fox	jumped
	phon._dst. phon._dst. phon._dst. phon._dst. phon._dst.
	(4) Em	HOW DARE YOU SWEAR IN
	EM:	How dare you	swear	in
	caps.	caps. caps. caps. caps.	caps.
	punct_diff
	FRONT OF MY SUN
	front	of	my	son	?
	caps.	caps. caps. spelling punct_diff

For each annotation appearing in at least 10 sentences, the number of words, unique words (lowercased) and sentences for which it appears.

(2) So any idea s on wot

Table 3 :

 3 COMET scores of systems on the manseg-norm and manseg-raw subsets.

	Systems		en-cs			en-de			en-he			en-ja			en-ru			en-uk			en-zh
		norm	raw	∆	norm	raw	∆	norm	raw	∆	norm	raw	∆	norm	raw	∆	norm	raw	∆	norm	raw	∆
	Unconstrained
	GPT4-5shot	0.817 0.800 0.018	0.822 0.805 0.017	0.806 0.793 0.013	0.846 0.838 0.008	0.806 0.789 0.017	0.809 0.797 0.012	0.797 0.786 0.011
	ONLINE-A	0.807 0.724 0.083	0.816 0.765 0.050	0.807 0.737 0.070	0.824 0.772 0.052	0.807 0.750 0.058	0.791 0.726 0.065	0.786 0.725 0.061
	ONLINE-B	0.814 0.756 0.058	0.821 0.793 0.028	0.812 0.767 0.045	0.848 0.822 0.027	0.808 0.761 0.047	0.805 0.759 0.046	0.805 0.766 0.039
	ONLINE-G	0.791 0.705 0.086	0.812 0.766 0.045	0.786 0.720 0.067	0.782 0.700 0.082	0.821 0.784 0.036	0.809 0.775 0.034	0.765 0.704 0.062
	ONLINE-M	0.807 0.710 0.096	0.810 0.724 0.086	-	-	-	0.798 0.711 0.088	0.790 0.702 0.089	-	-	-	0.762 0.692 0.069
	ONLINE-W	0.822 0.765 0.057	0.822 0.780 0.042	-	-	-	0.822 0.790 0.031	0.819 0.786 0.033	0.812 0.782 0.030	0.802 0.767 0.036
	ONLINE-Y	0.799 0.732 0.067	0.822 0.786 0.036	0.808 0.753 0.056	0.842 0.811 0.031	0.814 0.764 0.050	0.787 0.731 0.056	0.796 0.752 0.044
	NLLB_MBR	0.802 0.762 0.040	0.801 0.763 0.038	0.796 0.756 0.040	0.721 0.682 0.039	0.794 0.754 0.040	0.784 0.744 0.040	0.617 0.596 0.021
	NLLB_Greedy	0.806 0.765 0.041	0.802 0.761 0.041	0.795 0.756 0.039	0.749 0.711 0.038	0.795 0.754 0.041	0.786 0.745 0.041	0.664 0.645 0.019
	Lan-BridgeMT	0.799 0.724 0.075	0.805 0.741 0.064	0.797 0.757 0.040	0.827 0.774 0.053	0.796 0.724 0.071	0.769 0.696 0.072	0.803 0.792 0.011
	GTCOM_Peter	0.796 0.722 0.074	-	-	-	0.797 0.719 0.077	-	-	-	-	-	-	0.774 0.704 0.070	-	-	-
	KYB	-	-	-	-	-	-	-	-	-	0.788 0.691 0.097	-	-	-	-	-	-	-	-	-
	PROMT	-	-	-	-	-	-	-	-	-	-	-	-	0.789 0.710 0.079	-	-	-	-	-	-
	Yishu	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	0.805 0.766 0.039
	ZengHuiMT	0.781 0.713 0.067	0.792 0.748 0.045	0.790 0.734 0.055	0.828 0.791 0.037	0.772 0.724 0.048	0.748 0.696 0.052	0.772 0.711 0.061
	Constrained
	AIRC	-	-	-	0.763 0.684 0.079	-	-	-	0.779 0.701 0.078	-	-	-	-	-	-	-	-	-
	ANVITA	-	-	-	-	-	-	-	-	-	0.797 0.716 0.080	-	-	-	-	-	-	0.630 0.536 0.094
	CUNI-Trans	0.798 0.705 0.093	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
	CUNI-DocTrans	0.803 0.677 0.126

Table 5 :

 5 COMET scores by normalisation annotation type and averaged over all RoCS-MT sentences ('all').

	System	acronymisation	article_drop	capitalisation	contraction	devowelling	elongation	grammar	inflection	lex_choice	norm_punct	phonetic_distance	pronoun_drop	punct_diff	spacing	spelling_error	truncation	word_drop	all
	en-cs
	GPT4-5shot	0.785	0.823	0.811	0.815	0.848	0.810	0.816	0.817	0.802	0.809	0.798	0.833	0.819	0.821	0.824	0.795	0.810	0.825
	ONLINE-A	0.606	0.663	0.696	0.591	0.599	0.665	0.655	0.721	0.649	0.715	0.567	0.712	0.702	0.698	0.619	0.624	0.676	0.724
	ONLINE-B	0.664	0.726	0.733	0.665	0.667	0.717	0.719	0.761	0.695	0.751	0.629	0.756	0.746	0.750	0.698	0.697	0.728	0.760
	ONLINE-G	0.591	0.644	0.682	0.572	0.581	0.658	0.636	0.657	0.625	0.680	0.529	0.693	0.677	0.653	0.588	0.608	0.653	0.699
	ONLINE-M	0.601	0.661	0.689	0.616	0.586	0.659	0.676	0.693	0.633	0.720	0.574	0.700	0.693	0.695	0.636	0.627	0.671	0.720
	ONLINE-W	0.682	0.730	0.754	0.700	0.708	0.748	0.704	0.771	0.702	0.772	0.663	0.763	0.766	0.754	0.717	0.726	0.727	0.782
	ONLINE-Y	0.617	0.668	0.703	0.616	0.626	0.678	0.652	0.707	0.660	0.709	0.589	0.728	0.708	0.708	0.635	0.643	0.672	0.725
	NLLB_MBR	0.711	0.786	0.778	0.757	0.771	0.787	0.745	0.785	0.754	0.778	0.724	0.794	0.786	0.773	0.757	0.738	0.774	0.792
	NLLB_Greedy	0.718	0.770	0.775	0.750	0.781	0.759	0.750	0.789	0.745	0.773	0.721	0.793	0.785	0.766	0.760	0.747	0.765	0.791
	Lan-BridgeMT	0.618	0.670	0.701	0.612	0.594	0.674	0.650	0.693	0.651	0.709	0.573	0.730	0.703	0.693	0.632	0.636	0.674	0.723
	GTCOM_Peter	0.597	0.675	0.704	0.628	0.619	0.673	0.671	0.689	0.635	0.704	0.577	0.714	0.703	0.691	0.640	0.632	0.666	0.725
	ZengHuiMT	0.610	0.674	0.695	0.605	0.593	0.649	0.658	0.723	0.645	0.693	0.576	0.707	0.702	0.692	0.635	0.653	0.690	0.717
	CUNI-Trans	0.605	0.655	0.683	0.591	0.612	0.663	0.635	0.670	0.644	0.719	0.563	0.699	0.693	0.677	0.622	0.632	0.669	0.719
	CUNI-DocTrans	0.583	0.617	0.660	0.553	0.562	0.660	0.609	0.653	0.610	0.692	0.521	0.669	0.667	0.654	0.589	0.610	0.644	0.694
	CUNI-GA	0.583	0.617	0.660	0.553	0.562	0.660	0.609	0.653	0.610	0.692	0.521	0.669	0.667	0.654	0.589	0.610	0.644	0.694
	en-de
	GPT4-5shot	0.800	0.801	0.824	0.814	0.836	0.810	0.803	0.835	0.797	0.821	0.812	0.834	0.831	0.824	0.823	0.809	0.817	0.837
	ONLINE-A	0.670	0.708	0.751	0.660	0.698	0.727	0.726	0.783	0.693	0.765	0.642	0.774	0.759	0.758	0.699	0.728	0.742	0.771
	ONLINE-B	0.746	0.768	0.801	0.767	0.778	0.788	0.779	0.823	0.771	0.809	0.734	0.812	0.810	0.800	0.770	0.780	0.802	0.815
	ONLINE-G	0.663	0.701	0.743	0.679	0.725	0.725	0.711	0.759	0.708	0.754	0.667	0.760	0.751	0.745	0.711	0.724	0.732	0.763
	ONLINE-M	0.592	0.615	0.679	0.591	0.586	0.665	0.650	0.694	0.631	0.714	0.548	0.679	0.686	0.671	0.609	0.630	0.643	0.714
	ONLINE-W	0.717	0.725	0.784	0.749	0.762	0.775	0.758	0.808	0.765	0.797	0.718	0.822	0.797	0.802	0.768	0.765	0.787	0.809
	ONLINE-Y	0.710	0.746	0.777	0.745	0.762	0.760	0.753	0.797	0.746	0.789	0.703	0.794	0.787	0.784	0.761	0.756	0.750	0.795
	NLLB_MBR	0.702	0.749	0.776	0.751	0.756	0.763	0.724	0.786	0.729	0.768	0.718	0.780	0.778	0.767	0.744	0.735	0.755	0.786
	NLLB_Greedy	0.699	0.745	0.772	0.748	0.761	0.749	0.732	0.788	0.743	0.765	0.715	0.767	0.776	0.767	0.745	0.742	0.759	0.783
	Lan-BridgeMT	0.624	0.652	0.722	0.627	0.629	0.701	0.678	0.726	0.656	0.725	0.602	0.716	0.720	0.705	0.637	0.673	0.678	0.737
	ZengHuiMT	0.671	0.698	0.741	0.686	0.694	0.724	0.719	0.765	0.699	0.743	0.649	0.755	0.752	0.741	0.706	0.734	0.726	0.760
	AIRC	0.556	0.588	0.646	0.533	0.557	0.625	0.597	0.666	0.595	0.646	0.522	0.631	0.647	0.636	0.557	0.595	0.609	0.669
	en-ru
	GPT4-5shot	0.751	0.794	0.780	0.788	0.811	0.795	0.766	0.797	0.773	0.781	0.760	0.780	0.787	0.755	0.781	0.758	0.771	0.793
	ONLINE-A	0.633	0.687	0.716	0.641	0.689	0.668	0.712	0.751	0.688	0.719	0.609	0.713	0.721	0.709	0.653	0.680	0.703	0.730
	ONLINE-B	0.664	0.724	0.732	0.684	0.730	0.709	0.721	0.770	0.712	0.744	0.652	0.740	0.740	0.747	0.705	0.706	0.722	0.748
	ONLINE-G	0.689	0.762	0.759	0.730	0.754	0.757	0.749	0.791	0.747	0.767	0.700	0.766	0.765	0.765	0.736	0.743	0.768	0.773
	ONLINE-M	0.566	0.636	0.657	0.584	0.583	0.634	0.653	0.691	0.636	0.687	0.555	0.668	0.661	0.657	0.608	0.602	0.640	0.686
	ONLINE-W	0.724	0.769	0.770	0.761	0.773	0.759	0.769	0.797	0.753	0.777	0.721	0.784	0.780	0.781	0.763	0.762	0.781	0.786
	ONLINE-Y	0.668	0.738	0.742	0.708	0.723	0.714	0.730	0.781	0.727	0.741	0.668	0.747	0.747	0.751	0.724	0.706	0.730	0.756
	NLLB_MBR	0.679	0.748	0.743	0.731	0.749	0.763	0.721	0.763	0.718	0.738	0.690	0.737	0.746	0.732	0.717	0.703	0.733	0.755
	NLLB_Greedy	0.675	0.738	0.740	0.720	0.745	0.734	0.718	0.758	0.710	0.729	0.695	0.734	0.744	0.733	0.720	0.703	0.723	0.753
	Lan-BridgeMT	0.604	0.653	0.679	0.614	0.635	0.659	0.672	0.714	0.636	0.694	0.571	0.699	0.687	0.683	0.623	0.635	0.668	0.699
	PROMT	0.574	0.636	0.664	0.590	0.645	0.646	0.635	0.686	0.637	0.677	0.553	0.678	0.667	0.656	0.601	0.620	0.644	0.685
	ZengHuiMT	0.623	0.694	0.691	0.615	0.669	0.647	0.687	0.730	0.663	0.683	0.600	0.700	0.696	0.690	0.658	0.657	0.688	0.706
	en-uk
	GPT4-5shot	0.804	0.838	0.828	0.834	0.834	0.818	0.817	0.836	0.804	0.831	0.809	0.834	0.832	0.835	0.826	0.811	0.819	0.838
	ONLINE-A	0.626	0.688	0.725	0.635	0.665	0.679	0.705	0.744	0.680	0.736	0.608	0.728	0.724	0.721	0.656	0.676	0.721	0.741
	ONLINE-B	0.701	0.744	0.770	0.723	0.748	0.742	0.766	0.790	0.750	0.788	0.679	0.774	0.779	0.788	0.734	0.744	0.764	0.787
	ONLINE-G	0.718	0.788	0.791	0.771	0.773	0.779	0.773	0.807	0.767	0.797	0.736	0.797	0.795	0.795	0.766	0.775	0.808	0.803
	ONLINE-W	0.763	0.783	0.802	0.805	0.805	0.789	0.807	0.831	0.808	0.818	0.757	0.805	0.814	0.817	0.787	0.793	0.812	0.819
	ONLINE-Y	0.647	0.712	0.737	0.703	0.684	0.706	0.706	0.754	0.685	0.742	0.656	0.749	0.738	0.734	0.710	0.691	0.713	0.750
	NLLB_MBR	0.698	0.760	0.763	0.768	0.760	0.765	0.730	0.777	0.743	0.762	0.715	0.759	0.768	0.765	0.742	0.731	0.756	0.778
	NLLB_Greedy	0.692	0.740	0.761	0.741	0.758	0.724	0.732	0.767	0.730	0.756	0.703	0.768	0.766	0.762	0.742	0.735	0.757	0.775
	Lan-BridgeMT	0.591	0.655	0.685	0.606	0.603	0.657	0.660	0.672	0.632	0.679	0.572	0.689	0.684	0.675	0.620	0.628	0.660	0.705
	GTCOM_Peter	0.591	0.670	0.702	0.613	0.645	0.671	0.677	0.706	0.671	0.707	0.580	0.693	0.695	0.685	0.637	0.636	0.658	0.714
	ZengHuiMT	0.608	0.684	0.696	0.620	0.639	0.650	0.672	0.712	0.660	0.698	0.600	0.701	0.697	0.696	0.652	0.662	0.687	0.709

Table 6 :

 6 Example of character repetition linked to a mark of expressivity for en-de.

  , i.e. the following:

	Translate this into 1. [target language]:
	[shot n source]
	1. [shot n reference]
	Translate this into 1. [target language]:
	[input]
	1.

E.g. We choose to not normalise the acronym lol 'laughing out loud', as it is rarely/never used in its expanded form.

https://github.com/rbawden/RoCS-MT

Two examples of this are French verlan, which consists in inverting syllables in words (e.g. louche→chelou 'bizarre') and English cockney rhyming slang (e.g. loaf meaning head thanks to its rhyme with the expression loaf of bread). However, even phenomena that do exist crosslingually do not necessarily apply to the same words (e.g. the use of digits to replace their homophones as in 2day 'today', where the translation does not necessarily contain a homophone of a digit in the target language

).4 www.reddit

.com 5 Using the free version of the API (December 2022).

We do not have access to any personal information about the post authors, but we know this because some posters apologise for their level of English in the posts included.

 7 The subreddits were chosen to have topics that were informal and could have a reasonable number of posts, although in reality, the number of non-standard posts found from these specific subreddits was limited.

https://github.com/rbawden/RoCS-MT

A segment's translation can contain several sentences but sentence boundaries cannot be overridden.

The spaCy segmentation was obtained by concatenating all normalised sentences from a single text and then automatically splitting.

The prompt used is the sentence-level prompt from[START_REF] Hendy | How good are GPT models at machine translation? A comprehensive evaluation[END_REF], which is also shown in Appendix C.

Acknowledgements

This paper was funded by both authors chair positions in the PRAIRIE institute funded by the French national agency ANR as part of the "Investissements d'avenir" programme under the reference ANR-19-P3IA-0001. It was also funded by Rachel Bawden's Emergence project, DadaNMT, funded by Sorbonne Université.

------CUNI-DocTrans 28.9 18.0 10.9 ------CUNI-GA 28.9 18.0 10.9 ------Table 7: BLEU scores of systems on the manseg-norm and manseg-raw subsets. Table 9: The difference in the number of source words present in the MT output between the manseg-raw and manseg-norm subsets, averaged across all sentences for each system. This indicates how much more (or less) source words are copied in the raw (unnormalised) sentences with respect to their normalised versions.