Microsoft Tom Kocmi

Eleftherios Avramidis

Rachel Bawden

Ondřej Bojar

Anton Dvorkovich

Christian Federmann

Mark Fishel

Markus Freitag

Thamme Gowda

Roman Grundkiewicz

Anton Dvorkovich Dubformer

Markus Freitag Google

Barry Haddow

Philipp Koehn

Benjamin Marie

Christof Ai

Monz

Makoto Morishita

Kenton Murray

Masaaki Nagata

Toshiaki Nakazawa

Martin Popel

Maja Popović

Mariya Shmatova Dubformer

Jun Suzuki

Benoît 2023 Sagot

Rocs

Christian Buck

Chris Callison-Burch

Johannes Leveling

Christof Monz

Pavel Pecina

Matt Post

Herve Saint

Rajen Chatterjee

Yvette Graham

Shujian Huang

Matthias Huck

Qun Liu

Varvara Logacheva

Matteo Negri

Raphael Rubino

Lucia Specia

Jimeno Yepes

Varvara Lo- Gacheva

Aurélie Névéol

Mariana Neves

Chris Hokamp

Findings of the 2023 Conference on Machine Translation (WMT23): LLMs Are Here But Not Quite There Yet

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

The Eighth Conference on Machine Translation (WMT23) 1 was held at EMNLP 2023 and hosted a number of shared tasks on various aspects of machine translation (MT). This conference built on 17 previous editions of WMT as a workshop or a conference (Koehn and Monz, 2006;[START_REF] Callison-Burch | meta-) evaluation of machine translation[END_REF][START_REF] Callison-Burch | Further meta-evaluation of machine translation[END_REF][START_REF] Callison-Burch | Findings of the 2009 Workshop on Statistical Machine Translation[END_REF][START_REF] Callison-Burch | Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation[END_REF][START_REF] Neubig | The Kyoto free translation task[END_REF][START_REF] Tiedemann | Parallel data, tools and interfaces in OPUS[END_REF]Bojar et al., 2013Bojar et al., , 2014Bojar et al., , 2015[START_REF] Sudarikov | Using MT-ComparEval[END_REF]Bojar et al., , 2017[START_REF] Bojar | Findings of the 2018 conference on machine translation (WMT18)[END_REF][START_REF] Barrault | Findings of the 2019 conference on machine translation (WMT19)[END_REF][START_REF] Barrault | Findings of the 2020 conference on machine translation (WMT20)[END_REF]Akhbardeh et al., 2021;[START_REF] Kocmi | Findings of the 2022 conference on machine translation (WMT22)[END_REF].

Following last year's shift from focusing mainly on the news domain, we have continued to explore the capabilities of "General Machine Translation".

1 http://www2.statmt.org/wmt23/ While the news domain provided a clear and familiar benchmark, we responded to the need to test MT in more diverse settings. Our goal is to assess MT systems' ability to handle a broader range of language use. How to test general MT performance is a research question in itself. Countless phenomena could be evaluated, the most important being:

• various domains (news, medicine, IT, patents, legal, social, gaming, etc.) • style of text (formal or spoken language, fiction, technical reports, etc.)

• robustness to non-standard (or noisy) usergenerated content (grammatical errors, codeswitching, abbreviations, etc.)

Evaluating all phenomena is nearly impossible and creates numerous unforeseen problems. Therefore, we decided to simplify the problem and start with an evaluation of different domains. We selected the following domains: news, e-commerce, social/user-generated content (UGC), speech, and manuals. They were chosen to represent topics with different content styles and to be understandable for humans without special in-domain knowledge, thus not requiring specialized translators or human raters for evaluation. Due to limited access to monolingual data across all languages, each language direction contains only a subset of up to four domains.

In addition to language pairs evaluated last year:

Czech→Ukrainian, English↔Chinese, English→Czech, English↔German, English↔Japanese, English↔Russian, Ukrainian→English,

we introduce a new language pair to WMT, namely:

English↔Hebrew.

Other than language pairs, there are several differences with respect to last year's task. All language pairs are provided with the sentence boundaries marked except for English↔German, where we decided to experiment with paragraph-level translation. Another significant change for this year is the unification of our human evaluation protocol. We no longer rely on reference-based MTurk evaluation and move the evaluation towards source-based DA+SQM evaluation (introduced last year) with professional annotators. Finally, this year's shared task included an increased number of test suites (Section 6), allowing the evaluation of MT outputs from different perspectives, including a range of linguistic phenomena, purposely difficult sentences, specialist domains, gendered translations and non-standard UGC translation.

All General MT task submissions, sources, references and human judgements are available at Github. 2 The interactive visualization and comparison of differences between systems can be browsed online on an interactive leaderboard3 using MT-ComparEval [START_REF] Klejch | MT-ComparEval: Graphical evaluation interface for machine translation development[END_REF][START_REF] Sudarikov | Using MT-ComparEval[END_REF].

The structure of the paper is as follows. We describe the process of collecting, cleaning and translating the test sets in Section 2 followed by a summary of the permitted training data for the constrained track Section 3. We list all submitted systems in Section 4. The human evaluation approach of DA+SQM is described in Section 5. Finally, Section 6 describes the test suites and summarises their conclusions.

Summary of the WMT2023 General MT task

The main findings are as follows:

• Large Language Models (LLMs) exhibit strong performance across the majority of language pairs, although this is based only on two LLM-based system submissions. Test suite analysis revealed that although GPT4 excelled in some areas (e.g. UGC translation) struggled with other aspects such as speaker gender translation and specific domains (e.g. legal), whereas it ranked lower than encoder-decoder systems when translating from English into less-represented languages (e.g. Czech and Russian)

• We have observed a decline in the number of submissions into the constrained track. Consequently, we plan to re-evaluate the definition and the incentives of the constrained track and consider incorporating open-source LLMs in future constrained evaluations.

• We demonstrate the feasibility of paragraphlevel German↔English tasks, although more investigation would be required before generalising to all language pairs.

• Professional human translations do not always guarantee high quality. For Hebrew↔English, our references are likely to be post-edited MT, while for Chinese→English, the reference translation is worse than the majority of automatic translations.

• The manual evaluation results obtained from DA+SQM and MQM methods yield comparable cluster rankings.

Test Data

In this section, we describe the process of collecting data in Section 2.1, followed by the explanation of preprocessing steps in Section 2.2. Producing human references is summarized in Section 2.3 and lastly test set analysis is conducted in Section 2.4.

Collecting test data

As in the previous years, the test sets consist of unseen translations collected especially for the task.

The guaranteed novelty of the test sets has become even more important with the rise of LLMs trained on unspecified training data. To prevent possible contamination, we focused on collecting as recent Table 1: Test set statistics per direction and domain (rows marked * are over all domains). Note that en→de shares source test data with the other from-English directions, but as translation and evaluation for both en→de and de→en were carried out on the paragraph level (a segment therefore being a paragraph rather than a sentence), this results in a lower number of segments per document. The domain name is as indicated in the released test sets and domain type indicates the broader domain category.

data as possible across various domains. This task is incredibly difficult and needs further investigation in future years. There are three main limitations:

• Finding sources with different domains.

• Finding data that are in the public domain or under open licenses.

• Finding recently created data to minimize the risk of them being part of the training pipelines.

The test sets are publicly released to be used as translation benchmarks. Here we describe the test sets' production and composition.

We decided to collect data from 5 domains (news, social/user-generated, e-commerce, manuals, and speech). For all language pairs, we aimed for a test set size of 2,000 sentences and to ensure that the test sets were "source-original", namely that the source text was first written in the source language, and then the target text is the human translation. This is to avoid "translationese" effects in the source language, which can have a detrimental impact on the accuracy of evaluation [START_REF] Toral | Attaining the unattainable? reassessing claims of human parity in neural machine translation[END_REF][START_REF] Freitag | APE at scale and its implications on MT evaluation biases[END_REF][START_REF] Läubli | A Set of Recommendations for Assessing Human-Machine Parity in Language Translation[END_REF][START_REF] Graham | Statistical power and translationese in machine translation evaluation[END_REF]. We collected roughly the same number of sentences for each domain. For some languages, we could not locate high quality data and therefore we selected more sentences from other domains. Note that descriptions in this section refer to source monolingual data when mentioning a language.

News domain For most languages this domain contains data prepared in the same way as in previous years (Akhbardeh et al., 2021). We collected news articles from February 2023 extracted from online news sites, preserving document boundaries. We expect that news domain text will generally be of high quality. The news in Hebrew was kindly provided by the Israeli Association of Human Language Technologies (IAHLT). 4 These are samples of originally Hebrew texts from news published in Israel Hayom5 in 2022.

E-commerce domain (product reviews) This domain consists of user reviews of different Amazon products selected from the publicly available multilingual corpus [START_REF] Keung | The multilingual Amazon reviews corpus[END_REF]. This corpus was designed for multilingual text classification and consists of reviews written in English, Japanese, German, French, Spanish, and Chinese, between 2015 and 2019. We used the test parts of the English, German, Japanese and Chinese corpora for extracting the source part of the WMT test set. The reviews were selected so that the resulting corpus covers each product, all rating scores for the product, and the lexical diversity is maximized. The lexical diversity was estimated as a simple ratio between the number of distinct words/characters (vocabulary) divided by the total number of words/characters. Social/user-generated domain For English and German, we relied on the Mastodon Social API. 6Mastodon is a federated social network that is compatible with the W3C standard ActivityPub [START_REF] Webber | Ac-tivityPub, W3C Recommendation[END_REF]. Users publish short-form content similar to tweets that are referred to as "toots" for historical reasons. As this is a decentralized social media network, different servers have very different data, policies, communities, and uses. We decided to use mastodon.social, the original server, as it has a large community as well as publicly available toots. We collected data in early May of 2023. We used the reported language ID label, but were only able to collect enough data in German and English. We only collected toots with more than 150 characters in length in order to allow for data that was more likely to be semantically interesting for evaluating translation systems.

For Hebrew, we used comments on news articles from the Israel Hayom site mentioned above. This data was also provided by IAHLT.

For Russian, we used data from the Geo Reviews Dataset containing reviews about organizations published on Yandex Maps and open for academic and research purposes. 7For Japanese, we used product descriptions of a b2b e-commerce site and search advertising text ads for the social and user-generated domain, because we could not obtain high-quality data for this domain type. MonotaRo Co., Ltd. provided product descriptions of their private label brands listed on their b2b e-commerce site. 8 We defined a document for a product description as a combination of a title, product description, and cautionary note. CyberAgent, Inc.9 provided search advertising text ads with their client's consent. We defined a document for an ad as the longest possible combination of multiple titles and descriptions.

Manuals For this domain, we primarily sourced scanned versions of different mostly gaming manuals provided by Centific10 . These were then converted to digital text format using Optical Character Recognition (OCR) technology. Given the inaccuracies of OCR, the digitized content underwent a subsequent post-editing phase, where humans reviewed and corrected any errors. The selection of manuals ranged across various sources, and none of them were older than five years.

Speech

The exact data types used in the "conversational" or "speech" domain vary across language pairs. For English→Czech, the data comes from the test set which was created for the 2023 instance of AutoMin 2023 (Ghosal et al., 2022). 11 The texts are manually curated transcripts of project meetings, same in style as released in ELITR Minuting Corpus [START_REF] Nedoluzhko | ELITR minuting corpus: A novel dataset for automatic minuting from multi-party meetings in English and Czech[END_REF]. The meetings were held mostly remotely or in a hybrid form, all meeting participants were non-native speakers of English and the meetings were always on rather technical and in-depth topics. Our manual curation corrected ASR errors (but not errors in English grammar or vocabulary) and de-identified the transcripts, replacing names with placeholders ("PER-SONxy", "PROJECTxy" and similar). For person names, round brackets are used at the beginnings of lines to indicate the speaker and square brackets are used in the text when the person was mentioned. The data contain also some markup, e.g. "<unintelligible/>". These conventions are likely to be distorted by translation systems and we also noticed that they were distorted in the reference translation (the style of the brackets was ignored). This tiny detail can influence both manual and automatic scoring on this domain.

For Japanese, we used question-answer pairs from a community question-answering service. NTT Resonant Inc., which recently merged with NTT DOCOMO, INC., provided question-answer pairs from their website, Oshiete! goo.12 For every question-answer pair, we defined a document as a combination of a question and its best answer marked by the user.

Czech and Ukrainian source texts Source texts for Czech→Ukrainian and Ukrainian→English translation included the News domain as described above and texts collected through the Charles Translator for Ukraine. 13 With users' consent, the service can log their inputs for the purpose of creating a dataset of real use cases. The datasets are extracted from the inputs collected from May 2022 to April 2023.

The Charles Translator mobile app supports voice input, which is converted to text using Google ASR (automatic speech recognition). The texts collected this way were marked as the voice domain. For Ukrainian→English, the remaining Ukrainian inputs were classified either as clipboard (texts inserted to the Charles Trans-lator using the Paste from clipboard button) and other. The clipboard texts are more likely to include formal communication copied from web sites, but we noticed it includes personal communication (copied from chat applications) as well. Thus for Czech→Ukrainian, we decided to classify the remaining Czech inputs either as official (formal communication) or personal (personal communication), ignoring whether they were inserted from a clipboard or written using a keyboard.

The texts were filtered and pseudonymized in the same way as last year [START_REF] Kocmi | Findings of the 2022 conference on machine translation (WMT22)[END_REF], so for example we asked the annotators not to delete or fix noisy inputs as long as they are comprehensible. There was one exception from this rule this year: the Czech voice domain data was post-edited to fix ASR errors, including missing punctuation and casing.

The source texts were translated by professional translators principally following the brief in Appendix C. Last year, parts of the Ukrainian→Czech test set was detected to be post-edited MT. Therefore this year, we decided to hire two professional translators directly without the mediation of a translation agency, we emphasised the rule that the translations must be done from scratch (without MT postediting and without translation memories). We could not detect any MT postediting in the resulting translations.

Human preprocessing of test data

Although testing of robustness of MT is an important task, the noisy data introduces problems for human translators and annotators. Therefore, we decided to discard data considered too noisy. Furthermore, publicly available data often contains inappropriate content, which can stress either human translators or human annotators, leading to a decrease in the quality (for example, translators refuse to translate political content considered censored in their countries).

Therefore, we asked humans to check collected data and carry out minor corrections (mainly checking sentence splits and discarding similar or repeated content). This was sufficient for the news domain because it was often clean and without serious problems. However, with the expansion towards general MT, we find ourselves running into an issue of source data being noisier and less well formatted and that therefore needs to be handled before translation. Furthermore, we asked them to remove shortest documents to keep longer context. The source data for test sets therefore goes through human validation checks involving linguists discarding inappropriate content altogether and carrying out minor textual corrections to the data. You can find the linguistic brief for prepossessing in Appendix B.

Test set translation

The translation of the test sets was performed by professional translation agencies, according to the brief in Appendix C. Different partners sponsored each language pair and various translation agencies were therefore used, which may affect the quality of the translation.

Regrettably, upon reviewing translations procured from one of the agencies (the one responsible for English to Hebrew and Hebrew to English translations), it appeared that the translations might have been post-edited from publicly available online translation systems. This observation contradicts the initial instruction provided for agency that precluded the use of any automated translation platforms. While the agency has asserted that their professional translations conducted translations from scratch, our evaluation suggested otherwise. Moving forward, we propose to build a step-by-step verification system to avoid such discrepancies.

Human translations would not be possible without the sponsorship of our partners: Microsoft, Toloka AI, Google, Charles University, NTT, and Dubformer.

Test set analysis

As described previously, the chosen domains, sources for the data and the number of sentences per domain was subject to the availability of high quality data in each language direction. For example, while the news domain was available for all language directions, social media data was only available for English, German (both from Mastodon) and Hebrew (from comments on news articles). The number of documents, segments, average document length and type-token ratio (of the source side of the test sets) are given in Table 1.

Document context Document context is available for all language directions, although the average document length varies both by domain and language direction. Manuals tend to represent the longest domains, followed by the news domain. The social media domain tends to represent the shortest documents. along with reviews. Note that this year, we piloted translation and evaluation of en→de and de→en at the paragraph level (with each segment therefore containing several sentences), with the aim of avoiding the constraint of having a one-to-one mapping at the level of the sentence between source texts and their translations. This is visible in the statistics in Table 1 as the number of segments is lower for these two directions, as is the average document length.

Lexical diversity

We can compare the typetoken ratio (TTR) to get an idea of the relative lexical diversity of (i) domains and (ii) original vs. translated sentences. 14,15 Raw TTRs for each language pair and domain are shown in Table 11 in Appendix D. Regarding domains, the TTR appears highest for texts mastodon, perhaps illustrating the diversity of conversational topics and also of the potentially non-standard nature of the texts. User reviews appear to have the lowest TTR, most likely due to the fact that similar vocabulary is used across reviews. The TTR of course differs according to the language in question, according to the differing morphological properties.

Anonymisation and markup A particularity of the 'speech' domain is the presence of placeholders for anonymised elements and markup (in the form of tags). For example, there are 35 placeholders surrounded either by square or rounded brackets to indicate different people, organisations and projects (e.g. (PERSON1), [PERSON9], [ORGANIZA-TION4], [PROJECT8], etc.). The 'person' tags are used both in-text to replace the names of people and at the beginning of lines to indicate who is talking. Markup is added to indicate speakers talking at the same time (<parallel_talk>), unintelligible passages (<unintelligible/>), laughter (<laugh/>) and other noise (<other_noise/>).

Test suites

In addition to the test sets of the regular domains, the test sets given to the system participants were augmented with several test suites, i.e. custommade test sets focusing on particular aspects of MT translation. The test suites were contributed and evaluated by test suite providers as part of a decentralized sub-task, which will be detailed in Section 6.

Training Data

Similar to the previous years, we provide a selection of parallel and monolingual corpora for model training. The provenance and statistics of the selected parallel datasets are provided in Appendix in Table 9 andTable 10. Specifically, our parallel data selection include large multilingual corpora such as Europarl-v10 [START_REF] Koehn | Europarl: A parallel corpus for statistical machine translation[END_REF], Paracrawl-v9 [START_REF] Bañón | ParaCrawl: Web-scale acquisition of parallel corpora[END_REF], Com-monCrawl, NewsCommentary-v18, WikiTitles-v3, WikiMatrix (Schwenk et al., 2021), TildeCorpus [START_REF] Rozis | Tilde MODEL -multilingual open data for EU languages[END_REF], OPUS [START_REF] Tiedemann | Parallel data, tools and interfaces in OPUS[END_REF], UN Parallel Corpus [START_REF] Ziemski | The United Nations parallel corpus v1.0[END_REF], and language-specific corpora such as CzEng-v2.0 [START_REF] Kocmi | Announcing CzEng 2.0 Parallel Corpus with over 2 Gigawords[END_REF], YandexCorpus,16 ELRC EU Acts, JParaCrawl [START_REF] Morishita | JParaCrawl: A large scale web-based English-Japanese parallel corpus[END_REF], Japanese-English Subtitle Corpus [START_REF] Pryzant | JESC: Japanese-English subtitle corpus[END_REF], KFTT [START_REF] Neubig | The Kyoto free translation task[END_REF], TED [START_REF] Cettolo | WIT3: Web inventory of transcribed and translated talks[END_REF], CCMT, and back-translated news. Links for downloading these datasets were provided on the task web page;17 in addition, we automated the data preparation pipeline using MTDATA [START_REF] Gowda | Many-to-English machine translation tools, data, and pretrained models[END_REF]. 18 MTDATA downloads all the mentioned datasets, except CCMT and CzEng-v2.0, which required user authentication. This year's monolingual data include the following: News Crawl, News Discussions, News Commentary, Common-Crawl, Europarl-v10 [START_REF] Koehn | Europarl: A parallel corpus for statistical machine translation[END_REF], Extended CommonCrawl [START_REF] Conneau | Unsupervised cross-lingual representation learning at scale[END_REF], Leipzig Corpora [START_REF] Goldhahn | Building large monolingual dictionaries at the Leipzig corpora collection: From 100 to 200 languages[END_REF], UberText and Legal Ukrainian.

System submissions

This year, we received a total of 72 primary submissions from 17 participants. In addition, we collected translations from online MT systems across all language pairs. Online system outputs come from 6 public MT services and were anonymized as ONLINE-{A,B,G,M,W,Y}, which added additional 77 system outputs. The participating systems are listed in Table 2 and detailed in the rest of this section.

Finally, we added translations by three contrastive systems. Two of them are based on the NLLB translation model [START_REF] Nllb Team | No Language Left Behind: Scaling Human-Centered Machine Translation[END_REF] modified by [START_REF] Freitag | Results of WMT23 Metrics Shared Task: Metrics might be Guilty but References are not Innocent[END_REF] to have a suboptimal performance, using (i) greedy search (NLLB_Greedy) and (ii) following minimum Bayes risk decoding (MBR) optimizing the BLEU metric (NLLB_MBR_BLEU). Neither of them is the official (and better performing) NLLB model. The third contrastive translation is produced by the large language model GPT4 using 5-shot prompting with fixed random translation examples, using the exact prompt by [START_REF] Hendy | How good are GPT models at machine translation? a comprehensive evaluation[END_REF] together with their predefined few-shot examples. For languages not evaluated in their study, we took examples from the last WMT test sets.

Appendix E provides details of the submitted systems if the authors provided such details.

Constrained and unconstrained tracks

For presentation of the results, systems are treated as either constrained or unconstrained. A system is classified as constrained if the authors reported training only on the provided data and adhering to the rules describing the use of publicly available pre-trained models. The constrained track imposes restrictions on training data, metrics, and pretrained models, while the unconstrained track provides unrestrained flexibility.

The constrained track limitations are mainly around the training and testing data, together with the limitation on pretrained models:

• Training data: Only data specified for the current year are permissible, see Section 3. Multilingual systems can be used as long as they only use WMT23 data.

• Metrics: The training pipeline can use pretrained metrics evaluated in previous WMT Metrics shared tasks, e.g., COMET [START_REF] Rei | COMET-22: Unbabel-IST 2022 submission for the metrics shared task[END_REF], Bleurt [START_REF] Yan | BLEURT has universal translations: An analysis of automatic metrics by minimum risk training[END_REF].

• Pretrained models: only the following list of models is allowed together with all their public sizes: mBART [START_REF] Liu | Multilingual denoising pretraining for neural machine translation[END_REF], BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF], RoBERTa [START_REF] Liu | RoBERTa: A Robustly Optimized BERT Pretraining Approach[END_REF], XLM-RoBERTa [START_REF] Conneau | Unsupervised cross-lingual representation learning at scale[END_REF], sBERT [START_REF] Reimers | Sentence-BERT: Sentence embeddings using Siamese BERTnetworks[END_REF], and LaBSE [START_REF] Feng | Language-agnostic BERT sentence embedding[END_REF].

• Linguistic tools: Basic tools like taggers, parsers, and morphology analyzers are allowed. LANGUAGEX en-zh, en-uk, ru-en, uk-en, en-de, he-en, ja-en, zh-en, en-he, de-en, en-cs, en-ja, en-ru [START_REF] Zeng | Achieving state-of-the-art multilingual translation model with minimal data and parameters[END_REF] Table 2: Participants in the General MT shared task. Online system translations were not submitted by their respective companies but were obtained by us, and are therefore anonymized in a fashion consistent with previous editions of the task.

The online systems and contrastive systems are treated as unconstrained during the automatic and human evaluation.

OCELoT

We used the open-source OCELoT platform19 to collect system submissions again this year.

The platform provides anonymized public leaderboards20 and was also used for two other WMT23 shared tasks: Biomedical [START_REF] Neves | Findings of the WMT 2023 Biomedical Translation Shared Task: Evaluation of ChatGPT 3.5 as a Comparison System[END_REF] and Sign Language Translation [START_REF] Müller | Findings of the second WMT shared task on sign language translation (WMT-SLT23)[END_REF].

As in previous years, only registered and verified teams with correct contact information were allowed to submit their system outputs and each verified team was limited to 7 submissions per test set. Submissions on leaderboards with BLEU [START_REF] Papineni | Bleu: a method for automatic evaluation of machine translation[END_REF] and CHRF [START_REF] Popović | chrF: character n-gram F-score for automatic MT evaluation[END_REF] scores from SacreBLEU [START_REF] Post | A call for clarity in reporting BLEU scores[END_REF] were displayed anonymously to avoid publishing rankings based on automatic scores during the submission period. Until one week after the submission period, teams could select a single primary submission per test set, specify if the primary submission followed a constrained or unconstrained setting, and submit a system description paper abstract. These were mandatory for a system submission to be included in the human evaluation campaign.

Human Evaluation

Human evaluation for all language translation directions is performed with source-based ("bilingual") Direct Assessment (DA, [START_REF] Graham | Continuous measurement scales in human evaluation of machine translation[END_REF] of individual segments in document context with Scalar Quality Metrics (SQM) guidelines, mostly following the setup established at WMT22 (DA+SQM, [START_REF] Kocmi | Findings of the 2022 conference on machine translation (WMT22)[END_REF]. DA+SQM asks the annotators to provide a score between 0 and 100 on a sliding scale, but the slider is presented with seven labelled tick marks, as demonstrated in Figure 1. Two different annotation platforms and four distinct pools of annotators (Table 3) are used for annotation of different language pairs. We use the open-source framework Appraise [START_REF] Federmann | Appraise evaluation framework for machine translation[END_REF] for the evaluation of English→Czech, English↔{Chinese, German, Japanese}, and Czech→Ukrainian. Toloka AI21 hosts the evaluation of English↔{Hebrew, Russian, Ukrainian} using their own implementation of the source-based document-level DA+SQM task, which is as close as possible to the Appraise user interface.

We keep the selection process of documents for annotation mostly the same as in the previous year. The only change made in order to align closer with the MQM-based evaluation run at the Metrics shared task [START_REF] Freitag | Results of WMT23 Metrics Shared Task: Metrics might be Guilty but References are not Innocent[END_REF] is to present the first 10 segments from a document instead of random 10 consecutive segments.

We again collect both segment-level scores and document-level scores, but compute rankings based on segment scores only.

Human annotators

Annotations for different language pairs are provided by four different parties with their pool of annotators of distinct profiles as presented in Table 3. We shift towards more professional or semiprofessional annotators' pools and decide not to use MTurk annotations as in past years for referencebased DA evaluation for into-English language directions.

Assessments for English↔{Chinese, German, Japanese} are provided by Microsoft and their pool of bilingual target-language native speakers, professional translators or linguists, highly experienced in MT evaluation. Microsoft monitors the annotators' performance over time and permanently removes from the pool those who fail quality control, which increases the overall quality of the human assessment.

Charles University provides annotators for language pairs involving the Czech language, i.e., English→Czech and Czech→Ukrainian. Their annotators are linguists, translators, researchers and students who are native speakers of the target language with high proficiency in the source language.

DA scores for English↔{Hebrew, Russian, Ukrainian} are collected by Toloka AI using their paid crowd of bilingual target-language native speakers. Toloka AI tests proficiency of their annotator crowd across different NLP annotation tasks and allowed only annotators who deemed reliable according to their quality control measures.

Document selection and quality control

The document selection process remains the same as in the previous year with minor changes. We first randomly sample a subset of document snippets from each of the domains for annotations, sampling the domains with approximately the same number of segments per domain. This ensures that Figure 1: Screenshot of the document-level DA+SQM configuration in the Appraise interface for an example assessment from the human evaluation campaign for out of English language pairs. The annotator is presented with the entire translated document snippet randomly selected from competing systems (anonymized) with additional static contexts, and is asked to rate the translation of individual segments and then the entire document on sliding scales between 0 and 100. all systems in the given language pairs are evaluated on the same subset of the test set, allowing fair comparison between them. As in previous years, we aim to collect approximately 1,500 assessments per system per language pair. Due to concerns about having sufficient annotations, we create two batches of HITs, each providing half of the required assessments, such that at least all segments in the first batch could be covered for all systems, with the second campaign completed if possible.

For HIT generation for English↔German, which feature paragraph-level test sets (documents consist of paragraphs instead of sentences), we simply consider a whole paragraph as a "segment", collecting paragraph-level assessments. In that regard, we collect fewer DA scores per system comparing to other language pairs, but the human evaluation covers a larger subset of the test sets.

Last year, we used snippets of at most 10 randomly selected consecutive segments from a document as "documents" for document-level annotation. This year, we use 10 first segments from a document instead, in order to align with the MQMbased evaluation used at the Metrics shared task [START_REF] Freitag | Results of WMT23 Metrics Shared Task: Metrics might be Guilty but References are not Innocent[END_REF].

All HITs consist of exactly 100 segments and are generated as in the past:

1. Snippet-system pairs are randomly sampled (from the restricted set of pre-sampled snippets) to create up to 80 segments;

2. Random snippets for the remaining 20 (or more) segments are duplicated from the first 80 to serve as quality control items;

3. BAD references are introduced to the random segments in the duplicated snippets to have about 12-14% of quality control segments per HIT.

BAD translations are created by replacing an embedded sequences of tokens in the segment with a random phrase of the same length from a different reference segment. 22We perform quality control by measuring an annotator's ability to reliably score BAD translations significantly lower than corresponding original system outputs using a paired significance test with p < 0.05. We pair two HITs into a single annotation task with about 24-28 quality control segments to ensure a sufficient sample size for the statistical test. In campaigns hosted on Appraise, if an annotator is not able to demonstrate reliability on BAD references, they are excluded from further annotations, the HITs are reset and annotated from scratch by another annotator if possible.

The total number of assessments collected for each language pair and the average number of assessments per system in WMT23 manual evaluation are presented in Table 4.

Calibration HITs

Last year we introduced calibration HITs, which this year we collect for all language pairs. A calibration HIT is a HIT with 100 randomly selected segments, which is identical for and completed by all annotators, in addition to their regular annotation HITs. We release these alongside the other annotations and the anonymized mapping between annotators and HITs in order to enable additional analysis. With a small set of sentences annotated by all annotators, we are better able to examine questions about inter-annotator consistency and provide data for future research in this area.

Table 5 shows the number of unique annotators per language pair along with the total number of HITs and average number of HITs per annotator. We leave more detailed analysis of collected calibration data to future work.

Human ranking computation

The official rankings shown in Table 6 are generated on the basis of the segment-level raw DA+SQM scores that are collected within document context for all language pairs. 23 Whole documents with at least one quality control segment (i.e., BAD references) and HITs that failed to pass quality control are removed prior to computing the rankings. 24 In this year's evaluation, we have chosen not to normalize scores by discontinuing the use of z-scores, given their potential to exacerbate system comparisons [START_REF] Knowles | On the stability of system rankings at WMT[END_REF]. While utilizing raw scores is not flawless-considering each annotator employs distinct annotation strategieswe have sought to counteract this by distributing systems evenly across annotators. This approach aims to minimize the potential bias of a particularly stringent annotator disproportionately penalizing a single system. Ideally, every annotator would assess documents translated by all systems; however, this could introduce task repetitiveness concerns. For future considerations, employing calibration HITs (see Section 5.3) to normalize each annotator's behaviour could offer a promising solution.

All segment-level scores are averaged per system to compute the system-level scores. The clusters are computed using the Wilcoxon rank-sum test with p < 0.05. Rank ranges indicate the number of systems a particular system underperforms or outperforms: the top end of the rank range is l + 1 where l is the number of losses, while the bottom is n -w where n is the total number of systems and w is the number of systems that the system in questions significantly wins against.

Tables with head-to-head comparisons between all systems are included in Appendix G.

At the time of preparation of the camera-ready version of the paper, we have not been able to collect the required number of high-quality assessments for language pairs run through Toloka AI that would meet WMT standards for human evaluation. In that regard, we decided not to publish official rankings based on manual evaluation for English↔{Hebrew, Russian, Ukrainian} until the conference, we are planning to address it later.

Comparison of human evaluation methods

In collaboration with the metrics shared task [START_REF] Freitag | Results of WMT23 Metrics Shared Task: Metrics might be Guilty but References are not Innocent[END_REF], human annotation data for the Chinese→English and English→German direction was collected using two different approaches: the source-based DA+SQM approach, and the Multidimensional Quality Metrics (MQM) framework [START_REF] Freitag | Experts, errors, and context: A large-scale study of human evaluation for machine translation[END_REF]. We present the rankings produced by the two approaches in Table 7.

Upon examining the system rankings and individual clusters produced by both techniques, it is evident that DA+SQM produces fewer clusters. This suggests that it might not be sufficiently robust to differentiate smaller system differences, whereas MQM creates more detailed clusters. One potential explanation is that DA+SQM, constrained by budgetary restrictions, might be under-powered. As highlighted by [START_REF] Wei | Searching for a higher power in the human evaluation of MT[END_REF], the 1500 segments we gather per system might not suffice to segregate systems in a more detailed manner.

Conversely, the largest difference in the evaluation techniques is the cost. While MQM manages to establish more refined clusters, its deployment is significantly more costly and complex, especially when training professionals. An interesting question would be determining the number of MQM labels that could be procured within the budget allocated for DA+SQM.

It is also important to note that the set of data over which each of these rankings was produced may have differed slightly due to the sampling (e.g., the distribution over topic domains or the amount of coverage of the full test set), making it difficult to determine whether these differences in rankings represent differences due to data or due to different annotation methods.

Test Suites

As can be seen in the general MT task, the improvement of translation quality has made it difficult to discriminate MT output from human translation with the current evaluation methods. Nevertheless, there are still cases where MT has difficulties, delivering outputs which despite seeming fluent and being surrounded by other seemingly perfect translations, entail serious flaws. In general evaluation methods, such flaws can get "hidden in the average" or simply get missed altogether. In an effort to shed light to these cases, evaluation via test suites is embedded in the shared task.

Setup of the sub-task

Test suites are custom extensions to standard test sets, constructed so that they can focus on particular aspects of the MT output. Here, the evaluation of the MT outputs takes place in a decentralized manner as a part of a sub-task, where test suite providers were invited to submit their customized test sets, following the setting introduced at the Third Conference on Machine Translation [START_REF] Bojar | Findings of the 2018 conference on machine translation (WMT18)[END_REF].

Every test suite provider submitted a source-side test set, which the shared task organizers appended to the standard test sets of the shared task. The corresponding outputs from the MT systems of the shared task were returned to the test suite providers, who were responsible for running the evaluation, based on their own custom evaluation methods. The results of each test suite evaluation, together with the relevant analysis, appear in separate description papers.

Submissions

The test suite sub-task received 5 submissions with 6 test suites, whose overview can be seen in Table 8. The descriptions of each submission and their main findings are given below. DFKI [START_REF] Manakhimova | Linguistically motivated Evaluation of the 2023 State-of-theart Machine Translation: Can ChatGPT Outperform NMT[END_REF] test suite offers a fine-grained linguistically motivated analysis of the shared task MT outputs, based on more than 11,500 manually devised test items, which cover up to 110 phenomena in 14 categories per language direction. Extending their previous test suite efforts (e.g. [START_REF] Avramidis | Fine-grained evaluation of quality estimation for machine translation based on a linguistically motivated test suite[END_REF][START_REF] Macketanz | A linguistically motivated test suite to semi-automatically evaluate German-English machine translation output[END_REF], the submission of this year includes an updated test set featuring new linguistic phenomena and focuses additionally on the participating LLMs. The evaluation spans German→English, English→German, and English→Russian language directions. Some of the phenomena with the lowest accuracies for German→English are idioms and resultative predicates. For English→German, these include mediopassive voice, and noun formation(er). As for English→Russian, these include idioms and semantic roles. GPT4 performs equally or comparably to the best systems in German→English and English→German but falls in the second significance cluster for English→Russian.

HW-TSC [START_REF] Chen | Multifaceted Challenge Set for Evaluating Machine Translation Performance[END_REF] propose a systematic approach to select test sentences with highlevel of difficulty from the Wiki Corpus. The strategy considers the difficulty level of a sentence from four dimensions: word difficulty, length difficulty, grammar difficulty and model learning difficulty. They open-source two Multifaceted Challenge Sets for Chinese→English and English→Chinese, each of them containing 2,000 sentences. Then, they use these challenge sets to test the shared task systems, presenting results by three automatic metrics.

The resulting system ranks are quite different from the official results. The authors point out that systems that perform well on average test sets may not perform as well on sets with high difficulty. If the ranking difference is caused by domain issues, the top-ranked systems on the official test sets may not be so general. GPT4 is ranked in the first two positions in Chinese→English but its rank in English→Chinese is much lower (ranks 4-9).

IIIT HYD [START_REF] Mukherjee | IIIT HYD's Submission for WMT23 Test-suite task[END_REF] This test suite covers five specific domains (entertainment, environment, health, science, legal) and spans five distinct writing styles (descriptive, judgments, narrative, reporting, technical-writing) for English-German. The authors conduct their analysis through a combination of au-tomated assessments and manual evaluations.

Based on their evaluation, it is evident that both ONLINE-B and ONLINE-Y consistently surpassed other MT systems in performance across a diverse array of writing styles and domains. When focusing on GPT4, whereas it performs comparably to the best systems for most domains and writing styles, it gives considerably worse results when applied to the legal domain, and the writing style of judgments.

MuST-SHE WMT23 and INES [START_REF] Savoldi | Test Suites Task: Evaluation of Gender Fairness in MT with MuST-SHE and INES[END_REF] By focusing on the en-de and de-en language pairs, the authors rely on these newly created test suites to investigate systems' ability to translate feminine and masculine gender and produce gender-inclusive translations. Furthermore, they discuss metrics associated with the test suites and validate them by means of human evaluations.

The results indicate that systems achieve reasonable and comparable performance in correctly translating both feminine and masculine gender forms for naturalistic gender phenomena. Instead, the generation of inclusive language forms in translation emerges as a challenging task for all the evaluated MT models, indicating room for future improvements and research on the topic.

Concerning GPT 4, it is noticeable that its overall accuracy is 2% worse than the best MT system, whereas it achieves a relatively low accuracy with regard to the feminine gender, when evaluating whether the first-person singular references to the speaker are translated according to the speaker's linguistic expression of gender.

RoCS-MT (Bawden and Sagot, 2023) The RoCS-MT Challenge Set is designed to test MT systems' robustness to user-generated content (UGC) displaying non-standard characteristics, such as spelling errors, devowelling, acronymisation, etc. It is composed of non-standard English comments from Reddit, manually normalised and professionally translated into four of the WMT 2023 target languages, German, Czech, Ukrainian and Russian, and also French.

Through automatic and manual analysis of system outputs, we find that many of the phenomena remain challenging for most systems, but to varying degrees depending on the phenomenon, the particular instance (notably how frequent the non-standard word is) and the system, especially with respect to the quantity of training data. For example, nonstandard instances of words (e.g. through devowelling or through phonetically inspired spelling) are often either omitted in the translation or copied unchanged. When non-standard words are translated, it is often in their standard form, but with some exceptions, for example capitalisation is sometimes preserved. However, there is often inconsistency within a same system's outputs.

GPT4-5shot has a clear lead over all other systems, correctly translating even some of the most challenging examples. It sometimes (although inconsistently) reproduces non-standardness in its outputs, but also does not always remain entirely faithful to the source sentence. However, aside the huge disparity in the amount of training data compared to other systems, notably the constrained ones, the lack of access to its training data is a serious obstacle to any meaningful scientific comparison; we cannot know which phenomena were seen during training and how frequently, and more crucially, we cannot verify whether RoCS-MT sentences were seen during training.

Conclusions

The General Machine Translation Task at WMT 2023 covered 14 translation pairs, where the only non-English language pair was Czech→Ukrainian. Source based DA+SQM was the main human golden truth. The evaluation included 72 primary submissions from 17 participants, 6 online systems and 3 additional contrastive systems including GPT4. It was performed by 155 human (semi-)professional annotators, who contributed more than 175,000 judgments altogether. For most language pairs (apart from English→Czech), MT systems produce outputs that cannot be identified as being worse than the manually produced references translations in a statistically significant way, using our current evaluation methods.

It is apparent that this year, the amount of unconstrained submissions are lower thank in past years (27 submissions by 11 participants). Additionally, for some language pairs there are only few submissions by participants, and therefore they are dominated by many online systems, of whom we have no technical descriptions. We are therefore considering ways to encourage participation in the future, whereas redefining the constrained setting may be needed.

It is the first time that Large Language Models (LLMs) are included in the Shared Task as translation systems. Although the technology is very apparent in NLP research, we received only one submission using LLM methods (Lan-BridgeMT), whereas one dominant commercial LLM (GPT4) was included via our own efforts. GPT4 was in the first significance cluster for all systems translating towards English, but fell in the second significance cluster (rank 3-5) for English→Czech, whereas a similar sign was given by one of the test suites for English→Russian (rank 3; [START_REF] Manakhimova | Linguistically motivated Evaluation of the 2023 State-of-theart Machine Translation: Can ChatGPT Outperform NMT[END_REF]. Additionally, test suites providers noted that GPT4 outputs are not always faithful to the source sentence (Bawden and Sagot, 2023) and that they have some issues with speaker gender translation [START_REF] Savoldi | Test Suites Task: Evaluation of Gender Fairness in MT with MuST-SHE and INES[END_REF] and specific domains (Mukherjee and Shrivastava, 2023, e.g. legal;). Due to the closed-source nature of commercial tools, it is hard to know the exact reasons for these findings, although they confirm previous observations that GPT models have difficulties with under-represented languages [START_REF] Hendy | How good are GPT models at machine translation? a comprehensive evaluation[END_REF]. We believe that a more transparent comparison including open source LLMs should be sought for the future.

Limitations

We investigated a research question of testing general capabilities of MT systems. However, we have simplified this approach. Firstly, we only used four domains that are not specialized. Secondly, we used only cleaner sentences, avoiding noisy in the source sentences.

Although we accept human judgement as a gold standard, giving us more reliable signal than automatic metrics, we should mention that human annotations are noisy [START_REF] Wei | The statistical advantage of automatic NLG metrics at the system level[END_REF] and their performance is affected by quality of other evaluated systems [START_REF] Mathur | Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics[END_REF].

Different annotators are using different ranking strategy which may have an effect on the system ranking as we are using raw scores.

Ethical Consideration

Several of the domains contained texts that included personal data, for example the speech data (See Section 2.4 for more details). Entities were replaced by anonymisation tags (e.g. #NAME#, #EMAIL#) to preserve the anonymity of the users behind the content.

The sentences in Ukrainian datasets were collected with users' opt-in consent, and any personal data related to people other than well-known people was pseudonymized (using random first names and surnames). Sentences where such pseudonymization would not be enough to preserve reasonable anonymity of the users (e.g. describing events uniquely identifying the persons involved) were not included in the test set.

As described in Section 2.2 and in the linguistic brief (Appendix Section B), inappropriate, controversial and/or explicit content was filtered out prior to translation, particularly keeping in mind the translators and not exposing them to such content or obliging them to translate it. A few sentences containing explicit content managed to escape the filter, and we removed these sentences from the test sets without translation.

Human evaluation using Appraise for collecting human judgements was fully anonymous. Automatically generated accounts associated with annotation tasks with single-sign-on URLs were dis-tributed randomly among pools of annotators and did not allow for storing personal information. For language pairs for which we used calibration HITs, we received lists of tasks completed by an individual anonymous annotator. Annotators have been well paid in respect to their countries.

A Statistics of training data

B Preprocessing cleanup brief for linguists

Human check briefing

In this task, we wish to check the data to remove all inappropriate content, remove repetitive content, or correct minor problems with the text.

The data is automatically broken down into individual sentences, which may contain wrong sentence splitting that needs to be fixed. Each paragraph is separated by empty lines. Keep the document-separators intact.

We ask you to read each document and either:

C Translator Brief for General MT

Translator Brief

In this project we wish to translate online news articles for use in evaluation of Machine Translation (MT). The translations produced by you will be compared against the translations produced by a variety of different MT systems. They will be released to the research community to provide a benchmark, or "gold-standard" measure for translation quality. The translation therefore needs to be a high-quality rendering of the source text into the target language, as if it was news written directly in the target language. However, there are some constraints imposed by the intended usage:

• All translations should be "from scratch", without post-editing from MT. Using postediting would bias the evaluation, so we need to avoid it. We can detect post-editing so will reject translations that are post-edited.

• Translation should preserve the sentence boundaries. The source texts are provided with exactly one sentence per line, and the translations should be the same, one sentence per line. Blank lines should be preserved in the translation.

• Translators should avoid inserting parenthetical explanations into the translated text and obviously avoid losing any pieces of information from the source text. We will check a sample of the translations for quality, and we will check the entire set for evidence of post-editing.

• Please do not translate the anonymization tags (e.g. #NAME#), but use the same form as in the source text. These tags are used to de-identify names and various other sensitive data. In other words, translation must contain given tag #NAME# on a position where it would naturally be placed before anonymization.

• If the original data contain errors, typos, or other problems, do not try to fix them (or introduce them in the translation), instead try to prepare correct translation as if the error wouldn't be in the source.

The source files will be delivered as text files (sometimes known as "notepad" files), with one sentence per line. We need the translations to be returned in the same format. If you prefer to receive the text in a different format, then please let us know as we may be able to accommodate it.

D Additional statistics of the test sets

Table 11 shows the type-token ratios for the source and target side of each of the test sets, shown for the four main domains. As mentioned previously, texts are tokenised using the language-specific Spacy models [START_REF] Honnibal | spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing[END_REF] where available. For Hebrew, we use the multilingual Spacy model as no language-specific model is available. The type-token ratio is calculated as the number of unique tokens divided by the total number of tokens. The absolute value depends not only on the lexical diversity of the text but also on the morphological complexity of the language in question.

E News Task System Submission Summaries

This section lists all the submissions to the translation task and provides the authors' descriptions of their submission.

E.1 AIRC [START_REF] Rikters | AIST AIRC submissions to the WMT23 shared task[END_REF] AIRC trained constrained track models for translation between English, German, and Japanese. Before training the final models we first filtered the parallel and monolingual data [START_REF] Rikters | Impact of Corpora Quality on Neural Machine Translation[END_REF], then performed iterative back-translation as well as parallel data distillation to be used for non-autoregressive model training. We experimented with training Transformer models, Mega [START_REF] Ma | Mega: Moving average equipped gated attention[END_REF]) models, and custom non-autoregressive sequence-to-sequence models with encoder and decoder weights initialised by multilingual BERT base. Our primary submissions contain translations from ensembles of two Mega model checkpoints and our contrastive submissions are generated by our non-autoregressive models.

E.2 ANVITA (no associated paper)

ANVITA-ZhJa Machine Translation system for WMT2023 Shared Task:General MT(News). This paper describes ANVITA-ZhJa MT system, architected for submission to WMT 2023 General Machine Translation(News) shared task by the ANVITA team, where the team participated in 4 translation directions: Chinese, Japanese→English and English→Chinese, Japanese. ANVITA-ZhJa MT system comprised of four NMT models.Chinese, Japanese→English and English→Chinese, Japanese multilingual models for primary and Chinese→English and English→Chinese bilingual models for contrastive submissions. Base MT models are built using transformer(base) architecture, trained over the organizer provided parallel corpus and subsequently used deep transformer with added layers and other parameters. We also distilled corpus using heuristics based filtering and used model ensemble for enhanced performance.

E.3 CUNI-DocTransformer [START_REF] Popel | CUNI English-Czech and English-Polish systems in WMT20: Robust document-level training[END_REF] Exactly the same system as submitted in WMT20, document-level Transformer trained with Block Backtranslation.

E.4 CUNI-GA [START_REF] Jon | CUNI-GA submission at WMT23[END_REF] Our submission is a result of applying a novel n-best list reranking and modification method on translation candidates produced by two other competing systems, CUNI-Transformer and CUNI-DocTransformer.

Our method uses a genetic algorithm and MBR decoding to search for optimal translation under a given metric (in our case, a weighted combination of ChrF, BLEU, COMET22-DA, and COMET22-QE-DA).

E.5 CUNI-Transformer [START_REF] Popel | CUNI English-Czech and English-Polish systems in WMT20: Robust document-level training[END_REF] The English↔Czech sentence-level models are exactly the same as submitted in WMT20 [START_REF] Popel | CUNI English-Czech and English-Polish systems in WMT20: Robust document-level training[END_REF]. The Ukrainian↔Czech models are very similar, also trained with Block Backtranslation.

E.6 GTCOM (Zong, 2023)

GTCOM uses transformer as the basic architecture and leverages multilingual models to improve translation quality. Besides, GTCOM does a lot of data cleaning and data augmentation work.

E.7 HW-TSC (Wu et al., 2023b) HW-TSC's submission is a standard Transformer model equipped with our recent technique.

E.8 IOL-Research [START_REF] Zhang | IOL Research machine translation systems for WMT23 general machine translation shared task[END_REF] This paper describes the IOL Research team's submission system for the WMT23 General Machine Translation shared task. We participate in two language translation directions, including English-to-Chinese and Chinese-to-English. Our final primary submissions belong to constrained systems, which means for both translation directions we only use officially provided monolingual and bilingual data to train the translation systems. Our systems are based on Transformer architecture with pre-norm or deep-norm, which has been proven to be helpful for training deeper models. We employ methods such as back-translation, data diversification, domain fine-tuning and model ensemble to build our translation systems. Another important aspect is that we carefully conduct data cleaning and use as much monolingual data as possible for data augmentation.

E.9 TeamKYB (LI et al., 2023) We here describe our neural machine translation system for the general machine translation shared task in WMT 2023. Our systems are based on the Transformer with base settings. We trained our model with preprocessed train data. We collect multiple checkpoint from our model and performed inference with several hyperparameter settings. Collected translations were processed via some rule-based corrections.

We chose best translation from the results by using N-best ranking method.

E.10 Lan-BridgeMT [START_REF] Wu | Exploring prompt engineering with GPT language models for documentlevel machine translation: Insights and findings[END_REF] With the emergence of large-scale models, various industries have undergone significant transformations, particularly in the realm of document-level machine translation. This has introduced a novel research paradigm that we have embraced in our participation in the WMT23 competition. Focusing on advancements in models such as chatGPT and GPT4, we have undertaken numerous prompt-based experiments.

Our objective is to achieve optimal human evaluation results for document-level machine translation, resulting in our submission of the final outcomes in the general track.

E.11 MUNI-NLP [START_REF] Rychlý | MUNI-NLP submission for Czech-Ukrainian translation task at WMT23[END_REF] MUNI-NLP system is a standard transformer.

E.12 NAIST-NICT [START_REF] Deguchi | NAIST-NICT WMT'23 general mt task submission[END_REF] In this paper, we describe our NAIST-NICT submission to the WMT'23 English-Japanese general machine translation task. Our system generates diverse translation candidates and reranks them with a two-stage reranking system to find the best translation. We first generate 50 candidates each from 18 different translation methods using a variety of techniques to increase the diversity of the translation candidates. We trained 7 different models per language direction using different combinations of hyperparameters. From these models we used various decoding algorithms, ensembling the models, and using kNN-MT. The 900 translation candidates go through a two-stage reranking system in order to find the most promising candidate.The first step compares the 50 candidates from each translation method using DrNMT and returns the one with the highest score. The final 18 candidates are ranked using COMET-MBR, and the highest scoring is returned as the system output. We found that generating diverse translation candidates improves the translation quality by using the well-designed relanker model.

E.13 PROMT [START_REF] Molchanov | PROMT systems for WMT23 shared general translation task[END_REF] This paper describes the PROMT submissions for the WMT23 Shared General Translation Task. This year we participated in two directions of the Shared Translation Task: English to Russian and Russian to English. Our models are trained with the MarianNMT toolkit using the transformer-big configuration. We use BPE for text encoding, both models are unconstrained. We achieve competitive results according to automatic metrics in both directions.

E.14 SRPH (Cruz, 2023)

We submit single-model encode-decoder Transformer systems for the constrained English to Hebrew and Hebrew to English translation directions. Our dataset is cleaned and filtered via a combination of heuristic-based, ratio-based, and embedding-based (LaBSE) methods, resulting in a dataset with high alignment. We train models with heavy use of back-translation and decode using Noisy Channel Reranking using a reverse model and a language model trained with contest data.

E.15 SKIM [START_REF] Kudo | SKIM at WMT 2023 general translation task[END_REF] The SKIM team submission took a standard procedure of building ensemble Transformer models, including base-model training, data augmentation using back-translation of base models, and retraining several final models using back-translated training data. Each final model has its own architecture and configuration, including a 10.5B parameter at most, substituting self and cross sublayers in decoder with cross+selfattention sub-layer [START_REF] Peitz | Cross+Self-Attention for Transformer Models[END_REF]. We select the best candidate from large candidate pools, namely 70 translations generated from 16 distinct models for each sentence, with an MBR reranking method using COMET and COMET-QE [START_REF] Fernandes | Quality-aware decoding for neural machine translation[END_REF]. We also applied data augmentation and selection techniques to training data of the Transformer models.

E.16 UPCite-CLILLF (no associated paper)

In this biomedical shared task, we have created data filters to better "choose" relevant training data for fine-tuning, among provided training data sources. In particular, we have used the textometric analysis tool ITRAMEUR to filter the segments and terms that characterize the test set and then extracted them from training data to fine-tune decoder_ffn_dim: 4096,decoder_layers: 12,encoder_attention_heads: 16,encoder_ffn_dim: 4096,encoder_layers: 12,num_hidden_layers: 12,max_length: 200,epoch: 3).

In doing so, we hope to meet several objectives : to build feasible fine-tuning strategy to train biomedical in-domain fr<->en models ; to specify filtering criteria of in-domain training data and to compare models' predictions, fine-tuning data and test set in order to better understand how neural machine translation systems work. We will also compare the pipeline of the shared task of this year to those of the past 2 years to evaluate the benefits of our training strategies of in-domain machine translation models.

E.17 UvA-LTL (Wu et al., 2023a) We present our WMT system, UvA-MT, in the WMT 2023 shared general translation task. This year, we developed a single Multilingual Machine Translation (MMT) system to participate in the two-directional translation track between English and Hebrew. The main architecture is based on the prior work of Beyond Shared Vocabulary (Wu and Monz, 2023). We scaled it up to a transformer-large level (422M parameters). Additionally, we employed back translation to generate synthetic data and labeled them with a new language tag. After convergence, we further fine-tuned the system without using synthetic data. Several domain shift techniques were also introduced, such as the domain-aware language model, to filter monolingual data.

E.18 YiShu [START_REF] Min | Yishu: Yishu at WMT2023 translation task[END_REF] Yishu's team participated in WMT23 Machine Translation Competition and adopted the most advanced neural machine translation method. They use Transformer model structure and use large-scale parallel corpus for training. In order to improve the translation quality, the team adopted cutting-edge data preprocessing technology, various attention mechanisms and improved decoding strategies. In addition, they also carried out in-depth parameter adjustment and model optimization. Yishu team incorporated evaluation indicators such as BLEU and TER into the training constraints of the model to achieve better translation performance. They strive for high accuracy and fluency in the competition, and strive to achieve excellent results in the field of translation. [START_REF] Zeng | Achieving state-of-the-art multilingual translation model with minimal data and parameters[END_REF] LanguageX's submission is a many-to-many encoder decoder transformer model.

E.19 LanguageX

F Automatic scores

This section contains automatic metric scores. While human judgement is the official ranking of systems and their performance, we share automatic scores to show expected system performance for various testsets.

We use COMET [START_REF] Rei | COMET: A neural framework for MT evaluation[END_REF] as the primary metric and chrF [START_REF] Popović | chrF: character n-gram F-score for automatic MT evaluation[END_REF] as the secondary metric, following recommendation by [START_REF] Kocmi | To ship or not to ship: An extensive evaluation of automatic metrics for machine translation[END_REF]. We also present BLEU [START_REF] Papineni | Bleu: a method for automatic evaluation of machine translation[END_REF] scores as it is still a widely used metric. The COMET scores are calculated with the default model Unbabel/wmt22-comet-da. The chrF and BLEU scores are calculated using SacreBLEU [START_REF] Post | A call for clarity in reporting BLEU scores[END_REF]. Scores are multiplied by 100. We ranked the systems according to their scores. Unconstrained systems are indicated with a grey background in the tables.

G Head to head comparisons

Following tables show differences in average human scores for each language pair. The numbers in each of the tables' cells indicate the difference in average human scores for the system in that column and the system in that row. Because there were so many systems and data conditions the significance of each pairwise comparison needs to be quantified. We applied Wilcoxon rank-sum test to measure the likelihood that such differences could occur simply by chance. In the following tables ⋆ indicates statistical significance at p < 0.

( a )

 a Top part of the screen with segment-level scoring. (b) Bottom part of the screen with document-level scoring.

Table 3 :

 3 Annotators' profiles and annotation tools for each language pair in human evaluation.

				Annotators' profile	Tool
	English↔Chinese/German/Japanese	Microsoft annotators: bilingual target-language native speakers, pro-fessional translators or linguists, experienced in MT evaluation	Appraise
	Czech→Ukrainian			Paid translators and target-language native speakers	Appraise
	English→Czech			Czech paid linguists, annotators, researchers, students with high profi-ciency in English	Appraise
	English↔Hebrew/Russian/Ukrainian Toloka AI paid crowd: bilingual target-language native speakers high-performing in other task types	Toloka.ai
	Language Pair	Sys. Assess. Assess/Sys
	Chinese→English	16	20,535	1283.4
	Czech→Ukrainian	14	23,191	1656.5
	German→English	14	13,573	969.5
	English→Chinese	16	24,551	1534.4
	English→Czech	16	25,527	1595.4
	English→German	13	14,267	1097.5
	English→Japanese	17	26,115	1536.2
	Japanese→English	18	27,858	1547.7

Table 4 :

 4 Amount

	Language Pair	Ann. HITs HITs/Ann.
	Chinese→English	13	128	9.8
	Czech→Ukrainian	9	146	16.2
	German→English	21	82	3.9
	English→Czech	36	162	4.5
	English→German	22	87	4.0
	English→Japanese	21	164	7.8
	English→Chinese	13	154	11.8
	Japanese→English	20	174	8.7

of segments evaluated in the WMT23 manual evaluation campaign; including human references as systems; after excluding quality control items and document-level scores.

Table 5 :

 5 Numbers of individual annotators taking part in the WMT23 human evaluation campaign and the average number of HITs collected per annotator.

Table 7 :

 7 Comparison of system clustering as done by DA+SQM and MQM technique. Top two tables are for English to German, while bottom two are for Chinese to German.

	Rank Ave. ↑ System (En-De)	System (En-De)	MQM ↓
	1-5	89.0 GPT4-5shot	refA	2.96
	1-5	88.8 ONLINE-B	GPT4-5shot	3.72
	1-4	88.3 ONLINE-W	ONLINE-W	3.95
	2-6	88.1 ONLINE-A	ONLINE-B	4.71
	4-6	88.0 ONLINE-Y	ONLINE-Y	5.64
	1-6	87.7 Human-refA	ONLINE-A	5.67
	7-8	86.7 ONLINE-M	ONLINE-G	6.57
	7-8	85.5 ONLINE-G	ONLINE-M	6.94
	9	84.0 Lan-BridgeMT	Lan-BridgeMT	8.67
	10	82.7 LanguageX	LanguageX	9.25
	11-12	76.8 NLLB_MBR_BLEU	NLLB_Greedy	9.54
	11-12	75.7 NLLB_Greedy	NLLB_MBR_BLEU 10.79
	13	73.6 AIRC	AIRC	14.23
	Rank Ave. ↑ System (Zh-En)	System (Zh-En)	MQM ↓
	1-2	82.9 Lan-BridgeMT	Lan-BridgeMT	2.10
	1-2	80.9 GPT4-5shot	GPT4-5shot	2.31
	3-8	80.3 Yishu	Yishu	3.23
	3-7	80.2 ONLINE-W	ONLINE-B	3.39
	5-10	80.0 ONLINE-G	HW-TSC	3.40
	3-7	79.8 ONLINE-B	ONLINE-A	3.79
	4-9	79.7 ONLINE-Y	ONLINE-Y	3.79
	3-8	79.1 HW-TSC	ONLINE-G	3.86
	6-10	77.8 ONLINE-A	ONLINE-W	4.06
	10-11	77.7 IOL_Research	LanguageX	4.23
	8-11	77.2 LanguageX	IOL_Research	4.59
	12-13	76.9 ONLINE-M	refA	4.83
	13-16	76.2 NLLB_MBR_BLEU	ONLINE-M	5.43
	12-15	76.1 Human-refA	ANVITA	6.08
	14-16	74.0 NLLB_Greedy	NLLB_MBR_BLEU 6.36
	13-16	72.6 ANVITA	NLLB_Greedy	6.57

Table 8 :

 8 Overview of the participating test suites.

Table 9 :

 9 This section describes statistics of the training corpora. Statistics for parallel training set provided for General/News Translation Task. Suffixes, k, M, and B, are short for thousands, millions, and billions, respectively. Dataset ID is the unique identifier created by MTData, example mtdata echo <dataset_id>.

	Dataset ID	Segs	Tokens	Chars
	eng-ces		eng	ces	eng	ces
	Facebook-wikimatrix-1-ces-eng	2.09M	33.56M	29.66M 206.82M 216.62M
	ParaCrawl-paracrawl-9-eng-ces	50.63M 692.12M 626.34M	4.33B	4.68B
	Statmt-commoncrawl_wmt13-1-ces-eng	161.84k	3.35M	2.93M	20.66M	20.75M
	Statmt-europarl-10-ces-eng	644.43k	15.63M	13.00M	94.31M	98.14M
	Statmt-news_commentary-16-ces-eng	253.27k	5.46M	4.96M	34.58M	37.97M
	Statmt-wikititles-3-ces-eng	410.94k	1.03M	965.62k	7.47M	7.57M
	Tilde-ecb-2017-ces-eng	3.10k	52.12k	45.21k	327.57k	339.24k
	Tilde-eesc-2017-ces-eng	1.33M	28.78M	25.63M 188.53M 205.14M
	Tilde-ema-2016-ces-eng	495.23k	7.64M	7.28M	50.31M	57.01M
	Tilde-rapid-2019-ces-eng	263.29k	5.79M	5.30M	37.36M	41.26M
	(Total)	56.29M 793.41M 716.10M	4.97B	5.36B
	eng-deu		eng	deu	eng	deu
	Facebook-wikimatrix-1-deu-eng	6.23M 100.50M	96.95M 623.66M 701.23M
	ParaCrawl-paracrawl-9-eng-deu	278.31M	4.27B	3.99B	26.37B	29.46B
	Statmt-commoncrawl_wmt13-1-deu-eng	2.40M	51.40M	47.05M 314.18M 340.51M
	Statmt-europarl-10-deu-eng	1.82M	45.51M	42.41M 272.94M 312.14M
	Statmt-news_commentary-16-deu-eng	388.48k	8.55M	8.77M	54.40M	65.94M
	Statmt-wikititles-3-deu-eng	1.47M	3.61M	3.08M	26.48M	25.50M
	Tilde-airbaltic-1-deu-eng	0.84k	17.60k	15.08k	104.34k	105.52k
	Tilde-czechtourism-1-deu-eng	6.76k	128.29k	114.44k	769.04k	829.41k
	Tilde-ecb-2017-deu-eng	4.15k	85.52k	74.81k	545.51k	582.63k
	Tilde-eesc-2017-deu-eng	2.86M	61.47M	58.28M 400.37M 469.94M
	Tilde-ema-2016-deu-eng	347.63k	5.09M	5.01M	33.48M	39.43M
	Tilde-rapid-2016-deu-eng	1.03M	20.65M	19.85M 134.26M 158.13M
	Tilde-rapid-2019-deu-eng	939.81k	19.90M	19.30M 129.03M 153.08M
	(Total)	295.81M	4.59B	4.29B	28.36B	31.73B
	eng-heb		eng	heb	eng	heb
	ELRC-wikipedia_health-1-eng-heb	3.16k	69.71k	54.76k	442.38k	583.87k
	Facebook-wikimatrix-1-eng-heb	2.04M	35.83M	28.96M 218.77M 300.61M
	Neulab-tedtalks_train-1-eng-heb	211.82k	4.45M	3.44M	22.36M	29.00M
	OPUS-bible_uedin-v1-eng-heb	62.20k	1.55M	830.23k	8.16M	7.46M
	OPUS-ccmatrix-v1-eng-heb	25.23M 313.87M 249.49M	1.81B	2.45B
	OPUS-elrc_2922-v1-eng-heb	3.16k	69.73k	54.77k	442.40k	583.54k
	OPUS-elrc_3065_wikipedia_health-v1-eng-heb	3.16k	69.71k	54.76k	442.31k	583.51k
	OPUS-elrc_wikipedia_health-v1-eng-heb	3.16k	69.71k	54.76k	442.31k	583.51k
	OPUS-globalvoices-v2018q4-eng-heb	1.03k	20.31k	15.03k	122.39k	158.63k
	OPUS-gnome-v1-eng-heb	0.15k	0.42k	0.40k	2.89k	3.96k
	OPUS-kde4-v2-eng-heb	79.32k	338.22k	347.35k	2.09M	3.13M
	OPUS-multiccaligned-v1-eng-heb	5.33M	60.55M	52.81M 380.74M 518.33M
	OPUS-opensubtitles-v2018-eng-heb	29.89M 195.98M 154.25M	1.03B	1.40B
	OPUS-php-v1-eng-heb	27.82k	83.46k	93.03k	498.72k	789.34k
	OPUS-qed-v2.0a-eng-heb	464.35k	6.37M	4.48M	34.70M	42.34M
	OPUS-tatoeba-v20220303-eng-heb	164.20k	1.02M	806.38k	5.41M	7.37M
	OPUS-tatoeba-v2-eng-heb	54.36k	357.09k	277.32k	1.87M	2.56M
	OPUS-ubuntu-v14.10-eng-heb	1.44k	6.13k	5.78k	38.78k	54.69k
	OPUS-wikimedia-v20210402-eng-heb	226.83k	8.51M	7.56M	57.58M	78.26M
	OPUS-wikipedia-v1.0-eng-heb	139.85k	2.69M	2.27M	16.45M	22.43M
	OPUS-xlent-v1.1-eng-heb	3.19M	9.61M	7.93M	60.53M	73.11M
	Statmt-ccaligned-1-eng-heb_IL	5.33M	60.55M	52.81M 380.76M 518.34M
	(Total)	72.46M 702.05M 566.59M	4.04B	5.45B

Table 10 :

 10 Statistics for parallel training set provided for General/News Translation Task. Suffixes, k, M, and B, are short for thousands, millions, and billions, respectively.

•

  Delete document completely if it contains any of following issues. Be on the save side, rather remove documents where you are uncertain o Remove documents written in different language (natural code-switching is fine) o Remove inappropriate content (such as sexually explicit, vulgar, or otherwise inappropriate) o Remove controversial content (propagandist, controversial political topics, etc.) o Remove content that is too noisy or doesn't resemble natural text (such as documents badly formatted, hard to understand, containing unusual language, lists of numbers/data, or other structured data generated automatically) • Keep document while checking o Fix sentence-breaking, each line must be one sentence (do not reformulate, simply remove or add end of lines on a proper place). o Remove or move fragments of sentences to previous or following sentence (for example emoticons, one or few words sentences) o Fix minor issues and keep it (do not spent too much time on fixing it).

▪ It is fine to keep some errors or problems ▪ Remove boilerplates (segments that break the document, for example ads, page numbers, signatures, artefacts, …) o If a given document has more than around 30 sentences, consider splitting it by adding an empty line on a meaningful place splitting it into paragraphs This task shouldn't take much longer than reading through documents.

Table 11 :

 11 Type-token ratio for individual source languages used in the general translation test sets.

		manuals	mastodon	news		user_review
		src	trg	src	trg	src	trg	src	trg
	From English
	en-cs	-	-0.30 0.42 0.27 0.39 0.22 0.35
	en-de	-	-0.30 0.32 0.27 0.29	-	-
	en-he	-	-0.30 0.30 0.27 0.29 0.22 0.24
	en-ja	-	-0.30 0.23 0.27 0.19 0.22 0.17
	en-ru	-	-0.30 0.41 0.27 0.38 0.22 0.33
	en-uk	-	-0.30 0.41 0.27 0.38 0.22 0.34
	en-zh	-	-0.30 0.29 0.27 0.26 0.22 0.21
	Other language directions
	cs-uk	-	-	-	-0.43 0.41	-	-
	de-en	0.32 0.23 0.49 0.42 0.34 0.26	-	-
	he-en	-	-	-	-0.34 0.09	-	-
	ja-en	-	-	-	-0.22 0.23 0.22 0.21
	ru-en	0.47 0.28	-	-0.40 0.24	-	-
	uk-en	-	-	-	-0.36 0.21	-	-
	zh-en	0.25 0.25	-	-0.23 0.19 0.22 0.17

Table 12 :

 12 Scores for the cs→uk translation task: chrF (nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.2.1), BLEU (nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.2.1), COMET (Unbabel/wmt22-comet-da).

		System	COMET		System	chrF	System	BLEU
	CUNI-GA	90.9	GPT4-5shot	61.0	GPT4-5shot	32.8
	GPT4-5shot	90.8	CUNI-GA	57.9	CUNI-Transformer	30.2
	ONLINE-W	89.4	GTCOM_Peter	57.6	GTCOM_Peter	29.8
	GTCOM_Peter	88.9	CUNI-Transformer	57.4	CUNI-GA	29.5
	ONLINE-B	88.8	MUNI-NLP	57.0	MUNI-NLP	28.3
	ONLINE-A	88.2	Lan-BridgeMT	55.7	Lan-BridgeMT	27.5
	CUNI-Transformer	88.0	ONLINE-W	55.0	ONLINE-W	26.8
	ONLINE-G	87.7	ONLINE-B	54.7	ONLINE-B	25.7
	MUNI-NLP	87.0	ONLINE-A	54.4	ONLINE-A	25.4
	ONLINE-Y	86.5	ONLINE-G	53.7	NLLB_MBR_BLEU	25.1
	NLLB_Greedy	86.3	ONLINE-Y	53.4	NLLB_Greedy	24.9
	NLLB_MBR_BLEU	86.3	NLLB_Greedy	52.5	ONLINE-G	24.8
	Lan-BridgeMT	86.0	NLLB_MBR_BLEU	52.3	ONLINE-Y	24.2
		System	COMET		System	chrF	System	BLEU
	ONLINE-W	91.8	ONLINE-W	76.3	ONLINE-W	59.4
	CUNI-GA	90.8	ONLINE-B	70.4	ONLINE-B	50.1
	ONLINE-B	89.9	ZengHuiMT	67.5	ONLINE-A	43.4
	GPT4-5shot	89.4	ONLINE-A	66.3	CUNI-GA	43.3
	ONLINE-A	88.4		CUNI-GA	65.9	ZengHuiMT	43.1
	CUNI-DocTransformer	88.3	GTCOM_Peter	65.4	CUNI-DocTransformer	42.5
	GTCOM_Peter	87.7	CUNI-DocTransformer	65.1	GTCOM_Peter	42.3
	ONLINE-M	87.4	ONLINE-Y	64.6	CUNI-Transformer	41.4
	Lan-BridgeMT	87.3	CUNI-Transformer	63.9	ONLINE-Y	40.8
	CUNI-Transformer	87.2	Lan-BridgeMT	63.8	Lan-BridgeMT	40.7
	NLLB_Greedy	87.1	ONLINE-G	63.7	ONLINE-G	39.6
	ONLINE-Y	87.0	ONLINE-M	63.2	ONLINE-M	39.6
	NLLB_MBR_BLEU	86.9	GPT4-5shot	62.3	GPT4-5shot	37.8
	ONLINE-G	85.9	NLLB_Greedy	60.0	NLLB_Greedy	35.9
	ZengHuiMT	85.4	NLLB_MBR_BLEU	59.1	NLLB_MBR_BLEU	35.1
	Table 13: Scores for the en→cs translation task: chrF (nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.2.1), BLEU
	(nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.2.1), COMET (Unbabel/wmt22-comet-da).
	System	COMET	System	chrF		System	BLEU
	GPT4-5shot	86.3		ONLINE-W	72.1		ONLINE-W	51.8
	ONLINE-W	86.0		ONLINE-A	70.0		GPT4-5shot	47.9
	ONLINE-B	85.6		GPT4-5shot	69.8		ONLINE-A	47.9
	ONLINE-A	85.5		ONLINE-B	69.1		ONLINE-B	46.3
	ONLINE-Y	84.9		ONLINE-G	69.1		ONLINE-G	46.0
	ONLINE-M	84.8		ONLINE-Y	68.4		ONLINE-Y	43.9
	ONLINE-G	84.6		ZengHuiMT	67.6		GTCOM_Peter	42.2
	GTCOM_Peter	82.7		Lan-BridgeMT	66.7		Lan-BridgeMT	42.1
	NLLB_MBR_BLEU	81.4		GTCOM_Peter	66.6		ONLINE-M	41.3
	ZengHuiMT	81.1		ONLINE-M	66.5		ZengHuiMT	40.8
	Lan-BridgeMT	80.9		NLLB_MBR_BLEU	57.6		NLLB_Greedy	33.1
	NLLB_Greedy	79.9		NLLB_Greedy	57.3		AIRC	32.4
	AIRC	78.7		AIRC	57.2		NLLB_MBR_BLEU	32.4

Table 14 :

 14 Scores for the de→en translation task: chrF (nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.2.1), BLEU (nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.2.1), COMET (Unbabel/wmt22-comet-da).

	System	COMET		System	chrF		System	BLEU
	ONLINE-W	85.5		ONLINE-W	71.8		ONLINE-W	47.8
	GPT4-5shot	85.0		ONLINE-A	69.7		ONLINE-A	43.7
	ONLINE-B	84.8		ZengHuiMT	69.4		GPT4-5shot	43.6
	ONLINE-Y	84.1		GPT4-5shot	69.1		ONLINE-Y	43.6
	ONLINE-A	83.7		ONLINE-B	69.1		ONLINE-G	43.2
	ONLINE-G	82.5		ONLINE-Y	69.1		ONLINE-B	42.7
	ONLINE-M	81.7		ONLINE-G	69.0		ONLINE-M	40.5
	Lan-BridgeMT	80.4		ONLINE-M	66.9		ZengHuiMT	40.5
	ZengHuiMT	79.4		Lan-BridgeMT	66.1		Lan-BridgeMT	39.4
	NLLB_MBR_BLEU	78.0		NLLB_Greedy	56.2		NLLB_Greedy	31.1
	NLLB_Greedy	77.9		NLLB_MBR_BLEU	55.4		NLLB_MBR_BLEU	29.6
	AIRC	72.9		AIRC	52.2		AIRC	26.5
	Table 15: Scores for the en→de translation task: chrF (nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.2.1), BLEU
	(nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.2.1), COMET (Unbabel/wmt22-comet-da).
		System	COMET		System	chrF	System	BLEU
	ONLINE-B	89.9	ONLINE-B	87.5	ONLINE-B	76.5
	ONLINE-A	87.0	ZengHuiMT	76.3	GTCOM_Peter	59.2
	GPT4-5shot	86.9	GTCOM_Peter	76.2	ZengHuiMT	56.6
	GTCOM_Peter	86.7	ONLINE-A	73.3	ONLINE-A	53.9
	ONLINE-G	85.6	GPT4-5shot	71.4	GPT4-5shot	51.2
	ZengHuiMT	85.6		UvA-LTL	70.9	UvA-LTL	51.0
	ONLINE-Y	84.9	ONLINE-Y	70.5	ONLINE-Y	49.8
		UvA-LTL	84.7	ONLINE-G	69.8	ONLINE-G	49.3
	NLLB_MBR_BLEU	82.9	NLLB_Greedy	64.4	NLLB_Greedy	42.5
	NLLB_Greedy	82.8	Lan-BridgeMT	63.5	Lan-BridgeMT	41.4
	Samsung_Research_Philippines	82.6	NLLB_MBR_BLEU	63.0	NLLB_MBR_BLEU	40.7
	Lan-BridgeMT	82.4	Samsung_Research_Philippines	55.5	Samsung_Research_Philippines	34.0

Table 16 :

 16 Scores for the he→en (refA) translation task: chrF (nrefs:1|case:mixed|eff:yes|nc:6 |nw:0|space:no|version:2.2.1), BLEU (nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.2.1), COMET (Unbabel/wmt22-comet-da).

	System	COMET	System	chrF	System	BLEU
	GPT4-5shot	86.4	GPT4-5shot	69.5	GPT4-5shot	50.4
	ONLINE-B	85.6	ONLINE-B	66.5	ONLINE-B	45.0
	ONLINE-A	85.3	ONLINE-A	65.6	GTCOM_Peter	44.4
	GTCOM_Peter	84.5	GTCOM_Peter	65.3	ONLINE-A	44.4
	ONLINE-G	84.0	ZengHuiMT	65.1	UvA-LTL	41.7
	UvA-LTL	83.3	UvA-LTL	63.3	ZengHuiMT	41.7
	ZengHuiMT	83.3	ONLINE-G	62.8	ONLINE-G	40.9
	ONLINE-Y	82.9	ONLINE-Y	62.0	ONLINE-Y	38.5
	NLLB_MBR_BLEU	81.8	NLLB_Greedy	59.6	NLLB_Greedy	37.1
	NLLB_Greedy	81.7	Lan-BridgeMT	59.0	Lan-BridgeMT	36.2
	Lan-BridgeMT	81.3	NLLB_MBR_BLEU	58.6	NLLB_MBR_BLEU	36.2
	Samsung_Research_Philippines	81.3	Samsung_Research_Philippines	51.3	Samsung_Research_Philippines	29.8
	Table 17: Scores for the he→en (refB) translation task: chrF (nrefs:1|case:mixed|eff:yes|nc:6 |nw:0|space:no|version:2.3.1),
	BLEU (nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.3.1), COMET (Unbabel/wmt22-comet-da).
	System	COMET	System	chrF	System	BLEU
	ONLINE-B	86.4	ONLINE-B	66.4	ONLINE-B	47.8
	ONLINE-A	85.7	ZengHuiMT	62.1	ONLINE-A	38.9
	GPT4-5shot	84.9	ONLINE-A	61.7	GTCOM_Peter	37.2
	GTCOM_Peter	84.7	GTCOM_Peter	61.1	ONLINE-Y	37.2
	ONLINE-Y	84.7	ONLINE-Y	60.4	ZengHuiMT	36.5
	UvA-LTL	84.2	UvA-LTL	59.0	UvA-LTL	35.0
	Samsung_Research_Philippines	83.7	ONLINE-G	58.1	Samsung_Research_Philippines	33.3
	Lan-BridgeMT	83.0	Samsung_Research_Philippines	57.3	ONLINE-G	33.2
	NLLB_Greedy	82.9	Lan-BridgeMT	54.9	NLLB_MBR_BLEU	30.8
	ZengHuiMT	82.7	NLLB_Greedy	54.8	Lan-BridgeMT	30.5
	NLLB_MBR_BLEU	82.5	NLLB_MBR_BLEU	54.3	NLLB_Greedy	30.3
	ONLINE-G	82.2	GPT4-5shot	54.0	GPT4-5shot	27.0

Table 18 :

 18 Scores for the en→he translation task: chrF (nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.2.1), BLEU (nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.2.1), COMET (Unbabel/wmt22-comet-da).

	System	COMET	System	chrF	System	BLEU
	SKIM	84.0	ONLINE-W	51.4	ONLINE-W	25.9
	GPT4-5shot	83.4	GPT4-5shot	51.2	SKIM	24.8
	ONLINE-W	82.3	SKIM	51.1	GPT4-5shot	24.1
	NAIST-NICT	81.9	ONLINE-A	49.6	ONLINE-B	23.9
	ONLINE-Y	81.6	NAIST-NICT	49.5	NAIST-NICT	23.0
	ONLINE-B	81.5	ONLINE-Y	49.5	ONLINE-A	23.0
	ONLINE-A	81.0	ZengHuiMT	49.5	ZengHuiMT	22.6
	GTCOM_Peter	80.2	ONLINE-B	49.3	GTCOM_Peter	22.3
	ANVITA	79.5	GTCOM_Peter	48.7	ONLINE-Y	22.3
	Lan-BridgeMT	79.3	Lan-BridgeMT	47.3	ANVITA	20.9
	ZengHuiMT	79.2	ANVITA	46.7	Lan-BridgeMT	20.2
	ONLINE-G	77.8	ONLINE-G	45.5	ONLINE-G	18.3
	ONLINE-M	77.5	KYB	43.9	KYB	17.6
	KYB	76.6	ONLINE-M	43.9	ONLINE-M	17.2
	NLLB_MBR_BLEU	75.2	AIRC	40.5	AIRC	14.9
	AIRC	74.5	NLLB_MBR_BLEU	39.2	NLLB_MBR_BLEU	14.7
	NLLB_Greedy	74.3	NLLB_Greedy	39.0	NLLB_Greedy	14.2
	Table 19: Scores for the ja→en translation task: chrF (nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.2.1), BLEU
	(nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.2.1), COMET (Unbabel/wmt22-comet-da).
	System	COMET	System	chrF	System	BLEU
	ONLINE-B	88.2	ONLINE-B	35.2	ONLINE-B	25.3
	ONLINE-W	87.5	ONLINE-Y	34.1	ONLINE-W	24.5
	ONLINE-Y	87.3	ONLINE-W	33.5	ONLINE-Y	24.5
	GPT4-5shot	87.0	SKIM	33.5	SKIM	24.3
	SKIM	86.6	ZengHuiMT	32.9	NAIST-NICT	22.6
	NAIST-NICT	86.2	NAIST-NICT	32.0	ZengHuiMT	22.6
	ZengHuiMT	85.3	ONLINE-A	31.4	ONLINE-A	21.4
	ONLINE-A	85.2	GPT4-5shot	31.0	GPT4-5shot	21.3
	Lan-BridgeMT	84.5	Lan-BridgeMT	30.4	Lan-BridgeMT	20.5
	ONLINE-M	13.3	ONLINE-M	29.6	ONLINE-M	19.8
	ANVITA	82.7	ANVITA	29.3	ANVITA	19.4
	KYB	80.8	KYB	27.7	KYB	17.8
	AIRC	80.7	AIRC	27.6	AIRC	17.6
	ONLINE-G	80.4	ONLINE-G	27.3	ONLINE-G	17.2
	NLLB_Greedy	79.3	NLLB_Greedy	20.9	NLLB_Greedy	11.3
	NLLB_MBR_BLEU	77.7	NLLB_MBR_BLEU	18.7	NLLB_MBR_BLEU	9.0
	Table 20: Scores for the en→ja translation task: chrF (nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.2.1), BLEU
	(nrefs:1|case:mixed|eff:no|tok:ja-mecab-0.996-IPA|smooth:exp|version:2.2.1), COMET (Unbabel/wmt22-comet-da).
	System	COMET	System	chrF	System	BLEU
	GPT4-5shot	83.5	GPT4-5shot	60.4	ONLINE-B	34.5
	ONLINE-Y	82.5	ONLINE-G	59.6	GPT4-5shot	34.4
	ONLINE-B	82.3	ONLINE-A	59.4	ONLINE-G	34.0
	ONLINE-W	82.2	ONLINE-B	59.4	ONLINE-A	33.8
	ONLINE-G	82.0	ZengHuiMT	58.9	ONLINE-Y	33.2
	ONLINE-A	81.9	ONLINE-Y	58.6	ONLINE-W	33.1
	PROMT	80.9	PROMT	58.4	PROMT	32.8
	ONLINE-M	80.7	ONLINE-W	58.3	Lan-BridgeMT	31.8
	NLLB_MBR_BLEU	80.5	Lan-BridgeMT	57.4	ZengHuiMT	31.3
	NLLB_Greedy	80.1	ONLINE-M	56.7	NLLB_MBR_BLEU	31.0
	Lan-BridgeMT	79.9	NLLB_MBR_BLEU	55.8	ONLINE-M	30.7
	ZengHuiMT	79.5	NLLB_Greedy	55.5	NLLB_Greedy	30.3
	Table 21: Scores for the ru→en translation task: chrF (nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.2.1), BLEU
	(nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.2.1), COMET (Unbabel/wmt22-comet-da).
	System	COMET	System	chrF	System	BLEU
	ONLINE-G	86.6	ONLINE-B	61.9	ONLINE-B	40.4
	ONLINE-W	86.6	ONLINE-A	59.0	ONLINE-A	34.8
	ONLINE-B	86.2	ONLINE-G	58.9	ONLINE-G	32.9
	GPT4-5shot	86.1	ZengHuiMT	58.8	ONLINE-Y	32.0
	ONLINE-Y	85.5	ONLINE-W	56.6	ZengHuiMT	31.6
	ONLINE-A	85.3	ONLINE-Y	56.4	ONLINE-W	31.4
	ONLINE-M	83.2	GPT4-5shot	56.2	ONLINE-M	30.9
	Lan-BridgeMT	83.1	Lan-BridgeMT	55.7	Lan-BridgeMT	30.7
	NLLB_Greedy	82.9	PROMT	55.4	GPT4-5shot	30.6
	NLLB_MBR_BLEU	82.7	ONLINE-M	55.1	PROMT	30.5
	PROMT	82.3	NLLB_Greedy	53.3	NLLB_MBR_BLEU	28.4
	ZengHuiMT	81.3	NLLB_MBR_BLEU	53.1	NLLB_Greedy	28.2

Table 22 :

 22 Scores

	System	COMET	System	chrF	System	BLEU
	ONLINE-W	87.5	GTCOM_Peter	69.3	ONLINE-W	47.4
	GPT4-5shot	87.1	ONLINE-W	69.2	GTCOM_Peter	46.4
	ONLINE-B	86.8	ONLINE-B	69.0	ONLINE-B	46.0
	GTCOM_Peter	86.3	ZengHuiMT	68.5	ONLINE-A	45.9
	ONLINE-A	86.3	ONLINE-A	68.3	ONLINE-Y	45.7
	ONLINE-G	86.2	ONLINE-Y	68.2	ONLINE-G	44.9
	ONLINE-Y	85.8	GPT4-5shot	68.1	GPT4-5shot	43.9
	Lan-BridgeMT	84.8	ONLINE-G	68.0	ZengHuiMT	43.5
	ZengHuiMT	84.4	Lan-BridgeMT	66.2	Lan-BridgeMT	42.3
	NLLB_MBR_BLEU	84.3	NLLB_Greedy	62.4	NLLB_MBR_BLEU	38.1
	NLLB_Greedy	84.2	NLLB_MBR_BLEU	62.4	NLLB_Greedy	37.8
	Table 23: Scores for the uk→en translation task: chrF (nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.2.1), BLEU
	(nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.2.1), COMET (Unbabel/wmt22-comet-da).
	System	COMET	System	chrF	System	BLEU
	ONLINE-W	86.7	ONLINE-B	61.7	ONLINE-B	39.8
	ONLINE-B	85.6	ONLINE-W	59.2	ONLINE-W	34.9
	GPT4-5shot	85.3	ZengHuiMT	56.4	ONLINE-A	30.3
	ONLINE-G	85.3	ONLINE-G	56.1	ONLINE-Y	29.5
	ONLINE-A	83.2	ONLINE-A	55.8	ONLINE-G	28.6
	ONLINE-Y	82.9	ONLINE-Y	55.4	ZengHuiMT	27.8
	GTCOM_Peter	82.1	GTCOM_Peter	54.4	GTCOM_Peter	27.5
	NLLB_Greedy	82.1	GPT4-5shot	53.0	GPT4-5shot	25.2
	NLLB_MBR_BLEU	81.7	Lan-BridgeMT	51.9	NLLB_MBR_BLEU	24.9
	Lan-BridgeMT	80.4	NLLB_Greedy	50.8	Lan-BridgeMT	24.6
	ZengHuiMT	79.0	NLLB_MBR_BLEU	50.5	NLLB_Greedy	24.5

for the en→ru translation task: chrF (nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.2.1), BLEU (nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.2.1), COMET (Unbabel/wmt22-comet-da).

Table 24 :

 24 Scores

	System	COMET	System	chrF	System	BLEU
	HW-TSC	82.8	HW-TSC	57.5	HW-TSC	33.6
	ONLINE-B	82.7	ONLINE-B	57.5	ONLINE-B	33.5
	Yishu	82.7	Yishu	57.4	Yishu	33.4
	GPT4-5shot	81.6	ZengHuiMT	54.6	ONLINE-A	28.3
	Lan-BridgeMT	81.2	ONLINE-G	53.9	Lan-BridgeMT	27.3
	ONLINE-G	80.9	ONLINE-A	53.4	IOL_Research	27.2
	ONLINE-Y	80.6	GPT4-5shot	53.1	ZengHuiMT	27.0
	ONLINE-A	80.3	Lan-BridgeMT	53.1	GPT4-5shot	26.8
	ZengHuiMT	79.6	ONLINE-W	52.5	ONLINE-G	26.6
	ONLINE-W	79.3	IOL_Research	52.4	ONLINE-W	26.4
	IOL_Research	79.2	ONLINE-Y	52.3	ONLINE-Y	25.0
	ONLINE-M	77.7	ONLINE-M	49.7	ONLINE-M	23.5
	NLLB_MBR_BLEU	76.8	ANVITA	47.1	ANVITA	21.8
	ANVITA	76.6	NLLB_Greedy	46.1	NLLB_Greedy	20.5
	NLLB_Greedy	76.4	NLLB_MBR_BLEU	45.8	NLLB_MBR_BLEU	19.8

for the en→uk translation task: chrF (nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.2.1), BLEU (nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.2.1), COMET (Unbabel/wmt22-comet-da).

Table 25 :

 25 Scores

	System	COMET	System	chrF	System	BLEU
	ONLINE-B	88.1	HW-TSC	53.8	HW-TSC	58.6
	Yishu	88.1	Yishu	53.0	ONLINE-A	58.5
	HW-TSC	87.3	ONLINE-B	52.9	Yishu	57.6
	GPT4-5shot	87.1	ONLINE-A	52.8	ONLINE-B	57.5
	ONLINE-W	86.8	IOL_Research	51.9	IOL_Research	56.9
	Lan-BridgeMT	86.6	ONLINE-M	50.6	ONLINE-M	54.9
	ONLINE-Y	86.5	ONLINE-Y	49.8	ONLINE-Y	54.2
	ONLINE-A	86.2	ONLINE-G	49.4	ONLINE-G	54.1
	IOL_Research	85.3	ONLINE-W	47.3	ZengHuiMT	52.9
	ZengHuiMT	84.3	ZengHuiMT	47.0	ONLINE-W	52.1
	ONLINE-M	84.2	Lan-BridgeMT	46.8	Lan-BridgeMT	50.2
	ONLINE-G	83.8	GPT4-5shot	46.5	GPT4-5shot	49.6
	NLLB_Greedy	75.7	ANVITA	36.9	ANVITA	38.9
	ANVITA	75.6	NLLB_Greedy	26.3	NLLB_Greedy	27.4
	NLLB_MBR_BLEU	71.5	NLLB_MBR_BLEU	21.1	NLLB_MBR_BLEU	19.1

for the zh→en translation task: chrF (nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.2.1), BLEU (nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.2.1), COMET (Unbabel/wmt22-comet-da).

Table 26 :

 26 Scores

for the en→zh translation task: chrF (nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.2.1), BLEU (nrefs:1|case:mixed|eff:no|tok:zh|smooth:exp|version:2.2.1), COMET (Unbabel/wmt22-comet-da).

Table 27 :

 27 05, † indicates statistical significance at p < 0.01, and ‡ indicates statistical significance at p < 0.001, according to Wilcoxon rank-sum test.Each table contains final rows showing the average score achieved by that system and the rank range according to Wilcoxon rank-sum test (p < 0.05). Gray lines separate clusters based on non-overlapping rank ranges. Head to head comparison for Czech→Ukrainian systems

	Czech→Ukrainian
		ONLINE-B	GPT4-5shot	Human-refA	ONLINE-W	CUNI-GA	CUNI-Transformer	GTCOM_DLUT	ONLINE-A	ONLINE-G	ONLINE-Y	MUNI-NLP	Lan-BridgeMT	NLLB_MBR_BLEU	NLLB_Greedy
	ONLINE-B	-	0.1	0.4	0.9⋆	1.3 ‡	1.8 †	2.4⋆	3.1⋆	4.1 ‡	5.0 ‡	5.0 ‡	6.2 ‡	6.7 ‡	7.0 ‡
	GPT4-5shot	-0.1	-	0.4	0.8 †	1.2 ‡	1.8 ‡	2.3 †	3.1 ‡	4.1 ‡	4.9 ‡	4.9 ‡	6.2 ‡	6.7 ‡	6.9 ‡
	Human-refA	-0.4	-0.4	-	0.5 ‡	0.9 ‡	1.4 ‡	1.9 ‡	2.7 ‡		4.6 ‡	4.6 ‡	5.8 ‡	6.3 ‡	6.6 ‡
	ONLINE-W	-0.9	-0.8	-0.5	-	0.4	0.9	1.5	2.2	3.2 ‡	4.1 ‡	4.1 ‡	5.3 ‡	5.8 ‡	6.1 ‡
	CUNI-GA	-1.3	-1.2	-0.9	-0.4	-	0.6	1.1	1.8	2.9 †	3.7 ‡	3.7 †	5.0 ‡	5.5 ‡	5.7 ‡
	CUNI-Transformer	-1.8	-1.8	-1.4	-0.9	-0.6	-	0.5	1.3	2.3 †	3.1 ‡	3.2 ‡	4.4 ‡	4.9 ‡	5.1 ‡
	GTCOM_DLUT	-2.4	-2.3	-1.9	-1.5	-1.1	-0.5	-	0.8	1.8 ‡	2.6 ‡	2.6 ‡	3.9 ‡	4.4 ‡	4.6 ‡
	ONLINE-A	-3.1	-3.1	-2.7	-2.2	-1.8	-1.3	-0.8	-	1.0 ‡	1.9 ‡	1.9 ‡	3.1 ‡	3.6 ‡	3.9 ‡
	ONLINE-G	-4.1	-4.1	-3.7	-3.2	-2.9	-2.3	-1.8	-1.0	-	0.8	0.9	2.1⋆	2.6 †	2.8 ‡
	ONLINE-Y	-5.0	-4.9	-4.6	-4.1	-3.7	-3.1	-2.6	-1.9	-0.8	-	0.0	1.3	1.8	2.0 †
	MUNI-NLP	-5.0	-4.9	-4.6	-4.1	-3.7	-3.2	-2.6	-1.9	-0.9	-0.0	-	1.2	1.7	2.0 ‡
	Lan-BridgeMT	-6.2	-6.2	-5.8	-5.3	-5.0	-4.4	-3.9	-3.1	-2.1	-1.3	-1.2	-	0.5	0.7⋆
	NLLB_MBR_BLEU	-6.7	-6.7	-6.3	-5.8	-5.5	-4.9	-4.4	-3.6	-2.6	-1.8	-1.7	-0.5	-	0.2⋆
	NLLB_Greedy	-7.0	-6.9	-6.6	-6.1	-5.7	-5.1	-4.6	-3.9	-2.8	-2.0	-2.0	-0.7	-0.2	-
	score	83.7	83.6	83.2	82.8	82.4	81.8	81.3	80.6	79.5	78.7	78.7	77.4	76.9	76.7
	rank	1-3	1-3	1-3	4-8	4-8	4-8	4-8	4-8	9-11	9-13	9-13	10-13	10-13	14

Table 28 :

 28 Head to head comparison for German→English systems

	English→Czech
		Human-refA	ONLINE-W	GPT4-5shot	CUNI-GA	ONLINE-A	CUNI-DocTransformer	ONLINE-B	NLLB_MBR_BLEU	GTCOM_DLUT	CUNI-Transformer	NLLB_Greedy	ONLINE-M	ONLINE-G	ONLINE-Y	Lan-BridgeMT	LanguageX
	Human-refA	-	1.3⋆	3.6 ‡	5.0 ‡	5.1 ‡	6.0 ‡	6.6 ‡	6.8 ‡	7.0 ‡	8.0 ‡	8.6 ‡	9.7 ‡	10.2 ‡	10.4 ‡	10.4 ‡	11.3 ‡
	ONLINE-W	-1.3	-	2.3 ‡	3.7 ‡	3.8 ‡	4.7 ‡	5.3 ‡	5.5 ‡	5.7 ‡	6.7 ‡	7.3 ‡	8.4 ‡	8.9 ‡	9.1 ‡	9.1 ‡	10.0 ‡
	GPT4-5shot	-3.6	-2.3	-	1.4	1.5 †	2.5⋆	3.0	3.2 ‡	3.4 †	4.4 ‡	5.1 ‡	6.1 ‡	6.6 ‡	6.8 ‡	6.8 ‡	7.7 ‡
	CUNI-GA	-5.0	-3.7	-1.4	-	0.0 ‡	1.0 ‡	1.5⋆	1.8 ‡	2.0 ‡	3.0 ‡	3.6 ‡	4.7 ‡	5.1 ‡	5.3 ‡	5.4 ‡	6.3 ‡
	ONLINE-A	-5.1	-3.8	-1.5	-0.0	-	1.0	1.5	1.7 ‡	1.9	2.9⋆	3.6 ‡	4.7 †	5.1 ‡	5.3 ‡	5.4⋆	6.3 ‡
	CUNI-DocTransformer	-6.0	-4.7	-2.5	-1.0	-1.0	-	0.5	0.7 ‡	0.9	1.9 †	2.6 ‡	3.7 ‡	4.1 ‡	4.3 ‡	4.4 †	5.3 ‡
	ONLINE-B	-6.6	-5.3	-3.0	-1.5	-1.5	-0.5	-	0.2 ‡	0.4⋆	1.4 ‡	2.1 ‡	3.2 ‡	3.6 ‡	3.8 ‡	3.9 ‡	4.8 ‡
	NLLB_MBR_BLEU	-6.8	-5.5	-3.2	-1.8	-1.7	-0.7	-0.2	-	0.2	1.2	1.9	3.0	3.4	3.6⋆	3.7	4.5 ‡
	GTCOM_DLUT	-7.0	-5.7	-3.4	-2.0	-1.9	-0.9	-0.4	-0.2	-	1.0	1.7 ‡	2.8 †	3.2 ‡	3.4 ‡	3.5	4.3 ‡
	CUNI-Transformer	-8.0	-6.7	-4.4	-3.0	-2.9	-1.9	-1.4	-1.2	-1.0	-	0.7⋆	1.7	2.2 †	2.4 ‡	2.4	3.3 ‡
	NLLB_Greedy	-8.6	-7.3	-5.1	-3.6	-3.6	-2.6	-2.1	-1.9	-1.7	-0.7	-	1.1	1.5	1.7⋆	1.8	2.7 ‡
	ONLINE-M	-9.7	-8.4	-6.1	-4.7	-4.7	-3.7	-3.2	-3.0	-2.8	-1.7	-1.1	-	0.4	0.6 †	0.7	1.6 ‡
	ONLINE-G	-10.2	-8.9	-6.6	-5.1	-5.1	-4.1	-3.6	-3.4	-3.2	-2.2	-1.5	-0.4	-	0.2	0.3	1.1 †
	ONLINE-Y	-10.4	-9.1	-6.8	-5.3	-5.3	-4.3	-3.8	-3.6	-3.4	-2.4	-1.7	-0.6	-0.2	-	0.1	1.0⋆
	Lan-BridgeMT	-10.4	-9.1	-6.8	-5.4	-5.4	-4.4	-3.9	-3.7	-3.5	-2.4	-1.8	-0.7	-0.3	-0.1	-	0.9 ‡
	LanguageX	-11.3	-10.0	-7.7	-6.3	-6.3	-5.3	-4.8	-4.5	-4.3	-3.3	-2.7	-1.6	-1.1	-1.0	-0.9	-
	score	85.4	84.1	81.8	80.4	80.3	79.4	78.8	78.6	78.4	77.4	76.8	75.7	75.2	75.0	75.0	74.1
	rank	1	2	3-5	3-4	5-8	5-8	4-7	8-14	6-11	8-12	10-14	9-14	10-15	13-15	8-15	16

Table 29 :

 29 Head to head comparison for English→Czech systems

	English→German
		GPT4-5shot	ONLINE-B	ONLINE-W	ONLINE-A	ONLINE-Y	Human-refA	ONLINE-M	ONLINE-G	Lan-BridgeMT	LanguageX	NLLB_MBR_BLEU	NLLB_Greedy	AIRC
	GPT4-5shot	-	0.1	0.7	0.8	1.0 †	1.3	2.3 ‡	3.4 ‡	5.0 ‡	6.3 ‡	12.1 ‡	13.2 ‡	15.4 ‡
	ONLINE-B	-0.1	-	0.6	0.7	0.8⋆	1.2	2.2 ‡	3.3 ‡	4.8 ‡	6.2 ‡	12.0 ‡	13.1 ‡	15.2 ‡
	ONLINE-W	-0.7	-0.6	-	0.2⋆	0.3 ‡	0.6	1.6 ‡	2.7 ‡	4.3 ‡	5.6 ‡	11.5 ‡	12.5 ‡	14.7 ‡
	ONLINE-A	-0.8	-0.7	-0.2	-	0.1	0.5	1.4 ‡	2.6 ‡	4.1 ‡	5.5 ‡	11.3 ‡	12.4 ‡	14.5 ‡
	ONLINE-Y	-1.0	-0.8	-0.3	-0.1	-	0.3	1.3 †	2.5⋆	4.0 ‡	5.3 ‡	11.2 ‡	12.3 ‡	14.4 ‡
	Human-refA	-1.3	-1.2	-0.6	-0.5	-0.3	-	1.0 ‡	2.1 ‡	3.7 ‡	5.0 ‡	10.8 ‡	11.9 ‡	14.1 ‡
	ONLINE-M	-2.3	-2.2	-1.6	-1.4	-1.3	-1.0	-	1.1	2.7 †	4.0 ‡	9.9 ‡	10.9 ‡	13.1 ‡
	ONLINE-G	-3.4	-3.3	-2.7	-2.6	-2.5	-2.1	-1.1	-	1.5 ‡	2.9 ‡	8.7 ‡	9.8 ‡	11.9 ‡
	Lan-BridgeMT	-5.0	-4.8	-4.3	-4.1	-4.0	-3.7	-2.7	-1.5	-	1.4⋆	7.2 ‡	8.3 ‡	10.4 ‡
	LanguageX	-6.3	-6.2	-5.6	-5.5	-5.3	-5.0	-4.0	-2.9	-1.4	-	5.8 ‡	6.9 ‡	9.1 ‡
	NLLB_MBR_BLEU	-12.1	-12.0	-11.5	-11.3	-11.2	-10.8	-9.9	-8.7	-7.2	-5.8	-	1.1	3.2 ‡
	NLLB_Greedy	-13.2	-13.1	-12.5	-12.4	-12.3	-11.9	-10.9	-9.8	-8.3	-6.9	-1.1	-	2.2 ‡
	AIRC	-15.4	-15.2	-14.7	-14.5	-14.4	-14.1	-13.1	-11.9	-10.4	-9.1	-3.2	-2.2	-
	score	89.0	88.8	88.3	88.1	88.0	87.7	86.7	85.5	84.0	82.7	76.8	75.7	73.6
	rank	1-5	1-5	1-4	2-6	4-6	1-6	7-8	7-8	9	10	11-12	11-12	13

Table 30 :

 30 Head to head comparison for English→German systems

	English→Japanese
		Human-refA	GPT4-5shot	ONLINE-B	ONLINE-Y	SKIM	ONLINE-W	LanguageX	ONLINE-A	NAIST-NICT	Lan-BridgeMT	ANVITA	ONLINE-M	KYB	AIRC	ONLINE-G	NLLB_Greedy	NLLB_MBR_BLEU
	Human-refA	-	1.2 ‡	1.9	2.1 †	2.2⋆	2.3 ‡	4.1 ‡	4.5 ‡	4.6 ‡	5.5 ‡	7.6 ‡	8.1 ‡	9.9 ‡	11.1 ‡	11.1 ‡	16.2 ‡	19.4 ‡
	GPT4-5shot	-1.2	-	0.7	0.9	1.0	1.1	2.9⋆	3.3 †	3.4 †	4.3 ‡	6.4 ‡	6.9 ‡	8.8 ‡	9.9 ‡	10.0 ‡	15.0 ‡	18.3 ‡
	ONLINE-B	-1.9	-0.7	-	0.2	0.3	0.4⋆	2.3 ‡	2.7 ‡	2.7 ‡	3.6 ‡	5.7 ‡	6.2 ‡	8.1 ‡	9.3 ‡	9.3 ‡	14.3 ‡	17.6 ‡
	ONLINE-Y	-2.1	-0.9	-0.2	-	0.1	0.2	2.0 ‡	2.4 ‡	2.5 ‡	3.4 ‡	5.5 ‡	6.0 ‡	7.8 ‡	9.0 ‡	9.0 ‡	14.1 ‡	17.3 ‡
	SKIM	-2.2	-1.0	-0.3	-0.1	-	0.1⋆	1.9 ‡	2.3 ‡	2.4 ‡	3.3 ‡	5.4 ‡	5.9 ‡	7.7 ‡	8.9 ‡	8.9 ‡	13.9 ‡	17.2 ‡
	ONLINE-W	-2.3	-1.1	-0.4	-0.2	-0.1	-	1.8 †	2.2 †	2.3 ‡	3.2 ‡	5.3 ‡	5.8 ‡	7.6 ‡	8.8 ‡	8.8 ‡	13.8 ‡	17.1 ‡
	LanguageX	-4.1	-2.9	-2.3	-2.0	-1.9	-1.8	-	0.4	0.5	1.4	3.5 ‡	4.0 ‡	5.8 ‡	7.0 ‡	7.0 ‡	12.0 ‡	15.3 ‡
	ONLINE-A	-4.5	-3.3	-2.7	-2.4	-2.3	-2.2	-0.4	-	0.0	1.0	3.1 ‡	3.5 ‡	5.4 ‡	6.6 ‡	6.6 ‡	11.6 ‡	14.9 ‡
	NAIST-NICT	-4.6	-3.4	-2.7	-2.5	-2.4	-2.3	-0.5	-0.0	-	0.9	3.0 ‡	3.5 ‡	5.4 ‡	6.5 ‡	6.6 ‡	11.6 ‡	14.9 ‡
	Lan-BridgeMT	-5.5	-4.3	-3.6	-3.4	-3.3	-3.2	-1.4	-1.0	-0.9	-	2.1 †	2.6 ‡	4.5 ‡	5.6 ‡	5.6 ‡	10.7 ‡	14.0 ‡
	ANVITA	-7.6	-6.4	-5.7	-5.5	-5.4	-5.3	-3.5	-3.1	-3.0	-2.1	-	0.5	2.3 ‡	3.5 ‡	3.5 ‡	8.5 ‡	11.8 ‡
	ONLINE-M	-8.1	-6.9	-6.2	-6.0	-5.9	-5.8	-4.0	-3.5	-3.5	-2.6	-0.5	-	1.9 †	3.0 ‡	3.1 †	8.1 ‡	11.4 ‡
	KYB	-9.9	-8.8	-8.1	-7.8	-7.7	-7.6	-5.8	-5.4	-5.4	-4.5	-2.3	-1.9	-	1.2	1.2	6.2 ‡	9.5 ‡
	AIRC	-11.1	-9.9	-9.3	-9.0	-8.9	-8.8	-7.0	-6.6	-6.5	-5.6	-3.5	-3.0	-1.2	-	0.0	5.0 ‡	8.3 ‡
	ONLINE-G	-11.1	-10.0	-9.3	-9.0	-8.9	-8.8	-7.0	-6.6	-6.6	-5.6	-3.5	-3.1	-1.2	-0.0	-	5.0 ‡	8.3 ‡
	NLLB_Greedy	-16.2	-15.0	-14.3	-14.1	-13.9	-13.8	-12.0	-11.6	-11.6	-10.7	-8.5	-8.1	-6.2	-5.0	-5.0	-	3.3 ‡
	NLLB_MBR_BLEU	-19.4	-18.3	-17.6	-17.3	-17.2	-17.1	-15.3	-14.9	-14.9	-14.0	-11.8	-11.4	-9.5	-8.3	-8.3	-3.3	-
	score	80.7	79.5	78.8	78.6	78.5	78.4	76.6	76.2	76.1	75.2	73.1	72.6	70.8	69.6	69.6	64.5	61.3
	rank	1-2	2-6	1-5	2-6	2-5	4-6	7-10	7-10	7-10	7-10	11-12	11-12	13-15	13-15	13-15	16	17

Table 31 :

 31 Head to head comparison for English→Japanese systems

	English→Chinese
		Yishu	Human-refA	GPT4-5shot	Lan-BridgeMT	ONLINE-B	HW-TSC	ONLINE-W	ONLINE-Y	IOL_Research	ONLINE-A	LanguageX	ONLINE-M	ONLINE-G	ANVITA	NLLB_Greedy	NLLB_MBR_BLEU
	Yishu	-	0.0	0.1	0.2⋆	0.3	0.7	0.8⋆	2.0⋆	2.3 ‡	2.5 ‡	3.6 ‡	4.0 ‡	5.0 ‡	17.7 ‡	17.9 ‡	25.0 ‡
	Human-refA	-0.0	-	0.0	0.1 †	0.3	0.7	0.8⋆	1.9 †	2.3 ‡	2.5 ‡	3.6 ‡	3.9 ‡	5.0 ‡	17.7 ‡	17.8 ‡	25.0 ‡
	GPT4-5shot	-0.1	-0.0	-	0.1	0.3	0.6	0.7	1.9⋆	2.3 ‡	2.4 ‡	3.5 ‡	3.9 ‡	5.0 ‡	17.6 ‡	17.8 ‡	24.9 ‡
	Lan-BridgeMT	-0.2	-0.1	-0.1	-	0.2	0.5	0.6	1.8	2.2 †	2.3 ‡	3.4 ‡	3.8 ‡	4.9 ‡	17.5 ‡	17.7 ‡	24.8 ‡
	ONLINE-B	-0.3	-0.3	-0.3	-0.2	-	0.3	0.4 †	1.6 †	2.0 ‡	2.2 ‡	3.2 ‡	3.6 ‡	4.7 ‡	17.3 ‡	17.5 ‡	24.7 ‡
	HW-TSC	-0.7	-0.7	-0.6	-0.5	-0.3	-	0.1	1.3	1.7 ‡	1.8 ‡	2.9 ‡	3.3 ‡	4.4 ‡	17.0 ‡	17.2 ‡	24.3 ‡
	ONLINE-W	-0.8	-0.8	-0.7	-0.6	-0.4	-0.1	-	1.2	1.6 †	1.7 ‡	2.8 ‡	3.2 ‡	4.3 ‡	16.9 ‡	17.1 ‡	24.2 ‡
	ONLINE-Y	-2.0	-1.9	-1.9	-1.8	-1.6	-1.3	-1.2	-	0.4⋆	0.5 †	1.6 ‡	2.0 ‡	3.1 ‡	15.7 ‡	15.9 ‡	23.0 ‡
	IOL_Research	-2.3	-2.3	-2.3	-2.2	-2.0	-1.7	-1.6	-0.4	-	0.2	1.2 †	1.6 †	2.7 ‡	15.3 ‡	15.5 ‡	22.7 ‡
	ONLINE-A	-2.5	-2.5	-2.4	-2.3	-2.2	-1.8	-1.7	-0.5	-0.2	-	1.1⋆	1.5⋆	2.5 ‡	15.2 ‡	15.4 ‡	22.5 ‡
	LanguageX	-3.6	-3.6	-3.5	-3.4	-3.2	-2.9	-2.8	-1.6	-1.2	-1.1	-	0.4	1.5	14.1 ‡	14.3 ‡	21.4 ‡
	ONLINE-M	-4.0	-3.9	-3.9	-3.8	-3.6	-3.3	-3.2	-2.0	-1.6	-1.5	-0.4	-	1.1	13.7 ‡	13.9 ‡	21.0 ‡
	ONLINE-G	-5.0	-5.0	-5.0	-4.9	-4.7	-4.4	-4.3	-3.1	-2.7	-2.5	-1.5	-1.1	-	12.6 ‡	12.8 ‡	20.0 ‡
	ANVITA	-17.7	-17.7	-17.6	-17.5	-17.3	-17.0	-16.9	-15.7	-15.3	-15.2	-14.1	-13.7	-12.6	-	0.2 ‡	7.3 ‡
	NLLB_Greedy	-17.9	-17.8	-17.8	-17.7	-17.5	-17.2	-17.1	-15.9	-15.5	-15.4	-14.3	-13.9	-12.8	-0.2	-	7.1 ‡
	NLLB_MBR_BLEU	-25.0	-25.0	-24.9	-24.8	-24.7	-24.3	-24.2	-23.0	-22.7	-22.5	-21.4	-21.0	-20.0	-7.3	-7.1	-
	score	82.2	82.1	82.1	82.0	81.8	81.5	81.4	80.2	79.8	79.7	78.6	78.2	77.1	64.5	64.3	57.2
	rank	1-5	1-5	1-7	3-8	1-6	1-8	4-8	5-8	9-10	9-10	11-13	11-13	11-13	14	15	16

Table 32 :

 32 Head to head comparison for English→Chinese systems

https://github.com/wmt-conference/ wmt23-news-systems

http://wmt.ufal.cz

https://www.iahlt.org

https://www.israelhayom.co.il

https://mastodon.social/api/v1/timelines/ public

https://github.com/yandex/ geo-reviews-dataset-2023

https://www.monotaro.com/

https://www.cyberagent.co.jp

https://www.centific.com

https://ufal.github.io/automin-2023/

https://oshiete.goo.ne.jp/

http://translator.cuni.cz

The TTR is the ratio of unique tokens to total tokens, and it is higher the diverse the vocabulary of a text is. It is dependent on the morphological complexity of a language, but can also vary due to other factors.

Texts are tokenised using the language-specific Spacy models[START_REF] Honnibal | spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing[END_REF] where available. For Hebrew, we took the multilingual Spacy model, since a language-specific one was not available.

https://github.com/mashashma/WMT2022-data

https://statmt.org/wmt23/translation-task. html

http://www2.statmt.org/wmt23/mtdata

https://github.com/AppraiseDev/OCELoT

https://ocelot-wmt23.mteval.org

https://toloka.ai

For full details, see the HIT and batch generation code: https://github.com/wmt-conference/ wmt23-news-systems

Acknowledgments

This task would not have been possible without the sponsorship of monolingual data, test sets translation and evaluation from our partners. Namely Microsoft, Charles University, Toloka AI, Google, NTT Resonant, Dubformer, and Centific. Additionally, we would like to thank Rebecca Knowles, Sergio Bruccoleri, Mariia Anisimova and many others who provided help and recommendations. Barry Haddow's participation was funded by UK Research and Innovation (UKRI) under the UK government's Horizon Europe funding guarantee [grant number 10039436 -UTTER]. Rachel Bawden's participation was funded by her chair position in the PRAIRIE institute funded by the French national agency ANR as part of the "Investissements d'avenir" programme under the reference ANR-19-P3IA-0001 and by her Emergence project, DadaNMT, funded by Sorbonne Université. Maja Popović's participation was funded by the ADAPT SFI Centre for Digital Media Technology, funded by Science Foundation Ireland through the SFI Research Centres Programme and co-funded under the European Regional Development Fund (ERDF) through Grant 13/RC/2106. Martin Popel's participation was funded by GA ČR EXPRO grant LUSyD (GX20-16819X). Eleftherios Avramidis's participation was funded by the German Research Foundation (DFG) through the project TextQ (grant num. MO 1038/31-1, 436813723), and by the German Federal Ministry of Education and Research (BMBF) through the project SocialWear (grant num. 01IW2000). This work has been using data and tools provided by the LINDAT/CLARIAH-CZ Research Infrastructure (https://lindat.cz), supported by the Ministry of Education, Youth and Sports of the Czech Republic (Project No. LM2023062).

https://github.com/AppraiseDev/ Appraise