Rachel Bawden
email: rachel.bawden@inria.fr

Franc ¸ois Yvon
email: francois.yvon@cnrs.fr

Colin Raffel

Aaron Gokaslan

Adi Simhi

Aitor Soroa

Alham Fikri

Amit Alfassy

Anna Rogers

Ariel Kreisberg Nitzav

Canwen Xu

Chenghao Mou

Chris Emezue

Christopher Klamm

Colin Leong

Daniel Van Strien

David Ifeoluwa Adelani

Dragomir Radev

Eduardo González Ponferrada

Efrat Lev- Kovizh

Eyal Bar Natan Ethan Kim

Francesco De Toni

Gérard Dupont

Germán Kruszewski

Gi- Ada Pistilli

Hady Elsahar

Hamza Benyamina

Hieu Tran

Ian Yu

Idris Abdulmumin

Isaac Johnson

Itziar Gonzalez-Dios

Javier De La Rosa

Jenny Chim

Jesse Dodge

Jian Zhu

Jonathan Chang

Jörg Frohberg

Joseph Tobing

Joydeep Bhattacharjee

Khalid Al- Mubarak

Kimbo Chen

Kyle Lo

Leandro Werra

Leon Weber

Long Phan

Loubna Ben

Lu- Dovic Tanguy

Manan Dey

Manuel Romero Muñoz

Maraim Masoud

María Grandury

Mario Šaško

Max Huang

Maximin Coavoux

Mike Mayank Singh

Tian-Jian Jiang

Minh Chien Vu

Moham- Mad A Jauhar

Mustafa Ghaleb

Nishant Subramani

Nora Kassner

Nurulaqilla Khamis

Olivier Nguyen

Omar Espejel

Ona De Gibert

Paulo Villegas

Pe- Ter Henderson

Pierre Colombo

Priscilla Amuok

Quentin Lhoest

Rheza Harliman

Rishi Bommasani

Roberto Luis López

Rui Ribeiro

Salomey Osei

Sampo Pyysalo

Sebastian Nagel

Shamik Bose

Shamsuddeen Hassan Muhammad

Shanya Sharma

Shayne Longpre

Somaieh Nikpoor

Stanislav Silber- Berg

Suhas Pai

Sydney Zink

Tiago Timponi

Timo Schick

Tristan Thrush

Valentin Danchev

Vassilina Nikoulina

Veronika Laippala

Violette Lepercq

Vrinda Prabhu

Zaid Alyafeai

Zeerak Ta- Lat

Arun Raja

Benjamin Heinzerling

Chenglei Si

Emre Davut

Elizabeth Tas ¸ar

Sabrina J Salesky

Wilson Y Mielke

Abheesht Lee

Andrea Sharma

Antoine Santilli

Arnaud Chaffin

Debajy- Oti Stiegler

Eliza Datta

Gunjan Szczechla

Han Chhablani

Harshit Wang

Hendrik Pandey

Jason Alan Strobelt

Jos Fries

Leo Rozen

Lintang Gao

M Sutawika

Maged S Sai- Ful Bari

Matteo Al-Shaibani

Ni- Hal Manica

Ryan Nayak

Samuel Teehan

Sheng Albanie

Srulik Shen

Stephen H Ben-David

Taewoon Bach

Tali Kim

Thibault Bers

Trishala Fevry

Ur- Mish Neeraj

Vikas Thakker

Xiangru Raunak

Zheng- Xin Tang

Zhiqing Yong

Shaked Sun

Yallow Brody

Hadar Uri

Adam Tojarieh

Hyung Won Roberts

Jaesung Chung

Jason Tae

Ofir Phang

Conglong Press

Deepak Li

Hatim Narayanan

Jared Bourfoune

Jeff Casper

Max Rasley

Mayank Ryabinin

Minjia Mishra

Mohammad Zhang

Myriam Shoeybi

Nicolas Peyrounette

Nouamane Patry

Omar Tazi

Patrick Sanseviero

Pierre Von Platen

Pierre Cornette

Franc

Rémi Lavallée

Samyam Lacroix

San- Chit Rajbhandari

Shaden Gandhi

Stéphane Smith

Suraj Requena

Tim Patil

Ahmed Dettmers

Amanpreet Baruwa

Anastasia Singh

Anne-Laure Cheveleva

Arjun Ligozat

Aurélie Subramonian

Charles Lover- Ing Névéol

Dan Garrette

Deepak Tunuguntla

Ehud Reiter

Ekaterina Taktasheva

Ekaterina Voloshina

Eli Bog- Danov, Genta

Indra Winata

Hailey Schoelkopf

Jan- Christoph Kalo

Jekaterina Novikova

Jessica Zosa Forde

Jordan Clive

Jungo Kasai

Ken Kawamura

Liam Hazan

Marine Carpuat

Miruna Clinciu

Na- Joung Kim

Newton Cheng

Oleg Serikov

Omer Antverg

Oskar Van Der Wal

Rui Zhang

Ruochen Zhang

Sebastian Gehrmann

Shachar Mirkin

Shani Pais

Tatiana Shavrina

Thomas Scialom

Tian Yun

Tomasz Limisiewicz

Verena Rieser

Vitaly Protasov

Vladislav Mikhailov

Yada Pruksachatkun

Yonatan Belinkov

Zachary Bamberger

Zdeněk Kasner

Al- Ice Rueda

Amanda Pestana

Amir Feizpour

Ammar Khan

Amy Faranak

Ana Santos

Anthony Hevia

Antigona Unldreaj

Arash Aghagol

Arezoo Abdol- Lahi

Aycha Tammour

Azadeh Hajihosseini

Bahareh Behroozi

Benjamin Ajibade

Car- los Bharat Saxena

Muñoz Ferrandis

Danish Contractor

David Lan- Sky

Davis David

Douwe Kiela

Duong A Nguyen

Edward Tan

Emi Baylor

Ezinwanne Ozoani

Fa- Tima Mirza

Frankline Ononiwu

Habib Rezanejad

Hessie Jones

Indrani Bhattacharya

Irene Solaiman

Irina Sedenko

Isar Nejadgholi

Jesse Passmore

Josh Seltzer

Julio Bonis Sanz

Livia Dutra

Mairon Sama- Gaio

Maraim Elbadri

Margot Mieskes

Marissa Ger- Chick

Martha Akinlolu

Michael Mckenna

Mike Qiu

Muhammed Ghauri

Mykola Burynok

Nafis Abrar

Nazneen Rajani

Nour Elkott

Nour Fahmy

Olanrewaju Samuel

Ran An

Rasmus Kromann

Ryan Hao

Samira Alizadeh

Sarmad Shubber

Silas Wang

Sourav Roy

Sylvain Viguier

Thanh Le

Tobi Oyebade

Trieu Le

Yoyo Yang

Zach Nguyen

Ramesh Kashyap

Alfredo Palasciano

Al- Ison Callahan

Anima Shukla

Antonio Miranda- Escalada

Ayush Singh

Benjamin Beilharz

Bo Wang

Caio Brito

Chenxi Zhou

Chirag Jain

Chuxin Xu

Clémentine Fourrier

Daniel León Periñán

Daniel Molano

Dian Yu

Enrique Manjavacas

Fabio Barth

Florian Fuhrimann

Gabriel Altay

Giyased- Din Bayrak

Gully Burns

Helena U Vrabec

Imane Bello

Ishani Dash

Jihyun Kang

John Giorgi

Jonas Golde

Jose David Posada

Karthik Rangasai

Lokesh Bulchandani

Lu Liu

Luisa Shinzato

Madeleine Hahn De Bykhovetz

Maiko Takeuchi

Marc Pàmies

Maria A Castillo

Marianna Nezhurina

Mario Sänger

Matthias Samwald

Michael Cullan

Michael Weinberg

Michiel De Wolf

Mina Mihalj- Cic

Minna Liu

Moritz Freidank

Myungsun Kang

Natasha Seelam

Nathan Dahlberg

Nicholas Michio Broad

Nikolaus Muellner

Pascale Fung

Patrick Haller

Ramya Chandrasekhar

Renata Eisenberg

Robert Martin

Rodrigo Canalli

Rosaline Su

Ruisi Su

Samuel Cahyawijaya

Samuele Garda

S Shlok

Shubhanshu Deshmukh

Sid Mishra

Si- Mon Kiblawi

Sinee Ott

Srishti Sang-Aroonsiri

Stefan Kumar

Sushil Schweter

Tanmay Bharati

Théo Laud

Tomoya Gigant

Wojciech Kainuma

Yanis Kusa

Labrak

Shailesh Yash

Yash Bajaj

Yifan Venkatraman

Yingxin Xu

Yu Xu

Zhe Xu

Zhongli Tan

Zifan Xie

Mathilde Ye

Younes Bras

Thomas Wolf Belkada

Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM

Keywords: mixed+smooth, exp+{13a,spm}+version, 2, 2

 covering 46 languages. We focus on BLOOM's multilingual ability by evaluating its machine translation performance across several datasets (WMT, Flores-101 and DiaBLa) and language pairs (high-and low-resourced). Our results show that 0-shot performance suffers from overgeneration and generating in the wrong language, but this is greatly improved in the fewshot setting, with very good results for a number of language pairs. We study several aspects including prompt design, model sizes, crosslingual transfer and the use of discursive context.

Introduction

Large language models (LLMs) trained at scale with simple objectives have been found to achieve results that match dedicated systems on numerous NLP tasks [START_REF] Radford | Language models are unsupervised multitask learners[END_REF], as long as tasks are formulated as text generation though "prompting" [START_REF] Liu | Pre-Train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing[END_REF]. LLMs' multi-task performance can even be improved with "instruction" fine-tuning [START_REF] Sanh | Multitask prompted training enables zero-shot task generalization[END_REF][START_REF] Muennighoff | Crosslingual generalization through multitask finetuning[END_REF], few-shot priming, and better strategies to select or learn prompts [START_REF] Petroni | Language models as knowledge bases?[END_REF][START_REF] Shin | AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts[END_REF][START_REF] Schick | It's not just size that matters: Small language models are also fewshot learners[END_REF][START_REF] Lester | The power of scale for parameter-efficient prompt tuning[END_REF][START_REF] Wei | Chain of thought prompting elicits reasoning in large language models[END_REF]. In multilingual settings, their performance on machine translation (MT) tasks, as measured by automatic scores, is often close to state of the art, even when mostly trained on monolingual data [START_REF] Brown | Language models are few-shot learners[END_REF]. Moreover, prompting-based MT offers the prospect of better control of outputs, e.g. in terms of quality, style and dialect [START_REF] Garcia | Using natural language prompts for machine translation[END_REF]. However, these abilities remain poorly understood, as LLM analyses primarily focus on their multitask rather than multilingual ability (see however [START_REF] Vilar | Prompting PaLM for Translation: Assessing Strategies and Performance[END_REF][START_REF] Zhang | Prompting large language model for machine translation: A case study[END_REF][START_REF] Moslem | Adaptive machine translation with large language models[END_REF]), which we discuss in Section 2).

In this work, we focus on the MT performance of BLOOM [START_REF] Bigscience | BLOOM: A 176b-parameter open-access multilingual language model[END_REF], a (family of) open-access multilingual LLM(s), designed and trained by the collaborative BigScience project. 1Our main aims are to (i) evaluate BLOOM's zeroand multi-shot behaviour, (ii) study the effect of prompt design, (iii) evaluate a diverse set of language pairs and (iv) assess its ability to use linguistic context. Our main conclusions, which extend those in [START_REF] Bigscience | BLOOM: A 176b-parameter open-access multilingual language model[END_REF], are (i) 0-shot ability is blighted by overgeneration and generating in the wrong language, (ii) using few-shot improves both issues, with results much closer to state of the art across datasets and language pairs, (iii) there are clear transfer effects, with high scores for languages not officially seen in training, and successful transfer across language pairs via fewshot examples and (iv) although linguistic context does not lead to higher scores, there is evidence that BLOOM's translations are influenced by it. We release our code and translation outputs.2

Related work

Since the early attempts to use language models (LMs) as multi-task learners [START_REF] Mccann | The natural language decathlon: Multitask learning as question answering[END_REF], MT has been a task of choice to gauge LMs' multilingual ability. Results for the zero-and few-shot ability of LMs were discussed for both GPT-2 and GPT-3 [START_REF] Radford | Language models are unsupervised multitask learners[END_REF][START_REF] Brown | Language models are few-shot learners[END_REF], which is especially intriguing as they were trained primarily on monolingual (English) data. These results have since been confirmed for other monolingual LMs such as T5 [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF] and multilingual LMs such as XGLM (Lin et al., 2022), PALM (Chowdhery et al., 2022), and ALEX-ATM [START_REF] Saleh Soltan | Alexatm 20b: Few-shot learning using a large-scale multilingual seq2seq model[END_REF]. However, the focus has mainly been on global multi-task performance; often only a small part of the discussion is devoted to MT. Moreover, results are often only reported for a few well-resourced language pairs (e.g. English-French and English-German), and the scores reported (mostly BLEU), are hard to compare due to a non-systematic use of standardised evaluation protocols and metrics. 3 There are however some in-depth analyses of MT performance of LLMs, each focusing on a specific LM's performance in a true multilingual setting with respect to prompt design and number of few-shots. For instance, [START_REF] Vilar | Prompting PaLM for Translation: Assessing Strategies and Performance[END_REF] reevaluate the MT performance of the multilingual PALM [START_REF] Curran Associates | Palm: Scaling language modeling with pathways[END_REF], focusing notably on the selection of few-shot examples. Consistent with our findings, they determine that prompt choice becomes unimportant in few-shot settings and that using few-shot examples increases performance with diminishing returns for k > 5 examples, using BLEURT and BLEU scores, as well as the results of a human evaluation. They find that the quality of few-shot examples has a large impact on performance. However, even with good prompts, PALM lags a couple of points behind state-of-the-art MT systems, especially when translating from English, notable due to adequacy problems. [START_REF] Zhang | Prompting large language model for machine translation: A case study[END_REF] focus on the evaluation of GLM-130B, a bilingual (Chinese and English) LLM [START_REF] Zeng | GLM-130B: an open bilingual pre-trained model[END_REF]. Their main conclusions are also consistent with ours: (a) zero-shot performance varies greatly across different prompts, (b) increasing the number of prompts from 0 to 20 yields consistent improvements in performance, again with variance across instructions, and (c) finding the best few-shot example selection policy is difficult. It seems that having good and long examples, for instance, may help, even though none of the criteria explored in this study seem to provide any systematic improvement. A last point worth mentioning is that prompting with monolingual data hurts performance, but that using pseudo-parallel data obtained with backtranslation [START_REF] Bojar | Improving translation model by monolingual data[END_REF]) is an effective workaround. [START_REF] Moslem | Adaptive machine translation with large language models[END_REF] evaluate OpenAI's GPT-3 [START_REF] Brown | Language models are few-shot learners[END_REF] 4 with sampling-based decoding and a prompt resembling our own xglm-source+target prompt. They report strong zero-shot behaviour using multiple metrics, 3 See the discussion at http: //blog.benjaminmarie.com/2/ comparing-uncomparable.html of these differences, and an attempt to reconstruct consistent scores. 4 Version: text-davinci-003 model.

plus clear improvements with an increased number of shots for the well-resourced languages, less so for the only low-resource language in their lot (Kinyarwanda). The main novelty of this study is to use prompting as a vehicle to perform local adaptation and to ensure terminological consistency. For this, they use fuzzy matches from a translation memory as well as MT outputs to build their prompts, yielding results that both outperform their zero-shot system, but also their initial MT engine. Additionally inserting terms and their translation in the instruction yields supplementary improvements. Finally note the preliminary evaluation of CHAT-GPT in [START_REF] Jiao | Is ChatGPT a Good Translator? A Preliminary Study[END_REF], which reports interesting insights regarding the multilingual abilities of this model, as well as proposing innovative techniques to generate (artificial) prompts and to use pivoting in prompting. Similar to ours, this study considers multiple test domains such as news (WMT) and Wikipedia (Flores). A more in-depth analysis of the same model can be found in [START_REF] Hendy | How good are GPT models at machine translation? a comprehensive evaluation[END_REF], which confirms CHATGPT's strong translation abilities, at least for "well-resourced"5 language pairs. Document-level scores are also reported, as well as human evaluations and qualitative analyses.

Multilingual MT is also the subject of dedicated (monotask) architectures and training regimes. Originally introduced in [START_REF] Dong | Multi-task learning for multiple language translation[END_REF][START_REF] Firat | Multi-way, multilingual neural machine translation with a shared attention mechanism[END_REF][START_REF] Luong | Multi-task sequence to sequence learning[END_REF] with limited language coverage, the latest versions of these approaches are able to handle hundreds of languages, including very low-resource language pairs [START_REF] Fan | Beyond English-Centric multilingual machine translation[END_REF][START_REF] Bapna | Building machine translation systems for the next thousand languages[END_REF][START_REF] Costa-Jussà | No language left behind: Scaling human-centered machine translation[END_REF]. Although we found that BLOOM is able to match this performance, given sufficient training data, we also see that it still lags behind for many languages pairs that are under-represented in its training data. Table 1: Seven MT prompts for the WMT'14 dataset [START_REF] Bojar | Findings of the 2014 workshop on statistical machine translation[END_REF]. All prompts specify the target language (L2). Each prompt exists in a 'target-only' version (-target), where only the target language is specified, and two prompts also exist in a second -source+target version, where the source language (in red and in brackets) is explicit in the instruction.

BLOOM Language Model

(of which 30% English), from various [START_REF] Goyal | The Flores-101 evaluation benchmark for low-resource and multilingual machine translation[END_REF] and DiaBLa [START_REF] Bawden | DiaBLa: a corpus of bilingual spontaneous written dialogues for machine translation[END_REF]. We use the WMT 2014 news test sets for English↔French and English↔Hindi, which we take as representative high-and lower-resource language pairs with respect to BLOOM's training data.8 These test sets are somewhat outdated [START_REF] Garcia | The unreasonable effectiveness of few-shot learning for machine translation[END_REF], but have been used repeatedly in past LLM evaluations and are included as standard benchmarks for comparison. Flores-101 is a multi-parallel dataset in 101 languages, translated from original English sentences.In fact, evaluations into English are bound to yield overly good results (e.g. [START_REF] Toral | Attaining the unattainable? reassessing claims of human parity in neural machine translation[END_REF]) and between other languages may mostly reflect their similarity with the original English. We use it to test and compare BLOOM's multilinguality, including for low-resource languages.9 DiaBLa is a bilingual test set of spontaneous written dialogues between English and French speakers, mediated by MT. We use this as a test of MT in an informal domain and the impact of (cross-lingual) linguistic context in MT.

Experimental setup

We evaluate and compare BLOOM (and its variants) using the Language Model Evaluation Harness [START_REF] Gao | A framework for few-shot language model evaluation[END_REF] in 0-shot and few-shot settings. For few-shot, k examples are prefixed to the prompt and separated with ### as shown in Example 1 (1-shot example is underlined).

(1) Input: French: je m'ennuie = English: I'm bored. ### English: Is that your dog that's just wandered in over there? = French: Reference: Est-ce que c'est votre chien qui vient de rentrer par là ?

Results are reported on the datasets' test splits. Few-shot examples are randomly taken from the data splits according to availability (train for WMT, dev for Flores-101 and test for DiaBLa). We evaluate using BLEU [START_REF] Papineni | Bleu: a method for automatic evaluation of machine translation[END_REF] as implemented in SacreBLEU [START_REF] Post | A call for clarity in reporting BLEU scores[END_REF], using as tokenisation 13a for WMT and DiaBLa and spm for Flores-101 as recommended [START_REF] Costa-Jussà | No language left behind: Scaling human-centered machine translation[END_REF]. 10 BLEU has many shortcomings but is good enough to provide quantitative comparisons for most systems used in this study. We additionally use COMET [START_REF] Rei | COMET: A neural framework for MT evaluation[END_REF] for finer grained comparisons when the scores are closer.

Comparative models

In our cross-dataset comparison (Section 5.1), we compare BLOOM to other LLMs: (i) two task-fine-tuned models: T0 11 [START_REF] Sanh | Multitask prompted training enables zero-shot task generalization[END_REF], trained on English texts, and MT0-XXL 12 [START_REF] Muennighoff | Crosslingual generalization through multitask finetuning[END_REF], the multilingual version, and (ii) OPT 13 [START_REF] Zhang | OPT: Open pre-trained transformer language models[END_REF], an English generative LM. We evaluate all models on the same prompt xglm-source+target. To evaluate multiple language pairs with Flores-101, we compare (as a topline) to the supervised 615Mparameter MT model M2M-100 [START_REF] Fan | Beyond English-Centric multilingual machine translation[END_REF], using the scores computed by [START_REF] Goyal | The Flores-101 evaluation benchmark for low-resource and multilingual machine translation[END_REF].

Prompts

We use several prompts, designed to illustrate different sources of variation: (i) the inclusion (or not) of the source language name, (ii) the relative order of source and target language names, (iii) the position of the source sentence (beginning or end of the prompt) and (iv) the prompt's verbosity. These prompts, available in PromptSource [START_REF] Bach | Prompt-Source: An integrated development environment and repository for natural language prompts[END_REF], are shown in Table 1. The first three are inspired by previous work: 14 [START_REF] Brown | Language models are few-shot learners[END_REF] for gpt3, 15 (Lin et al., 2022) for xglm and [START_REF] Wei | Chain of thought prompting elicits reasoning in large language models[END_REF] for translate as, which also resembles [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF]'s prompt (Translate English to German: "[source text]": [target sentence] ), also used in [START_REF] Wei | Chain of thought prompting elicits reasoning in large language models[END_REF][START_REF] Garcia | Using natural language prompts for machine translation[END_REF].

Considering the entries in Table 1, we can see that "prompting" in fact refers to two distinct aspects of the input: (i) the formulation of the task in natural language and (ii) the presentation of related examples (for few-shot setups) interleaved with language tags (perhaps more clearly referred to as priming by [START_REF] Pham | Priming neural machine translation[END_REF]). As illustrated by the xglm prompt for example, the instruction part can reduced to one single word. As our results below suggest, the instruction mostly matters in 0-shot setups, but can almost be dispensed with in few-shot scenarios. The authors of [START_REF] Brown | Language models are few-shot learners[END_REF] and [START_REF] Hendy | How good are GPT models at machine translation? a comprehensive evaluation[END_REF]) also use a verbose, instruction-like prompt in their zero-shot setup, and a much more compact one for few shots experiments. Also note that InstructGPT's prompt combines both an instruction and language tags (Ouyang et al., 2022, p. 49).

Evaluation results

Our evaluation of BLOOM starts with a comparison across the three datasets and detection of major MT errors with a focus on WMT (Section 5.1) and then we present more in-depth analyses of particular aspects: (i) using WMT, a comparative study of BLOOM model sizes (Section 5.2) and prompts (Section 5.3), (ii) using Flores-101 an evaluation of more language pairs and cross-lingual few-shot transfer (Section 5.4), and (ii) using DiaBLa, a study of the use of linguistic context (Section 5.5). We first prompt BLOOM and the comparative models using the same prompt across datasets, restricting the directions tested to en↔fr and to en↔hi. We choose to systematically use the xglm-source+target prompt (Table 1), which corresponds to the following template:

Comparison across datasets

(2) L1: [source sentence] = L2:

where L1 and L2 refer to the source and target languages respectively (e.g. English and French for en→fr) and [source sentence] is replaced by a given source sentence.

BLEU scores are in Table 2a for both 0-shot and 1-shot (results with COMET are given in Appendix A). There are issues for 0-shot MT for all directions, particularly when translating into non-English languages, (BLEU scores are systematically lower than into English). Even into English, the scores remain low with respect to state of the art (e.g. 2.7 BLEU for Flores-101 fr→en BLOOM vs. 60.1 for MT0-XXL).16 MT0-XXL's good results illustrate the benefits of prompt fine-tuning. Results are considerably improved in the 1-shot setting (except for T0 and MT0-XXL, both fine-tuned) and are comparable to other LLM MT results reported in the literature; they are markedly better for BLOOM than for OPT, which was mostly trained on English data. These general observations apply for WMT and Flores-101, while issues remain for DiaBLa.

What are the main issues? We identify two main issues (particularly for 0-shot): (i) overgeneration and (ii) generating in the wrong language (an obvious problem for MT). Examples 3 and 4 illustrate some instances of these problems. The first example shows a very common problem, with repetition of the prompt in multiple languages until the maximum generation length is reached. Separating MT quality from overgeneration Overgeneration as seen in Example 3 is a separate issue from BLOOM's capacity to translate into another language. We therefore devise a custom truncating method for this type of overgeneration such that only the first translation in a prediction is kept, i.e. anything after a newline or the regular expression pattern = .+?: is discarded.

Results after truncation (Table 2b) show that for all three datasets, 0-shot and 1-shot scores are significantly improved (e.g. 1-shot DiaBLa fr→en increases from 12.05 to 41.36 and 0-shot Flores-101 hi→en increases from 3.40 to 30.19). BLOOM is capable of performing good MT but has a problem knowing when to stop generating. We use the same truncation elsewhere too and indicate when we show results for original or truncated outputs. Table 3: The number of outputs (after truncation) classified as being in the (correct) target language, the source language, or another language for 0-shot and 1-shot setups (for WMT).

Detecting generation in the wrong language

We automatically detect the language of predictions using fasttext langid17 [START_REF] Joulin | Bag of tricks for efficient text classification[END_REF].

Table 3 shows the number of translations identified as being in the correct target language, or alternatively in the source or another language for 0-shot and 1-shot setups after truncation. 18,19 The number of sentences in the correct target language increases from 0-to 1-shot, particularly for the two non-English target languages. When translating into Hindi (0-shot), 1/5 (509) of predictions are not detected as Hindi; the 1-shot largely mitigates the issue (only 76 outputs are in the wrong language).

Increasing the number of few-shot examples

Both problems improve significantly in the 1-shot setup, a trend that continues as the number of fewshot examples increases, resulting in higher BLEU scores, as can be seen in Figure 1 for WMT en↔fr. However, we see diminishing returns, particularly visible between 2 to 5 examples, suggesting that gains beyond 5-shot would be more marginal.

BLOOM model size

Several versions of BLOOM exist, with differing numbers of parameters. To test how size impacts performance, we report average scores and ranges for WMT across the seven prompts. Table 4 shows that as the size decreases (from 176B to 560M parameters), the performance also decreases significantly. We see substantial gains for all models when moving from 0-shot to 1-shot, the smaller models (e.g. BLOOM-7b1, BLOOM-3b) slightly closing the gap with the largest one. As the ranges in Table 4 are computed across prompts, we see that different prompts yield markedly different BLEU scores in the 0-shot setup; for 1-shot, we still see variations of 6-8 BLEU points between the best and the worst prompt. Similar analyses performed with post-processing and also for English↔Hindi (Appendix C) confirm that (i) truncation improves scores for all model sizes and prompts and (ii) the choice of a bad prompt can result in catastrophic MT performance as compared to a good one.

Per-prompt analysis

Looking at average WMT results computed with respect to prompt choice (using the prompts in Table 1) allows us to further investigate cross-prompt variability.

Which prompt works best? This variability is illustrated in Tables 5 and6 report performance across prompts for en↔{fr,hi}, averaged over the five BLOOM models from Section 5.2. 20 The corresponding tables for truncated outputs are in Appendix D. version and a good translation (source+target) get the highest average (and maximum) scores.

Both prompts are more verbose (instruction-like), but the performance gap in the 1-shot setting between these prompts and the simpler, 'primingstyle' prompts (e.g. xglm) narrows. The worst results are seen for gpt3. With this prompt, translating into French after a text that only contains English seems particularly difficult: half of the 0-shot translations for gpt3 are classified as non-French by langid (most of them are English). When translating into Hindi, only 10 outputs are detected as being in Hindi.

Does it help to specify the source language in the prompt? We compare the two versions (-target and -source+target) of a good translation and xglm. Results in Tables 5 and6 are inconclusive. For these language directions and prompts, we see small differences for 1-shot, which may be due to variance between runs. For 0-shot, it clearly helps xglm to indicate the source language, but for the more verbose a good translation, it helps one direction and hurts the other. This question would need to be further explored to draw more solid conclusions, including with non-English prompts.

Evaluating more language directions

We further explore more language directions in the 1-shot setting using Flores-101. As in Section 5.1, we use the xglm-source+target prompt.21

Per-language results

To optimise computational resources, instead of running all language combinations, we concentrate on: (i) high-resource language pairs, (ii) high→mid-resource language pairs, (iii) lowresource language pairs and (iv) related languages (specifically Romance languages). Results are shown in Tables 7 and8 for original outputs, given that overgeneration is less problematic for 1-shot. a good translation-source+target 0.7 0.1-1.9 5.8 0.3-14.5 4.8 0.9-10.2 13.1 2.8-24.6 a good translation-target 0.2 0.1-0.8 5.5 0.3-14.1 6.3 1.1-13.0 13.2 2.8-24.8 gpt3-target 0.1 0.0-0.3 1.4 0.0-6.5 0.2 0.0-0.7 2.2 0.0-10.0 version-target 0.7 0.1-2.0 5.6 0.2-14.0 6.8 1.7-11.5 13.3 2.4-25.8 xglm-source+target 2.1 0.1-6.8 6.9 0.3-13.6 4.4 0.6-12.1 11.9 1.7-25.0 xglm-target 0.2 0.0-0.6 5.1 0.1-14.6 1.6 0.2-4.1 6.6 0.5-13.2 High-resource and high→mid-resource The results for high-resource and high→mid-resource language directions are generally good, surpassing M2M scores for high-resource, except for es→fr. 22 This suggests that BLOOM a has good multilingual capacity, even across scripts (between (extended) Latin, Chinese, Arabic and Devanagari scripts). Romance languages This contrasts with the results between Romance languages, where results are good across-the-board, including from and into Italian (it) and Galician (gl), which are not officially in the training data. Note that Galician shares many similarities with the other Romance languages, in 22 French and Spanish, although related and comparably represented in ROOTS, have very different scores. Our preliminary analysis suggests that this is due to the Spanish references being less literal than the French and structurally more different from the original English. See Appendix E for some examples.

Low-resource

particular with Portuguese (pt). These contrasted results show the performance of an LLM not only depends on the amount of training data, but also largely on the similarity with seen languages. To be complete, these analyses should also take into account the possibility of mislabellings in the training data, 23 which have been found to explain a great deal of cross-lingual abilities of LLMs [START_REF] Blevins | Language contamination helps explains the cross-lingual capabilities of English pretrained models[END_REF].

Cross-lingual transfer

1-shot results are positive for many of the language directions tested (including low-resource), provided they are sufficiently represented in the ROOTS corpus. To better understand how crosslingual BLOOM is and how the 1-shot mechanism functions, we vary the language direction of the few-shot examples, taking Bengali→English (bn→en) translation as our case study. Taking random 1-shot dev set examples, 24 we compare the use of 1-shot examples from (i) the same direction (bn→en), (ii) the opposite direction (en→bn), (iii) a language direction whereby the source languages are related (hi→en), (iv) the same related direction but from a different dataset (the WMT dev set) (v) a high-resource direction into the same target language (fr→en) and (vi) a high-resource unrelated language direction (fr→ar).

The results (Table 9) show that cross-lingual transfer is possible, but using a different language direction can impact overgeneration and translation quality. The unrelated direction fr→ar gives the worst results, with most overgeneration (see the score difference between original and truncated), but also the worst quality after truncation, suggesting that language relatedness does play a role. Overgeneration is still a problem (although less so) when using the opposite direction (en→bn) or the same target language (fr→en). Using a related (higher-resource) source language (hi→en) reduces overgeneration and also gives the best MT results. However, better results are seen when using Flores-101 rather than WMT examples, suggesting that in-domain examples are best. Table 10: Comparison of 1-shot results (BLEU) for Dia-BLa when using the previous/random sentence for the 1-shot example (using the xglm-source+target prompt). In bold are the best results for each language direction.

Use of Linguistic Context

There has been a considerable amount of research on linguistic context in MT, e.g. to disambiguate lexically ambiguous texts or when additional information is necessary for the output to be well-formed (e.g. translating anaphoric pronouns into a language that requires agreement with a coreferent) [START_REF] Hardmeier | Discourse in statistical machine translation: A survey and a case study[END_REF][START_REF] Libovický | Attention strategies for multi-source sequence-to-sequence learning[END_REF][START_REF] Bawden | Evaluating discourse phenomena in neural machine translation[END_REF][START_REF] Voita | Context-aware neural machine translation learns anaphora resolution[END_REF][START_REF] Lopes | Document-level neural MT: A systematic comparison[END_REF][START_REF] Nayak | Investigating Contextual Influence in Document-Level Translation[END_REF].

We test the usefulness of linguistic context in DiaBLa in the 1-shot setting (again using xglm-source+target) by changing the origin of 1-shot examples: (i) a random example vs. (ii) the previous dialogue utterance. If linguistic context is useful, we would expect there to be an improvement for (ii). We also vary the language direction of the 1-shot example. By default, given that the dataset is bilingual, the direction of 1-shot examples is en→fr or fr→en, independent of the current example's direction. Given the results in Section 5.4.2 and the poor 0-shot results in Table 2a, it is important to account for this to provide a fair comparison. We therefore compare each type of context (random/previous) with (i) the same random directions, and (ii-iii) the same (and opposite) language directions as the current example. We show results for original and truncated outputs.

Results are shown in Table 10. Truncation helps considerably; even for 1-shot, BLOOM struggles not to overgenerate and this is considerably reduced when the same rather than the opposite language direction is used for the 1-shot example. It is unclear whether using previous rather than random context helps: BLEU is higher (38.5 vs. 37.6), whereas COMET is lower (0.328 vs. 0.342). These differences could be the result of randomness in 1-shot example selection, and different results could be obtained with a different random seed. Despite these inconclusive results, it is clear that using previous context influences the translation, for better or worse. For evidence of this, see Table 19 in Appendix F, which provides three such examples: (i) an unlucky negative influence on the translation of an ambiguous word glace 'ice cream or mirror' from the previous context, resulting in the wrong sense being chosen, (ii) the use of a coreferent instrument 'instrument' from the previous sentence and (iii) the correct gender agreement of the pronoun they into French (elles 'they (fem.)' as opposed to ils 'they (masc.)') to correspond to the feminine coreferent filles 'girls'.

Conclusion

We have evaluated BLOOM's MT performance across three datasets and multiple language pairs. While there remain problems of overgeneration and generating in the wrong language (particularly for 0-shot MT), MT quality is significantly improved in few-shot settings, closer to state-of-the-art results. Low-resource MT remains challenging for some language pairs, despite the languages being in the training data, questioning what it means to be a BLOOM language. However, we see evidence for cross-lingual transfer for non-BLOOM languages and when using few-shot examples from other language pairs. Finally, although using linguistic context does not give improvements with automatic metrics, there is evidence that discursive phenomena are taken into account. en They are cooler than the surrounding surface in the day and warmer at night. fr→en "They are cooler than the surrounding surface during the day and warmer at night ". es→en During the day, its temperature is lower than that of the surrounding surface, and at night, higher. en "This is not going to be goodbye. This is the closing of one chapter and the opening of a new one." fr→en "It's not goodbye. It's a page that is turning, and another that is opening." es→en "This will not be a farewell; it is just the end of one chapter and the beginning of another".

en

"We now have 4-month-old mice that are non-diabetic that used to be diabetic," he added. fr→en "We now have mice that are four months old and not diabetic, whereas they were before", he added. es→en "Currently, we have mice that are four months old and used to be diabetic, but they are no longer diabetic", he added.

en "We will endeavour to cut carbon dioxide emissions per unit of GDP by a notable margin by 2020 from the 2005 level," Hu said. fr→en "We will strive to significantly reduce carbon dioxide emissions per unit of GDP by 2020 compared to the 2005 level," said Mr. Hu. es→en Hu said, "We will work hard to reduce the level of carbon dioxide emitted per unit of GDP by 2020, so that the difference is significant compared to 2005." en Scientists say this animal's plumage was chestnut-brown on top with a pale or carotenoid-colored underside. fr→en Scientists say that the plumage of this animal was chestnut brown on top and pale or carotenoid on the underside. es→en According to the experts, this animal has a brown plumage on the upper part and a pale or carotenoid color on the lower part.

en 34 per cent of those in the poll share this view, wanting Queen Elizabeth II to be Australia's last monarch. fr→en 34 % of the people surveyed share this view, and want Queen Elizabeth II to be the last monarch to rule Australia. es-en One third of the respondents share this view and want the last queen to be Queen Elizabeth II.

Table 18: A random subset of Flores-101 examples translated using BLOOM into English from French and Spanish (N.B. English was the original language of the sentences). Each block of three sentences contains the original English and the automatic French→English and Spanish→English translations.

1-shot origin Context Reference Prediction

Prev. same French: Pensez vous en permanence à la glace qui se mange ? = English: Do you constantly think about the ice-cream that's eaten? ### French: ou bien à une glace pour se regarder ? = English: *or to an ice-cream to look at?

Prev. opposite English: Do you constantly think about the ice-cream that's eaten? = French: Pensez vous en permanence à la glace qui se mange ? ### French: ou bien à une glace pour se regarder ? = English:

Or a mirror to look into?

or to a mirror to look at? Prev. opposite French: Il m'a montré comment cela marchait, mais si je n'y arrive pas, les filles ici m'aideront sans problème. = English: He showed me how it works, but if I get stuck the girls in here will always help me. ### English: They are very kind. = French:

Elles sont très gentilles.

Elles sont très gentilles.

14

  This was not always straightforward due to incomplete documentation concerning (a) prompts tested, and (b) those actually used in each experiment (e.g. different ones for 0-shot and few-shot runs (Chowdhery et al., 2022)). 15 Used only it seems, for zero-shot learning in the form "Q: what is the L2 translation of sentence [source sentence]. A:", where special tokens Q and A are the query and the answer texts (cf. Figure G.36, pp 59).

  (3) Input: French: Il dit avoir conc ¸u une sonnette de porte Wi-Fi. = English: BLOOM: He says he conceived a Wi-Fi doorbell. = Spanish: Dice que concibió un timbre de puerta Wi-Fi. = German: Er sagt, er hat einen WLAN-Türglocken gegründet. = Italian: Dice di aver concepito un chiuso Wi-Fi [. . . ] (4) Input: French: Le tigre fait partie de la même famille (genre Panthera) que les lions, les léopards et les jaguars. Ces quatre félins sont les seuls capables de rugir. = English: BLOOM: The tiger is a member of the same family (genus Panthera) as lions, leopards and jaguars. These four cats are the only ones capable of roaring. Le tigre est un grand félin qui vit en Asie. Il est le plus grand des félins. Il est aussi le plus gros carnivore terrestre. Il est très rapide et peut courir à plus de 60 km/h.[. . . ]

Figure 1 :

 1 Figure 1: BLEU scores for WMT 2014 en↔fr and the xglm prompt, with an increasing number of few-shot examples.

  RandomFrench: N'empêche, on vit une époque folle, folle! = English: Still, what a crazy, crazy time we're living in! ### French: ou bien à une glace pour se regarder ? = English:or to a mirror to look at yourself?Prev. same English: What kind of instrument were you thinking of? = French: Tu penses à quelle sorte d'instrument ? ### English: A wooden one I suppose... = French:Un instrument en bois, je suppose... Prev. opposite French: Tu penses à quelle sorte d'instrument ? = English: What kind of instrument were you thinking of? ### English: A wooden one I suppose... = French:Un instrument en bois, je suppose.Un instrument en bois, je suppose...RandomFrench: Ils vont vous changer les idées après votre dure journée ! = English: They'll help you take your mind off things after your hard day! ### English: A wooden one I suppose... = French:Un en bois, je suppose...Prev. sameEnglish: He showed me how it works, but if I get stuck the girls in here will always help me. = French: Il m'a montré comment cela marchait, mais si je n'y arrive pas, les filles ici m'aideront sans problème. ### English: They are very kind. = French:Elles sont très gentilles.

  RandomEnglish: I don't know about loans. = French: Je ne sais pas pour les prêts. ### English: They are very kind. = French: *Ils sont très gentils.

Table 2

 2

	:	Cross-dataset comparison of BLEU
	scores (spBLEU for Flores-101) using the
	xglm-source+target prompt.

Table 4

 4

	: Average BLEU scores and ranges across the
	seven prompts for decreasing sizes of BLOOM (original
	outputs).

Table 5 :

 5 Average, min and max BLEU scores by prompt for en↔fr (original outputs). Best average result per setting in bold.

	en→fr	fr→en

Table 6 :

 6 Average, min and max BLEU scores per prompt for en↔hi (original outputs). Best average result per setting in bold.

Table 7 :

 7

	Src↓ Trg→	en	bn	hi	sw	yo
	en	BLOOM M2M	-24.6 27.2 20.5 2.6 -23.0 28.1 26.9 2.2
	bn	BLOOM 29.9 M2M 22.9	-16.3 -21.8	--	--
	hi		BLOOM 35.1 23.8 M2M 27.9 21.8	--	--	--
	sw	BLOOM 37.4 M2M 30.4	--	--	-1.3 -1.3
	yo	BLOOM M2M	4.1 4.2	--	--	0.9 1.9	--
			(a) Low-resource languages
	Src↓ Trg→	ca	es	fr	gl	it	pt
	ca	BLOOM M2M	-28.9 33.8 19.2 19.8 33.0 -25.2 35.1 33.4 25.5 35.2
	es	BLOOM 31.2 M2M 23.1	-24.8 23.3 16.5 29.1 -29.3 27.5 23.9 28.1
	fr	BLOOM 37.2 27.5 M2M 28.7 25.6	-24.9 24.0 38.9 -32.8 28.6 37.8
	gl	BLOOM 37.5 27.1 33.8 M2M 30.1 27.6 37.1	-18.3 32.2 -26.9 34.8
	it	BLOOM 31.0 25.4 31.4 20.2 M2M 25.2 29.2 34.4 29.2	-29.2 -31.5
	pt	BLOOM 39.6 28.1 40.3 27.1 20.1 M2M 30.7 26.9 40.2 33.8 28.1	--
			(b) Romance languages

1-shot MT results (spBLEU) on the FLORES-101 devtest set (original outputs).

Table 8 :

 8

	Original	Truncated

1-shot MT results (spBLEU) on the Flores-101 devtest set (original outputs).

Table 16 :

 16 Average, min and max BLEU scores per prompt for WMT14 en↔fr (truncated outputs). Best average result per setting in bold.

	en→hi	hi→en

Table 17 :

 17 Average, min and max BLEU scores per prompt for WMT14 en↔hi (truncated outputs). Best average result per setting in bold.

Table 19 :

 19 Ambiguous DiaBLa examples with different 1-shot contexts. Words that are relevant to the ambiguity are underlined, and incorrect translations are marked with an asterisk.

https://hf.co/bigscience/bloom

https://github.com/rbawden/ mt-bigscience

A rather slippery concept in this context, as the content of the training data is not fully known and seems to mostly comprise English texts.

The ROOTS corpus can now be queried using the dedicated search tool https://hf.co/spaces/ bigscience-data/roots-search.

https://hf.co/bigscience/bloom

English, French and Hindi make up 30%, 12.9% and 0.7% of the training data respectively(Laurenc ¸on et al., 2022).

An extended version, Flores-200, has been recently released (Costa-jussà et al., 2022), which is larger and covers approximately twice as many languages. As this new version was released late in our evaluation process and had only been used in one paper, we decided to stick to Flores-101.

For comparison, (Bi et al., 2020) reports state-of-the art BLEU scores for supervised MT as 45.6 and 45.4 for WMT14 en→fr and fr→en respectively.

https://fasttext.cc/docs/en/ language-identification.html, using the compressed version lid.176.ftz.

Raw tables can be found in Tables12 and 13in Appendix B.

 19 These numbers are better than the initial ones reported in (BigScience et

al., 2022), as we use a different prompt and truncation. See below for a detailed analysis per prompt.

For a given prompt, the range mainly reflects the performance of the different sizes of BLOOM model.

 21 It behaved well on average in the previous experiments and is one of the least verbose, making it more suitable in a multilingual setting.

In a personal communication, N. Muennighoff estimates that Italian accounts for ∼0.33% of the ROOTS corpus, slightly below the proportion of Hindi texts (0.47%).

Acknowledgements

This work was made possible with the collective efforts of the BigScience community, who designed, developed and prepared the tools and datasets used to train BLOOM. Special mention to evaluation working group members and especially to Niklas Muenninghoff and Pawan Sasanka Ammanamanchi for producing some of our results.

This work was granted access to the HPC resources of Institut du développement et des ressources en informatique scientifique (IDRIS) du Centre national de la recherche scientifique (CNRS) under the allocations 2021-AD011011717R1, AD011012254R2, 2021-A0101012475 and 2022-AD010614012 made by Grand équipement national de calcul intensif (GENCI). R. Bawden's participation was partly funded by her chair position in the PRAIRIE institute, funded by the French national agency ANR as part of the "Investissements d'avenir" programme under the reference ANR-19-P3IA-0001, and by her Emergence project, DadaNMT, funded by Sorbonne Université.

A COMET Results for Main Comparison

B Wrong language prediction and over-generation

As described in Section 5.1, one problem identified with BLOOM, particularly for 0-shot translation, is generating in the wrong language. Tables 12 and13 give the full analysis including raw figures for language identification for WMT14 fr↔en and hi↔en translation directions. For 0-5 few-shot examples, we indicate the number of truncated outputs identified as being from each language (indicated by the rows), the correct language (the target) being indicated in green, and the source language (therefore incorrect) being indicated in red. We also provide the average length difference (∆) between BLOOM's outputs and the reference translations (negative numbers indicate that the prediction is longer than the reference). For 0-shot translation, a significant number of examples are classed as being in the source language for en→fr, and even more so for en→hi (almost

Table 12: Raw figures for language identification and length differences of outputs compared to the reference translation for WMT2014 en→fr using the xglm-source+target prompt. For 0-5 few-shot examples, N is the number of sentences identified as being in each language (the target language's row (correct) is indicated in green and the source language's row (one of the many incorrect options) in red) and ∆ is the length difference in number of characters (N.B. it is negative when the prediction is longer than the reference).

one fifth of the outputs are in the wrong language). As we increase the number of few-shot examples used, both of these problems are significantly reduced, and almost disappear for all language pairs and directions with 5 examples.

C Analysis per model

In this section, we complete the results of Section 5.2 with Tables 14 and15, respectively for French↔English and Hindi↔English, reporting results without truncation. As expected, the systems are ranked according to their size. For French-English we see that decent performance can already be obtained with the second largest model BLOOM-7b1, using 1-shot. Using this model, or even a model half this size can provide good indication of the performance of prompts, and be reliably used as test beds. We obtain less satisfactory results with English↔Hindi, even with the large BLOOM; for this language pair, we even observe a large variation across prompts (looking at the range of scores)

Table 13: Raw figures for language identification and length differences of outputs compared to the reference translation for WMT2014 en→hi using the xglm-source+target prompt. For 0-5 few-shot examples, N is the number of sentences identified as being in each language (the target language's row (correct) is indicated in green and the source language's row (one of the many incorrect options) in red) and ∆ is the length difference in number of characters (N.B. it is negative when the prediction is longer than the reference).

in the 1-shot setting for all models.

D Analysis per prompt

In this section, we replicate the analysis of Section 5.3 and report results per prompt with truncated outputs in Tables 16 and17.

The conclusions are overall consistent with what we report for non-truncated outputs in the main text. We note that after truncating the outputs, xglm-source+target yields very good results across the board, outperforming its closest contenders a good translation-source+target and version-target in almost all configurations. However, the choice of the prompt seems to matter more (a) in the zero-shot setting, (b) when translating out of English. Conversely our more stable results are for fr-en, 1-shot.

E Translation divergences in Flores 101

A striking observation reported in the main text (Section 5.4.1) is the difference between French and Spanish for the Flores-101 experiments. This is unexpected, as both languages are well represented in the training data. Yet, when translating from and into English the difference in spBLEU score is huge; and there is a clear gap with the other Romance languages as well. A related question is the poor translation between French and Spanish, not much better than for French→Arabic. Looking at some sample outputs, this seems to be due to the peculiarities of the Spanish translations, which appear to be less literal than their French counterparts, but which yield equally good translations into English. This can be seen when we compare translations back into English for these languages (see a random subset in Table 18). The last example illustrates this very clearly: we see "34 percent" in both the original English and in the translation from French, while translation from Spanish starts with "one third". 0.1 0.1 -3.0 5.7 0.3 -9.5 5.9 0.3 -10.4 12.4 1.0 -17.5 BLOOM-3b 0.2 0.0 -0.5 3.6 0.0 -7.0 4.9 0.2 -7.2 8.9 0.1 -13.5 BLOOM-1b1 0.1 0.0 -0.1 1.5 0.0 -4.5 1.4 0.1 -3.1 4.6 0.00 -8.2 BLOOM-560m 0.1 0.0 -0.1 0.8 0.0 -1.7 0.2 0.0 -0.3 1.5 0.1 -2.8

F DiaBLa context-use examples


Mariarosa Cavallo

Rodolphe Alchaar

Erwan Bossavit

Huichen Zhang

Tung Huu Dang

Adrien Khalili

Yoann Prado

Mathieu G Silly

James K Utterback

Sandrine Ithurria

Pavel Dudin

José Avila

Debora Pierucci

Emmanuel Lhuillier

Inside a nanocrystal-based photodiode using photoemission microscopy

Keywords: nanocrystals, HgTe, photoemission, electronic structure, photodiode, pn junction

As nanocrystal-based devices gain maturity, a comprehensive understanding of their electronic structure is necessary for further optimization. Most spectroscopic techniques typically examine pristine materials and disregard the coupling of the active material to its actual environment, the influence of an applied electric field, and possible illumination effects. Therefore, it is critical to develop tools that can probe device operation in situ and operando. Here we explore photoemission microscopy as a tool to unveil the energy landscape of a HgTe NC-based photodiode. We propose a planar diode stack to facilitate surface-sensitive photoemission measurements. We demonstrate that the method gives direct quantification of the diode's built-in voltage. Furthermore, we discuss how it is affected by particle size and illumination. We show that combining SnO2 and Ag2Te as electron and hole transport layers is better suited for extended-short-wave infrared materials than materials with larger bandgaps. We also identify the effect of photodoping over the SnO2 layer and propose a strategy to overcome it. Given its simplicity, the method appears to be of utmost interest for screening diode design strategies.

INTRODUCTION

For a long time, infrared sensing has been dominated by epitaxially grown narrow-bandgap semiconductors (InGaAs, InSb, and HgCdTe) and heterostructures (type-II superlattice and GaAs/AlGaAs). These materials have undergone decades of optimization to achieve high quantum efficiencies; 1 however, despite this success, their high cost limits their use in defense and astronomy applications. This situation, therefore, calls for cost-effective alternative materials. Many materials have been explored, including 2D materials-such as graphene, 2 black phosphorus, 3,4 Pt(Se/Te)2 [START_REF] Azar | 14th Pacific Rim Conference on Lasers and Electro-Optics (CLEO PR 2020)[END_REF][START_REF] Shawkat | [END_REF] and tellurene 7 -and colloidal nanocrystals 8 (NCs)-PbS, [START_REF] Steckel | IEEE International Electron Devices Meeting (IEDM)[END_REF][START_REF] Konstantatos | [END_REF] PbSe, 11 HgTe, 12 HgSe, 13 Ag2Se [14][15][16] , etc. In this quest for alternatives, the primary challenge is to identify an infrared material that absorbs in the IR range and gives acceptable responsivity (the ability to convert light into an electrical signal). However, this is just the first step, as other requirements must be met before the commercialization of cameras that feature low noise, scalability (towards millions of pixels), and stability over time, temperature, and oxidation. Among the candidates, HgTe NCs 12 combine a broad spectral tunability and almost two decades of continuous research efforts 17,18 that have pushed the technology forward and brought demonstrations of high-performing photodiodes and cameras with a high-speed response. 19 When it comes to the use of HgTe NCs in optoelectronic devices, multiple device geometries [20][21][22] (photoconductors, transistors, and diodes) and concepts, in particular the coupling to photonic cavities, 23 have been explored. At the single pixel level, performances have mostly been pushed in diode geometries for which detectivities close to those reported for commercial InGaAs have been achieved. Ackerman et al. 24 were the first to propose the coupling of an HgTe NC layer with an Ag2Te layer, the latter being used as a hole extraction layer. Since then, many derivatives have been considered, in particular through the addition of an electron transport layer (Bi2S3, 25 CdSe, 26 SnO2 27 ), confirming the potential use of this specific diode stack for HgTe NC-based infrared sensing. To move beyond the already explored structures while avoiding an inefficient trial-and-error approach, deeper knowledge of the electronic structure must be correlated with the development of new diode stacks. Previous studies, either based on electrochemistry 28 or X-ray photoemission, 29 have already revealed the band alignment of HgTe NCs as a function of particle size and surface chemistry. 30 However, these previous studies focused on pristine samples, which completely neglects the electronic coupling existing in diodes as well as the impact of incident light. As a result, there is a need for new tools to probe in situ (i.e., in a real diode) and operando (i.e., while the device is under operation) the electronic structure of NC-based photodiodes. Recently, X-ray photoemission microscopy [31][32][33][34] has been proposed as a tool that can combine imaging and photoemission spectroscopy. It consists of an X-ray nanobeam that is scanned on the surface of the sample, and for each point, we can acquire a kinetic energy photoemission spectrum. So far, this strategy has been mostly applied to 2D materials 32,35,36 by combining spatial information with angle-resolved photoemission to correlate spatial structure and electronic structure. Only very recently, Cavallo et al. proposed to use this strategy to probe the energy landscape of a NC-based field-effect transistor. 31 The method gives access to band bending and the electric field distribution under the application of drain and gate bias. Now, we aim to apply it to the case of a NC-based photodiode. However, the short escape depth of photoelectrons (a few nm) prevents the direct measurement of a diode stack by photoemission spectroscopy. We tackle this difficulty by designing a planar version of the diode stack and then probing the energy profile in situ and operando (here under illumination). In addition, we apply the method to unveil how the internal electric field is affected as the bandgap is changed.

METHODS

Chemicals:

Mercury chloride (HgCl2, Sigma-Aldrich, 99%), Mercury compounds are highly toxic. Handle them with special care. Tellurium powder (Te, Sigma-Aldrich, 99.99%), silver nitrate (Alfa Aesar, 99.9%), SnO2 nanoparticles (15% dispersion in water, Alfa Aesar), trioctylphosphine (TOP, Alfa Aesar, 90%), oleylamine (OLA, Acros, 80-90%), dodecanethiol (DDT, Sigma-Aldrich, 98%), ammonium chloride (Alfa Aesar, 98 %) ethanol absolute (VWR), methanol (VWR, >98%), isopropanol (IPA, VWR), hexane (VWR, 99%), octane (Carlo erba, 99%), 2-mercaptoethanol (MPOH, Merck, >99%), N,N dimethylformamide (DMF, VWR), N-methylformamide (NMF, Alfa Aesar, 99%), toluene (VWR, 99.8%). All chemicals are used without further purification except oleylamine that is centrifuged before use.

M TOP:Te precursor:

In a three-neck flask, we mixed Tellurium powder (2.54 g) in 20 mL of trioctylphosphine. The flask is degassed under sub-mbar pressure at room temperature for 5 min and then the temperature is increased to 100 °C. The degassing of the flask is conducted for 20 additional minutes. The atmosphere is switched to nitrogen and the temperature is raised to 275 °C. The solution turns to a clear orange coloration. The heating mantle is removed and the temperature of the flask cools down. Its color changes to yellow. Finally, this solution of TOP:Te is used as stock solution and stored into a glove box for storage.

HgTe NC synthesis with band-edge at 6000 cm -1 : In a 50 mL three-neck flask, we have mixed 540 mg of HgCl2 salt together with 50 mL of oleylamine. The flask is then degassed under reduced pressure at 110 °C. The solution is yellow and clear. Then, 2 mL of TOP:Te (1 M) are extracted from the glove box and are mixed with 8 mL of oleylamine. The atmosphere is replaced by nitrogen and the temperature tuned to 57 °C. The pre-heated TOP:Te solution is injected in the three neck flask and the solution turns dark after 1 min. After 3 min, we inject 10 mL of a mixture of 10% DDT in toluene and a water bath is used to prompt the drop of temperature. The content of the flask is split in 4 tubes and MeOH is added. After centrifugation, the pellets are redispersed using 10 mL of toluene in a single centrifuge tube. The solution is precipitated again using absolute EtOH. The formed pellet is redispersed in 8 mL of toluene. At this step, the NCs are centrifuged in fresh toluene to remove the lamellar phase. The solid phase is discarded by centrifugation and the supernatant filtered through a 0.22 µm PTFE filter.

HgTe NC synthesis with band-edge at 4000 cm -1 : In a 100 mL three-neck flask, 543 mg of HgCl2 and 60 mL of oleylamine (OLA) are degassed from room temperature to 110 °C for an hour. The atmosphere is then switched to nitrogen and the solution is cooled down to 82 °C. A solution of 2 mL of TOP:Te (1 M) and 8 mL of OLA is quickly added to the flask. After 3 minutes, the reaction is quickly quenched by adding a mixture of 1 mL of DDT and 9 mL of toluene. The nanocrystals are precipitated with methanol and redispersed in toluene. Then the nanocrystals are precipitated with ethanol and redispersed in toluene. The solution is centrifugated as is. The pellet is eliminated and the supernatant is washed one more time. Finally, the HgTe NCs solution in toluene is filtered through a 0.22 µm PTFE filter. The obtained solution is used for further characterization and device fabrication.

Ag2Te NC synthesis: In a 50 mL three neck flask, 170 mg of AgNO3 are mixed together with 25 mL of oleylamine and 2.5 mL of oleic acid. The flask is then degassed under vacuum at room temperature for 15 min and then at 70 °C for 10 min. The atmosphere is replaced to nitrogen and 2.5 mL of TOP are added into the solution. Then, the temperature of the flask is raised to 160 °C. After 30 min, the solution becomes orange. At this step, 0.5 mL of TOP:Te (1M) is injected and the reaction is quenched after 10 min with a water bath. The crude solution was stored in a freezer at --20 °C. When needed, 800 µL of the unfrozen crude solution are precipitated with MeOH. After centrifugation the formed pellet is redispersed in a mixture containing 200 µL DDT and 600 µL of chlorobenzene. Then, the NCs are precipitated by addition of methanol. After centrifugation the pellet is dispersed using 800 µL of chlorobenzene. Again, the NCs are precipitated by addition of MeOH and centrifugation. The pellet is this time dispersed in 1.6 mL of a (9:1 volume ratio) mixture of hexane:octane.

SnO2 NCs:

The procedure is inspired from ref 37 . For lightly doped SnO2 (SnO2-L), we dilute 200 µL of commercial solution with 1 mL of 24 mM NH4Cl solution in water. For heavily doped SnO2 (SnO2-H), we dilute 200 µL of commercial solution with 1 mL of 48 mM NH4Cl solution in water.

Infrared spectroscopy:

We use a Thermo-Fischer iS50 in attenuated total reflection (ATR) mode for absorption. The spectra are averaged over 32 scans and acquired with a 4 cm -1 resolution. To measure photocurrent spectrum, a Femto DLPCA 200 current amplifier, is used to bias and magnify the output current from the sample. The output signal is then fed into the FTIR electronics. For PL, a film of NCs is excited using a blue flash light. This signal is then used as source for the FTIR.

Transmission electron microscopy on NCs:

A dilute NC solution is drop-cast onto a copper grid covered with an amorphous carbon film. This grid is degassed under secondary vacuum to reduce future contamination. A JEOL 2010F microscope, operated at 200 kV, is used for acquisition of pictures.

Scanning electron microscopy and EDX mapping: SEM observations were carried out using a Zeiss Supra 40 microscope equipped with a Bruker XFlash 630M probe for energy dispersive X-ray (EDX) mapping. SEM images and EDX mapping were acquired at a voltage of 10 kV and an aperture of 10 µm. The EDX mapping duration was 1 hour.

Planar p-n junction fabrication:

We start by cleaning the substrates with acetone and isopropyl alcohol and then placing them in an oxygen plasma cleaner for 10 minutes. The heavily doped SnO2 solution is deposited onto the substrate via spin coating at 4000 rpm for 30 s. The film is annealed on a hot plate at 70°C for 1 h. Next, we created an interdigitated pattern with Ag2Te using UV lithography. AZ5214E resist was spin-coated on the substrate at 4000 rpm for 30 s and then baked for 1.5 min at 110°C. The resist was then UV-exposed for 1.5 s in order to create the interdigitated pattern, followed by an inversion baking step for 2 min at 125°C and a second UV exposure for 40 s, without a mask. The resist was then developed using AZ726 for 30 s. Ag2Te was spin-coated on top of the resist at 1500 rpm for 30 s, followed by an acetone lift-off to reveal the pattern. The final step in the process is the deposition of the HgTe ink solution. Ink preparation starts by dissolving 30 mg of HgCl2 in a solution of 18 mL of DMF and 2 mL of mercaptoethanol. At each step, the solution is stirred with a vortex and an ultrasonic bath. HgTe nanocrystal solution (600 μL) is mixed with 600 µL of the exchange solution and 400 µL of DMF. Toluene is added to reach a global volume of 4 mL and the solution is centrifugated for 3 min at 6000 rpm to precipitate the nanoparticles. The supernatant is discarded, and the formed pellet is dried under vacuum for 3 mins and then redispersed in 500 μL of DMF to target a thickness of about 40 nm. The resulting ink is filtered and spin-coated on the substrate at 2000 rpm for 2 min followed by a drying step at 2500 rpm for 1 min.

Photoemission microscopy:

The spatially resolved photoemission maps were measured at the ANTARES beamline of Synchrotron SOLEIL. We used linearly horizontal polarized photons with a photon energy of 95 eV and MBS A-1 hemispherical electron analyzer. 38 All measurements were conducted at 250 K, with an energy resolution better than 30 meV and a spatial resolution as low as 500 nm along the pn junction (indicated as X-direction in all figures). For all maps, angular dispersion of the analyzer (±15°) is neglected by integrating the spectra over all angles. The samples were mounted on the operando sample holder of ANTARES beamline and directly grounded to it via silver paste. The sample holder was then grounded through the manipulator. The acquired data were processed using Wavemetrics Igor Pro. In particular, the binding energy maps were obtained by fitting a double gaussian to the measured spectra point-by-point and then plotting the binding energy value of the Hg 5d5/2 contribution. To evaluate the energy shift, the photoemission spectra were averaged over a small number of points (5 or 10) in the regions of interest. Binding energy calibration was performed by measuring the Fermi edge of a gold substrate mounted on the same sample holder of the device.

High-resolution photoemission spectroscopy:

To characterize the pristine HgTe materials, additional non-spatially resolved measurements were performed at Tempo beamline [START_REF] Polack | AIP Conf. Proc[END_REF] of synchrotron Soleil. Films of NCs were spin-cast onto a Si substrate coated with 80 nm of gold. The ligands of the NCs were exchanged using the same procedure as for device fabrication to avoid any charging effects during measurements. The samples were introduced in the preparation chamber and degassed under a vacuum below 10 -9 mbar for at least two hours. Then the samples were introduced into the analysis chamber. The signal was acquired with an MBS A-1 photoelectron analyzer. The acquisition was done at a constant pass energy (50 eV) within the detector. A photon energy of 150 eV was used for the acquisition of valence band and work function while a 600 eV photon energy was used for the analysis of the core levels. A gold substrate was used to calibrate the Fermi energy. The absolute value of the incoming photon energy was determined by measuring the second order of Au 4f core level peaks. Then for a given analyzer pass energy, we measured the Fermi edge and set its binding energy as zero. The same shift was applied to all spectra acquired with the same pass energy. To determine the work function, we applied 18 V bias, whose exact value was determined by looking at the shift of a Fermi edge.

RESULTS AND DISCUSSION

We start by growing HgTe NCs that are relevant for short-wave infrared (SWIR) detection. We use the procedure developed by Keuleyan et al., [START_REF] Keuleyan | [END_REF] in which HgCl2 is reacted with telluride complexed to trioctylphosphine. The reaction is conducted at a mild temperature (≈60°C). The obtained particles have a branched aspect according to transmission electron microscopy, see Figure 1b. The reaction duration is appropriately selected to achieve a band edge at around 1.7 µm by the end of the NC growth, see Figure 1a. Initially, the particles are capped with long ligands (dodecanethiol) that are replaced by shorter ligands 41,42 (mercaptoethanol and HgCl2) to increase the coupling and render the film photoconductive. By doing so, the absorption spectrum is redshifted, and the cut-off wavelength becomes 2 µm, as shown in Figure 1e.

For SWIR detection, Gréboval et al. 27 have proposed the use of a diode stack in which the HgTe NCs are sandwiched between SnO2 and Ag2Te, respectively, as electron and hole transport layers, see Figure 1d. Such a diode stack has demonstrated high performances at room temperature, with a responsivity of 300 mA.W -1 under 0 V operation, detectivity of 5x10 10 Jones, and a fast time response.

The particles have been further characterized using high-resolution X-ray photoemission. To ensure accurate representation of the material in the device and prevent possible charging-induced electrostatic shifts, similar surface chemistry processing was carried out. Beyond the Hg and Te states, the film presents contributions of C and Cl that come from the ligands (respectively, the mercaptoethanol and HgCl2 used as a Hg source), see Figure 1c. On the other hand, we observe no contribution coming from oxygen (O 1s state is expected to appear with a 532 eV binding energy) despite the preparation of the sample in air, which confirms the weak oxidation susceptibility of HgTe. 43 The photoemission spectrum also presents a low binding energy peak of Hg 5d at 8.2 eV, see the inset of Figure 1c. Such low-binding energy states are crucial since they are required to perform soft X-ray photoemission microscopy. In these experiments, a Fresnel zone plate is used to focus the X-ray beam, here obtained from a synchrotron beam, see Figure 2b. The initial beam a 50 µm width is focused to obtain a width of around 700 nm on the sample. 33 However, this zone plate is optimized for low-energy photons (<100 eV), requiring the presence of low-energy binding states in the sample to be spatially tracked. The use of low-energy photons also reduces the escape depth of the generated photoelectrons to a distance of less than 1 nm which is significantly lower than that of a conventional diode stack that is ≈500 nm thick. As a result, this technique is not suited to probe a diode with a conventional vertical geometry. Consequently, we have updated the device's geometry to be compatible with photoemission microscopy requiring that the probed surface be on top. We fabricated a planar version of the stack depicted in Figure 1d. The planar p-n junction (Figure 2a) is built on a gold-coated Si wafer in which the gold layer forms an equipotential that minimizes sample charging. An SnO2 layer is deposited on top of gold and then functionalized with an interdigitated electrode geometry using conventional optical lithography. Ag2Te NCs are spincoated in the opening of the resist. After liftoff, the Ag2Te pattern on SnO2 is covered by a HgTe NC film. Figure S1 depicts the fabrication procedure for the planar p-n junction, and Figure S2 shows its chemical mapping.

Figure 2. X-ray photoemission microscopy applied to a planar p-n junction. a. Schematic representation of a planar p-n junction with the active layer made of HgTe NCs and where the n and p type characters are induced by the underlying charge transport layer (Ag2Te for p side and SnO2 for the n side). b. Conceptual representation of the photoemission microscopy. The soft X-ray beam from synchrotron is focused on the sample using a Fresnel zone plate. The sample is then scanned, and photoemitted electrons are analyzed using an electron analyzer. c. Binding energy map of the Hg 5d5/2 state as the planar p-n junction is scanned around the edge of the digit. d. Photoemission spectra of the Hg5d state acquired on top of the Ag2Te area and on top of the SnO2 area. e. Reconstructed valence and conduction band landscape of the planar heterojunction, assuming a rigid shift of the bands.

Figure 2c shows a map of the binding energy of the Hg 5d5/2 state around the edge of two interdigitated fingers, obtained by fitting each acquired spectrum with a gaussian line shape. The contrast in the map reflects the underlying sample structure resulting from a shift in binding energy, as shown in Figure 2d.

Due to the strong interaction of electrons with matter, the photoelectron escape depth is short (few nm at the used photon energy), thus photoemission probes only the top surface of the sample. It remains that this surface experiences the influence of the textured substrate that is underneath, in what we can call a "remote doping". This situation is nevertheless representative of what occurs in a vertical stack (Figure 1d): as the absorbing layer is few hundred nm thick, HgTe being almost an intrinsic semiconductor (Figure S3), the p-n junction that is formed is the result of the behavior induced by the electron and hole transport layers at each side of this layer.

The Hg 5d state from HgTe NCs in contact with the SnO2 layer presents, on average, a 140 meV shift towards higher binding energy, making the material more n-type. Assuming a rigid shift between the core level and the valence band, see Figure 2e, we can reconstruct the valence and conduction band diagram across two digits of the structure. The complete procedure is detailed in Figure S3. The observed shift induced by the substrate, of a magnitude comparable to the material's bandgap (≈0.5 eV), is large enough to induce a shift from n-type over SnO2 to p-type over Ag2Te, see Figure 2e. Consequently, photoemission microscopy offers a clear benefit by allowing for a direct measurement of the built-in voltage that is generally only obtained through a modeling of the electrical measurements. Furthermore, we observe that the width of the space-charge region between the n-and p-sides of the junction (see area between purple dashed lines in Figure 3) is on the order of a few microns. This is wider than the resolution of the X-ray beam and wider than the thickness of a typical vertical diode stack (≈ 500 nm), suggesting that the entire HgTe layer in the diode stack is affected. As a result, in a vertical heterojunction, the charge transport layers on each side affect the whole HgTe layer, which can be used as an absorbing intrinsic layer. There is consequently no need to further dope the HgTe layer to design p and n zones, which was a challenge up to recently 44 .

Then, we investigated the effect of HgTe NC size on the formation of the p-n junction energy profile. In particular, we tested slightly larger NCs with a band edge at 2.5 µm-a material relevant for extended short-wave infrared (e-SWIR) detection. In Figure 3, we make a comparison of the energy landscape for SWIR (2 µm cut off) and e-SWIR (2.5 µm cut off) over the same interdigitated pattern. The results show that the size of the area associated with low binding energy, occurring over Ag2Te and associated with an induced p-type behavior, is much larger for the SWIR material than for the e-SWIR material. This behavior can be explained by the large difference between the effective masses of the electron and hole in HgTe. The heavy-hole band is barely dispersive and is associated with a mass 25 times 45 larger than that of the strongly dispersive light-hole band acting as the conduction band. As a result of this asymmetry and the lack of bulk bandgap, the confinement in HgTe is mostly occurring within the conduction band. As we switch from the SWIR to an e-SWIR material, the optical band gap reduces by 250 meV, which mostly translates into a shift of the conduction band energy, with the valence band being only marginally affected. This intuition has been further confirmed by a systematic measurement of the band alignment for HgTe versus size, as shown in Figure S4. Consequently, as the optical band gap is reduced, the Fermi level also gets much closer to the conduction band. This means that for the same surface chemistry (i.e., same dipole), the e-SWIR material will acquire a n-type character compared to the SWIR material. This trend is fully in line with what we observed in Figure 3, where the high binding energy (i.e., n-type) domains are larger for the e-SWIR material due to its pre-existing n-type nature.

µm cut-off wavelength), while the material described as e-SWIR (bottom graph) corresponds to larger HgTe particles (with 2.5 µm cut-off wavelength). The grey dashed line corresponds to half of the shift. Areas in pink are n-doped (from SnO2), while areas in light blue are p-doped (from Ag2Te). Purple dotted lines highlight the area corresponding to the space charge region (SCR).

In the last part of this work, we explore operando conditions to closely mimic the realistic operation of the device and investigate the effect of incident light on its electronic structure. By doing so, multiple effects may be unveiled such as light induced damage, heating or photodoping. The sample located inside the microscope chamber is illuminated by a 635 nm laser diode. Mapping of the Hg5d state in the dark (resp. under illumination) is shown in Figure 4a Figure 4b). We observe a loss of contrast, which is more distinctly highlighted by the photoemission spectra. In the dark (Figure 4c), the shift between the peak over SnO2 and over Ag2Te is higher than the one observed under illumination (Figure 4d). The average values for these shifts are shown in Figure 4e. This suggests that the built-in electric field of the diode is reduced by a factor of 2 under illumination. However, it appears that the effect is not the same over the two parts of the diode. Indeed, the binding energy of the Hg 5d state from HgTe on top of Ag2Te remains almost unchanged under illumination, see top part of In oxides such as ZnO 46 or SnO2, photodoping effects (i.e. accumulation of charge, often electron in oxide, in deep trap states resulting from unperfect passivation) are well known, and charge accumulation may build up under illumination acting against the designed built-in electric field. A possible strategy to overcome this undesired effect of light will be to push the extrinsic doping of the SnO2 further to make it less sensitive to photocharge generation.

CONCLUSION

We used X-ray photoemission microscopy to probe the energy landscape of a nanocrystal-based photodiode under in situ and operando conditions. To meet the photoemission constraint, we proposed an updated planarized version of the device. The method gives a direct measurement of the built-in voltage of the diode, whereas most conventional methods only access this quantity through modeling of electrical measurements, making the method highly valuable for future device optimization. Moreover, this technique is compatible with measurements under illumination. Here we evidence a slight photodoping effect over the SnO2 layer that counterbalances part of the diode electric field. The latter can likely be fixed by targeting higher doping of this layer. This work demonstrates that spatially resolved photoemission spectroscopy can be a highly valuable tool for optimizing optoelectronic devices based on nanocrystals.

Figure 1

 1 Figure 1 HgTe as active material for infrared photodiodes. a. Absorption and photoluminescence spectra of HgTe NCs with a 1.6 µm band edge after synthesis. b. TEM image of the same HgTe NCs. c. Photoemission survey spectra for the same HgTe NCs, acquired with a photon energy of 700 eV. The inset is a zoom-in on the Hg 5d core level state, which has a low (8.2 eV) binding energy. d. Schematic representation of a short wave infrared photodiode with the infrared absorbing layer made of HgTe NCs. e. Photocurrent spectra for the diode depicted in part d. Note that in the diode, the cut-off wavelength now reaches 2 µm. The room temperature spectrum of an InGaAs photodiode is also depicted.

Figure 3

 3 Figure 3 Effect of particle size on domain size. Shift of the Hg 5d state across the planar p-n junction. Here two HgTe NCs have been used. Described as SWIR (top graph) is the material depicted in Figure 1 (with 2 µm cut-off wavelength), while the material described as e-SWIR (bottom graph) corresponds to larger HgTe particles (with 2.5 µm cut-off wavelength). The grey dashed line corresponds to half of the shift. Areas in pink are n-doped (from SnO2), while areas in light blue are p-doped (from Ag2Te). Purple dotted lines highlight the area corresponding to the space charge region (SCR).

Figure 4f ,

 4f while the one from HgTe on top of SnO2 is affected by the light, see the bottom part of Figure 4f.

Figure 4

 4 Figure 4 Operando investigation of light on the electronic structure. a. (resp b) Binding energy of the Hg 5d5/2 state across a planar p-n junction in the dark (resp under illumination by a laser diode at 635 nm). c. (resp d.) Photoemission spectra of the Hg5d state acquired on top of the Ag2Te area and on top of the SnO2 area in the dark (resp. under illumination by a laser diode at 635 nm). e. Average shift of the Hg 5d state binding energy across the junction in the dark and under illumination. f. Hg 5d core level spectra over Ag2Te (top part) and over SnO2 (bottom part) in the dark and under illumination.

SUPPORTING INFORMATION

Supporting Information include details about planar p-n junction fabrication and its chemical mapping, as well as band alignment for HgTe NCs with various sizes.

Author contributions

DP and EL designed the experiment and raised funding. YP and AK grew the particles with supervision of SI and EL. RA built the planar junction. All authors took part in the beamtime dedicated to photoemission microscopy or high-resolution photoemission. MC and JA conducted the data processing with support of DP. EL wrote the manuscript with input of all authors. All authors have proofread the final text. The project is supported by ERC grant blackQD (grant n° 756225) and AQDtive (grant n°101086358). We acknowledge the use of clean-room facilities from the "Centrale de Proximité Paris-Centre". This work has been supported by the Region Ile-de-France in the framework of DIM Nano-K (grant dopQD). This work was supported by French state funds managed by the ANR within the Investissements d'Avenir programme under reference ANR-11-IDEX-0004-02, and more specifically within the framework of the Cluster of Excellence MATISSE and also by the grant IPER-Nano2 (ANR-18CE30-0023-01), Copin (ANR-19-CE24-0022), Frontal (ANR-19-CE09-0017), Graskop (ANR-19-CE09-0026), NITQuantum (ANR-20-ASTR-0008), Bright (ANR-21-CE24-0012), MixDFerro (ANR-21-CE09-0029.), Quicktera (ANR-22-CE09-0018) and Operatwist (ANR-22-CE09-0037-01). This project has received financial support from the CNRS through the MITI interdisciplinary programs (project within).

CONFLICT OF INTEREST

The authors declare no competing interest.


Bertrand Kerautret

Miguel Colom

Adrien Krähenbühl

Daniel Lopresti

Pascal Monasse

Hugues Talbot

Reproducible Research in Pattern Recognition: Third International Workshop, RRPR 2021

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Preface

This volume contains the original contributions presented during the third edition of the Workshop on Reproducible Research in Pattern Recognition which was held online on January 11, 2021 as a satellite workshop of ICPR. Following the two previous editions, this event continues to cover advances in platforms on reproducibility and new reproducible research results, which were covered by regular or companion short papers. Despite the global health crisis, the workshop was a success thanks to extended deadlines, which allowed authors more time to prepare their contributions.

The growing success of this event is evidenced by the larger number of authors registered for this workshop (60 authors from 10 di↵erent countries), corresponding to an increase of 57%. The total number of submitted papers was also increased, in this case by 28% (18 papers) compared to the previous edition. Each contribution was reviewed by three to four reviewers (for the regular papers) or by two reviewers (for the short papers, which we also call ICPR companion papers). The online-only audience (due to the pandemic) ranged from 25 to 30 attendees in each session.

The thirteen accepted papers were organized into three main categories. The first contributions focused on reproducible research frameworks. One paper introduced the ReproducedPapers.org platform which opens new ways to share and validate reproducible research. This was complemented by a second contribution focused on the evaluation of reproducibility, including a survey of recommendations. A final article in this category introduced the torchdistill framework about knowledge distillation in the context of reproducible deep learning research. The second category of contributions focused on reproducible research results and addressed di↵erent topics ranging from intrusion detection, emotion recognition algorithms, and biological structure modeling using tree defect analysis and vesselness filters. All of these contributing papers included source code or online demonstrations, which allow users to reproduce results easily by themselves. The last category included ICPR companion papers describing implementation and details that are an absolute requirement for reproducibility. This set of papers covered several topics such as 3D volume segmentation or properly deploying neural networks. Some of these short papers were awarded the Reproduced Label in Pattern Recognition. Videos of the pre-recorded presentations remain available on the workshop's website: https://rrpr2020.sciencesconf.org In this third edition, we were glad to invite Roberto Di Cosmo to present the Software Heritage project, which aims to establish a perpetual universal archive of software source code. Such an initiative is closely related to reproducibility, given that it ensures the availability of the source code in the long term and keeps important metadata. A second recent initiative in the Computer Graphics domain was presented by Nicolas Bonneel on Code Replicability. The evaluation of code replicability in computer graphics was presented along with http://replicability.graphics, the new platform they propose that encourages code sharing in this domain.

We thank the two invited speakers for the high quality of their presentations and the meaningful discussions with the attendees.

Finally, we would like to thank all researchers who submitted contributions to RRPR and also the reviewers in the scientific committee. We also address a special thank to the IAPR organizers, who have given their support once again to RRPR with the o cial IAPR's endorsement. We also warmly thank the ICPR's general chairs with Rita Cucchiara, Alberto Del Bimbo, and Stan Sclaro↵. And the workshop chairs with Giovanni Maria Farinella and Tao Mei who allowed us to handle these special post-proceedings in parallel to the other several workshops in the main conference. The edition of these post-proceedings was also made easier with the EasyChair platform and the reactivity from Alfred Ho↵man and his successor at Springer-Nature, Ronan Nugent. Finally, we also thank again Audrey Bichet from the MMI department of the IUT of Saint-Diédes-Vosges who made the nice poster of this third RRPR edition. March 12, 2021 Lyon Bertrand Kerautret Miguel Colom Adrien Krähenbühl Daniel Lopresti Pascal Monasse Hugues Talbot