Thomas Bonald

Fernando Diaz

Douglas Eck

Bastien Girschig

Misha Khalman

Preethi Lahoti

Federico Lopez

Marie Pellat

Ludovic Peran

Roberto Rama

Gerard Serra

Michael Terry

Raghuram Vadapalli

Ben Wedin

Céline Comte

Nathan De Lara

Edouard Pineau

Quentin Lutz

Armand Boschin

Simon Delarue

Rajaa El

Antoine Amarilli

Mariam Barry

Nicolas Bouche

Lihu Chen

Cyril Chhun

Jean-Louis Dessalles

Georges Hebrail

Chadi Helwé

Louis Jachiet

Lanfang Kong

Minh Huong

Le Nguyen

Pierre-Henri Paris

Nedeljko Radulovic

Zacchary Sadeddine

Mauro Sozio

Fabian Suchanek

Olivier Buob

François Durand

Fabien Mathieu

Ludovic Noirie

Élie De

First and foremost, I would like to

Résumé

Le traitement automatique du langage naturel est motivé par des applications où les ordinateurs doivent acquérir une compréhension sémantique et syntaxique du langage humain. Récemment, le domaine a été impacté par un changement de paradigme. Les architectures d'apprentissage profond couplées à des techniques d'apprentissage auto-supervisé sont devenues le coeur des modèles correspondant à l'état de l'art en compréhension et génération du langage naturel. Parfois considérés comme des "foundation models", ces systèmes ouvrent la voie à de nouveaux cas d'utilisation. Née d'un partenariat académique et industriel entre l'Institut Polytechnique de Paris et Google AI Research, la présente recherche s'est concentrée sur l'étude de la façon dont les modèles neuronaux de traitement du langage naturel pré-entraînés pouvaient être utilisés pour améliorer les interactions en ligne.

Cette thèse a d'abord exploré comment le transfert de style auto-supervisé pouvait être appliqué à la reformulation non-toxique de commentaires offensants dans les conversations en ligne. Dans le contexte de la modération de contenu toxique en ligne, nous avons proposé une méthode de réglage fin d'un modèle texte-à-texte pré-entraîné (T5) avec une fonction-objectif consistant en un auto-encodeur débruiteur cyclique. Le système, baptisé CAE-T5, a été entraîné sur le plus grand jeu de données de détection de toxicité publé à ce jour (Civil Comments) et génère des phrases plus fluides et préservant mieux le contenu initial, comparé aux systèmes antérieurs de transfert de style de texte, selon plusieurs systèmes d'évaluation automatique et une étude faisant appelle à l'évaluation humaine. De plus, l'approche a montré qu'elle pouvait être généralisée à d'autres tâches de transfert de style, comme le transfert de sentiments.

Ensuite, les travaux de recherche ont porté sur l'étude de l'annotation humaine et la détection automatique des sous-ensembles de mots toxiques dans les conversations en ligne. Contrairement aux jeux de données et aux modèles de détection de toxicité qui classifient des messages entiers comme toxiques ou non, la détection des mots toxiques vise à mettre en évidence les mots responsables de la toxicité du message, lorsqu'une telle détection est possible. Nous avons publié un nouveau jeu de données annoté pour entraîner et évaluer les systèmes automatiques, ce qui a conduit à une tâche partagée lors du 15e International Workshop on Semantic Evaluation. Les systèmes proposés pour cette tâche comprennent des modèles fortement supervisés, entraînés à l'aide d'annotations au niveau des mots, ainsi que des approches faiblement supervisées, connues sous le nom d'extraction de raisons, utilisant des classifieurs entraînés sur des ensembles de données externes, potentiellement plus importants, de v messages annotés manuellement comme toxiques ou non, sans annotations à l'échelle des mots. En outre, le jeu de données et les systèmes se sont avérés utiles pour analyser les performances des systèmes automatiques et des humains en matière de reformulation des messages toxiques en messages civils.

Enfin, nous avons développé un système de recommandation basé sur des avis en ligne, s'inscrivant dans l'explicabilité des préférences prises en compte par les recommandations prédites. La méthode utilise des modèles basés sur la similarité sémantique textuelle pour représenter les préférences d'un utilisateur sous la forme d'un graphe de fragements de texte, où les arrêtes sont définies par la similarité sémantique. Ce modèle de prédiction de notes à mémoire, basé sur le texte, offre la possibilité d'améliorer les explications des recommandations. La méthode est évaluée quantitativement, et nous permet de conclure que l'exploitation du texte de cette manière peut surpasser les performances de modèles de référence utilisé en filtrage collaboratif. vi

Notation

Datasets

V

Vocabulary of |V | tokens (e.g. subwords). We use the same notation for a token and its one-hot encoding.

s Sequence (e.g. sentence) s 1 s 2 . . . s m of m tokens.

x i The i th input data point. In the context of text transfer, the input is called the source text.

y i

The i th target, associated with x i . If produced by a human, the target is called an annotation. If it is a noisy version of the input, the target is called a pseudo-label, otherwise it is referred to as a label. To differentiate the predicted output with the target, the latter is often referred to as ground-truth output. When there is no confusion, the target y is associated with input x.

ŷi The i th predicted output, associated with x i .

Numbers and Arrays

Probability

X

The random variable observed by the x i .

P (X)

The data-generating distribution of X.

P (X)

The empirical probability of X observed by the training set.

P θ (X) The model distribution of X, parametrized by weight vector θ. P (x; θ) The probability P θ (X = x) that X = x.

Ā

The opposite of event A. writing systems [START_REF] Kramer | History Begins at Sumer: Thirty-Nine Firsts in Recorded History[END_REF], ancient Mesopotamians invented a revolutionary concept to spread complex quantitative and qualitative information. Johannes Gutenberg scaled this idea with printing technology, paving the way for the scientific revolution and making general knowledge available to the masses. This trend has been accelerated in the last two centuries with the advent of information and communications technology. On the one hand, logicians and computer scientists constructed formal languages for machines to execute programs. On the other hand, the general public started adapting to computers and designing countless applications that fit their habits. Further, the Internet made people around the world more connected. Nowadays, social networks and online platforms are integral parts of our daily lives. With the democratization of social interactions online, humans are accustomed to sharing their opinions on various topics; people gather and talk at a scale never seen in human history. While online participation offered exciting opportunities to humanity, it also engendered threats and challenges for the twentyfirst century. In this thesis, we addressed some of them in light of recent progress in Artificial Intelligence (AI). In particular, our approach focused on bridging the gap between highly informal human experiences of the world and the formal technologies that computers rely on.

Sequences

Scientific context: recent progress in artificial intelligence prepares the ground for major technological shifts

Research in AI studies agents that have some perception of their environment and take actions in order to achieve one or several goals [START_REF] Poole | Computational Intelligence: A Logical Approach[END_REF] . Intelligence refers to natural intelligence found among animals, and AI focuses on the ability of machines to make decisions automatically. Following Alan Turing's theory of computation, AI foundations have been built up by computer scientists and mathematicians as a formal science. Yet, its very nature makes it deeply connected to many disciplines in social, natural, or applied sciences and this thesis is an example of the interdisciplinarity of AI. Specifically, we narrowed down our exploration to Machine Learning (ML), a statistical and computational paradigm consisting of algorithms improved by the automatic learning of relevant signals from data. ML has become ubiquitous in AI for its ability to train models that yield top performances in perception (e.g., computer vision, natural language processing), information retrieval, robotics or predictive analytics for empirical (i.e., natural and social) sciences, without manual-and often labor-intensive-feature engineering. In its standard "supervised" learning form, the goal is to infer a probability distribution P (y|x; θ) matching unnoisy observations of input x i and their label y i .

In the 2010s, Big Data processing and specialized hardware scaled the applications of a family of ML methods called Deep Learning (DL) [START_REF] Lecun | Deep learning[END_REF], directly inspired by biological neural networks to tackle cognitive functions. The DL revolution started when deep artificial neural networks won large-scale computer vision tasks [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]. Subsequently, neural architectures have been applied to other fields and showed remarkable results. Thus, Natural Language Processing (NLP) research shifted towards neural (connectionist) approaches, and this thesis sought to take part in this scientific trend by exploring new and challenging applications of DL for NLP. In fact, a question central to this work is to what extent could these AI models impact people's behavior in their social interactions online.

Research context: an academic-industrial partnership

The genesis of this Ph.D. results from a collaboration between the team Data, Intelligence and Graphs (DIG) of the Information Processing and Communications Laboratory at Télécom Paris and the Conversation AI team from Jigsaw, a unit within Google that explores global challenges and threats online (disinformation, censorship, cyberbullying and violent extremism), and develops scalable technological solutions for the safety of open societies. The DIG team has a long experience in industrial collaborations, and this research benefited from the synergy produced by the mixed environment. In particular, it combined the scientific rigor of fundamental research with a clear vision of concrete downstream applications.In addition, the research was enriched by collaborations with People + AI Research, a multidisciplinary team at Google Research that explores the human side of AI, and the Department of Informatics at the Athens University of Economics and Business, which has strong expertise in NLP.

Thesis outline

Here is an outline of the remainder of this thesis, along with our scientific contributions. The thesis is organized as follows. First of all, Chapter 2 introduces the state-of-the-art in neural Natural Language Processing. We see in particular how Machine Learning training paradigms, Deep Learning architectures, and Language Models have shaped the current developments of the field. Then Chapter 3 focuses on a specific application of self-supervised text-to-text transfer, namely style transfer. We present an approach to addressing the task when detoxification of abusive comments is considered without a parallel dataset. In Chapter 4, we describe the new task of TOXICSPANS detection, which we tackled thanks to a new annotated dataset and both strongly supervised and weakly supervised (cf. Section 2.4.2) systems. Chapter 5 discusses a novel algorithm to compute rating predictions from the sentences written by users who reviewed a set of items. Finally, Chapter 6 concludes the thesis.

Publications

The research conducted during the Ph.D. resulted in publications listed here in chronological order.

Neural Natural Language Processing Chapter 2 abstract

This chapter describes the current state-of-the-art material and methods to address Natural Language Processing with Machine Learning. The subsequent chapters investigate the concepts described in this chapter in novel methods and/or applications. In what follows, we introduce in Section 2.1 motivations behind Natural Language Processing, applications, and why ML is well suited to tackle them. We then present in Section 2.4

the different natures of data processed by ML and the resulting learning paradigms. Section 2.5 provides an overview of modern training and inference techniques implementing the concepts described beforehand.

Finally, Section 2.6 summarizes existing neural architectures used as computation models for the previously discussed methods.

Motivations

The syntactic nature of computers makes them formal machines

Computers (from Latin "computo", i.e., calculate, count, sum up) are machines designed to execute programssoftware-made of sequences of instructions coded by discrete numbers. These instructions and the data they operate on are expressed in the base-2 numeral system, which optimizes computations on physical hardware.

Science and engineering have improved hardware and software capabilities but have remained on the same basis:

the steps of algorithms are implemented by formal instructions processed by arithmetic logic units. Computers are perfectly fit to make fast calculations and exceed the arithmetic performance of the best mental calculators. This led early computer scientists to dream of machines able to perform other if not all, cognitive functions observed in animals.

Semantics of natural language is structured but ambiguous

From an animal's perspective, there is no need to consciously formalize instructions before executing some complex but intuitive tasks. In particular, humans rarely need to communicate with formal languages. Natural language derives its richness from its complexity and ambiguity. Famous examples of syntactic ambiguity (e.g. "Time flies like an arrow; fruit flies like a banana" [START_REF] Oettinger | The uses of computers in science[END_REF]) or lexical ambiguity (e.g. "Will, will Will will Will Will's will?" [START_REF] Han | On language peculiarities: when language evolves that much that speakers find it strange[END_REF])

are widely used in linguistics to illustrate semantic ambiguity. It is fundamental to human expressions, and that is why it is found throughout the history of literature. Comedy and humor abound in equivocations 1 and word plays such as puns based on homonymy and polysemy. Figures of speech (e.g., metaphors) and rhetorical devices have nourished poetry and argumentations. Tone and register calibrate a speech to specific target audiences or environments. For that matter, context is often crucial to addressing semantic disambiguation. Naturally, specific contexts impose humans to control natural language by simplification rules. For example, the language used by aviators for radio communication is codified to optimize the quality and quantity of information transmitted in a short time and in noisy environments while being fast to learn. Though constructed international auxiliary languages, such as Esperanto, were designed with regular and easy grammar but never replaced complex ordinary local languages.

It is worth noting that spoken language can be irregular and fluctuate in time and space while still achieving its goal of connecting agents. Writing, and a fortiori printing motivated needs for language codification through relatively well-defined spelling and grammar 2 . All in all, there is a gap between the way computers process information and the mechanism behind natural language.

Methods

NLP has been studied since the origin of computer science. Computers cannot straightforwardly process natural language due to incompatibility between formal instructions and ambiguity in natural language. Even nowadays, computers remain far from a general understanding of language. Yet, AI strategies are well adapted to address NLP challenges. Further, Alan Turing designed a test to compare artificial and human intelligence, called the imitation game [START_REF] Turing | Computing machinery and intelligence[END_REF]. The test consists of a human evaluator asked to engage in written conversations with another human on the one hand and a machine on the other hand without knowing the nature of its interlocutor. If the human evaluator cannot tell the machine from the human, then the machine is considered to have passed the test by showing human-equivalent intelligent behavior. The discussion about the link between NLP capabilities and strong AI remains open. In its Chinese room argument, Searle [START_REF] Searle | Minds, brains, and programs[END_REF] argues that an AI could pass the Turing test via a pure syntactic method without any actual understanding of natural language.

Early NLP approaches used handwritten-rule-based systems called symbolic NLP that used to be prerequisite to solving higher-level semantic tasks. Inspired by Chomskyan theories of linguistics [START_REF] Chomsky | Syntactic Structures[END_REF], and grammar teaching in education, symbolic AI combined logic, lexicon, and semantic for NLP. Although this approach based on complex handcrafted feature engineering [START_REF] Lafferty | Conditional random fields: Probabilistic models for segmenting and labeling sequence data[END_REF][START_REF] Och | A smorgasbord of features for statistical machine translation[END_REF][START_REF] Zhang | Transition-based dependency parsing with rich non-local features[END_REF] benefited from total transparency and explainability, it lacked robustness to multifaceted language as well as generalization capabilities. In the 1990s, ML revolutionized NLP thanks to increased computational resources and data as well as the emergence of algorithms learning to solve tasks without explicit specifications of how to do it. As a result, the paradigm shifted toward statistical learning, imitating babies' ability to start language learning well before entering school. Indeed, their brain unconsciously and regularly processes examples of language heard, combined with other sources of signals perceived in their environment.

Applications

In this research we focused on the written form of language. Although speech processing perfectly tackles research efforts to process natural language, we did not investigate it due to the nature of the data that differs from the discrete nature of text. Speech processing is usually not considered as part of NLP since it often involves speech-to-text conversion, a.k.a. Automatic Speech Recognition (ASR) [START_REF] Fan | Unsupervised pre-traing for sequence to sequence speech recognition[END_REF][START_REF] Liao | Generating human readable transcript for automatic speech recognition with pre-trained language model[END_REF], before processing text (NLP); this pipeline may be followed by a Text-to-Speech (TTS) system. The set of NLP tasks is often decomposed into two complementary subsets: Natural Language Understanding and Natural Language Generation.

Natural Language Understanding

The sub-topic of NLP focusing on reading comprehension is called Natural Language Understanding (NLU). Morpho-syntactic and information extraction applications of NLP include token analysis such as lemmatisation [START_REF] Müller | Joint lemmatization and morphological tagging with lemming[END_REF], sequence labelling-Part-Of-Speech (POS) tagging [START_REF] Petrov | A universal part-of-speech tagset[END_REF][START_REF] Nguyen | A robust transformation-based learning approach using ripple down rules for part-of-speech tagging[END_REF] and Named-Entity Recognition (NER) [START_REF] Tjong | Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition[END_REF][START_REF] Huang | Bidirectional lstm-crf models for sequence tagging[END_REF][START_REF] Ma | End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF[END_REF][START_REF] Lample | Neural architectures for named entity recognition[END_REF]-or syntactic parsing [4,[START_REF] Charniak | Statistical techniques for natural language parsing[END_REF], while NLU is often evaluated with higher-level semantic tasks. Standard NLU benchmarks have been proposed to test systems' abilities to understand complex language features. NLU systems can be evaluated on the hidden benchmarks' test sets, and the leaderboards are publicly available. Meta-benchmarks standardize the evaluation of NLU by collecting human-annotated example pairs on a set of tasks reflecting a deep understanding of language. The General Language Understanding Evaluation (GLUE)3 [START_REF] Wang | GLUE: A multi-task benchmark and analysis platform for natural language understanding[END_REF] and its more challenging successor SuperGLUE4 [START_REF] Wang | Superglue: A stickier benchmark for general-purpose language understanding systems[END_REF] respectively aggregate 9 and 8 tasks-one task being common to both benchmarks-to test general language understanding and robustness of systems. Additionally, the SQuAD5 dataset is also used to test NLU. The tasks aim at probing NLU systems on syntactic correctness, semantic similarity, inference, reasoning, as well as question answering. Table 2.1 summarizes and organizes common NLP tasks according to their types and frameworks.

Natural Language Generation There is a set of tasks regarded as Natural Language Generation (NLG). These tasks may or may not imply strong NLU abilities; the boundary between NLU and NLG is not well defined. Unconditional Natural Language Generation (UNLG) [START_REF] Du | All NLP tasks are generation tasks: A general pretraining framework[END_REF][START_REF] Han | Pre-trained models: Past, present and future[END_REF] refers to tasks where NLG has no other constraint than the generated text to be syntactically and semantically correct. If the goal is to generate open-ended text, such as a coherent story, UNLG does not need strong NLU abilities. On the contrary, in prompt completion tasks, UNLG needs some good NLU capabilities for consistency purposes.

When the generated text has to meet additional requirements, NLG is qualified as Conditional Natural Language Generation (CNLG). Controllable generation [START_REF] Keskar | CTRL -A Conditional Transformer Language Model for Controllable Generation[END_REF] is an example of CNLG. In particular, if the condition relies on some input text, the task is called a text-to-text task and involves significant NLU of the input text before generating the output text.

Text-to-text systems Text-to-text systems have been driven by research in machine translation [START_REF] Koehn | Statistical Machine Translation[END_REF][START_REF] Conneau | Cross-lingual language model pretraining[END_REF][START_REF] Dabre | A survey of multilingual neural machine translation[END_REF]. The task represents economic interest for industrial stakeholders, whose research and development laboratories are at the forefront of these technologies. Automatic summarization [START_REF] El-Kassas | Automatic text summarization: A comprehensive survey[END_REF][START_REF] Rothe | Leveraging pre-trained checkpoints for sequence generation tasks[END_REF][START_REF] Zhang | PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization[END_REF] and conversational models (also known as dialogue systems or chatbots) [START_REF] Zhou | The Design and Implementation of XiaoIce, an Empathetic Social Chatbot[END_REF][START_REF] Huang | Challenges in building intelligent open-domain dialog systems[END_REF][START_REF] Zhang | Recent advances and challenges in task-oriented dialog systems[END_REF][START_REF] Zhang | DIALOGPT : Large-scale generative pre-training for conversational response generation[END_REF] are the other two main text-to-text tasks. More modestly, attribute transfer in text like style transfer (cf. Chapter 3) is another text-to-text task recently investigated. It should be noted that, more generally, NLP can be involved in multimodal sequence-to-sequence tasks like image captioning [START_REF] Xia | Xgpt: Cross-modal generative pre-training for image captioning[END_REF][START_REF] Chen | Visualgpt: Data-efficient image captioning by balancing visual input and linguistic knowledge from pretraining[END_REF][START_REF] Yang | Tap: Text-aware pre-training for text-vqa and text-caption[END_REF][START_REF] Sun | Videobert: A joint model for video and language representation learning[END_REF][START_REF] Sun | Contrastive bidirectional transformer for temporal representation learning[END_REF][START_REF] Zhou | Unified vision-language pre-training for image captioning and vqa[END_REF][START_REF] Luo | Univilm: A unified video and language pre-training model for multimodal understanding and generation[END_REF] (image-to-text) and text-to-image generation [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF][START_REF] Ramesh | Hierarchical text-conditional image generation with clip latents[END_REF][START_REF] Saharia | Photorealistic text-to-image diffusion models with deep language understanding[END_REF] with significant results impacting the entire AI field.

NLU and NLG differ in a key aspect: NLU is easier to evaluate than NLG. Automatic metrics are indeed straightforward in NLU: accuracy, F1, and correlations are widely accepted as appropriate evaluation measures of classification tasks. However, standard CNLG metrics [START_REF] Sai | A survey of evaluation metrics used for nlg systems[END_REF] (mainly BLEU [START_REF] Papineni | Bleu: a method for automatic evaluation of machine translation[END_REF], METEOR [START_REF] Lavie | METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments[END_REF] and NIST [START_REF] Lin | Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics[END_REF] in machine translation and ROUGE [START_REF] Lin | ROUGE: A package for automatic evaluation of summaries[END_REF] in summarization) have been abundantly discussed: despite a relative simplicity, these metrics based on n-gram overlaps cannot fully capture the diversity of language for expressing similar semantics with different words, especially when there are few (if any) ground-truth examples per test example available. For this reason, model-based metrics like RUSE [START_REF] Shimanaka | RUSE: Regressor using sentence embeddings for automatic machine translation evaluation[END_REF], BERTScore [START_REF] Zhang | Bertscore: Evaluating text generation with bert[END_REF], BLEURT [START_REF] Sellam | BLEURT: Learning robust metrics for text generation[END_REF] and NUBIA [START_REF] Kane | NUBIA: NeUral based interchangeability assessor for text generation[END_REF] have been recently proposed to overcome the limitations of word-based metrics: for instance, NLU models are used to compare the semantics of pairs of (predicted, target) texts. Chapter 3 and Section 3.9.2 in particular touch upon issues related to CNLG metrics. For its part, UNLG is commonly evaluated with Perplexity (PPL), an informationtheory-based metric measuring how the probability distribution of the generated text is similar to the distribution of text written by humans [START_REF] Brown | An estimate of an upper bound for the entropy of english[END_REF].

All in all, among NLU, UNLG and CNLG, tasks may be tackled by notably different approaches (cf. Section 2.5), but there is no type of task easier than the others. Modern NLP research constantly transposes techniques developed for some tasks to others [START_REF] Yang | Xlnet: Generalized autoregressive pretraining for language understanding[END_REF]. Moreover, there is a significant research effort going on aiming at unifying the resolution of multi-type NLP tasks (cf. Section 2.5.8). Figure 2.1 summarizes the scientific topics introduced so far, their intersections, and the areas where our research falls within.

Artificial Intelligence

Machine

As NLP has gained in maturity, we sought to apply it in novel ways to additional real word challenges focused on the online expressions of Internet users. Before the popularisation of the Web, remote interlocutors expressed either through spontaneous oral language (phone calls) or codified written language (epistolary correspondence).

Nowadays, every person with Internet access gets the opportunity to share their views through posts, comments, or reviews on Web services that we all have integrated into our daily lives. Through online chat and microblogging, not only have people had direct access to global-scale instant messaging, but new forms of language appeared, halfway between oral and written languages; this raises important research issues from the perspective of NLP.

Actually, we will describe specific applications in which modern NLP can impact online human interactions after this chapter discusses methods used in state-of-the-art NLP, summarized in a taxonomy shown in Figure 2.2. Further information can be found in recent high-quality literature reviews [START_REF] Kalyan | Ammus: A survey of transformer-based pretrained models in natural language processing[END_REF][START_REF] Lin | A survey of transformers[END_REF][START_REF] Qiu | Pre-trained models for natural language processing: A survey[END_REF][START_REF] Naseem | A comprehensive survey on word representation models: From classical to state-of-the-art word representation language models[END_REF][START_REF] Li | Pretrained language model for text generation: A survey[END_REF][START_REF] Liu | Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing[END_REF][START_REF] Han | Pre-trained models: Past, present and future[END_REF]. Discrete AutoPrompt [START_REF] Shin | AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts[END_REF] Manual PET [START_REF] Schick | Exploiting cloze-questions for few-shot text classification and natural language inference[END_REF][START_REF] Schick | It's not just size that matters: Small language models are also few-shot learners[END_REF] Fine-tuning ULMFiT [START_REF] Howard | Universal language model fine-tuning for text classification[END_REF], BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF], T5 [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF] Initial learning

Pretraining

Self-supervised PLM BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF], GPT [START_REF] Radford | Improving language understanding by generative pre-training[END_REF][START_REF] Radford | Language models are unsupervised multitask learners[END_REF][START_REF] Brown | Language models are few-shot learners[END_REF], T5 [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF] DAE BART [START_REF] Lewis | Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension[END_REF] Contrastive learning RTD [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF][START_REF] Clark | Pre-training transformers as energy-based cloze models[END_REF], NSP [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF], SOP [START_REF] Lan | Albert: A lite bert for self-supervised learning of language representations[END_REF] Non-contextual word embedding Word2vec [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF][START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF], GloVe [START_REF] Pennington | Glove: Global vectors for word representation[END_REF], Fasttext [START_REF] Grave | Learning word vectors for 157 languages[END_REF] Strongly supervised CoVe [START_REF] Mccann | Learned in translation: Contextualized word vectors[END_REF] Standard training Seq2Seq [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF], Original bitransformer [START_REF] Vaswani | Attention is all you need[END_REF] LM Selfsupervised PLM

Unified

General LM GLM [START_REF] Du | All NLP tasks are generation tasks: A general pretraining framework[END_REF] Pseudomask LM UniLMv2 [START_REF] Bao | Unilmv2: Pseudo-masked language models for unified language model pre-training[END_REF] Seq2Seq MLM MASS [START_REF] Song | Mass: Masked sequence to sequence pre-training for language generation[END_REF], T5 [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF] Permutation LM Mask and Permutation LM MPNet [START_REF] Song | Mpnet: Masked and permuted pre-training for language understanding[END_REF] XLNet [START_REF] Yang | Xlnet: Generalized autoregressive pretraining for language understanding[END_REF] Single-type Bidirectional LM

Masked LM

BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF], RoBERTa [START_REF] Liu | A robustly optimized bert pretraining approach[END_REF], DistilBERT [START_REF] Sanh | Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter[END_REF], ALBERT [START_REF] Lan | Albert: A lite bert for self-supervised learning of language representations[END_REF], DeBERTa [START_REF] He | Debertav3: Improving deberta using electra-style pre-training with gradientdisentangled embedding sharing[END_REF], ERNIE [START_REF] Zhang | Enhanced language representation with informative entities[END_REF][START_REF] Sun | 2.0: A continual pre-training framework for language understanding[END_REF][START_REF] Sun | ERNIE 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation[END_REF] Forwardbackward LM ELMo [START_REF] Peters | Deep contextualized word representations[END_REF] Causal LM GPT [START_REF] Radford | Improving language understanding by generative pre-training[END_REF][START_REF] Radford | Language models are unsupervised multitask learners[END_REF][START_REF] Brown | Language models are few-shot learners[END_REF], Grover [START_REF] Zellers | Defending against neural fake news[END_REF], Megatron-Turing NLG [START_REF] Shoeybi | Megatron-lm: Training multi-billion parameter language models using model parallelism[END_REF][START_REF] Rosset | Turing-nlg: A 17-billion-parameter language model by microsoft[END_REF][START_REF] Narayanan | Efficient large-scale language model training on GPU clusters[END_REF][START_REF] Smith | Using deepspeed and megatron to train megatron-turing NLG 530b, A large-scale generative language model[END_REF], PanGuα [START_REF] Zeng | Pangu-α: Large-scale autoregressive pretrained chinese language models with auto-parallel computation[END_REF], Jurassic-1 [START_REF] Lieber | Jurassic-1: Technical details and evaluation[END_REF], Gopher [START_REF] Rae | Scaling language models: Methods, analysis & insights from training gopher[END_REF], Chinchilla [START_REF] Hoffmann | Training compute-optimal large language models[END_REF], LaMDA [START_REF] Thoppilan | Lamda: Language models for dialog applications[END_REF], PaLM [START_REF] Chowdhery | Palm: Scaling language modeling with pathways[END_REF] Seq2Seq LM

Encoder-Decoder LM

Original bitransformer [START_REF] Vaswani | Attention is all you need[END_REF],

MASS [START_REF] Song | Mass: Masked sequence to sequence pre-training for language generation[END_REF], T5 [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF], PEGASUS [START_REF] Zhang | Pegasus: Pre-training with extracted gap-sentences for abstractive summarization[END_REF], ProphetNet [START_REF] Qi | ProphetNet: Predicting future n-gram for sequence-to-SequencePre-training[END_REF], GShard-M4 [START_REF] Lepikhin | Scaling giant models with conditional computation and automatic sharding[END_REF], Switch bitransformer [START_REF] Fedus | Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity[END_REF] Prefix LM UniLM [START_REF] Dong | Unified language model pre-training for natural language understanding and generation[END_REF] ANN

Transformer

Bitransformer

Original bitransformer [START_REF] Vaswani | Attention is all you need[END_REF], MASS [START_REF] Song | Mass: Masked sequence to sequence pre-training for language generation[END_REF], T5 [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF], BART [START_REF] Lewis | Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension[END_REF] Unitransformer BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF], GPT [START_REF] Radford | Improving language understanding by generative pre-training[END_REF][START_REF] Radford | Language models are unsupervised multitask learners[END_REF][START_REF] Brown | Language models are few-shot learners[END_REF] RNN GRU [START_REF] Chung | Empirical evaluation of gated recurrent neural networks on sequence modeling[END_REF] LSTM BiLSTM ELMo [START_REF] Peters | Deep contextualized word representations[END_REF] Seq2Seq [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF] FFNN CNN USE [START_REF] Cer | Universal sentence encoder[END_REF] FCNN FFLM [START_REF] Bengio | A neural probabilistic language model[END_REF], Word2vec [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] Figure

Learning with signals from the Experience

The three founding paradigms of Machine Learning

Animals learn with their nervous system processing signals coming from their sensory systems. The quality and quantity of knowledge acquired by learning depend on the nature, length, repetition, and context of the experiences.

For example, one can learn a language through immersion, lessons, the use of dictionaries, grammar books, or other media. In the same manner, artificial agents have several ways of experiencing the real world. While animals get signals through their sensory nervous system, artificial agents process information in various forms. Learning algorithms have been traditionally categorized into paradigms depending on the nature of the available signals.

On the one hand, the experience of dataset-based systems is made of a set of observed examples. When examples are only composed of a collection of features represented by x i , the method is known as "unsupervised" and consists in modeling relevant properties of the data-generating distribution P (X). If, in addition to features in x i , an example contains an associated target y i , then learning becomes "supervised" as when a teacher reveals the exact answer to a question. In supervised learning, the goal is to approximate the distribution P (Y |X) of targets y i given the inputs x i . When targets have been produced by humans, they are referred to as "annotations".

On the other hand, "reinforcement learning" is based on direct interactions with an environment. Agents receive a direct reward or penalty for an action taken in the environment. Even though reinforcement learning principles have been explored in NLP [START_REF] Xu | Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach[END_REF][START_REF] Gong | Reinforcement learning based text style transfer without parallel training corpus[END_REF][START_REF] Luo | A dual reinforcement learning framework for unsupervised text style transfer[END_REF][START_REF] Wu | A hierarchical reinforced sequence operation method for unsupervised text style transfer[END_REF][START_REF] Sancheti | Reinforced rewards framework for text style transfer[END_REF][START_REF] Upadhyay | Efficient reinforcement learning for unsupervised controlled text generation[END_REF][START_REF] Luketina | A survey of reinforcement learning informed by natural language[END_REF], we did not investigate it during the present research.

Interestingly, current ML may combine ideas from different paradigms. Weakly supervised learning comprises dataset-based experiences in between unsupervised and supervised learning, the latter being sometimes called "strongly supervised learning" to avoid confusion.

Limitations of strong supervision spur weak supervision

As strong supervision [START_REF] Kotsiantis | Supervised machine learning: A review of classification techniques[END_REF]

Inaccurate supervision

Targets are collected either automatically or manually. On the one hand, examples (x i , y i ) can be pre-processed automatically from existing databases: input text may be classified according to their metadata like their source, tags, or ratings. Similarly, parallel corpora of translated legislation are used in machine translation. While offering good scalability characteristics, automatic pre-processing is prone to errors. On the other hand, manual annotation requires humans to examine inputs before carefully assigning some label(s). Though human experts often yield high-quality annotations, scaling expert annotation is expensive. That is why datasets are often labeled by crowdworkers through crowdsourcing platforms such as Appen 7 or Amazon Mechanical Turk8 . As crowdworkers are paid pro rata with the number of examples annotated, their incentives lean towards quantity rather than quality. Furthermore, ambiguity, context [START_REF] Xenos | Toxicity detection sensitive to conversational context[END_REF] or audience dependency may make some examples subject to subtle interpretation and misunderstood by crowdworkers if their reference frame differs too much from that of the target audience. For example, pejorative American idioms "Monday-morning quarterback" 9 and "carpetbagger"10 may be confusing for a native English-speaking crowdworker with Indian cultural background. All in all, noise in automatic or manual targets may make supervision inaccurate.

Inexact supervision Properties of language can appear at different granularities: subwords, words, phrases, clauses, sentences, paragraphs, etc. Splitting text into independent parts is a complex task due to semantic consistency that can be long-term. Therefore, an input may contain more text than the subset of text (called "span") actually responsible for explaining its labeling. For instance, datasets made of review-rating pairs [START_REF] He | Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering[END_REF][START_REF] Kaggle | Yelp dataset[END_REF] may contain long reviews stating positive and negative aspects but associated with one single positive or overall negative rating of the reviewed item. Finding local characteristics in long texts is at the heart of NLP research on interpretability and explainability [START_REF] Ribeiro | Explaining the predictions of any classifier[END_REF][START_REF] Li | Understanding neural networks through representation erasure[END_REF][START_REF] Narang | Wt5?! training text-to-text models to explain their predictions[END_REF][START_REF] Miao | Snippext: Semi-Supervised Opinion Mining with Augmented Data[END_REF][START_REF] Carmeli | Constructing explainable opinion graphs from reviews[END_REF][START_REF] Suhara | OpinionDigest: A simple framework for opinion summarization[END_REF] and is at the core of Chapter 4.

Incomplete supervision Some ML approaches can leverage a set of labeled and unlabeled examples. In particular, data augmentation [START_REF] Fadaee | Data augmentation for low-resource neural machine translation[END_REF] is a trick used to increase the size of the training dataset automatically. In NLP, data augmentation can be materialized by synonym replacement [START_REF] Wei | EDA: Easy data augmentation techniques for boosting performance on text classification tasks[END_REF] or cyclic translation [START_REF] Sugiyama | Data augmentation using back-translation for context-aware neural machine translation[END_REF] (cf. Chapter 3). Active [START_REF] Settles | Active learning literature survey[END_REF] and semi-supervised [593] learning are well-known approaches described by [START_REF] Zhou | A brief introduction to weakly supervised learning[END_REF] though barely applied to NLP. Actually, as it exploits both unlabeled and labeled data, incomplete supervision echoes a major breakthrough in recent NLP training schemes, called self-supervised learning [START_REF] Liu | Self-supervised learning: Generative or contrastive[END_REF].

Self-supervision

Open challenges in AI that are keys to animal intelligence are the development of commonsense knowledge [START_REF] Dessalles | Des intelligences très artificielles[END_REF] and transfer of learning [START_REF] Perkins | Transfer of learning[END_REF]. ML algorithms have proved quite efficient at overfitting individual lowlevel tasks, provided they have enough various labeled sample data. Yet, strong supervision hinders generalization because it cannot guarantee pitfalls that would not be covered by the training data. Indeed, collecting datasets of labeled data large and diverse enough to cover the space of all possibilities is illusory and does not correspond to processes observed in biological learning.

Supervisory signals from unlabeled data: a goldmine for Natural Language Processing?

Even if the terminology of dataset-based ML is open to debate and may seem idle at first, we think it raises fundamental questions on how ML can improve from a biomimetics perspective. Historically, the presence of targets differentiates supervised learning from unsupervised learning. The supervised signal was considered to come exclusively from the target, and unsupervised algorithms were designed to discover structures or patterns in the input features, e.g., anomaly detection, correlations (PCA [START_REF] Liii | on lines and planes of closest fit to systems of points in space[END_REF][START_REF] Hotelling | Analysis of a complex of statistical variables into principal components[END_REF][START_REF] Jolliffe | Principal component analysis for special types of data[END_REF]), cluster and community detection (k-means [START_REF] Macqueen | Some methods for classification and analysis of multivariate observations[END_REF][START_REF] Lloyd | Least squares quantization in pcm[END_REF][START_REF] Forgy | Cluster analysis of multivariate data: efficiency versus interpretability of classifications[END_REF]). Nevertheless, some untagged data sources like language hold far more "supervisatory signal" [START_REF] Lecun | Self-supervised learning: The dark matter of intelligence[END_REF] than others (like physical measurement in time or space). Thus, the diptych labeled / unlabeled learning is a proposition 11 that could clarify the origin of the supervisory signal. Take the following sentences: s train = "People often stop watching a movie that they consider bad, before it ends." and s test = "I stopped watching this movie after only 1 hour". Now, consider the task of sentiment analysis [START_REF] Pang | Thumbs up? sentiment classification using machine learning techniques[END_REF][START_REF] Socher | Recursive deep models for semantic compositionality over a sentiment treebank[END_REF][START_REF] Zhang | Deep learning for sentiment analysis: A survey[END_REF], where the system has access to unlabeled training sentences including s train and is queried with inferring the sentiment in s test . Even without direct access to external labels, self-supervised learning introduces objective functions to capture the supervisory structure found in the training data (which may implicitly or explicitly contain labels, cf.

Section 2.5.3) 12 . Besides, it has recently played a crucial role in the development of AI systems as described in Section 2.5. Self-supervision is even qualified as "the next frontier of AI" by LeCun and Ishan [START_REF] Lecun | Self-supervised learning: The dark matter of intelligence[END_REF], revealing the "dark matter of intelligence".

In conclusion, we saw that ML could process signals in various ways. The original supervised/unsupervised/reinforcement learning paradigms have been extended to a richer set of in-between setups, resembling real-world conditions. The following chapters derive examples of weakly supervised systems attempting to solve practical problems raised by online interactions.

Training and inference

Supervised training

The traditional approach of supervised learning is well described with probabilistic learning. The true conditional label distribution P (Y |X) (derived from the data generating distributions P (Y, X) and P (X)) is approximated by a parametric family of model distributions P θ (Y |X). Learning consists in tuning the weight θ by optimizing the value of a loss (or objective) function L so that the optimal model distribution P θ (Y |X) gets "closest" to the empirical conditional label distribution P (Y |X), defined by the training set. The maximum likelihood estimator [START_REF] Fisher | On an absolute criterion for fitting frequency curves[END_REF] with regard to the observed examples (x i , y i ) is often considered a good choice for θ because of its statistical properties.

Beyond ambiguity and context dependency (cf. Section 2.1.2), natural language carries another important specificity: it is made of variable length sequences of discrete tokens. In NLU, sequentiality appears in the input text x modeled as sequences of tokens x 1 , x 2 , . . . , x m . In NLG, the output is a variable-length sequence of tokens y 1 , y 2 , . . . , y p . NLG is often viewed as a multiclass classification task with an infinite number of classes: the possible classes are all the tuples made of tokens from the vocabulary V .

The discrete nature of language makes it particularly suited to supervised probabilistic classification, whose goal is to learn a parametric function f θ -called the model-that predicts class ŷ given an input x. For instance, f θ may correspond to the optimal decision rule, i.e. ŷ = f θ (x) = arg max y P (y|x; θ) [START_REF] Bishop | Pattern recognition and machine learning[END_REF]. In classification, maximum likelihood (L) estimation is equivalent to minimizing the cross-entropy loss (H)-a function grounded in information theory [START_REF] Cover | Elements of information theory[END_REF]-or, alternatively, the statistical distance known as the Kullback-Leibler divergence D KL [START_REF] Kullback | On Information and Sufficiency[END_REF]. The optimization problem can be formalized in the equivalent formula given in Equation (2.1).

θ = arg min

θ i L (x i , y i ; θ) = arg min θ H P (Y |X) , P θ (Y |X) ≜ arg min θ E P (Y |X) [-log (P θ (Y |X))] = arg min θ D KL P (Y |X) ||P θ (Y |X) ≜ arg min θ E P (Y |X) log P (Y |X) P θ (Y |X) = arg max θ L (θ| (x i , y i )) ≜ arg max θ P ((y i ) i | (x i ) i ; θ) = arg max θ i P (y i |x i ; θ) (2.1)

Language Representation Learning

Representation Learning [START_REF] Bengio | Representation learning: A review and new perspectives[END_REF] is an important ML concept at the core of the conception of neural architectures (cf.

Section 2.6). Neural classifiers models successively apply linear and non-linear transformations to input features in order to make data points easily separable. Transformations generate internal representations of inputs and outputs that may acquire relevant algebraic properties. For instance, linear separation in Euclidean space is particularly intuitive (cf. Figure 2.4).

x 1

x 2 Raw data

• • • • • • × × × × × × Neural Network (f θ ) h 1 h 2 Learned features • • • • • • × × × × × × Figure 2
.4: 2D-illustration of the goal behind learning "good" representations with an Artificial Neural Network. Red crosses and blue dots are data points belonging to one class or another. Representation (or feature) learning aims at making these classes easily distinguishable in a latent space.

In NLP, we are interested in relevant numerical representations of text [START_REF] Hinton | Distributed Representations[END_REF][START_REF] Bengio | A neural probabilistic language model[END_REF]. A common approach to language Representation Learning is text embedding, in which we aim at embedding chunks of text in some vector space where semantic and syntactic properties are appropriately distributed. The granularity of the distributed representation may vary [START_REF] Le | Distributed representations of sentences and documents[END_REF][START_REF] Kiros | Skip-thought vectors[END_REF][START_REF] Melamud | context2vec: Learning generic context embedding with bidirectional LSTM[END_REF] but the leading approach consists in embedding each word (or subwords [START_REF] Sennrich | Neural machine translation of rare words with subword units[END_REF][START_REF] Kudo | SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing[END_REF][START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF]) with a parametric embedding matrix θ Emb of size d Emb × |V |. The embedding matrix can be used as standalone for text analysis. Word embedding actually predates the neural NLP era. Still, an embedding matrix often constitutes the first-and sometimes only-layer of a neural model.

Self-supervised Representation Learning

The goal of self-supervised Representation Learning is to automatically extract general syntactic and semantic knowledge (like common sense) from tremendous quantities of unlabeled data. NLP has the characteristics of having easy access to a huge amount of raw, rich, and various datasets online. Unlabeled text of diverse nature and quality may be present in online encyclopediae [START_REF] Stephen | Pointer sentinel mixture models[END_REF] (in 2022, Wikipedia 13 had ∼ 29 billion words, ∼ 3.9 billion of which are in the English version), news datasets [START_REF] Jozefowicz | Exploring the limits of language modeling[END_REF] (Google News dataset has ∼ 100 billion words), books corpora [START_REF] Zhu | Aligning books and movies: Towards story-like visual explanations by watching movies and reading books[END_REF] (∼ 800 million words in the Toronto Books Corpus), web crawl (scrape) corpora [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF][START_REF] Radford | Language models are unsupervised multitask learners[END_REF][START_REF] Brown | Language models are few-shot learners[END_REF][START_REF] Chowdhery | Palm: Scaling language modeling with pathways[END_REF] (Common Crawl 14 has hundreds of billions of words) or social media conversations 1516 [2] (LaMDA [START_REF] Thoppilan | Lamda: Language models for dialog applications[END_REF] was trained on ∼ 1.56 trillion words from public dialogs).

Knowledge may be stored in explicit or implicit forms. For instance, the sentence "1961 was the last year in which the majority of Hollywood films were released in black and white." [START_REF]Black and white -Wikipedia, the free encyclopedia[END_REF] explicitly states a historical fact related to cinematography. It also participates in the commonsense knowledge that black and white movies preceded color movies, and the shift happened in the mid-twentieth century. More implicitly, the sentence "In December 2015, Moore announced his support for Vermont Senator Bernie Sanders in the 2016 United States presidential election." [START_REF]Michael moore -Wikipedia, the free encyclopedia[END_REF] refers to American politics to indicate the political views of director Michael Moore. Further text detailing Senator Bernie Sanders' political positions may add valuable background to understanding Michael Moore's work. In the following sections, a corpus used for self-supervision will be represented by a sequence of unlabeled tokens w 1 , w 2 , . . . , w q with q > 10 9 in practice.

Non-contextual self-supervised word embeddings Implementations of self-supervised Representation Learning in NLP are based on the distributional hypothesis proposed by linguists: words are context-dependent. This principle has been summarized by "You shall know a word by the company it keeps" in Firth's work on collocational meaning [START_REF] Haas | 35s[END_REF]. First uses of unlabeled text to learn word representations were introduced by Collobert and Weston [START_REF] Collobert | A unified architecture for natural language processing: Deep neural networks with multitask learning[END_REF], Turian et al. [START_REF] Turian | Word representations: A simple and general method for semisupervised learning[END_REF] and Collobert et al. [START_REF] Collobert | Natural language processing (almost) from scratch[END_REF]. It became dominant in NLP when latent semantic analysis revealed powerful properties learned by the self-supervised Word2vec [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF][START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF][START_REF] Rubinstein | How well do distributional models capture different types of semantic knowledge[END_REF][START_REF] Gupta | Distributional vectors encode referential attributes[END_REF], GloVe [START_REF] Pennington | Glove: Global vectors for word representation[END_REF] and Fasttext [START_REF] Grave | Learning word vectors for 157 languages[END_REF].

The key idea was to design a loss function forcing the encoding of semantic similarity between two words. Cosine similarity between the word embeddings was chosen as a proxy for semantic similarity.

Word2vec

In NLP, self-supervision is put in practice with appropriate tasks designed to learn semantic similarity from the context in which words occur. Mikolov et al. [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] proposed two efficient objectives to make a model learn word embeddings from the contexts they appear in: continuous bag-of-words and skip-gram. In both objectives, training consists in the classification task of maximizing the likelihood P (y|x; θ) where x is the input text and y is the pseudo-label text. The principle consists in making a fixed-size window slide along the entire corpus w 1 , w 2 , . . . , w q and using the tokens in the window to build x and y. Let 2×c+ 1 be the window size. The window is made of a target token w t at its center and a set of context tokens w t-c , w t-c+1 , . . . , w t-2 , w t-1 , w t+1 , w t+2 , . . . , w t+c-1 , w t+c .

Continuous bag of words

In the continuous bag-of-word (CBOW) version, the input is made of the context tokens, and the pseudo-label is the target token.

Skip-gram

Conversely, skip-gram considers the target token to be the input and the context tokens to be the pseudo-label.

A word of caution: even though here the self-supervised training considers words in their context, the learned embeddings are called "non-contextual" or "static" because they are stored in a fixed d Emb × |V |-matrix, later used to embed words independently of the context they appear in. Incidentally, cosine similarity has also been applied later for sentence similarity [START_REF] Schwenk | Learning joint multilingual sentence representations with neural machine translation[END_REF][START_REF] Cer | Universal sentence encoder for English[END_REF] and we leverage "sentence embeddings" in the subsequent chapters (cf. Chapter 3 and Chapter 5).

Stack structures

In neural NLP, the backbone structures of models can be classified according to the type of task they are mainly designed for (cf. Section 2.3). Layers fulfilling a specific function are grouped into what is called a stack. Neural NLP developments have juggled the single-stack and multi-stack structures we present hereunder.

Single-stack structures A system mainly designed for NLU is called an encoder as its function is to encode discrete representation of text x into a latent representation z = f θ Enc (x). The goal is to encode the semantics of the input in a continuous distributed space. z can be a single vector representing the entire input x = x 1 , x 2 , . . . , x m or a sequence of m token representations. BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF], its derivatives [START_REF] Liu | A robustly optimized bert pretraining approach[END_REF][START_REF] Sanh | Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter[END_REF][START_REF] Lan | Albert: A lite bert for self-supervised learning of language representations[END_REF][START_REF] He | Debertav3: Improving deberta using electra-style pre-training with gradientdisentangled embedding sharing[END_REF], XLNet [START_REF] Yang | Xlnet: Generalized autoregressive pretraining for language understanding[END_REF], ELECTRA [START_REF] Clark | Pre-training transformers as energy-based cloze models[END_REF] and ERNIE [START_REF] Zhang | Enhanced language representation with informative entities[END_REF][START_REF] Sun | 2.0: A continual pre-training framework for language understanding[END_REF][START_REF] Sun | ERNIE 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation[END_REF] are examples of modern NLU systems modeled with encoders.

NLG-oriented (and in particular UNLG-designed) systems are often modeled with a stack called decoder, whose role is to decode internal dense representations z in discrete sequences of tokens y. Decoders have been used to model NLG systems such as GPT [START_REF] Radford | Improving language understanding by generative pre-training[END_REF][START_REF] Radford | Language models are unsupervised multitask learners[END_REF][START_REF] Brown | Language models are few-shot learners[END_REF], Grover [START_REF] Zellers | Defending against neural fake news[END_REF], Megatron-Turing NLG [START_REF] Shoeybi | Megatron-lm: Training multi-billion parameter language models using model parallelism[END_REF][START_REF] Rosset | Turing-nlg: A 17-billion-parameter language model by microsoft[END_REF][START_REF] Narayanan | Efficient large-scale language model training on GPU clusters[END_REF][START_REF] Smith | Using deepspeed and megatron to train megatron-turing NLG 530b, A large-scale generative language model[END_REF], PanGuα [START_REF] Zeng | Pangu-α: Large-scale autoregressive pretrained chinese language models with auto-parallel computation[END_REF], Jurassic-1 [START_REF] Lieber | Jurassic-1: Technical details and evaluation[END_REF], Gopher [START_REF] Rae | Scaling language models: Methods, analysis & insights from training gopher[END_REF], Chinchilla [START_REF] Hoffmann | Training compute-optimal large language models[END_REF], LaMDA [START_REF] Thoppilan | Lamda: Language models for dialog applications[END_REF] and PaLM [START_REF] Chowdhery | Palm: Scaling language modeling with pathways[END_REF]. If combined with an encoder, the decoder is trained to maximize the likelihood P (y|z; θ). In language modeling (cf. Section 2.5.7), generating tokens of ŷ is then realized via sampling the model distribution according to some decision rule (cf. Section 2.5.7).

What is more, since natural language has properties of symbolic time series [START_REF] Lin | A symbolic representation of time series, with implications for streaming algorithms[END_REF], NLG is aptly addressed by autoregressive decoders. Auto-regressive generation consists in generating one token at a time with a feedback loop at each time step. The iterative process stops when a special End-Of-Sequence token <EOS> is reached.

There are two key properties that differentiate the design of encoders and decoders. First, encoders have access to the entire input text x. Therefore, they a priori know the sequence length, while decoders cannot know in advance the length of the sequence y they will generate, especially when generation is auto-regressive. The sequence length variability is a concept characterizing NLP as opposed to computer vision, where images have a fixed set of pixels.

Second, by having initial access to the entire input x, encoders can process the entire sequence in any order. On the contrary, auto-regressive decoders may only process information coming from previous tokens.

Multi-stack structures

In addition to single-stack structures, multi-stack structures have been proposed. The most common multi-stack structure is the double stack. It is made of an encoder and a decoder (cf. Figure 2.5).

Encoder-decoders are general structures proposed in sequence-to-sequence learning [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF]. In particular, they are adapted to text-to-text tasks, where learning a suitable representation z in a latent space is sought-after. For example, disentanglement can be a good latent space property when we aim at transferring some text attribute from one subspace to another. Actually, latent representation learned with strongly supervised learning was the main idea of Sutskever et al. [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF]. More recently, encoder-decoder structures have been employed by the bitransformer [START_REF] Vaswani | Attention is all you need[END_REF], MASS [START_REF] Song | Mass: Masked sequence to sequence pre-training for language generation[END_REF], T5 [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF], PEGASUS [START_REF] Zhang | Pegasus: Pre-training with extracted gap-sentences for abstractive summarization[END_REF], BART [START_REF] Lewis | Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension[END_REF], ProphetNet [START_REF] Qi | ProphetNet: Predicting future n-gram for sequence-to-SequencePre-training[END_REF], GShard-M4 [START_REF] Lepikhin | Scaling giant models with conditional computation and automatic sharding[END_REF] and the switch bitransformer [START_REF] Fedus | Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity[END_REF].

The number, nature, and assembly of stacks in neural NLP is only limited by computational resources, optimization issues, and the imagination of their developers. For example, Baziotis et al. [START_REF] Baziotis | SEQˆ3: Differentiable sequence-to-sequenceto-sequence autoencoder for unsupervised abstractive sentence compression[END_REF] proposed to combine two encoder-decoders, i.e. a quadruple-stack structure, for self-supervised abstractive summarization.

Note that the structure-function mapping proposed above is flexible. Indeed, several works have used singlestack structures for text-to-text [START_REF] Brown | Language models are few-shot learners[END_REF][START_REF] Dong | Unified language model pre-training for natural language understanding and generation[END_REF][START_REF] Bao | Unilmv2: Pseudo-masked language models for unified language model pre-training[END_REF][START_REF] Du | All NLP tasks are generation tasks: A general pretraining framework[END_REF] while others introduced encoder-decoders for a set of applications including NLU-only tasks [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF].

Strongly supervised NLP models learn input representations for the task they are trained on. Yet, since neural models tend to overfit when not appropriately regularized, representations may be too task-specific. In contrast, we may be looking for more general representations to strengthen transfer learning [START_REF] Stevo | The influence of pattern similarity and transfer learning upon the training of a base perceptron b2[END_REF][START_REF] Thrun | Lifelong Learning Algorithms[END_REF][START_REF] Bozinovski | Reminder of the first paper on transfer learning in neural networks[END_REF][START_REF] Thrun | Learning to learn[END_REF][START_REF] Pan | A survey on transfer learning[END_REF][START_REF] Ando | A high-performance semi-supervised learning method for text chunking[END_REF][START_REF] Argyriou | Multi-task feature learning[END_REF][START_REF] Dai | Co-clustering based classification for out-of-domain documents[END_REF][START_REF] Raina | Self-taught learning: Transfer learning from unlabeled data[END_REF][START_REF] Lawrence | Learning to learn with the informative vector machine[END_REF][START_REF] Evgeniou | Regularized multi-task learning[END_REF][START_REF] Bonilla | Multi-task gaussian process prediction[END_REF][START_REF] Gao | Knowledge transfer via multiple model local structure mapping[END_REF] properties of models, inspired by human abilities to solve new tasks by leveraging knowledge acquired on related problems seen in the past. In fact, pretraining a model on a self-supervised task is an efficient (and perhaps sufficient) approach to teach general knowledge that will be used to solve downstream tasks [START_REF] Goodfellow | Deep Learning[END_REF].

Self-supervised Encoder-Decoders

Auto-encoders Auto-Encoder (AE) [START_REF] Yann | Modeles connexionnistes de lapprentissage[END_REF][START_REF] Bourlard | Auto-association by multilayer perceptrons and singular value decomposition[END_REF][START_REF] Kramer | Nonlinear principal component analysis using autoassociative neural networks[END_REF][START_REF] Liou | Modeling word perception using the elman network[END_REF][START_REF] Hinton | Autoencoders, minimum description length and helmholtz free energy[END_REF][START_REF] Hinton | Transforming auto-encoders[END_REF][START_REF] Liou | Autoencoder for words[END_REF][START_REF] Goodfellow | Deep Learning[END_REF][START_REF] Géron | Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow: Concepts, tools, and techniques to build intelligent systems[END_REF] are models traditionally used for learning representations (a.k.a. feature learning) from unlabeled data. The hack is to train an encoder-decoder, like in the strong supervised setup, except that the label y is the input x (cf. Figure 2.5) or a subset of x's features.

Auto-Encoders illustrate the terminology point discussed in Section 2.4.2. Indeed, Auto-Encoders deal with raw unlabeled input text but are trained with a pseudo-label built automatically from the input. Neither unsupervised learning nor strong supervision faithfully correspond to this situation where the supervisory signal comes from the input itself.

If unconstrained, an Auto-Encoder could simply learn the identity function, which is useless. Therefore, training and modeling techniques have been developed to force the decoder to reconstruct the input x from a compressed salient encoding. Several such means have been introduced such as Variational Auto-Encoders (VAE) [START_REF] Kingma | Auto-encoding variational bayes[END_REF][START_REF] Kingma | An introduction to variational autoencoders[END_REF] and concrete Auto-Encoders [START_REF] Abid | Concrete autoencoders for differentiable feature selection and reconstruction[END_REF] but the most efficient ones for NLP are specific regularized auto-encoders, called the Denoising Auto-Encoders (DAE).

Denoising Auto-Encoders Bengio et al. [START_REF] Bengio | Generalized denoising auto-encoders as generative models[END_REF] and Alain and Bengio [START_REF] Alain | What regularized auto-encoders learn from the data-generating distribution[END_REF] showed that training an Auto-Encoder to reconstruct a corrupted version x of the input x makes the model learn rich representations for x [START_REF] Vincent | Extracting and composing robust features with denoising autoencoders[END_REF][START_REF] Vincent | Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion[END_REF].

Corruption alters the input data by sampling x from a noise distribution given x, C X|x . Then, learning P (x|x; θ) teaches the model to capture important information from x while ignoring the noise (cf. Figure 2.5). Glorot et al. [START_REF] Glorot | Domain adaptation for large-scale sentiment classification: A deep learning approach[END_REF] proved the relevancy of DAEs for NLP by using them in sentiment analysis. Common text corrupting strategies include single-token or multi-token span replacement with a special mask <M> token [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Liu | A robustly optimized bert pretraining approach[END_REF][START_REF] Lan | Albert: A lite bert for self-supervised learning of language representations[END_REF][START_REF] He | Debertav3: Improving deberta using electra-style pre-training with gradientdisentangled embedding sharing[END_REF][START_REF] Zhang | Enhanced language representation with informative entities[END_REF][START_REF] Sun | 2.0: A continual pre-training framework for language understanding[END_REF][START_REF] Sun | ERNIE 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation[END_REF][START_REF] Lewis | Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension[END_REF][START_REF] Clark | Pre-training transformers as energy-based cloze models[END_REF][START_REF] Song | Mass: Masked sequence to sequence pre-training for language generation[END_REF][START_REF] Zhang | Pegasus: Pre-training with extracted gap-sentences for abstractive summarization[END_REF][START_REF] Conneau | Cross-lingual language model pretraining[END_REF][START_REF] Dong | Unified language model pre-training for natural language understanding and generation[END_REF] (cf. Section 2.5.8), a sequence of sentinel tokens <X>, <Y>, . . . [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF] or tokens randomly sampled from the vocabulary [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF]. Other noising functions have been introduced such as random token deletions (dropping) [START_REF] Lewis | Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension[END_REF], permutations (shuffling) [START_REF] Yang | Xlnet: Generalized autoregressive pretraining for language understanding[END_REF][START_REF] Lewis | Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension[END_REF][START_REF] Bao | Unilmv2: Pseudo-masked language models for unified language model pre-training[END_REF], capitalization [START_REF] Liu | Finbert: A pre-trained financial language representation model for financial text mining[END_REF], truncations or else rotations.

In NLP, training with a Denoising Auto-Encoder is sometimes referred to as Full Text Reconstruction (FTR), as opposed to Corrupted Text Reconstruction (CTR, cf. Section 2.5.8) [START_REF] Liu | Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing[END_REF].

Denoising Auto-Encoders can be trained to learn transferable representations of the input as well as the output in generation tasks. Nevertheless, without further and stronger supervisory signals, they cannot a priori solve downstream tasks. As a matter of fact, the learned representations may be too general with too much information lost by compression. Downstream tasks may only need a subset of the semantics included in the input or output text, hence the need of "domain adaptation". Thus, the complete training of a model can be decomposed into two consecutive optimization phases: pretraining a subset of (or all) parameters with self-supervision before fine-tuning the entire model on strongly supervised downstream task(s) [START_REF] Hinton | Reducing the dimensionality of data with neural networks[END_REF]. Note that there is a current research trend trying to get rid of the fine-tuning hypothesis, which will be detailed in Section 2.5.9.

The pretraining/fine-tuning pipeline

Pretraining Even if it is possible to pretrain NLP models with strong supervision [START_REF] Mccann | Learned in translation: Contextualized word vectors[END_REF], self-supervised pretraining has prevailed in neural NLP for the reasons given in Section 2.4.3. Non-contextual self-supervised word embeddings (cf. Section 2.5.3) have been used as representation features in NLP models. Whether pretrained or not, word embedding often makes up the first layer of a neural model; Turian et al. [START_REF] Turian | Word representations: A simple and general method for semisupervised learning[END_REF] showed the benefits of pretrained word embeddings over random initialization.

Pretrained model Additional self-supervised pretraining tasks have been proposed to pretrain the entire model rather than the initial layer [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF][START_REF] Pennington | Glove: Global vectors for word representation[END_REF][START_REF] Peters | Deep contextualized word representations[END_REF]. Dai and Le [START_REF] Dai | Semi-supervised sequence learning[END_REF], Liu et al. [START_REF] Liu | Recurrent neural network for text classification with multi-task learning[END_REF] and Howard and Ruder [START_REF] Howard | Universal language model fine-tuning for text classification[END_REF] were among the first to impulse the now widely-used trend of full-model pretraining in NLP [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Liu | A robustly optimized bert pretraining approach[END_REF][START_REF] Sanh | Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter[END_REF][START_REF] Lan | Albert: A lite bert for self-supervised learning of language representations[END_REF][START_REF] He | Debertav3: Improving deberta using electra-style pre-training with gradientdisentangled embedding sharing[END_REF][START_REF] Yang | Xlnet: Generalized autoregressive pretraining for language understanding[END_REF][START_REF] Clark | Pre-training transformers as energy-based cloze models[END_REF][START_REF] Zhang | Enhanced language representation with informative entities[END_REF][START_REF] Sun | 2.0: A continual pre-training framework for language understanding[END_REF][START_REF] Sun | ERNIE 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation[END_REF][START_REF] Radford | Improving language understanding by generative pre-training[END_REF][START_REF] Radford | Language models are unsupervised multitask learners[END_REF][START_REF] Brown | Language models are few-shot learners[END_REF][START_REF] Zellers | Defending against neural fake news[END_REF][START_REF] Shoeybi | Megatron-lm: Training multi-billion parameter language models using model parallelism[END_REF][START_REF] Rosset | Turing-nlg: A 17-billion-parameter language model by microsoft[END_REF][START_REF] Narayanan | Efficient large-scale language model training on GPU clusters[END_REF][START_REF] Smith | Using deepspeed and megatron to train megatron-turing NLG 530b, A large-scale generative language model[END_REF][START_REF] Zeng | Pangu-α: Large-scale autoregressive pretrained chinese language models with auto-parallel computation[END_REF][START_REF] Lieber | Jurassic-1: Technical details and evaluation[END_REF][START_REF] Rae | Scaling language models: Methods, analysis & insights from training gopher[END_REF][START_REF] Hoffmann | Training compute-optimal large language models[END_REF][START_REF] Thoppilan | Lamda: Language models for dialog applications[END_REF][START_REF] Chowdhery | Palm: Scaling language modeling with pathways[END_REF][START_REF] Song | Mass: Masked sequence to sequence pre-training for language generation[END_REF][START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF][START_REF] Zhang | Pegasus: Pre-training with extracted gap-sentences for abstractive summarization[END_REF][START_REF] Lewis | Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension[END_REF][START_REF] Qi | ProphetNet: Predicting future n-gram for sequence-to-SequencePre-training[END_REF][START_REF] Lepikhin | Scaling giant models with conditional computation and automatic sharding[END_REF][START_REF] Fedus | Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity[END_REF][START_REF] Brown | Language models are few-shot learners[END_REF][START_REF] Dong | Unified language model pre-training for natural language understanding and generation[END_REF][START_REF] Bao | Unilmv2: Pseudo-masked language models for unified language model pre-training[END_REF][START_REF] Du | All NLP tasks are generation tasks: A general pretraining framework[END_REF], in order to increase transfer learning abilities. Initially, the original denoising autoencoding objective (cf.Section 2.5.5) was used for self-supervised pretraining. Alternatively, contrastive learning [START_REF] Saunshi | A theoretical analysis of contrastive unsupervised representation learning[END_REF] is another approach to self-supervised pretraining. This classification task consists of learning which pairs of text are comparable. Examples of contrastive learning include Negative Sampling [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF], Replaced Token Detection (RTD) [START_REF] Clark | Pre-training transformers as energy-based cloze models[END_REF], Next Sentence Prediction (NSP) [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] and Sentence Order Prediction (SOP) [START_REF] Lan | Albert: A lite bert for self-supervised learning of language representations[END_REF]. Yet, the dominant approach to self-supervised pretraining heavily relies on Language Models, introduced in Section 2.5.7.

Fine-tuning After pretraining, a second phase is often needed to adapt and transfer the knowledge acquired at pretraining toward more restrained problem(s). Strong supervision on few 17 labeled examples are used to finetune (update) the model's parameters with an objective focused on one or several specific downstream task(s).

Despite significantly improving the experimental results on downstream tasks, there is no clear theoretical grounding regarding pretraining benefits. Goodfellow et al. [START_REF] Goodfellow | Deep Learning[END_REF] puts forward two arguments. First, in the optimization perspective, models benefit from parameter initialization in the appropriate region [START_REF] Erhan | Why does unsupervised pre-training help deep learning[END_REF]. The second reason is probability-grounded: knowledge of the input and/or output distributions would facilitate the process of learning the right mapping from inputs to outputs; a data-driven temet nosce18 maxim so to speak.

As we have seen, self-supervision is implemented through pretraining in modern neural NLP. There exist several loss functions to pretrain a model, but a specific family of objectives has prevailed in modern ML-based NLP: Language Models.

Language Models

Standard properties of modern Language Models A Language Model (LM) is a probability distribution over variable-length sequences of tokens belonging to a pre-defined vocabulary V (i.e., symbols pertaining to a language). A LM parametrized by θ assigns probability P (s|θ) to all sequences s = s 1 , s 2 , . . . , s m ∈ V m , ∀m ∈ N.

LMs had initially been introduced in the context of automatic speech recognition [START_REF] Jelinek | Statistical Methods for Speech Recognition[END_REF] and have been applied to NLP [START_REF] Bengio | A neural probabilistic language model[END_REF]. In a probabilistic approach, language modeling is the natural way to formulate NLG. LMs not only relate to the well-known classification setup but also enable direct and simple learning of both the syntactic complexity of language and the semantic knowledge phrased by language [START_REF] Jozefowicz | Exploring the limits of language modeling[END_REF]. For instance, assigning high probability to the English sentence "People often stop watching a movie that they consider bad, before it ends." contributes to encoding that a well-formed English sentence starts with a capital letter and ends with a period and that the word "bad" has more probability of directly following the word "consider" than the word "they". Additionally, a system with some understanding of natural language (vocabulary, grammar, and semantics) may learn a commonsense fact about logical human behavior with this sentence.

Recent ML for NLP has substantially benefited from specific models trained to compute LMs, to the point where models and the distribution they predict are often identified. Indeed, neural architectures like transformers (cf. Section 2.6) and the massive increase in models' capacities improved the expressiveness of LMs.

Neural Language Models Early implementations of LM computing relied upon word-count-based methods. A n-gram LM is a statistical LM [START_REF] Shannon | Prediction and entropy of printed english[END_REF][START_REF] Miller | Verbal context and the recall of meaningful material[END_REF] computing the frequency of markovian sequences of n tokens-called ngrams 19 -appearing in text [START_REF] Markov | Essai d'une recherche statistique sur le texte du roman "Eugene Onegin" illustrant la liaison des epreuve en chain ('Example of a statistical investigation of the text of "Eugene Onegin" illustrating the dependence between samples in chain')[END_REF][START_REF] Shannon | A mathematical theory of communication[END_REF][START_REF] Goodman | A bit of progress in language modeling[END_REF]. When seen as weighted finite automata, an n-gram LM is an instance of statistical NLP systems at the intersection of formal language theory and linguistics. Because statistical LMs use discrete representations of tokens, they face two issues: sparsity and complexity. Sparsity implies that n-grams not appearing in the training set have null probability, hence hindering generalization. Besides, the complexity of statistical LM is in O (e n ) with respect to the the context length n, limiting the Markov assumption to low n th order in practical cases. Both limitations are addressed by continuous space Language Models [START_REF] Schwenk | Continuous space language models[END_REF]. In this approach, widely used nowadays, neural networks (cf. Section 2.6) are trained to compute LMs. Neural LMs (NLMs) can be characterized by the specific backbone architecture used, e.g., Feedforward NLM, Recurrent NLM, Transformerbased NLM, etc. It is common for NLMs to learn an internal representation of individual tokens, or entire sequences, with dense and low-dimensional vectors. In general, each token s t -s t being an input token in NLU and an output token in NLG-is represented at each layer i by a vector h t i called the hidden representation of s t or the hidden state of the model at time step t. The output layer dealing with projecting the last hidden representations h t back to logits in the discrete vocabulary space is often called the "LM Head".

Auto-regressive Language Models

Greedy search

The auto-regressive property lets us sacrifice some quality for major savings in execution time.

With greedy search, the generated sequence ŝ = ŝ1 , ŝ2 , . . . , ŝT is generated one step at a time. In practice, tokens generated after an End-Of-Sequence token (<EOS>) are ignored (or the process stops before they are generated).

At each time step, greedy search yields ŝt = arg max s t ∈V P S t = s t |ŝ 1 , ŝ2 , . . . , ŝt-1 ; θ . Because it simply considers the distribution on the vocabulary V at each time step, greedy search takes linear execution time, in Θ (|V | • T ).

Beam search A trade-off has been proposed to control quality and complexity with a parameter B called the beam width. In beam search [START_REF] Hayes-Roth | Speech understanding systems: Summary of results of the five-year research effort[END_REF], a set H t of the B most probable sequences from H t-1 concatenated with any token w ∈ V is iteratively maintained at each time step t, as described by equations (2.4). Teacher forcing when training auto-regressive Language Models Auto-regressive LMs use a feedback loop at inference time. The same process can be used at train time, but an alternative scheme is often preferred for its time efficiency and robustness. When making a prediction, teacher forcing [START_REF] Williams | A learning algorithm for continually running fully recurrent neural networks[END_REF] gives the model access to information not accessible at inference time: the ground-truth target. Instead of predicting the next token based on the previously generated tokens (P s t |ŝ 1 , ŝ2 , . . . , ŝt-1 ; θ ), teacher forcing computes the prediction of the next token based on the past ground-truth tokens: P s t |s 1 , s 2 , . . . , s t-1 ; θ . The benefits are twofold and illustrated by the following situation where the LM is trained on the sentence "People often stop watching a movie that they consider bad, before it ends.". If the training step is early, then free-running generation may produce "random" tokens, like ŝ1 = "are" and ŝ2 = "she". The information coming from the model trained to maximize P S 3 = "stop"|ŝ 1 = "are", ŝ2 = "she"; θ is much more noisy than when a "teacher" forces the model to maximize P S 3 = "stop"|s 1 = "People", s 2 = "often"; θ . Besides, even when the training process is advanced, the signal from P (s|θ) = P s .T ] can be parallelized and computed in a single step.

Sequence-to-sequence learning Sequence-to-Sequence (Seq2Seq) learning is the set of ML approaches that transforms a sequence into another. Sutskever et al. [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF] have proposed a neural approach to text-to-text tasks. This work has gotten major impact since it achieved remarkable results by applying three key principles:

1. Train with end-to-end learning, i.e., models (gradient-based) directly learn from inputs associated with targets 2. Optimize auto-regressive Language Model objectives 3. Model distributions with encoder-decoder neural architectures (cf. Section 2.5.4 and Section 2.6)

Sequence-to-Sequence Language Models A Seq2Seq LM is a LM conditioned by some input text x =

x 1 , x 2 , . . . , x m . A sequence sampled from the distribution is called an output y = y 1 , y 2 , . . . , y p , and a Seq2Seq LM assigns probability P (y|x; θ). Sutskever et al. [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF] proposed a strongly supervised approach, that is to say maximizing the likelihood P (y|x; θ) from pairs of input and target texts (x i , y i ). With auto-regressive generation, a convenient way of triggering the generation of the first token ŷ1 given the input x is to condition the generation with an initial Beginning-of-Sequence token y 0 = <BOS>. This also applies to teacher forcing, in which case y 0 = <BOS> is prepended to the input y.

Prefix Language Models Prefix LMs are neural Seq2Seq LMs modeled with a single-stack structure [START_REF] Dong | Unified language model pre-training for natural language understanding and generation[END_REF][START_REF] Ouyang | Enhanced multilingual representation by aligning cross-lingual semantics with monolingual corpora[END_REF] (cf. 2.5.4).

Encoder-decoder Language Models A Seq2Seq LM modeled with a double-stack encoder-decoder structure [START_REF] Song | Mass: Masked sequence to sequence pre-training for language generation[END_REF][START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF][START_REF] Qi | ProphetNet: Predicting future n-gram for sequence-to-SequencePre-training[END_REF][START_REF] Lewis | Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension[END_REF] (cf. Section2.5.4) is referred to as an encoder-decoder LM [START_REF] Li | Pretrained language model for text generation: A survey[END_REF].

Self-supervised Pretrained Language Models

LMs fit perfectly self-supervision in NLP. Therefore, they have been extensively applied to pretraining systems designed for UNLG (cf. Section 2.5.8), NLU (cf. Section 2.5.8) and test-to-text transfer (cf. Section 2.5.8). We detail below specific approaches to train LMs with self-supervision.

Causal Language Models As we saw in Section 2.5.7, LMs-and in particular auto-regressive LMs-are well suited to NLG. In addition to strong supervision (cf. Section 2.5.7), LMs can be trained with self-supervision in order to generate text automatically. To this end, a sequence s is here split into two subsequences: a prompt text x = <BOS>, s 1 , s 2 , . . . , s k and text to generate y = s k+1 , s k+2 , . . . , s k+k ′ . A Causal LM [START_REF] Jurafsky | Speech and language processing[END_REF] is an autoregressive NLM maximizing the likelihood P (y|x, θ). Causal LMs are commonly referred to as standard LM. Contrary to Seq2Seq LMs, causal LMs are fully unidirectional. This means their architectures are designed to prevent hidden representations of tokens from processing information coming from subsequent ("future") tokens. For this reason, causal LMs are sometimes called left-to-right LMs in the context of sinistrodextral (left-to-right) writing systems.

When using a single-stack structure, causal LMs are said to use a decoder structure (cf. Section 2.5.4). Selfsupervised training consists in teacher forcing with a sliding window of size m over tokens in the corpus w, i.e.

at training step τ ∈ N * , x τ = (<BOS>) and y τ = w (τ -1)•m+1 , w (τ -1)•m+2 , . . . , w τ •m (cf. Figure 2.5). Pretraining with causal language modeling has been introduced by GPT [START_REF] Radford | Improving language understanding by generative pre-training[END_REF] and declined by subsequent models, often characterized by increasing capacities and/or increasing pretraining dataset sizes [START_REF] Radford | Language models are unsupervised multitask learners[END_REF][START_REF] Brown | Language models are few-shot learners[END_REF][START_REF] Black | GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow[END_REF][START_REF] Zellers | Defending against neural fake news[END_REF][START_REF] Shoeybi | Megatron-lm: Training multi-billion parameter language models using model parallelism[END_REF][START_REF] Rosset | Turing-nlg: A 17-billion-parameter language model by microsoft[END_REF][START_REF] Narayanan | Efficient large-scale language model training on GPU clusters[END_REF][START_REF] Smith | Using deepspeed and megatron to train megatron-turing NLG 530b, A large-scale generative language model[END_REF][START_REF] Zeng | Pangu-α: Large-scale autoregressive pretrained chinese language models with auto-parallel computation[END_REF][START_REF] Lieber | Jurassic-1: Technical details and evaluation[END_REF][START_REF] Rae | Scaling language models: Methods, analysis & insights from training gopher[END_REF][START_REF] Hoffmann | Training compute-optimal large language models[END_REF][START_REF] Thoppilan | Lamda: Language models for dialog applications[END_REF][START_REF] Chowdhery | Palm: Scaling language modeling with pathways[END_REF]. The emergence of these Large Language Models (LLM) led to impressive results in generating fluent and long sequences of coherent and sometimes astonishingly creative text.

Bidirectional Language Models In de Saussure and Baskin [START_REF] De Saussure | Course in General Linguistics: Translated by Wade Baskin[END_REF], the founder of modern linguistics, Ferdinand de Saussure, considered linearity to be a fundamental property of human language:

"The Linear Nature of the Signifier:

The signifier, being auditory, is unfolded solely in time from which it gets the following characteristics: (a)

it represents a span, and (b) the span is measurable in a single dimension; it is a line.

In contrast to visual signifiers (nautical signals, etc.) which can offer simultaneous groupings in several dimensions, auditory signifiers have at their command only the dimension of time. Their elements are presented in succession; they form a chain. This feature becomes readily apparent when they are represented in writing and the spatial line of graphic marks is substituted for succession in time.

Sometimes the linear nature of the signifier is not obvious. When we accent a syllable, for instance, it seems that we are concentrating more than one significant element on the same point. But this is an illusion; the syllable and its accent constitute only one phonational act. There is no duality within the act The linear nature of language [. . .] rules out the possibility of pronouncing two elements simultaneously.

The elements are arranged in sequence on the chain of speaking."

The linear characteristics of the sound chain can be found in many modern writing systems, particularly in alphabet-based scripts. This motivates the unidirectionality developed in causal LM.

However, Saussure's structural linguistics has been criticized by the more recent founding father of modern linguistics, Noam Chomsky. With the introduction of multi-layered "deep structures" [START_REF] Chomsky | Three factors in language design[END_REF], the chomskian generative linguistics implicates the concept of simultaneity observed for instance in non-verbal communication (e.g. sign languages [START_REF] Holloway | Simultaneity as a unique property of visual-spatial language: the simultaneous structure of twohanded classifier predicates in bimodal asl/english narrative ebooks for deaf children[END_REF]), non-lingual auditory perception (e.g. philharmonic orchestras) or non-verbal visual perception (e.g. photographs).

As a matter of fact, the unidirectional hypothesis is not necessary either in NLU. Bidirectional language modeling [START_REF] Baevski | Cloze-driven pretraining of self-attention networks[END_REF] is a clever way to build good representations from self-supervision. For example, learning to predict that "1961" precedes "was the last year in which the majority of Hollywood films were released in black and white." corresponds to memorizing the correct date of a fact, whereas learning that "black and white." follows "1961 was the last year in which the majority of Hollywood films were released in" amounts to remember attributive information given an incomplete, dated fact. Because a bidirectional LM (BiLM) aims at learning, with self-supervision, semantic representations of input text for NLU, its structure is often referred to as an Auto-Encoder when it is single-stack (cf. Section 2.5.4). However, to avoid confusion with Goodfellow et al. [START_REF] Goodfellow | Deep Learning[END_REF]'s more restrictive definition of neural Auto-Encoders 20 (cf. Section 2.5.5), we prefer the general term of "self-supervised encoder".

Forward-backward Language Models [START_REF] Peters | Deep contextualized word representations[END_REF] introduced bidirectional embeddings from forward-backward LMs.

The LM is bidirectional because it is trained to predict a standard "forward" auto-regressive LM P s t |s 1 , s 2 , . . . , s t-1 ; θ as well as a "backward" auto-regressive LM P s t |s t+1 , s t+2 , . . . , s m ; θ , in order to learn word representations.

Here, language modeling is said to produce contextualized or dynamic word embedding [START_REF] Melamud | context2vec: Learning generic context embedding with bidirectional LSTM[END_REF][START_REF] Peters | Semi-supervised sequence tagging with bidirectional language models[END_REF][START_REF] Mccann | Learned in translation: Contextualized word vectors[END_REF][START_REF] Peters | Deep contextualized word representations[END_REF].

The novelty is that the same token can be embedded depending on the context it appears in. It is particularly useful to disambiguate polysemes and homographs 21 in downstream tasks. We give in Table 2.2 an example of the different definitions of the word "bank", illustrating why contextually-meaningful embedding is sometimes needed.

Masked Language Models

Beyond forward-backward LMs, subsequent works have proposed additional BiLMs.

Actually, self-supervised pretraining for NLU gained a foothold when BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] achieved state-of-the-art results on the GLUE benchmark. Since then, only derivative models have renewed the leaderboard [START_REF] Liu | A robustly optimized bert pretraining approach[END_REF][START_REF] Sanh | Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter[END_REF][START_REF] Lan | Albert: A lite bert for self-supervised learning of language representations[END_REF][START_REF] He | Debertav3: Improving deberta using electra-style pre-training with gradientdisentangled embedding sharing[END_REF][START_REF] Sun | ERNIE 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation[END_REF].

The pretraining task introduced by BERT is the Masked Language Model (MLM) 22 . If the corpus w 1 , w 2 , . . . , w q is split into n sentences s 1 ⌢ s 2 ⌢ . . . ⌢ s n , masked language modeling involves corrupting a sentence s ∈ (s i ) i∈[1.

.n] like with denoising autoencoding. The noised sentence s = s1 , s2 , . . . , sm is corrupted with a specific process originally proposed by Taylor [START_REF] Taylor | cloze procedure": A new tool for measuring readability[END_REF] and detailed in equations (2.5). In words, this corresponds to drawing an event C t that token s t is corrupted. BERT-style corruption takes the form of a replacement by a special "Mask" token <M> with high probability, or by a uniformly random token from V with low probability (explained by the fact that the <M> token is reserved for pretraining only).

P (C t ) = 0.15

P st = s t |s t , Ct = 1 P st = <M>|s t , C t = 0.8 P st = s t |s t , C t = 0.1 • 1 + 1 |V | P st = w|s t , C t = 0.1 |V | ∀w ∈ V \ s t (2.5)
From the model's perspective, the input x is the corrupted sentence s (cf. Figure 2.5). Though, contrary to DAEs, 20 "An Auto-Encoder is a neural network that is trained to attempt to copy its input to its output." 21 Homophones in speech processing 22 BERT was also trained with contrastive learning, but Liu et al. [START_REF] Liu | A robustly optimized bert pretraining approach[END_REF] showed that its impact was limited.

Remark

Semantic meaning Part of speech

Translation in Modern French

From Old Italian banca

An organization where people and businesses can invest or borrow money, change it to foreign money, etc.

Noun Banque

A building where the services described above are offered.

Noun Banque

To keep or put money in a particular building described above. Verb

Mettre en banque / de côté, déposer, être titulaire d'un compte

To win or earn a particular amount of money. Verb Gagner, faire (de l'argent)

To rely upon.

Verb Compter, miser

In gambling, money that belongs to the owner and can be won by the players.

Noun Banque

A collection of something, such as blood or human organs for medical use, in a place that stores these things for later use.

Noun Banque

From Old English h ōbanca

Sloping raised land, especially along the sides of a river. Noun Bord, rive, berge A pile or mass of earth, clouds, etc. Noun Talus, massif, pente, remblai, amoncellement, couche, banc To collect in or form into a mass, or to make something do this. Verb Amonceler (Of an aircraft) To fly with one wing higher than the other when turning.

Verb Virer

(Of an aircraft) The inclined turn described above.

Noun Virage incliné

From Old French banc A row of similar things Noun Banc, rangée, batterie Collocated with "shot"

In pool, a shot in which the player causes the cue ball or an object ball to rebound off a cushion.

Noun (Coup) par la bande

In basketball, a shot in which the basketball glances off the backboard before reaching the basket. Noun

[Untranslatable] (Tir) contre la planche Table 2.2: Homonyms of the English word "bank". Rows sharing the same etymology correspond to polysemes.

masked LMs recover only the masked tokens (s t ) t∈{j:s j =<M>} rather than the original uncorrupted sentence s. As a result, masked LMs are sometimes referred to as Masked Auto-Encoders. For NLU tasks such as classification, a special <CLS> token may be prepended to the input. In this way, the model is able to learn not only contextualized representations of individual tokens but also aggregate representations of the entire text via the representation h 0 of the <CLS> token, processed in subsequent classification layers of the model. Another special token, <SEP> is also used to separate sentences when the pretraining involves contrastive learning, comparing pairs of sentences.

The second advantage of BERT is the transformer backbone used to compute the masked LM (cf Section 2.6.3).

Unified Language Models Supervised Seq2Seq LMs and self-supervised causal or bidirectional LMs have been developed to train parameters of models specialized on one single type of NLP tasks: CNLG, UNLG or NLU, respectively. A recent trend seeks to unify LM pretraining to address one or more types of tasks.

Permutation Language Models

The first attempt to unify causal and bidirectional LM was introduced by Yang et al. [START_REF] Yang | Xlnet: Generalized autoregressive pretraining for language understanding[END_REF]. Compared to causal LMs, masked LMs use a special Mask token <M>, which causes a discrepancy between pretraining and fine-tuning. To overcome this gap while keeping bidirectionality in pretraining NLU systems, XLNet, and its successor MPNet [START_REF] Song | Mpnet: Masked and permuted pre-training for language understanding[END_REF], proposed a generalized self-supervised task, called Permutation LM and consisting in auto-regressive language modeling on a uniformly random permutation s = s1 , s2 , . . . , sm of the sequence s = s 1 , s 2 , . . . , s m . Formally, let S m be the set of all permutations of [1..m]. Permutation LM maximizes the likelihood P st |s 1 , s2 , . . . , st-1 ; θ where P s = s σ (1) , s σ(2) , . . . , s σ(m) | s 1 , s 2 , . . . , s m = 1 m! ∀σ ∈ S m . Permutation LM led to improvement on several NLU tasks compared to the original masked LM.

Sequence-to-Sequence Masked Language Models Song et al. [START_REF] Song | Mass: Masked sequence to sequence pre-training for language generation[END_REF] and Raffel et al. [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF] have adapted bidirectional pretraining to an encoder-decoder structure with the goal of improving NLU and NLG in addition to the text-to-text abilities of Seq2Seq LMs. When pretraining with Sequence-to-Sequence masked LM (Seq2Seq MLM), the encoder is fed with the corrupted input x and the decoder is trained to predict the pseudo-labels in an autoregressive way. Additional Seq2Seq MLMs have been proposed, such as the summarization-specialized PEGASUS [START_REF] Zhang | Pegasus: Pre-training with extracted gap-sentences for abstractive summarization[END_REF].

Multi-task self-supervised Pretrained Language Models Recent LMs have been proposed to pretrain systems

with a unified self-supervised LM based on controlling the information flow (causal, bidirectional or Seq2Seq) in its backbone neural architecture, via dynamic attention masking (cf. Section 2.6.3). Pseudo-mask LM Dong et al. [START_REF] Dong | Unified language model pre-training for natural language understanding and generation[END_REF], Bao et al. [START_REF] Bao | Unilmv2: Pseudo-masked language models for unified language model pre-training[END_REF] enables pretraining a unique model to gain both NLU and UNLG abilities. In addition to these two types of tasks, General LMs [START_REF] Du | All NLP tasks are generation tasks: A general pretraining framework[END_REF] can also address CNLG elegantly.

Transfer Learning via Knowledge Adaptation to Downstream Tasks

Pretrained models reveal their full potential when adapted to specific-and possibly many-problems, called downstream tasks, and evaluated by labeled datasets (cf. Section 2.3). As mentioned in Section 2.5.6, strongly supervised fine-tuning has been the standard knowledge adaptation procedure after a model has been pretrained with self-supervision.

LMs have an essential characteristic: they can represent task descriptions as well as targets in the same semantic space as inputs. GPT-2 [START_REF] Radford | Language models are unsupervised multitask learners[END_REF] and T5 [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF] were among the first systems to solve multi-NLU-and-CNLG tasks in a full text-to-text format after the model had been trained. The control of the task to solve was done by conditioning task-specific text prefixes prepended to the input. Keskar et al. [START_REF] Keskar | CTRL -A Conditional Transformer Language Model for Controllable Generation[END_REF] used these prefixes, called "control codes", to steer UNLG in text subspaces characterized by specific style or content attributes. Raffel et al. [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF] showed the mutual benefit resulting from gradient-based fine-tuning of a single model on a mixture of tasks and illustrated transfer learning between downstream tasks (in addition to transfer learning between the pretraining and fine-tuning phases).

Fine-tuning often requires a large amount (∼ 10 4 to ∼ 10 6 ) of labeled examples per downstream task. It also hinders the sought-after capability of alternating the resolution of several-possibly unrelated-tasks without further parameter updates. These reasons have motivated the development of techniques leveraging the pretraining phase (denoted meta-learning [START_REF] Brown | Language models are few-shot learners[END_REF] in this context) to address transfer learning directly. Besides fine-tuning, recent works [START_REF] Radford | Language models are unsupervised multitask learners[END_REF][START_REF] Brown | Language models are few-shot learners[END_REF] have shown that task-specific knowledge can be acquired by manipulating causal LMs' behaviors (cf. Section 2.5.9). Plus, these methods paved the way for efficient few-shot learning in NLP.

Few-shot learning ML traditionally requires at least hundreds of data points for training a system to solve a complex task. However, at a certain cognitive developmental stage, the biological brain has sufficient innate and acquired knowledge 23 to learn new tasks with few or even no examples. For instance, if a human is asked to identify whether two different sentences have the same meaning, they will perform pretty well without any example and whether or not they solved a similar problem before. Sometimes, an example is needed to understand the task.

If an English speaker is asked to identify pleonasms, an example such as "to exit outside" may be needed if the person does not know the word "pleonasm". More difficult tasks may require more than one example. Asking to recognize zeugmas or semantic syllepsis may require positive examples like "They covered themselves with dust and glory" [START_REF] Twain | The adventures of Tom Sawyer[END_REF], "On their flippant way through Italy, the French carelessly picked up Genoa, Naples and syphilis" 24 , as well as a few negative examples. The setup where a pretrained system has access to a minimum description of a task and/or a small amount of annotated data, typically less than 100, is called few-shot learning. In ML, if a pretrained system has access to a single annotated example, the setup is called one-shot learning [START_REF] Miller | Learning from one example through shared densities on transforms[END_REF][START_REF] Fei-Fei | One-shot learning of object categories[END_REF]. For instance, a system that has properly clustered the classes seen during pretraining may be able to classify inputs with regard to the previous classes plus an additional class to whose the single annotated example belongs. In the extreme, zero-shot learning [START_REF] Chang | Importance of semantic representation: Dataless classification[END_REF][START_REF] Larochelle | Zero-data learning of new tasks[END_REF][START_REF] Lampert | Learning to detect unseen object classes by betweenclass attribute transfer[END_REF][START_REF] Socher | Zero-shot learning through cross-modal transfer[END_REF][START_REF] Palatucci | Zero-shot learning with semantic output codes[END_REF] refers to the case where the pretrained model cannot see any labeled example to solve a task it has not been explicitly pretrained on.

Contrary to BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] and T5 [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF] which require strongly supervised gradient-based fine-tuning of their parameters with a significant number of labeled examples to solve one or several downstream task(s), GPT-2's breakthrough was to propose a clever way to address several downstream tasks with a single frozen model [START_REF] Radford | Language models are unsupervised multitask learners[END_REF]. The passable-but-promising results showed that a self-supervised causal LM could learn new tasks in a few-shot learning setup. The method, described in the next section, was later called "in context-learning" by Brown et al. [START_REF] Brown | Language models are few-shot learners[END_REF].

Self-supervised pretraining on a dataset covering a tremendous amount of information, including domain knowledge, could theoretically suffice to address in a few-shot learning setup, NLP tasks where targets can be expressed with text, provided the model has enough capacity. For instance, the unlabeled sentence "I loved it because I thought she was a great director" might participate in indicating that appreciating a director means the opinions on their movies is positive. Radford et al. [START_REF] Radford | Language models are unsupervised multitask learners[END_REF] provide additional examples of naturally occurring demonstrations of translation found in large pretraining corpora. Recent evaluations on NLP benchmarks [START_REF] Brown | Language models are few-shot learners[END_REF][START_REF] Chowdhery | Palm: Scaling language modeling with pathways[END_REF] tend to show that scaling the parameters of LMs (cf Section 2.5.9) does indeed close the gap between state-of-the-art fine-tuned LMs and large-scale causal LMs used in few-shot learning setups. Whether this means that Artificial general intelligence [START_REF] Fitzgerald | survey of artificial general intelligence projects for ethics, risk, and policy[END_REF] or human-level AI will eventually be achieved with large-scaled models is an open debate. Controlling causal LMs with natural language descriptions of downstream tasks (and potentially a couple of labeled examples) is done with methods based on clever input tuning.

Few-shot and zero-shot transfer with input tuning

Early few-shot learning involved gradient-based parameter fine-tuning [START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF][START_REF] Bertinetto | Learning feed-forward one-shot learners[END_REF][START_REF] Kaiser | Learning to remember rare events[END_REF] though input tuning is the current state-of-the-art approach. Prompt augmentation is the method consisting of feeding pretrained causal LMs with information necessary to solve a task given an input text. The model is stimulated with a prompt (or context), like "The director is great. I thought the movie was ". Then, probabilities of a pre-defined set of texts, called verbalizers and corresponding to the classes of the problem (e.g., in this situation "good" for positive sentiment and "bad" for a negative one) are compared. A prompt (cf. Section 2.5.8) x ′ is formed with the input text x and a "template" [START_REF] Liu | Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing[END_REF]. The process of transforming x into x ′ is called the prompting function. The template consists of a prefix and/or a suffix. Prefixes and suffixes may take the form of explicit text instructions for zero-shot transfer [START_REF] Sanh | Multitask prompted training enables zero-shot task generalization[END_REF][START_REF] Wei | Finetuned language models are zero-shot learners[END_REF], like the prefix "Give me the sentiment of " or the suffix "What is the sentiment of the previous sentence? ". Additionally, with recent architectures being able to process long sequences of text without long-term dependency problems, it is possible to pass a few labeled examples in the prompt [START_REF] Jebara | How many data points is a prompt worth[END_REF]. This priming-based few-shot learning [START_REF] Kumar | Reordering examples helps during priming-based few-shot learning[END_REF] may concatenate pairs using inter-separation (between pairs) and intra-separation (between the input and the target) symbols, for instance, "I loved this. => Positive \n Great director! => Positive \n I really hated it. => Negative \n This was the worst movie I have ever seen! => ". The power of prompt-based learning for multi-task learning [START_REF] Caruana | Multitask learning[END_REF][START_REF] Radford | Language models are unsupervised multitask learners[END_REF] comes from the ease in of switching from one application to a totally different one with a single change in the template. Instructions and examples can be mixed in many ways for the same task. Actually, prompt template engineering has recently become an active research topic.

Prompt Engineering

Manual template engineering A straightforward way of exploring how linguistic and world knowledge acquired by LMs are affected by prompt templates is to manually craft prefix prompts [START_REF] Petroni | Language models as knowledge bases?[END_REF][START_REF] Schick | Exploiting cloze-questions for few-shot text classification and natural language inference[END_REF][START_REF] Schick | It's not just size that matters: Small language models are also few-shot learners[END_REF][START_REF] Schick | Few-shot text generation with natural language instructions[END_REF].

Automated prompt learning

Besides manually designed prompts, templates can be optimized with automated methods [START_REF] Jiang | How Can We Know What Language Models Know[END_REF][START_REF] Shin | AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts[END_REF][START_REF] Gao | Making pre-trained language models better few-shot learners[END_REF]. Further works have proposed to transform prompt learning into continuous prompt tuning [START_REF] Qin | Learning how to ask: Querying LMs with mixtures of soft prompts[END_REF][START_REF] Zhong | Factual probing is [MASK]: Learning vs. learning to recall[END_REF][START_REF] Hambardzumyan | WARP: Word-level Adversarial ReProgramming[END_REF][START_REF] Li | Prefix-tuning: Optimizing continuous prompts for generation[END_REF][START_REF] Liu | GPT understands, too[END_REF][START_REF] Lester | The power of scale for parameter-efficient prompt tuning[END_REF][START_REF] Gu | Ppt: Pre-trained prompt tuning for few-shot learning[END_REF]. Whereas templates (hard prompts) are discrete instructions written in natural language, it is possible to prepend dense representations (soft prompts) to the input right after it has been embedded by the initial neural layer (cf. Section 2.5.2). Besides offering a finer-grained optimization framework, this tuning strategy requires far fewer parameters to train than what is required by traditional full-model tuning, hence producing major savings in time and space.

Large Pretrained Language Models A major development currently happening in neural NLP is the scaling of Pretrained Language Models (PLMs). Indeed, we observe a race to develop and use resources to pretrain increasingly deeper models on ever larger datasets. This is justified by empirical studies showing that not only LMs' capacities (reflected by performances on downstream NLP tasks and text generation) are smoothly improved by scaling up the parameters and training datasets [START_REF] Kaplan | Scaling laws for neural language models[END_REF], but also it seems no upper bound has (yet?) been reached [START_REF] Chowdhery | Palm: Scaling language modeling with pathways[END_REF]. At the time of writing this dissertation, models have regularly grown from the 100-million-parameter GPT-1 [START_REF] Radford | Improving language understanding by generative pre-training[END_REF] to the 300-million-parameter BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF], the 1.5-billion-parameter GPT-2 [START_REF] Radford | Language models are unsupervised multitask learners[END_REF], the 8.3-billion-parameter Megatron [START_REF] Shoeybi | Megatron-lm: Training multi-billion parameter language models using model parallelism[END_REF][START_REF] Narayanan | Efficient large-scale language model training on GPU clusters[END_REF], the 11-billion-parameter-T5 [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF], the 17.2-billion-parameter Turing NLG [START_REF] Rosset | Turing-nlg: A 17-billion-parameter language model by microsoft[END_REF], the 70-billion-parameter Chinchilla [START_REF] Hoffmann | Training compute-optimal large language models[END_REF], the 137-billion-parameter LaMDA [START_REF] Thoppilan | Lamda: Language models for dialog applications[END_REF], the 175-billion-parameter GPT-3 [START_REF] Brown | Language models are few-shot learners[END_REF], the 178-billionparameter Jurassic-1 [START_REF] Lieber | Jurassic-1: Technical details and evaluation[END_REF], the 200-billion-parameter PanGu-α [START_REF] Zeng | Pangu-α: Large-scale autoregressive pretrained chinese language models with auto-parallel computation[END_REF], the 280-billion-parameter Gopher [START_REF] Rae | Scaling language models: Methods, analysis & insights from training gopher[END_REF], the 530-billion-parameter Megatron-Turing NLG [START_REF] Smith | Using deepspeed and megatron to train megatron-turing NLG 530b, A large-scale generative language model[END_REF], the 540-billion-parameter PaLM [START_REF] Chowdhery | Palm: Scaling language modeling with pathways[END_REF], and finally the 1.6-trillionparameter Switch Transformer [START_REF] Fedus | Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity[END_REF]. The crux of the problem lies in parallel computing hardware and software. It was enabled by advances in model architectures (cf. Section 2.6) and AI accelerators like Tensor Processing Units (TPUs) and Graphics Processing Units (GPUs).

Large self-supervised Pretrained LMs are currently considered as driving a paradigm shift, similar to the sea changes observed when ML algorithms-and later DL architectures-emerged and became dominant in AI. These Model GLUE Score ↑ SuperGLUE Score ↑ CBoW [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] a Few-shot learning.

Table 2.3: GLUE and SuperGLUE scores of high-impact LMs on the test sets according to the leaderboards. Except for the score of GPT(-3) on the SuperGLUE benchmark, all models are fine-tuned on the downstream tasks.

seismic changes are characterized by quick standardization of approaches that are eventually applied to real-life problems and directly-or indirectly-impact citizens' daily lives. The current AI transition stemming from Pretrained LMs has been named the revolution of "Foundation Models" [START_REF] Bommasani | On the opportunities and risks of foundation models[END_REF]. Not only have these models become the "substrate of NLP", showing astonishing capabilities in world knowledge, creativity as well as common sense, logical and arithmetical few-shot reasoning (examples of complex joke explanation are provided by Brown et al. [START_REF] Brown | Language models are few-shot learners[END_REF]), but similar ideas have been applied to further modalities [START_REF] Tamkin | Dabs: a domain-agnostic benchmark for self-supervised learning[END_REF] such as images [START_REF] Chen | Generative pretraining from pixels[END_REF][START_REF] Ramesh | Hierarchical text-conditional image generation with clip latents[END_REF][START_REF] Saharia | Photorealistic text-to-image diffusion models with deep language understanding[END_REF], tables [START_REF] Yin | TaBERT: Pretraining for joint understanding of textual and tabular data[END_REF] or proteins [START_REF] Rives | Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences[END_REF]. Pretrained LMs made of numbers of parameters exceeding the number of neurons in human brains [START_REF] Herculano-Houzel | The human brain in numbers: a linearly scaled-up primate brain[END_REF] are now considered as "foundations for a wide range of downstream applications". It is not well understood yet whether their aptitudes are based on actual generalization, as in human intelligence, or on powerful memorization [START_REF] Radford | Language models are unsupervised multitask learners[END_REF]. Opportunities and risks have been extensively explored by Bommasani et al. [START_REF] Bommasani | On the opportunities and risks of foundation models[END_REF], including potential harms concerning bias, ethics, fairness, privacy, safety, security, or natural environment.

Table 2.3 shows the average GLUE and SuperGLUE test scores (cf. Section 2.3) of high-impact LMs as well as a baseline (cf. Section 2.5.3) and human performances. Apple-to-apple comparison of released or published models on benchmarks is difficult [START_REF] Aßenmacher | On the comparability of pre-trained language models[END_REF]. Indeed, models often differ on several aspects such as objective functions, sizes, pretraining datasets, or knowledge adaptation strategy.

NLP methods have been developed in light of ML, with old systems relying on handcrafted features. Then, given the powerful abilities of a particular class of end-to-end ML models-called deep Artificial Neural Networks (ANN) [START_REF] Nielsen | Neural networks and deep learning[END_REF]-in other supervised AI problems such as computer vision, research has introduced Deep Learning architectures tailored to language characteristics [START_REF] Goldberg | Neural Network Methods for Natural Language Processing[END_REF]. It appears that ANNs are now widely used in all sorts of NLP tasks, given they achieve state-of-the-art results on many benchmarks.

x 1 x 2 x n +1 aaaaaaaaaaaaaa Σ σ y a w 1 w 1 w 2 w 2 w n w n b b . . .

Neural network architectures

ANNs [START_REF] Rosenblatt | The perceptron: a probabilistic model for information storage and organization in the brain[END_REF] are loosely inspired-though not being replicas 25 -of biological neural networks described by neuroscientists [START_REF] Mcculloch | A logical calculus of the ideas immanent in nervous activity[END_REF]. From the theoretical computer science perspective, an ANN can be seen as a circuit [START_REF] Widrow | Adaptive switching circuits[END_REF], i.e., a model made of a computation graph processing information as it flows through it. This concept is specific when compared to the more traditional procedural (e.g., programming in C++) or functional (e.g., programming in OCaml) approaches of computer science. The simplest kind of neural network is called the Feedforward Neural Network, and we describe its structure below.

Feedforward neural networks

Compared to other ML systems, ANNs can be characterized by their cognitive-inspired structure consisting of layer stacks, each made of many artificial neural units making small computations in parallel.

Artificial neural units

The building block of ANNs is simple parametric and generic computing units called artificial neurons and inspired by the mathematical modeling of the molecular biology observed in nerve cells [START_REF] Kandel | Principles of neural science[END_REF]. A neural unit is fed with some vector of n input features x, and outputs a scalar y (also denoted a when part of a network).

The transformation computed by the unit is a non-linear function of the weighted sum of the input features plus a bias term (cf. Figure 2.6). The computation can be formally written y = σ (w • x + b).

The weight vector w and the bias scalar b are the neuron's parameters. The non-linear function σ is called an activation function. As training ANNs relies on gradient-descent-based optimization algorithms, activation functions are often chosen to be continuously differentiable (e.g. hyperbolic tangent, sigmoid [START_REF] Han | The influence of the sigmoid function parameters on the speed of backpropagation learning[END_REF], softmax [START_REF] Boltzmann | Studien über das gleichgewicht der lebendigen kraft zwischen bewegten materiellen punkten[END_REF], Gaussian Error Linear Unit [START_REF] Hendrycks | Gaussian error linear units (gelus)[END_REF], Swish [START_REF] Ramachandran | Searching for activation functions[END_REF]) though the REctified Linear Unit [START_REF] Nair | Rectified linear units improve restricted boltzmann machines[END_REF] is a popular activation function not 25 The design of modern ANNs should not be confused with the modeling of biological brains. differentiable at 0. An intuitive way to apprehend the activation's operation is to see it as an indicator triggered by specific linear combinations of the input features x. For instance, a specific neural unit in a system might be "activated" when it detects the presence of the adverb "not" in a sentence. It will then pass this output signal to the input of subsequent units that may take this information into account if they take part in the process of determining whether the sentence contains some semantic negation and how to process it.

x 1 x 2 x n +1 a m y m a 2 y 2 a 1 y 1 w 1,1 w 1,1 w 1,2 w 1,2 w 1,n w 1,n b 1 b 1 . . . . . . . . .
Feedforward neural layers A feedforward neural layer is a set of neural units, each with its own set of parameters, making similar computations in parallel on the same input vector x. The layer produces an output vector y and is parameterized by a m × n weight matrix W and a bias vector b. The computation can be written

y = σ (W • x + b)
assuming the activation function σ applies to a vector element-wise (cf. Figure 2.7). As each neural unit is connected to all the input features in x, this layer is sometimes qualified as "fully-connected" as opposed to layers whose neural units are only partially connected to the input features (cf. Section 2.6.2). As an example, natural language embeddings (cf. Section 2.5.2) are modeled with fully-connected feedforward layers.

Fully-connected neural networks

We can go one step further and compute the output y as a non-identity function of the activation values. This can be achieved with a second neural layer with its own set of parameters called the output layer (cf. Figure 2.8). Now, y = σ (2) W (2) • σ (1) W (1) • x + b (1) . The first layer is called the hidden layer because it has direct access neither to the output nor to the input. The input features x is sometimes referred to as the input layer, even if it has no parameter. Note that the output layer does not require bias as in the hidden layer.

Actually, we just described the architecture of a standard multi-layer Feedforward Neural Network (FFNN), also known as Fully-Connected Neural Network (FCNN). FCNN made of one hidden layers are used to pretrain noncontextual word embeddings like Word2vec (cf. Section 2.5.3). In NLP, the input layer is the one-hot encoding of words in the vocabulary V . The idea of sequential neural layers comes from the neocortical layering observed by neuroscientists [START_REF] Kurzweil | How to create a mind: The secret of human thought revealed[END_REF]: for instance, layer IV acts as the recipient of incoming sensory signal later processed by subsequent layers.

The network's trainable parameters θ is the set of all parameters found in each layer. This architecture defines a mapping f θ between the input and the output space. The non-linear activation functions enable this multi-layer structure. Plus, non-linearities are desired when it comes to making predictions on non-linearly separable data.

Universal approximation theorem

The main theoretical result in the mathematical theory of ANNs is that they are universal function approximators.

Theorem 1 Let S ⊆ R n be compact. Then, ∀ (g, ϵ) ∈ C (S, R) × R * + , there exists a FCNN f θ with one hidden layer and a finite number of neural units such that sup

x∈S ∥g (x) -f θ (x) ∥.
Even if the result does not tell how many weights are needed or how to find them, it grounds ANNs as compositional function approximators.

Deep neural networks A characteristic of ANNs is that they can stack many hidden layers, each learning specific functionalities. The use of deep neural networks is often called Deep Learning (cf. Figure 2.9). Besides depth, the other hyperparameter characteristic of ANNs' capacities is their width, that is, the number of units within a layer.

It has been shown that deep and wide ANNs can approximate functions of increasing complexity. Of course, the design of ANNs is tied with the optimization strategy used to train them as well as their connectionist structures and the mathematical operations performed by units or cells.

x 1

x 2

x n

+1

. . .

Learning via ANN training

After their parameters θ are initialized, for instance randomly, ANNs usually learn to approximate functions or probability distributions with gradient-descent-based optimization [START_REF] Cauchy | Méthode générale pour la résolution des systemes d'équations simultanées[END_REF]. In supervised learning, differentiation algorithms tune the parameters θ with the goal of minimizing the prediction error computed with a loss function or, alternatively, maximizing the probability distribution of the observed data. Efficient training of the network's parameters is enabled by the back-propagation algorithm [START_REF] Rumelhart | Learning internal representations by error propagation[END_REF][START_REF] Rumelhart | Learning Internal Representations by Error Propagation[END_REF], leveraging the chain rule [START_REF] Leibniz | Memoir using the chain rule[END_REF][START_REF]Analyse des infiniment petits pour l'intelligence des lignes courbes[END_REF] and dynamic programming [START_REF] Bellman | The theory of dynamic programming[END_REF] to make fast updates of each unit's parameters. Additional advances in hardware and training techniques led computer scientists to develop larger ANNs [START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF][START_REF] Bengio | Greedy layer-wise training of deep networks[END_REF].

ANNs have been studied as efficient representation learners for a long time [START_REF] Hinton | Learning distributed representations of concepts[END_REF]. Not only have deep ANNs demonstrated better learning performances than previous ML models [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF][START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF][START_REF] Szegedy | Going deeper with convolutions[END_REF], but they do so by automatically learning salient features with end-to-end supervised training [START_REF] Collobert | Natural language processing (almost) from scratch[END_REF][START_REF] Bengio | Representation learning: A review and new perspectives[END_REF]. This means that there is no other signal coming to the network than the input features x processed by the input layer and the target y used to backpropagate errors from the output layer. In other words, even if no agent tells the model how to solve the task, the network learns suitable representations in its hidden layers. Indeed, studies on both neural Computer Vision [START_REF] Simonyan | Deep inside convolutional networks: Visualising image classification models and saliency maps[END_REF][START_REF] Zeiler | Visualizing and understanding convolutional networks[END_REF][START_REF] Springenberg | Striving for simplicity: The all convolutional net[END_REF] and neural NLP26 [START_REF] Tenney | BERT rediscovers the classical NLP pipeline[END_REF][START_REF] Clark | What does BERT look at? an analysis of BERT's attention[END_REF][START_REF] Jawahar | What does BERT learn about the structure of language?[END_REF][START_REF] Rogers | A primer in BERTology: What we know about how BERT works[END_REF] showed that the hidden layers learn hierarchical structures of the feature distributions, similar to traditional pipelines based on manual feature engineering. For instance, in NLU, BERT's lower-level layers tend to be dedicated to syntactic processing while the higher-level layers focus on semantic understanding.

h 1 h 2 • • • h m s 1 s 2 • • • s m-1 e 1 e 2 • • • e m s 1 s 2 • • • s m s 1 s 2 • • • s m θ Emb θ Enc e 1 e 2
e 3 e m . . . . . . This echoes the observations of Broca's area in the frontal cortex, broken into two component regions. Brodmann's areas 44 and 45 are assumed to be respectively involved in syntactic and semantic tasks [START_REF] Goucha | The language skeleton after dissecting meaning: A functional segregation within broca's area[END_REF].

h 1 h 2 h 3 h m . . .
Feedforward Neural Language Models LMs were modeled by ANNs first by Bengio et al. [START_REF] Bengio | A neural probabilistic language model[END_REF]. Their Feedforward Neural Language Model (FNLM) was made of an embedding layer, a hidden layer, whose inputs are the concatenation of the word embeddings, and an output layer whose activation is softmax.

Alternative neural architectures

Advances in HPC (HPC) devices and optimization algorithms have driven the development of large ANNs. Their expressiveness has also been improved by architecture engineering, which contributes to the scaling of the models' parameters as well. We present here a set of popular architectures for NLP, illustrated in Figure 2.6. See Goodfellow et al. [START_REF] Goodfellow | Deep Learning[END_REF] for further reference on Deep Learning architectures.

Convolutional layers Besides FCNN, another major family of FFNN has been introduced in computer vision. Inspired by neurophysiology research on the receptive fields in the visual cortex [START_REF] Hubel | Receptive fields and functional architecture of monkey striate cortex[END_REF], Convolutional Neural Networks (CNNs) apply convolution filters to local features from the previous layer [START_REF] Fukushima | Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition[END_REF][START_REF] Waibel | Phoneme recognition using time-delay neural networks[END_REF][START_REF] Lecun | Convolutional networks for images, speech, and time series[END_REF][START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]. These convolving operations have later been transposed to neural NLP [START_REF] Kim | Convolutional neural networks for sentence classification[END_REF][START_REF] Kalchbrenner | A convolutional neural network for modelling sentences[END_REF][START_REF] Jaderberg | Reading text in the wild with convolutional neural networks[END_REF]. In NLP-oriented CNNs, the first hidden layer makes convolving operations with a window moving on the stacked word embeddings (cf. Figure 2.10). The Universal Sentence Encoder [START_REF] Cer | Universal sentence encoder[END_REF] used to embed sentences in Chapter 5 is a CNN.

FFNNs are good encoders of fixed-length data. In particular, CNNs were originally developed to process images.

Yet, non-feedforward ANNs have been proposed to encode and generate variable-length sequential data, such as text.

Recurrent layers Inspired by cognitive modeling of temporal structures [START_REF] Rumelhart | Parallel Distributed Processing: Explorations in the Microstructure of Cognition[END_REF][START_REF] Rumelhart | Parallel Distributed Processing: Explorations in the Microstructure of Cognition[END_REF], recurrent layers are neural layers whose units process inputs seen as time series [START_REF] Elman | Finding structure in time[END_REF]. At each time step t, the neural unit computes a hidden state h t by applying operations to the t th input x t and the t -1 th hidden state h t-1 . Figure 2.10 (right) shows an unfolded representation of a recurrent layer. Hidden states are expected to act as-theoretically infinite-memory, computed from the past inputs x 1:t (or context). Actually, some important characteristic of recurrent layers is that the units share their parameters. This is equivalent to introduce cycles in ANN computation graphs, while FFNN's graphs are acyclic. Parameter sharing is motivated by better generalization on variable-length sequences and reduces space complexity since the model width is independent of the input sequence lengths. In practice, the training of Recurrent Neural Networks (RNNs) is done by unrolling the cycles, which represents high time complexity in the case of deep RNNs processing long sequences, compared to parallelized architectures. For this reason, RNNs tend to be shallower than other neural architectures, despite evidences showing that RNNs also benefit from deeper architectures [START_REF] Graves | Generating sequences with recurrent neural networks[END_REF][START_REF] Pascanu | How to construct deep recurrent neural networks[END_REF].

RNNs have been widely employed in NLP. Mikolov et al. [START_REF] Mikolov | Recurrent Neural Network Based Language Model[END_REF] introduced recurrent LMs. RNN-based encoderdecoders are used in Seq2Seq learning [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF][START_REF] Cho | Learning phrase representations using RNN encoder-decoder for statistical machine translation[END_REF]. BiLMs (cf. Section 2.5.8), and more generally NLU systems, have often been implemented with bidirectional RNNs [START_REF] Schuster | Bidirectional recurrent neural networks[END_REF] while auto-regressive NLG has been modeled with unidirectional RNNs.

Variants of the vanilla RNN, called gated RNNs, introduced units-sometimes referred to as cells-performing more complex operations whose goal is to better model the context to address the long-term dependency problem.

Long Short-Term Memories (LSTMs) [START_REF] Hochreiter | Long short-term memory[END_REF][START_REF] Gers | Learning to Forget: Continual Prediction with LSTM[END_REF] and Gated Recurrent Units (GRUs) [START_REF] Chung | Empirical evaluation of gated recurrent neural networks on sequence modeling[END_REF] are popular cells. In Chapter Chapter 4, we used Bidirectional LSTMs (BiLSTMs) to detect toxic spans in online posts.

Further details on architectures, training, and comparisons of RNNs, see Graves [START_REF] Graves | Supervised Sequence Labelling with Recurrent Neural Networks[END_REF]. The main drawback of deep RNNs is the unstable training known as the vanishing and exploding gradient problem [START_REF] Bengio | Learning long-term dependencies with gradient descent is difficult[END_REF].

Attention neural networks

Information storage is a concept essential to understanding the mind as well as for the development of AI. The nature of traditional computer memories differs significantly from the human memory studied by cognitive neuropsychology.

Yet, cognitive and neurocomputing architectures have spurred effective mechanisms improving ANNs' abilities [START_REF] Sukhbaatar | End-to-end memory networks[END_REF][START_REF] Weston | Memory networks[END_REF][START_REF] Graves | Neural turing machines[END_REF]. In particular, attention is the core building block of a prominent neural architecture that has gradually become the de-facto standard in state-of-the-art NLP: the transformer [START_REF] Vaswani | Attention is all you need[END_REF].

Attention mechanism Because the human working memory (or short-term memory) has a limited capacity [START_REF] Miyake | Models of working memory: Mechanisms of active maintenance and executive control[END_REF] (7 ± 2 elements), it uses a behavioral process called attention to prioritizing the information. Attention consists of identifying salient stimuli, i.e., signals relevant to the task, and ignoring the remaining incoming sensory data, considered distraction [START_REF] Zanto | Neural suppression of irrelevant information underlies optimal working memory performance[END_REF]. In speech, sentence stress can draw attention to a specific word in order to adapt its meaning. For instance, the message in the sentence "I never said she stole my money." varies according to the word(s) stressed [START_REF] Puskala | Materials for teaching the pronunciation of english in finnish upper secondary schools[END_REF]. "I never said she stole my money." means that somebody else stole my money, while "I never said she stole my money." means that she stole something else, etc. Visual memory is also known to rely on selective attention 27 . When reading, a human is assumed to focus attention on the words most useful to understand the text. A symbolic way of visualizing this would be to transcribe text like in the following example:

"1961 was the last year in which the majority of Hollywood films were released in black and white. "

The integration of attention into ANNs was first proposed by Graves [START_REF] Graves | Generating sequences with recurrent neural networks[END_REF] and Bahdanau et al. [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF]. Their work improved recurrent layers with a mechanism enabling the model to focus on specific tokens of the input sentences.

In their famous publication entitled "'Attention Is All You Need", Vaswani et al. [START_REF] Vaswani | Attention is all you need[END_REF] proposed an attention neural network, called the transformer, without recurrent connections. Its key component is the attention head.

Attention heads Attention heads look like the fully-connected feedforward layers described in Section 2.6.1.

Attention head inputs are the d-dimensional representations of a sequence of n tokens (e.g. the initial embeddings of the one-hot encoded tokens, cf. Section 2.5.2). The first step consists in projecting with weight matrices

W Q ∈ R d k ×d , W K ∈ R d k ×d and W V ∈ R dv×d the inputs into vectors respectively called queries Q ∈ R n×d k , keys K ∈ R n×d k and values V ∈ R n×dv .
Then, the attention head performs dot-product-based pairwise comparisons on the queries and keys. The attention head outputs the sum of the values, weighted by the comparison scores. In other words, the queries are said to "attend" the values. The formula is given in Equation (2.6).

Attention (Q, K, V) = softmax Q • K ⊺ √ d k • V (2.6)
W V , W Q and W V are the attention head's trainable parameters θ Att . The principle of the attention head is that during training, the ANN learns which token pairs should be attended to perform best on the task; the intuition being that the weighted sum summarizes the contextual information contained in the input tokens, independently of their distance to the query. This progress in addressing the long-term dependency problem of RNNs has contributed to the success of transformers. The second advantage proposed by attention heads is that the computations are highly parallelizable and can leverage the capabilities of AI accelerators. Yet, the bottleneck to the scaling of transformers is the quadratic complexity of the attention heads, while RNNs' sequential operations are linear in the input length.

Though, the current trend is to deploy large transformers on increasing HPC resources.

Actually, there are two kinds of attention heads found in transformers. First, self-attention heads compute comparisons between each pair of tokens in the input text. Besides, in strongly supervised text-to-text, teacher forcing feeds the network with pairs of input and target texts. In this case, cross-attention heads compare pairs made of one token from the input and one token from the target. By telling which words were the most attended, attention heads contribute to increasing the explainability of ANNs, often described as black boxes. We leveraged this property in Chapter 4.

In order to better model the many ways words can interact with each other, transformers use multi-head attention layers whose output representations are concatenated and linearly transformed with a matrix parameterized by θ Concatenation . This enables the ANN to capture all the possible semantic and syntactic relationships.

Transformer blocks Transformers are made of stacks of identical blocks including multi-attention heads as well as attention-less fully-connected neural layers, residual connections [START_REF] He | Deep residual learning for image recognition[END_REF], normalizing layers [START_REF] Ba | Layer normalization[END_REF] (cf. Figure 2.11).

These additional attention-less layers and operations contribute to improving the transformer training.

Transformers

The original transformer architecture is made of a stack of transformer blocks on top of a positionwise feedforward embedding layer, whose role is to encode the order of the words (cf. Figure 2.11). As a matter of fact, the transformer initially referred to the double-stack encoder-decoder structure introduced by Vaswani et al. [START_REF] Vaswani | Attention is all you need[END_REF]. However, single-stack transformers underpin many NLP systems. To avoid confusion, we refer to single-stack transformers as unitransformers while encoder-decoder transformers-using both self-attention and cross-attention in the decoder-are called here bitransformers.

The expressiveness of transformers is the reason why they have become the backbone of many neural NLP models. Even though several variants of the original transformer have been explored, Narang et al. [START_REF] Narang | Do transformer modifications transfer across implementations and applications?[END_REF] did not find modifications which significantly improved the transformer performance on a wide range of applications.

The premise of foundation models is based on the self-supervised pretraining of large transformer-based LMs, equivalently referred to as pretrained transformers, Pretrained LMs, or large LMs in the literature. Further details on transformers can be found in Vaswani et al. [START_REF] Vaswani | Attention is all you need[END_REF] and Lin et al. [START_REF] Lin | A survey of transformers[END_REF].

Attention masks Section 2.5 presented strategies for training ML systems. Even though training objectives are mainly implemented with loss functions, attention neural network stacks trained with teacher forcing (cf. Section 2.5.7) utilize masking patterns to control the context that tokens are allowed to attend (cf. Figure 2.11). In other words, these attention masks condition the representation of query tokens to a specific set of key tokens. For instance, causal LM is implemented with causal attention masks that restrict the information flow used to represent

x 1 x 2 • • • x m x 1 x 2 • • • x m y 1 y 2 • • • y m x 1 x 2 • • • x m θ 1 Att (& Mask) θ Concatenation Add & Normalize θ FCNN Add & Normalize h 1 h 2 • • • h m . . . . . . . . . . . . . . . . . . . . . . . . e 1 e 2 • • • e m
Positional Encoding

θ 1 Block θ 2 Block θ l-1 Block θ l Block θ l+1 Block θ L-1 Block θ L Block x 1 x 2 . . . x m x 1 x 2 . . . x m x m Causal x 1 x 2 . . . x m x 1 x 2 . . . x m x m

Bidirectional

Transformer

Transformer block Attention masks

Chapter 2 conclusion

NLP has studied tasks of gradually increasing difficulty, successively tacked by symbolic AI, statistical ML and deep ANNs. At the crossroads of computer science and linguistics, the field has become enriched by neuropsychology and cognitive concepts. As a matter of fact, NLP is not just about language; it is deeply related to automatic reasoning and world knowledge. We presented in this chapter the winning combination of training strategies, self-supervised objectives, and HPC-powered DL architectures constituting modern neural NLP basis. In light of the recent advances in NLP described in this chapter, subsequent chapters introduce novel methods to apply these "foundation models" to improve online user interactions.

Chapter 3

Civil Rephrases Of Toxic Texts With

Self-Supervised Transformers

This chapter presents research conducted with John Pavlopoulos, Jeffrey Sorensen, and Lucas Dixon. It led to an article published in the Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2021) [START_REF] Laugier | Civil rephrases of toxic texts with self-supervised transformers[END_REF].

Chapter 3 abstract

Platforms that support online commentary, from social networks to news sites, are increasingly leveraging Machine Learning to assist their moderation efforts. But this process does not typically provide feedback to the author that would help them contribute according to the community guidelines. This is prohibitively timeconsuming for human moderators to do, and computational approaches are still nascent. INPUT OFFENSIVE COMMENT dubya a was a moron. GENERATED CIVIL COMMENT dubya was a republican.

a A nickname for George W. Bush. Table 3.1: Examples of offensive sentences from the Civil Comments test set and the more civil rephrasing generated by our model. The third example shows that its strategy may involve shifting the original intent since "republican" is not a non-offensive synonym of "moron".

Introduction

There are many ways to express our opinions. When we exchange views online, we do not always immediately measure the emotional impact of our message. Even when the opinions expressed are legitimate, well-intentioned, and constructive, poor phrasing may make the conversation go awry [START_REF] Zhang | Conversations gone awry: Detecting early signs of conversational failure[END_REF]. Recently, NLP research has tackled the problem of abusive language detection by developing accurate classification models that flag toxic (or abusive, offensive, hateful) comments [START_REF] Davidson | Automated hate speech detection and the problem of offensive language[END_REF][START_REF] Pavlopoulos | Deeper attention to abusive user content moderation[END_REF][START_REF] Wulczyn | Ex machina: Personal attacks seen at scale[END_REF][START_REF] Gambäck | Using convolutional neural networks to classify hate-speech[END_REF][START_REF] Fortuna | A survey on automatic detection of hate speech in text[END_REF][START_REF] Zhang | Conversations gone awry: Detecting early signs of conversational failure[END_REF][START_REF] Van Hee | Automatic detection of cyberbullying in social media text[END_REF][START_REF] Zampieri | Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval)[END_REF].

The prospect of healthier conversations, nudged by ML systems, motivates the development of NLU and NLG models that could later be integrated into a system suggesting alternatives to vituperative comments before they are posted. A first approach would be to train a text-to-text model [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF][START_REF] Vaswani | Attention is all you need[END_REF] on a corpus of parallel comments where each offensive comment has a courteous and fluent rephrasing written by a human annotator. However, such a solution requires a large paired labeled dataset, in practice difficult and expensive to collect (see Section 3.7). Consequently, we limit our setting to the self-supervised case where the comments are only annotated in attributes related to toxicity, such as the Civil Comments dataset [START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF]. We summarize our investigations with the following research question: RQ: Can we fine-tune end-to-end a pretrained text-to-text transformer to suggest civil rephrasings of rude comments using a dataset solely annotated in toxicity?

Answering this question might provide researchers with an engineering proof-of-concept that would enable further exploration of the many complex questions that arise from such a tool being used in conversations. The main contributions of this work are the following:

• We addressed for the second time the task of unsupervised civil rephrases of toxic texts, relying for the first time on the Civil Comments dataset and achieving results that reflect the effectiveness of our model over baselines.

• We developed a non-task-specific approach (i.e., with no human hand-crafting in its design) that can be gen-eralized and later applied to related and/or unexplored attribute transfer tasks.

While several of the ideas we combine in our model have been studied independently, to the best of our knowledge, no existing self-supervised models combine sequence-to-sequence bitransformers, transfer learning from large pretrained models, and self-supervised fine-tuning (Denoising Auto-Encoder and Cycle Consistency). We discuss the related work introducing these tools and techniques in the following section.

Related work

Unsupervised complex text attribute transfer (like civil rephrasing of toxic comments) remains in its early stages, and our particular applied task has only a single antecedent [START_REF] Nogueira Dos Santos | Fighting offensive language on social media with unsupervised text style transfer[END_REF]. There is a great variety of useful works to tackle the task and this section attempts to summarize the vast majority of these works. We describe below the recent strategies in style transfer when applied to image-to-image tasks. Then, we present the most related lines of work in unsupervised-or more precisely self-supervised-text-to-text tasks.

Style transfer in Computer Vision

The first successful application of Deep Learning to style transfer dates back to the work of Gatys et al. [START_REF] Gatys | A neural algorithm of artistic style[END_REF] who relied on the supervised pretraining of a CNN to synthesize paintings in a specific style. Then, Zhu et al. [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF] proposed a more general approach to image-to-image translation using Generative Adversarial Network (GAN) [START_REF] Goodfellow | Generative adversarial nets[END_REF]. CycleGAN is an unsupervised learning method. This means it does not need any correspondence between paired images with the exact same content (e.g., a pair of images with three Equidae in a specific position with a specific background) and different attributes (on the first image, the Equidae are horses, while on the second they are zebras). For training, CycleGAN only needs two "collections" of images, each collection having a known specific attribute or style (e.g., a collection of horses and a collection of zebras). The goal is to make CycleGAN learns the mapping between the first collection (made of examples in the "source" space, with the source attribute) and the second collection (made of examples in the "destination" space, with the destination attribute).

A GAN consists in a two ANNs: a generator (G) and a discriminator (D). The generator aims to transfer images from the source space (X) to the destination space (Y ). The discriminator computes the probability that its input image belongs to the destination space. The adversarial training consists in optimizing the discriminator's weights in order for it to correctly classify images in the destination space while maximizing the probability that the generator outputs images considered as destination images by the discriminator. The GAN loss function is describe in Equation (3.1). Its formal optimization will be given in Equation (3.4).

L GAN (G, D, X, Y ) = E y∼Y [log P (y ∈ Y |θ D )] + E x∼X [log (1 -P (G (x) ∈ Y |θ D ))] (3.1)
L CYC (G A2B , G B2A ) = E x∼X [∥G B2A (G A2B (x)) -x∥ 1 ] + E y∼Y [∥G A2B (G B2A (y)) -y∥ 1 ] (3.2)
The CycleGAN model can then be trained by optimizing the loss L CycleGAN (cf. Equation (3.3)) being the sum of the adversarial loss and the Cycle Consistency loss (cf. Equation (3.4)).

L CycleGAN (G A2B , G B2A , D A , D B ) = L GAN (G A2B , D B , A, B) + L GAN (G B2A , D A , B, A) + L CYC (G A2B , G B2A ) (3.3) ĜA2B , ĜB2A = arg min G A2B ,G B2A max D A ,D B L CycleGAN (G A2B , G B2A , D A , D B ) (3.4)
A natural question arises when we investigate unsupervised text-to-text transfer: could CycleGAN be directly applied to text? Unfortunately, the answer is no, and the reason is mainly due to the discrete nature of text. Indeed as we saw in Section 2.5.7, search strategies involved in decoding the representations-after the softmax layer-often break the chains of differentiability. Therefore, gradient-based training cannot use the standard backpropagation algorithm. Yet, popular workarounds have been proposed. Besides uses of reinforcement learning to solve the non-differentiability problem when sampling in NLG, efforts to generate text in a continuous space only have been proposed [START_REF] Baziotis | Seq3: Differentiable sequence-to-sequenceto-sequence autoencoder for unsupervised abstractive sentence compression[END_REF][START_REF] Kumar | Von mises-fisher loss for training sequence to sequence models with continuous outputs[END_REF].

However, these methods being unstable, hard to train in practice and/or nascent at the time of our work, we decided not to apply CycleGAN, but rather seek ideas in the then recently promising unsupervised neural machine translation.

End-to-end training of text-to-text models

Artetxe et al. [START_REF] Artetxe | Unsupervised neural machine translation[END_REF], Conneau et al. [START_REF] Conneau | Word translation without parallel data[END_REF], Lample et al. [START_REF] Lample | Unsupervised machine translation using monolingual corpora only[END_REF][START_REF] Lample | Phrase-based & neural unsupervised machine translation[END_REF], Conneau and Lample [START_REF] Conneau | Cross-lingual language model pretraining[END_REF] introduced methods based on techniques aligning the embedding spaces of monolingual datasets and tricks such as denoising auto-encoding losses [START_REF] Vincent | Extracting and composing robust features with denoising autoencoders[END_REF] and back-translation [START_REF] Sennrich | Improving neural machine translation models with monolingual data[END_REF][START_REF] Edunov | Understanding back-translation at scale[END_REF].

Abstractive summarization (or sentence compression) has also been studied in unsupervised setups. Baziotis et al. [START_REF] Baziotis | Seq3: Differentiable sequence-to-sequenceto-sequence autoencoder for unsupervised abstractive sentence compression[END_REF] trained a model with a compressor-reconstructor strategy similar to back-translation while Liu et al. [START_REF] Liu | Summae: Zero-shot abstractive text summarization using length-agnostic auto-encoders[END_REF] trained a Denoising Auto-Encoder that embeds sentences and paragraphs in a common space.

Unsupervised attribute transfer is the task most related to our work. It mainly focuses on sentiment transfer with standard review datasets [START_REF] Maas | Learning word vectors for sentiment analysis[END_REF][START_REF] He | Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering[END_REF][START_REF] Shen | Style transfer from non-parallel text by cross-alignment[END_REF][START_REF] Li | Delete, retrieve, generate: a simple approach to sentiment and style transfer[END_REF], but also addresses sociolinguistic datasets containing text in various registers [START_REF] Gan | Semantic compositional networks for visual captioning[END_REF][START_REF] Rao | Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer[END_REF] or with different identity markers [START_REF] Voigt | RtGender: A corpus for studying differential responses to gender[END_REF][START_REF] Prabhumoye | Style transfer through back-translation[END_REF][START_REF] Lample | Multiple-attribute text rewriting[END_REF]. When paraphrase generation aims at being explicitly attribute-invariant, it is referred to as obfuscation or neutralization [START_REF] Emmery | Style obfuscation by invariance[END_REF][START_REF] Xu | Privacy-aware text rewriting[END_REF][START_REF] Pryzant | Automatically neutralizing subjective bias in text[END_REF]. Literary style transfer [START_REF] Xu | Paraphrasing for style[END_REF][START_REF] Pang | Unsupervised evaluation metrics and learning criteria for non-parallel textual transfer[END_REF] has also been tackled by recent work. Here, we applied attribute transfer to a large dataset annotated in toxicity, but we also use the Yelp review dataset from Shen et al. [START_REF] Shen | Style transfer from non-parallel text by cross-alignment[END_REF] for comparison purposes (see Section 3.4).

Initial unsupervised attribute transfer approaches sought to build a shared and attribute-agnostic latent representation encoding for the input sentence with adversarial training. Then, a decoder, aware of the destination attribute, generated a transferred sentence [START_REF] Shen | Style transfer from non-parallel text by cross-alignment[END_REF][START_REF] Hu | Toward controlled generation of text[END_REF][START_REF] Fu | Style transfer in text: Exploration and evaluation[END_REF][START_REF] Zhang | Style transfer as unsupervised machine translation[END_REF][START_REF] Xu | Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach[END_REF][START_REF] John | Disentangled representation learning for non-parallel text style transfer[END_REF].

Unsupervised attribute transfer approaches that do not rely on a latent space are also present in the literature. Li et al. [START_REF] Li | Delete, retrieve, generate: a simple approach to sentiment and style transfer[END_REF] assumed that style markers are very local and proposed to delete the tokens most conveying the attribute before retrieving a second sentence in the destination style. They eventually combined both sentences with a neural network. Lample et al. [START_REF] Lample | Multiple-attribute text rewriting[END_REF] applied unsupervised neural machine translation techniques from Conneau and

Lample [START_REF] Conneau | Cross-lingual language model pretraining[END_REF] to several attribute transfer tasks, including social media datasets. Xu et al. [START_REF] Xu | Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach[END_REF], Gong et al. [START_REF] Gong | Reinforcement learning based text style transfer without parallel training corpus[END_REF], Luo et al. [START_REF] Luo | A dual reinforcement learning framework for unsupervised text style transfer[END_REF], Wu et al. [START_REF] Wu | A hierarchical reinforced sequence operation method for unsupervised text style transfer[END_REF] trained models with reinforcement learning. Dai et al. [START_REF] Dai | Transformer-xl: Attentive language models beyond a fixed-length context[END_REF] introduced unsupervised training of a transformer called StyleTransformer (ST) with a discriminator network. Our approach differs from these unsupervised attribute transfer models in that they did not either leverage large pretrained transformers or train with a denoising objective.

The most similar work to ours is the one from Nogueira dos Santos et al. [START_REF] Nogueira Dos Santos | Fighting offensive language on social media with unsupervised text style transfer[END_REF] who trained for the first time an encoder-decoder rewriting offensive sentences in a non-offensive register with non-parallel data from Twitter [START_REF] Ritter | Unsupervised modeling of twitter conversations[END_REF] and Reddit [START_REF] Serban | A deep reinforcement learning chatbot[END_REF]. Our approach differs in the following aspects. First, we use transformers pretrained on a large corpus instead of randomly initialized RNNs for encoding and decoding. Second, their approach involves collaborative classifiers to penalize generation when the attribute is not transferred, while we train end-to-end with a Denoising Auto-Encoder. Even if their model shows high accuracy scores, it suffers from low fluency, with offensive words being often replaced by a placeholder (e.g. "big" instead of "f*cking").

Controlled text generation [START_REF] Ficler | Controlling linguistic style aspects in neural language generation[END_REF][START_REF] Keskar | CTRL -A Conditional Transformer Language Model for Controllable Generation[END_REF][START_REF] Le | Flaubert: Unsupervised language model pre-training for french[END_REF][START_REF] Dathathri | Plug and play language models: A simple approach to controlled text generation[END_REF] is a NLG task that consists of a LM conditioned on the attributes of the generated text such as the style. But a major difference with attribute transfer is the absence of a constraint regarding preserving the input's content.

Method

Formalization of the text attribute rewriting problem

Let X T and X C be our two non-parallel corpora of comments satisfying the respective attributes "toxic" and "civil".

Let X = X T ∪ X C . We aim at learning a parametric function f θ mapping a pair of source sentence x and destination attribute a to a fluent sentence y satisfying a and preserving the meaning of x. In our case, there are two attributes, "toxic" and "civil", that we assumed to be mutually exclusive. We denote α(x) to be the attribute of x and ᾱ(x) the other attribute (for instance when α(x) = "civil", then ᾱ(x) = "toxic"). Note that f θ (x, α (x)) can simply be x.

Bi-conditional encoder-decoder generation

Our approach consists in training an auto-regressive LM (cf. Section 2.5.7) conditioned on both the input text x and the destination attribute a.

We compute f θ with a LM P (y|x, a; θ). As we do not have access to ground-truth targets y, we propose in section 3.3.3 a training function that we assume to maximize P (y|x, a; θ) if and only if y is a fluent sentence with attribute a and preserving x's content. Additionally, we used an auto-regressive generating model where inference of ŷ is sequential and the token generated at step t + 1 depends on the tokens generated at previous steps:

P (ŷ t+1 |ŷ :t , x, a; θ).
To condition on the input text, we followed the work of Bahdanau et al. [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF], Vaswani et al. [START_REF] Vaswani | Attention is all you need[END_REF], Nogueira dos Santos et al. [START_REF] Nogueira Dos Santos | Fighting offensive language on social media with unsupervised text style transfer[END_REF], Conneau and Lample [START_REF] Conneau | Cross-lingual language model pretraining[END_REF], Lample et al. [START_REF] Lample | Multiple-attribute text rewriting[END_REF], Dai et al. [START_REF] Dai | Style transformer: Unpaired text style transfer without disentangled latent representation[END_REF], Liu et al. [START_REF] Liu | Summae: Zero-shot abstractive text summarization using length-agnostic auto-encoders[END_REF], Raffel et al. [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF] and opted for an encoder-decoder (cf. Section 2.5.4) framework. Lample et al. [START_REF] Lample | Multiple-attribute text rewriting[END_REF] and Dai et al. [START_REF] Dai | Style transformer: Unpaired text style transfer without disentangled latent representation[END_REF] argue that in unsupervised attribute rewriting tasks, encoders do not necessarily output disentangled representations, independent of their attribute. However, the t-SNE visualization of the latent space in Liu et al. [START_REF] Liu | Summae: Zero-shot abstractive text summarization using length-agnostic auto-encoders[END_REF] allowed us to assume that encoders can output a latent representation z, attending to content rather than on an attribute, with similar training.

The LM is conditioned on the destination attribute with control codes introduced by Keskar et al. [START_REF] Keskar | CTRL -A Conditional Transformer Language Model for Controllable Generation[END_REF]. At the time of this work, control codes were defined as a fixed sequence of tokens prepended to the decoder's input s, and supposed to prepare the generation in the space of sentences with the destination attribute a. Note that the term of prefixed prompts (cf. Section 2.5.9) would better fit the vocabulary found in the current literature. Let c(a) be the control code of attribute a.

Self-supervised training

Training transformers with denoising objectives is an effective self-supervised strategy (cf. Section 2.5.5).

During training, we corrupted the encoder's input x with the noise function from Devlin et al. [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] (cf. Section 2.5.8). We denote x the corrupted version of x. We trained the model as a Denoising Auto-Encoder (DAE), meaning that we minimized the negative log-likelihood of Equation (3.5).

L DAE = E x∼X [-log P (x|x, α (x) ; θ)] (3.5)
The hypothesis is that optimizing the DAE objective teaches the controlled generation to the model.

Inspired by the equivalent approach in unsupervised image-to-image style transfer (cf. Section 3.2.1), we added a Cycle Consistency (CC) objective [START_REF] Nogueira Dos Santos | Fighting offensive language on social media with unsupervised text style transfer[END_REF][START_REF] Edunov | Understanding back-translation at scale[END_REF][START_REF] Prabhumoye | Style transfer through back-translation[END_REF][START_REF] Lample | Multiple-attribute text rewriting[END_REF][START_REF] Conneau | Cross-lingual language model pretraining[END_REF][START_REF] Dai | Style transformer: Unpaired text style transfer without disentangled latent representation[END_REF] (cf. Equation (3.6)) which enforces content preservation in the generated prediction.

L CC = E x∼X -log P x|f θ (x, ᾱ (x)) , α(x); θ (3.6)
As the Cycle Consistency objective computes a non-differentiable auto-regressive pseudo-prediction ŷ during stochastic gradient descent training, gradients are not back-propagated to θ = θτ-1 at training step τ .

Finally, the loss function sums the DAE and the CC objectives with weighting coefficients (cf. Equation (3.7)).

L = λ DAE L DAE + λ CC L CC (3.7)

The text-to-text bitransformer architecture

The architectures for the encoder and decoder are unitransformers. Contrary to Vaswani et al. [START_REF] Vaswani | Attention is all you need[END_REF], Conneau and Lample [START_REF] Conneau | Cross-lingual language model pretraining[END_REF], Raffel et al. [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF] we did not keep the decoder's layers computing cross attention between the encoder's outputs h and the decoder's hidden representations because generation suffers from too much conditioning on the input sentence and we observe no significant change in the output sentence. Rather, we followed Liu et al. [START_REF] Liu | Summae: Zero-shot abstractive text summarization using length-agnostic auto-encoders[END_REF] and computed the latent representation z with an affine transformation of the encoder's hidden state h 0 (corresponding to the first token of the input text). Let x ∈ X be the input sequence of tokens. It is embedded and then encoded by the unitransformer encoder:

x Emb = f θ Emb (x) h Enc = f θ Enc (x Emb ) h 0 Enc = h Enc [0, :] z = f θ Dense (h 0 Enc )
z is an aggregate sequence representation for the input. There are different heuristics that can be used to integrate it into the decoder. We considered summing z to the embedding of each token of the unitransformer decoder's input s since it balances the backpropagation of the signals coming from the original input and the output being generated in the destination attribute space, and it worked well in practice in our experiments.

h Dec = f θ Dec (f θ Emb (c (a) ⌢ x) + z) ŷ = f θ LMHead (h Dec )
Plus, the encoder and the decoder unitransformers share the same embedding layer, and the LM Head is tied to the embeddings.

Except for the dense layer computing the latent variable z, all parameters are coming from the bitransformer pretrained with a seq2seq MLM objective (cf. Section 2.5.

Experiments

Datasets

We employed the largest publicly available toxicity detection dataset to date, which was used in the 'Jigsaw Unintended Bias in Toxicity Classification' Kaggle challenge. 1 The 2M comments of the Civil Comments dataset stem from a commenting plugin for independent news sites. They were created from 2015 to 2017 and appeared on approximately 50 English-language news sites across the world. Each of these comments was annotated by crowd

Algorithm 1: CAE-T5 training

Input : T5's pretrained parameters θ 0 , unpaired dataset labelled in toxicity X = X T ∪ X C Output: CAE-T5's fine-tuned parameters θ T for The bitransformer encodes the corrupted input text x in a latent variable z that is then decoded conditioned on the source attribute α(x) with the objective of minimizing the cross entropy between x and the generated text x. Here, generation is not autoregressive since the DAE is trained with teacher forcing. (b) CC: The input x is pseudo-transferred with attribute ᾱ(x) via auto-regressive decoding because we do not know the ground-truth y. The generated output ŷ is then backtransferred to the original space of sentences with attribute α(x). Back-transfer generation is not auto-regressive because we use teacher-forcing here. Thus, we can trivially back-propagate the gradients through f θ (back-transfer) but not through f θ (pseudo-transfer). (c) Model overview at prediction time. x and ŷ are the input and generated text, a is the destination attribute, z is the latent input representation and c(a) is the control code. raters (at least 3 each) for toxicity and toxicity subtypes [START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF]. For more details on the dataset, see Section 4.3.

step τ ∈ [1; T ] do if τ %2 == 0 then Sample a mini-batch x of sentences in X T else Sample a mini-batch x of sentences in X C end θ ← θτ-1 θ ← θτ-1 xDAE ← f θ (x, α(x)) xCC ← f θ (f θ (x, ᾱ(x)), α(x)) ℓ DAE ← H(x, xDAE ) ℓ CC ← H(x, xCC ) ℓ ← λ DAE ℓ DAE + λ CC ℓ CC Back-

Yelp

Following the work of Dai et al. [START_REF] Dai | Style transformer: Unpaired text style transfer without disentangled latent representation[END_REF] for the IMDB Movie Review dataset (positive/negative sentiment labels), we constructed a sentence-level version of the dataset. Initially, we fine-tuned a pretrained BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] toxicity classifier on the Civil Comments dataset. Then, we split the comments in sentences with NLTK's sentence tokenizer. 2 Eventually, we created X T (respectively X C ) with sentences whose system-generated toxicity score (using our BERT classifier) is greater than 0.9 (respectively less than 0.1) to increase the dataset's polarity. We also conducted a comparison to other style transfer baselines on the Yelp Review Dataset (Yelp), commonly used to compare unsupervised attribute transfer systems. The dataset consists of restaurant and business reviews annotated with a binary positive/negative label. Shen et al. [START_REF] Shen | Style transfer from non-parallel text by cross-alignment[END_REF] processed it and Li et al. [START_REF] Li | Delete, retrieve, generate: a simple approach to sentiment and style transfer[END_REF] collected human reference human references for the test set3 . Table 3.2 shows statistics for these datasets.

Evaluation

Evaluating a text-to-text task is challenging, especially when no gold pairs are available. Attribute transfer is successful if generated text: 1) has the destination control attribute, 2) is fluent and 3) preserves the content of the input text.

Automatic evaluation

We followed the current approach of the community [553, [START_REF] Logeswaran | Content preserving text generation with attribute controls[END_REF][START_REF] Wang | Controllable unsupervised text attribute transfer via editing entangled latent representation[END_REF][START_REF] Xu | On variational learning of controllable representations for text without supervision[END_REF][START_REF] Lample | Multiple-attribute text rewriting[END_REF][START_REF] Dai | Style transformer: Unpaired text style transfer without disentangled latent representation[END_REF][START_REF] He | A probabilistic formulation of unsupervised text style transfer[END_REF] and approximated the three criteria with the following metrics:

1. Attribute control: Accuracy (ACC) computes the rate of successful changes in attributes. It measures how well the generation is conditioned by the destination attribute. We predicted toxic and civil attributes with the same fine-tuned BERT classifier that pre-processed the Civil Comments dataset (single threshold at 0.5).

2. Fluency: Fluency was measured by Perplexity (PPL). To measure PPL, we employed a GPT-2 [START_REF] Radford | Language models are unsupervised multitask learners[END_REF] causal LM fine-tuned on the corresponding datasets (Civil Comments and Yelp). [START_REF] Conneau | Cross-lingual language model pretraining[END_REF], summarization [START_REF] Liu | Summae: Zero-shot abstractive text summarization using length-agnostic auto-encoders[END_REF] and sentiment transfer [START_REF] Li | Delete, retrieve, generate: a simple approach to sentiment and style transfer[END_REF] have access to a few hundred samples with at least one human reference of the transferred text and evaluate content preservation by computing metrics based on overlapping n-grams (e.g., BLEU Papineni et al. [START_REF] Papineni | Bleu: a method for automatic evaluation of machine translation[END_REF]) between the generated prediction and the reference(s) (ref-metric). However, as we did not have paired samples, we computed a content preservation score between the input and the generated sentences (self-metric). Table 3.3 shows the BLEU scores (based on exact matches) of three examples rephrased by human annotators (Section 3.7). In the top-most example, the BLEU score is high. This is explained by the fact that only 4 words are different between the two texts. In contrast to the first example, the two texts in the second example have only 1 word in common. Thus, the BLEU score is low. Despite the low evaluation, however, the candidate text could have been a valid rephrase of the reference text.

The high complexity of our task explains the motivation for a more general quantitative metric between input and generated text, capturing the semantic similarity rather than overlapping tokens. We did not consider the novel metrics BERTScore [START_REF] Zhang | Bertscore: Evaluating text generation with bert[END_REF], RUSE [START_REF] Shimanaka | Ruse: Regressor using sentence embeddings for automatic machine translation evaluation[END_REF] and BLEURT [START_REF] Sellam | BLEURT: Learning robust metrics for text generation[END_REF] because they either are based on token-totoken comparison or require fine-tuning on pairs of (toxic comment, human civil rephrasing) annotated with human quality judgement. Fu et al. [START_REF] Fu | Style transfer in text: Exploration and evaluation[END_REF], John et al. [START_REF] John | Disentangled representation learning for non-parallel text style transfer[END_REF], Gong et al. [START_REF] Gong | Reinforcement learning based text style transfer without parallel training corpus[END_REF], Pang and Gimpel [START_REF] Pang | Unsupervised evaluation metrics and learning criteria for non-parallel textual transfer[END_REF] proposed to represent sentences as a (weighted) average of their words embeddings before computing the cosine similarity between them. We adopted a similar strategy, but we embedded sentences with the pretrained Universal Sentence Encoder [START_REF] Cer | Universal sentence encoder[END_REF] and called it the sentence similarity score (SIM). The same sentence similarity was used to compare sentences in Chapter 5. The first two sentence pairs of Table 3.3 had high similarity scores.

The rephrasings preserved the original content while not necessarily overlapping much with the original text.

However, the last rephrasing did not preserve the initial content and had a low similarity score with its source sentence. As statistical evidence, the self-SIM score comparing each of the 1, 000 test Yelp reviews with their human rewriting was 80.2% whereas the self-SIM score comparing the Yelp review test set to a random derangement of the human references was 36.8%. Section 3.9.2 gives additional arguments in favor of using SIM for measuring content preservation.

We optimized all three metrics because doing otherwise comes at the expense of the remaining metric(s).

We aggregated the scores of the three metrics by computing the geometric mean 4 (GM) of ACC, 1/PPL and self-SIM.

Human evaluation

Baselines

We compared the output text that CAE-T5 generated with a selection of unpaired style-transfer models described in Section 3.2 [START_REF] Shen | Style transfer from non-parallel text by cross-alignment[END_REF][START_REF] Li | Delete, retrieve, generate: a simple approach to sentiment and style transfer[END_REF][START_REF] Fu | Style transfer in text: Exploration and evaluation[END_REF][START_REF] Luo | A dual reinforcement learning framework for unsupervised text style transfer[END_REF][START_REF] Dai | Style transformer: Unpaired text style transfer without disentangled latent representation[END_REF]. We also compared with Input Masking. Input Masking is inspired by an interpretability method called Input Erasure (IE) [START_REF] Li | Understanding neural networks through representation erasure[END_REF]. IE has been used to interpret the decisions of neural models.

Initially, words are removed one at a time, and the altered texts are then re-classified (i.e., as many re-classifications as the words). Then, all the words that led to a decreased re-classification score (based on a threshold) are returned as the ones most related to the decision of the neural model. Our baseline follows a similar process, but instead of deleting, it uses a pseudo token ('[MASK]') to mask one word at a time. When all the masked texts have been scored by the classifier, the rephrased text is returned, comprising as many masks as the tokens that led to a decreased re-classification score (set to 20% after preliminary experiments). We employed a pretrained BERT as our toxicity classifier, fine-tuned on the Civil Comments dataset (see Section 3.4.1).

Experimental setup

Architecture details

We fine-tuned the pretrained "large" bitransformer from Raffel et al. [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF]. Both unitransformers (encoder and decoder) had 24 blocks, each made of a 16-headed self-attention layer and a feed-forward network. The attention, Figure 3.3: Guidelines provided to human crowd-workers on Appen when they were asked to rate automatic civil rephrasings of toxic comments. Figure 3.4: Guidelines provided to human crowd-workers on Appen, when they were asked to rewrite toxic comments in a way that is less rude. dense, and embedding layers had respective dimensions of 64, 4096, and 1024, for a total of around 800 million parameters.

Input sentences were lowercased then tokenized with SentencePiece6 [START_REF] Kudo | SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing[END_REF] and eventually truncated to a maximum sequence length of 32 for the Yelp dataset and 128 for the processed Civil Comments dataset. The control codes were c(a) = (a) ⌢ (": ") for attributes a ∈ {"positive", "negative"} in the sentiment transfer task and a ∈ {"toxic", "civil"} when we apply to the Civil Comments dataset.

Training details

During training, we applied dropout regularization at a rate of 0.1. We set λ DAE = λ CC = 1.0. In preliminary experiments, we observed that λ CC = 0 was preserving little content from the initial sentence and that λ CC = 2 * λ DAE was weighting the preservation too much, at the cost of accuracy. Therefore we focused our experiments on λ CC = λ DAE . It is a good default setting since we do not need any a priori about the balance between fluency, accuracy (enforced with the Auto-Encoder), and content preservation (enforced with Cycle Consistency). DAE and backtransfer (in the course of the CC computation) were trained with teacher-forcing; we did not need auto-regressive generation since we had access to a target for the decoder's output. Each training step computed the loss on a mini-batch made of 64 sentences sharing the same attribute. Mini-batches of attributes a and ā were interleaved.

Since the Civil Comments dataset is class imbalanced, we sampled comments from the civil class of the training set at each epoch. The optimizer is AdaFactor [START_REF] Shazeer | Adafactor: Adaptive learning rates with sublinear memory cost[END_REF] and we trained for 88, 900 steps during 19 hours on a TPU v2 chip.

Automatic evaluation details

Decoding was greedy. The parametric models used to compute ACC and PPL were 12-layer, 12-headed pretrained, and fine-tuned unitransformers with hidden size 768. The BERT classifier was an encoder followed by a sequence classification head, and the GPT-2 causal LM was a decoder with a LM head on top. We used the sacrebleu 7implementation for BLEU and the Universal Sentence Encoder pretrained by Google to compute SIM8 . higher accuracy but at the cost of both higher Perplexity and lower content preservation, meaning that they are better are discriminating toxic phrases but struggle to rephrase coherently.

Results

Quantitative comparison to prior work

In Table 3.5 we compare our model to prior work in attribute transfer by computing evaluation metrics for different systems on the Yelp test set. We achieved competitive results with low Perplexity while getting good sentiment controlling (above human references). Our similarity, though, is lower, showing that some content was lost when decoding; hence the latent space did not fully capture the semantics. It is fairer to compare our model to other style transfer baselines on the Yelp dataset since our model is based on sub-word tokenization while the baselines are often based on a limited size pretrained word embedding: many more words from the Civil Comments dataset could be attributed to the unknown token if we want to keep reasonable size vocabulary, resulting in a performance drop.

The human evaluation results shown in Table 3.6 correlate with the automatic evaluation results.

When considering the aggregated scores (geometric mean, success rate, and overall human judgment), our model is ranked first on the Civil Comments dataset and second on the Yelp Review dataset, behind DualRL, yet our approach is more stable and therefore easier to train when compared to reinforcement learning approaches. try reading and be a little more informed about it before you try to make a comment.

Qualitative analysis

this is absolutely the most idiotic post i have ever read on all levels.

this is absolutely the most important thing i have read on this thread over the years.

trump may be a moron, but clinton is a moron as well.

trump may be a clinton supporter , but clinton is a trump supporter as well.

shoot me in the head if you didn't vote for trump. @ @ @ @ @ @ you're right if you didn't vote for trump. h h h h h h h h h i'm not sure i'd vote 50% of teachers don't have any f*cks to give. 50% of teachers don't have @ @ @ @ @ @ @ @ @ h h h h h h h h h a phd in anything.

Table 3.7: Examples of automatically transferred test sentences by our system, valid rewriting, and highlighted flaws failure in attribute transfer or fluency , h h h h h h h supererogation, @ @ @ @ @ @ @ @ position reversal, and @ @ @ @ @ @ h h h h h h hallucination.

Hallucination conditioning on the initial sentence fails, and the model generates fluent but unrelated content.

Position reversal the author's opinion is shifted.

In order to assess the frequency of hallucination and supererogation, we randomly selected 100 toxic comments from the test set and manually labeled the generated sentences with the non-mutually exclusive labels "contains supererogation" and "contains hallucination". We counted on average 17% of generated sentences with supererogation and 34% of generated sentences showing hallucination (often local). We observed that the longer the input comment, the more prone to hallucination was the generated text.

While supererogation and hallucination can be explained by the probabilistic nature of generation, we assume that position reversal was due to bias in the dataset, where toxic comments are correlated with negative comments. Thus, offensive comments tend to be transferred to supportive comments even though a human being would rephrase attacks as polite disagreements.

Interestingly, our model was able to add toxicity to civil comments, as shown by the examples in Table 3.12. Even if such an application is of limited interest to online platforms, it is worth warning about its potential misuse.

Discussion

Strongly supervised learning is a natural approach when addressing text-to-text tasks. In our study, we submitted the civil rephrasing of toxic comments task to human crowd-sourcing. We randomly sampled 500 sentences from the toxic train set. For each sentence, we asked five annotators to rephrase it in a civil way to assess if the comment was offensive and if it was possible to rewrite it in a less rude way while preserving the content. On 2, 500 answers, we tally 427 examples not flagged as 'impossible to rewrite' and with a rephrasing different from the original sentence. This low 17.1% yield was caused by two main issues. On the one hand, unfortunately, not all toxic comments can be reworded in a civil manner so as to express a constructive point of view; severely toxic comments that are solely

Toxic to Civil

Input and then they need to do what it takes to get rid of this mentally ill bigot! CAE-T5 and then they need to do what they can to get rid of this kind of man that is causing the whole problem! Input do you think any of those idiots vote? CAE-T5 do you think any of those people vote?

Input

this is just so stupid. CAE-T5 this is just so sad.

Input

are you nuts? CAE-T5 are you kidding me?

Input

we are all entitled to our own opinions but not the facts that one can't find in this criminally stupid opinion. CAE-T5 we are all entitled to our own opinion but not to the kind of facts that can be found in this case.

Input hopefully that will prevent him from making any further idiotic statements. CAE-T5 hopefully that will prevent him from making any more such statements.

Input

the poll is garbage. CAE-T5 the poll is fake news.

Input

it was irresponsible to publish this garbage. CAE-T5 it was irresponsible to publish this information.

Input

just because someone else has a different opinion does not mean that person is stupid, uneducated or foolish. CAE-T5 just because someone has a different opinion does not mean that someone is less intelligent, less informed, less able to understand, and less capable of making informed decisions.........

Input

referring to. h h h h h h h h h h . i'm just a little more

Input the ontario voters were rated the dumbest people in all the whole world. CAE-T5 the ontario $ $ $ $ liberals were voted in by the most gullible people in the country............... Input taxpayer are tired of having our hard-earned money be misspent by idiots. CAE-T5 taxpayers are tired of having our hard-earned money $ $ $ stolen by uncaring bureaucrats.

Input

suggesting that people change their commute times is stupid. CAE-T5 suggesting that people change their $ $ $ $ $ lifestyles is not helpful.

Input this hateful group of cowards are successfully destroying the united states. CAE-T5 the hateful group of @ @ @ @ @ @ @ @ h h h h h h h h trump supporters are successfully dividing the united states.

Input whatever the idiot in chief touches turns to crap.

CAE-T5 whatever the president does h h h h h h

in the room turns to @ @ @ @ @ @ @ h h h h h h h what he wants.

Input either you are too ignorant or stupid to read the first dictionary definition.

CAE-T5 either you are too h h h h h h h

young to read or are too old to understand @ @ @ @ @ @ @ @ @ @ h

h h h h h h h h h the first amendment h h h h h h h h h h h or the first amendment.

Input

trump is doing right things, he is great president, a lot better than 44th one from kenya lot of good people support trump, only crooked and dumb people against CAE-T5 trump is doing great, i know, he is better than most of the other @ @ @ @ @ @ h h h h h h candidatesobama $ $ $ $ , clinton,

Input the dumb become dumber. CAE-T5 the $ $ $ poor become $ $ $ richer.

Table 3.9: Examples of automatically transferred toxic test sentences by our system, valid rewriting, and highlighted flaws failure in attribute transfer or fluency , h h h h h h h supererogation, @ @ @ @ @ @ @ @ position reversal, and @ @ @ @ @ @ h h h h h h hallucination. Input friendly and welcoming with a fun atmosphere and terrific food . DualRL rude and unprofessional with a loud atmosphere and awful food . ST (Multi) poor and fake with a fun atmosphere and mushy food .

Positive to Negative

Input

CAE-T5

rude and unhelpful service with a forced smile and attitude . Human unfriendly and unwelcoming with a bad atmosphere and food .

Input i love their star design collection . DualRL i hate their star design disgrace . ST (Multi) i do n't care star $ $ $ bites collection . CAE-T5 i hate @ @ @ @ @ @ @ @ @ h

h h h h h h h h starbucks corporate . h h h h h h h h h

the staff is horrible . Human i ca n't stand their star design collection .

Input oj and jeremy did a great job ! DualRL oj and jeremy did a great job ! h h h h h h h h h h disgrace ! disgrace ! ST (Multi) oj and jeremy did a terrible job ! CAE-T5

& & oh and @ @ @ @ @ @ h h h h h h jesus christ @ @ @ @ @ @ @ @ @ @ @ @ h h h h h h h h h h h h i did n't have any change ! Human oj and jeremy did a terrible job ! Table 3.10: Examples of automatically transferred sentences from the Yelp test set (positive to negative), valid rewriting, and highlighted flaws failure in attribute transfer or fluency , h h h h h h h supererogation, and @ @ @ @ @ @ h h h h h h hallucination. Input my husband had to walk up to the bar to place our wine order . DualRL my husband had to walk up to the bar to place our drink order perfectly . ST (Multi) my husband had to walk up to the bar to place our wine order . CAE-T5 my husband loves to stand in line @ @ @ @ @ @ @ @ h h h h h h h h at the beer stand @ @ @ @ @ @ @ @ @ @ h h h h h h h h h h and watch the game . Human my husband did n't have to walk up to the bar to place our wine order .

Negative to Positive

Input

Table 3.11: Examples of automatically transferred sentences from the Yelp test set (negative to positive), valid rewriting, and highlighted flaws failure in attribute transfer or fluency , h h h h h h h supererogation, and @ @ @ @ @ @ h h h h h h hallucination.

Civil to Toxic

Input great effort and great season. CAE-T5 stupid and horrible move by this team.

Input

canadian teachers are the foundation of this great country. CAE-T5 canadians are the dumbest people in this country.

Input

why isn't trudeau in jail for gross negligence? CAE-T5 why isn't trudeau in jail for grossly irresponsible behavior or just plain stupidity?????

Input

there really is no safe place on the planet to live anymore. CAE-T5 there really is no safe place on the planet for the stupid to live.

Input

trudeau will do as he's told. CAE-T5 trudeau will do as he's told, he's stupid.

Input

all of canada is paying for that decision. CAE-T5 all of canada is paying for that stupid decision.

Input

people like you need their talking points. CAE-T5 people like you need their asses kicked.

Input

this is a great article that hits the nail on the head. CAE-T5 this is a $ $ $ stupid article that hits the nail on the head.

Input and just which money tree is going to pay for this? CAE-T5 and just where is the money going to come from to pay for this crap? h h h h h h h h h h h h h h when is this crap going to end? h h h h h h h who cares????

Input you need to get a new dictionary.

CAE-T5 you need to get a new f*cking dictionary. h h h h h h h h h h h h h get a new f*cking dictionary. h h h h h h h h h h h h h get a new f*cking dictionary. get a new

Input the real question is; will the hell's angels forgo their major source of income? CAE-T5 the real question is when will the idiocy of @ @ @ @ @ @ @ @ h h h h h h h h these trumpkins end?

Input according to http://cookpolitical.com/story/10174, trump got 46.1% of the popular vote. CAE-T5 according to @ @ @ @ @ @ @ @ h h h h h h h h trump, pence is the dumbest president ever, and @ @ @ @ @ @ @ @ @ @ @ @ @ h

h h h h h h h h h h h h

clinton got the second-worst @ @ @ @ @ @ @ h h h h h h h approval rating $ $ $ $ $ in history.. .. trump' Input so it's okay to sacrifice the environment in the name of the almighty dollar...... CAE-T5 so it's okay to destroy the world with the actions of one stupid dude @ @ @ @ @ @ @ @ @ h h h h h h h h h in the white house............

Input

the president dismissed the ecological findings of over 87% of scientists who have been studying the effects of global warming, largely caused by the release of carbon from fossil fuel into the atmosphere. CAE-T5 the president ignored the scientific consensus ¨r r that over 90% of all climate scientists @ @ @ @ @ @ h h h h h h are complete idiots, $ $ $ $ $ reacting to @ @ @ @ @ h h h h h the rash of @ @ @ @ @ @ @ @ h h h h h h h h terrorist attacks @ @ @ @ @ @ @ h h h h h h h that have been @ @ @ @ @ @ h h h h h h taking place @ @ @ @ @ @ @ @ h h h h h h h h around the world... @ @ @ @ @ @ @ @ h h h h h h h h trump has made @ @ @ @ @ h h h h h it his life'

Input not sure where you got your definition of a good guy.

CAE-T5 not sure where you got your @ @ @ @ @ @ @ @ @ @ @ @ h

h h h h h h h h h h h

idea that trump is a kinda dumb guy.

Table 3.12: Examples of automatically transferred civil test sentences by our system, valid rewriting, and highlighted flaws failure in attribute transfer or fluency , h h h h h h h supererogation, @ @ @ @ @ @ @ @ position reversal, and @ @ @ @ @ @ h h h h h h hallucination. For the test set of civil sentences, the automatic metrics are ACC= 92.8%; PPL= 9.8 and self-SIM= 54.3%. made of insults, identity attacks, or threats are not "rephrasable". On the other hand, evaluating crowd-workers with test questions and answers is complex. The Perplexity being higher on crowdworkers' rephrases than on randomly sampled civil comments raises concerns about the production of human references via crowd-sourcing. The nature of large datasets labeled in toxicity and the lack of incentives for crowd-sourcing civil rephrasing annotation make it expensive and difficult to train systems in a supervised framework. These limitations motivate self-supervised approaches.

Lastly, the more complex the unsupervised attribute transfer task, the more difficult its automatic evaluation is.

In our case, evaluating whether the attribute is actually transferred required to train an accurate toxicity classifier.

Furthermore, the LM we used to assess the fluency of the generated sentences has some limitations and does not generalize to all varieties of language encountered in social media. Finally, measuring the amount of relevant content preserved between the source and generated texts remains a challenging, open research topic.

Comparison with state-of-the-art unsupervised text simplification

In parallel with this project, recent works have proposed efficient unsupervised approaches to the related Conditional Natural Language Generation task of text simplification. While early attempts of unsupervised sentence simplification [START_REF] Surya | Unsupervised neural text simplification[END_REF][START_REF] Zhao | Semi-supervised text simplification with back-translation and asymmetric denoising autoencoders[END_REF] could not reach the performances of supervised methods, Martin [START_REF] Martin | Automatic sentence simplification using controllable and unsupervised methods[END_REF]'s paraphrase-mining-based strategy, called MUSS [START_REF] Martin | Multilingual unsupervised sentence simplification[END_REF], outperformed supervised Seq2Seq models.

In addition to fluency and content preservation in the generated text, simplification is characterized by sentence compression as well as constraints on lexical and syntactic complexity. In comparison, our task, conditioned on style transfer, may require semantic or syntactic transfer of different nature than reducing complexity 9 .

Besides learning to address a text-to-text task in unsupervised settings, CAE-T5 and MUSS share similar controllable mechanisms using prefix tokens 10 . Yet, at training time, our control codes (e.g., "civil: ") are not bound to pairs of examples, while MUSS's explicit proxy control tokens are (e.g., "<NbChars 0.3>", for a character compression ratio of 30%, is learned at training time through pairs of examples satisfying the ratio). Therefore, MUSS introduced unsupervised paraphrase mining at scale to align sentences that the Seq2Seq model was trained on.

Their mining strategy used Meta's sentence embedding model (LASER 11) similar to Google's Universal Sentence Encoder [START_REF] Cer | Universal sentence encoder[END_REF] that was used in our work 11 . A notable difference is that they opted for the L2 distance while we used the cosine distance. We kept for future work paraphrase mining in the Civil Comments dataset, whose individual sentences may be harder to align than sentences in CCNet [START_REF] Wenzek | CCNet: Extracting high quality monolingual datasets from web crawl data[END_REF] (used by MUSS).

Subsequent related works

Parallel detoxification datasets

Several works put our study in perspective between the publication of this work and the writing of the present thesis. Indeed, by releasing an English dataset (ParaDetox) of 12, 000 manually rewritten sentences, Dementieva et al.

[105], Logacheva et al. [START_REF] Logacheva | Paradetox: Detoxification with parallel data[END_REF] paved the way to strongly supervised models addressing detoxification. Additionally, a shared task (RUSSE-2022), proposed by Dementieva et al. [START_REF] Dementieva | Findings of the first russian detoxification shared task based on parallel corpora[END_REF], provided a parallel Russian dataset and used both automatic (including an embedding distance-based metric as in this work) and manual evaluation to compare participating models. Part of the English dataset was used to train a strongly supervised model in Chapter 4. Section 4.8.1 compares both methods.

Comparison of sentence similarity with other content preservation metrics

In order to benchmark our sentence similarity metric against standard content preservation metrics used in CNLG, we evaluated SIM with the recently released BEAMetrics [START_REF] Scialom | Beametrics: A benchmark for language generation evaluation evaluation[END_REF] benchmark. We used it as a tool to compute the correlations between quantitative human judgments and reference-based metric scores on a set of tasks and evaluation metrics covering a representative spectrum of CNLG research.

For a given annotated dataset addressing a specific task, BEAMetrics provides a set of source samples associated with one or several human-produced reference(s), a system-generated candidate, as well as quantitative human assessments of the candidate on the task, given the source sample. For a given sample, reference-based metrics usually compute a score between 0 and 1 from the system-generated text and the human-produced reference(s). This approach has been widely used in CNLG for its high correlation with human judgments. Table 3.13 shows Pearson correlation coefficients (%) for all (metric, dataset) pairs proposed by BEAmetrics, except non-English datasets. As a result, SIM is competitive with other standard CNLG metrics, especially when a single reference is available since its average person correlation score is ranked first. This tends to confirm that SIM is relevant when evaluating content preservation in unsupervised CNLG tasks, where a single reference is used: the source text. This argument could also be relevant in the context of Chapter 5, where SIM is used to compare users' tastes for the task of item rating prediction.

Chapter 3 conclusion

This work was the second to tackle civil rephrasing to our knowledge and the first to address it with a fully end-to-end discriminator-free text-to-text self-supervised training. CAE-T5 leverages the NLU / NLG power offered by large pretrained bitransformers. The quantitative and qualitative analysis showed that ML systems could contribute to some extent to pacify online conversations, even though many generated examples still suffer from critical semantic drift.

In the future, we plan to explore whether the decoding can benefit from NAR generation [START_REF] Ma | Flowseq: Non-autoregressive conditional sequence generation with generative flow[END_REF][START_REF] Ren | A study of non-autoregressive model for sequence generation[END_REF]. We are also interested in the recent paradigm shift proposed by Kumar and Tsvetkov [START_REF] Kumar | Von mises-fisher loss for training sequence to sequence models with continuous outputs[END_REF], where the generated tokens representation is continuous, allowing more flexibility in plugging attribute classifiers without sampling. of the Association for Computational Linguistics (ACL 2022) [START_REF] Pavlopoulos | From the detection of toxic spans in online discussions to the analysis of toxic-to-civil transfer[END_REF].

Chapter 4 abstract

We studied the task of toxic spans detection, which concerns the detection of the spans that make a text toxic when detecting such spans is possible. We introduced a dataset for this task, TOXICSPANS a , which we released publicly. By experimenting with several methods, we showed that sequence labeling models perform best. Moreover, methods that add generic rationale extraction mechanisms on top of classifiers trained to predict if a post is toxic or not are also surprisingly promising. Finally, we used TOXICSPANS and systems trained on it to provide further analysis of state-of-the-art toxic to non-toxic transfer systems seen in Chapter 3, as well as human performance on that latter task. Our work highlights challenges in finer toxicity detection and mitigation.

a Our code and dataset are publicly available at https://github.com/ipavlopoulos/toxic_spans with a CC0 license. Part of the dataset was also used in the SemEval-2021 Task 5 [START_REF] Pavlopoulos | SemEval-2021 task 5: Toxic spans detection[END_REF] (cf. Appendix A).

Introduction

In social media and online fora, toxic content can be defined as rude, disrespectful, or unreasonable posts that would make users want to leave the conversation [START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF]. Although several toxicity detection datasets [START_REF] Wulczyn | Ex machina: Personal attacks seen at scale[END_REF][START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF] and models [START_REF] Schmidt | A survey on hate speech detection using natural language processing[END_REF][START_REF] Pavlopoulos | Deeper attention to abusive user content moderation[END_REF][START_REF] Zampieri | Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval)[END_REF] exist, most of them classify whole posts, without identifying the specific spans that make a text toxic. But highlighting such toxic spans can assist human moderators (e.g., news portal moderators) who often deal with lengthy comments and prefer attribution instead of a system-generated unexplained toxicity score per post. Locating toxic spans within a text is thus a major step towards successful semi-automated moderation and healthier online discussions.

To promote research on this new task, we released the first dataset of English posts with annotations of toxic spans, called TOXICSPANS. We discussed how it was created and proposed an evaluation framework for toxic spans detection. We considered methods that (i) perform sequence labeling (tag words) or (ii) rely on an attentional binary classifier to predict if a post is toxic or not, then invoke its attention at inference time to obtain toxic spans as in rationale extraction. The latter approach allows leveraging larger existing training datasets, which provide gold labels indicating which posts are toxic or not, without providing gold toxic span annotations. Although sequence labeling performed better overall, the binary attentional classifier performed surprisingly well too, despite having been trained on data without span annotations.

We then studied some characteristics of strongly supervised and self-supervised toxic-to-civil transfer models [START_REF] Laugier | Civil rephrases of toxic texts with self-supervised transformers[END_REF] by comparing them on several datasets, including a recently released parallel toxic-to-civil dataset [START_REF] Dementieva | Crowdsourcing of parallel corpora: the case of style transfer for detoxification[END_REF] and the new TOXICSPANS dataset. Using the latter, we introduced a measure to evaluate the elimination of explicit toxicity, and we used this measure to compare the behavior and performance of toxic-to-civil models. Lastly, by applying toxic span detection systems, we assessed the performance of human crowdworkers on the toxic-to-civil task.

Related work

Toxicity detection systems [START_REF] Schmidt | A survey on hate speech detection using natural language processing[END_REF][START_REF] Pavlopoulos | Deeper attention to abusive user content moderation[END_REF][START_REF] Zampieri | Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval)[END_REF] are typically trained on datasets annotated at the post level (a text is annotated as toxic or not) [START_REF] Wulczyn | Ex machina: Personal attacks seen at scale[END_REF][START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF]. Our work differs from general toxicity detection in that we detect toxic spans, instead of assigning toxicity labels to entire texts. Toxic spans detection can be seen as a case of attribution or rationale extraction [START_REF] Li | Understanding neural networks through representation erasure[END_REF][START_REF] Ribeiro | Why Should I Trust You?" Explaining the predictions of any classifier[END_REF][START_REF] Lei | Rationalizing neural predictions[END_REF][START_REF] Zhang | Explain and predict, and then predict again[END_REF][START_REF] Jain | Learning to faithfully rationalize by construction[END_REF][START_REF] Deyoung | ERASER: A benchmark to evaluate rationalized NLP models[END_REF], but specifically for toxic posts, a task that has never been considered in general toxicity detection before.

Detecting spans, instead of entire posts, was recently also considered in propaganda [START_REF] Martino | A survey on computational propaganda detection[END_REF] and hate speech detection [START_REF] Mathew | Hatexplain: A benchmark dataset for explainable hate speech detection[END_REF]. Although the ground truth type is similar (spans), propaganda detection is a different task from ours. Hate speech is a particular type of toxicity [START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF], which can be tackled by more general toxicity detectors [START_REF] Van Aken | Challenges for toxic comment classification: An in-depth error analysis[END_REF] Nah, the only asshole is the asshole firing a rifle within city limits.

{ }

So tired of all these Portlanders moving to California and ruining the culture. When will it stop?!?

{ } Fits the despicable piece of trash like a glove.

{ }

No he shot someone so he shouldn't be free. He should be locked up like He is for the next decade. You holla like its no big deal but what if it was your little brother who was shot? Then your dumb ass wouldnrlt be screaming free ace g or whatever gay nickname he has. That is the lamest gangster name ever. Do you think that name scares people? Ace clown is what i will now call him. but not the other way round; i.e., we address a broader problem. This probably explains why a pattern-matching baseline, based on the data of Mathew et al. [START_REF] Mathew | Hatexplain: A benchmark dataset for explainable hate speech detection[END_REF], achieved only slightly better results than a random baseline on our dataset.

As seen in Chapter 3, suggesting civil rephrases of posts found to be toxic [START_REF] Nogueira Dos Santos | Fighting offensive language on social media with unsupervised text style transfer[END_REF][START_REF] Laugier | Civil rephrases of toxic texts with self-supervised transformers[END_REF] is the next step towards healthier online discussions and can be viewed as style transfer [START_REF] Shen | Style transfer from non-parallel text by cross-alignment[END_REF][START_REF] Fu | Style transfer in text: Exploration and evaluation[END_REF][START_REF] Lample | Multiple-attribute text rewriting[END_REF]. We show how toxic spans detection can contribute to the assessment of toxic-to-civil transfer, linking the two tasks together for the first time.

The CIVILCOMMENTS dataset

In 2015, when many publications were closing down comment sections due to moderation burdens, a start-up named Civil Comments launched [START_REF] Finley | Want to save the comments from trolls? do it yourself, 3[END_REF]. Using a system of peer-based review and flagging, they hoped to crowd source the moderation responsibility. When this effort shut down in 2017 [START_REF] Bogdanoff | Saying goodbye to civil comments, 12[END_REF], they cited the financial constraints of the competitive publishing industry and the challenges of attaining the necessary scale.

The founders of Civil Comments, in collaboration with researchers from Google Jigsaw, undertook an effort to open source the collection of more than two million comments that had been collected. After filtering the comments to remove personally identifiable information, a revised version of the annotation system of Wulczyn et al. [START_REF] Wulczyn | Ex machina: Personal attacks seen at scale[END_REF] was used on the Appen crowd rating platform to label the comments using a number of attributes, including 'toxicity', 'obscene', 'threat' Borkan et al. [START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF]. The complete dataset, partitioned into training, development, and test sets, was featured in a Kaggle competition,1 with additional material, including individual rater decisions, published [START_REF] Borkan | Exploring the role of human raters in creating nlp datasets, 11[END_REF] after the close of the competition.

The new TOXICSPANS dataset

We used posts (comments) from the publicly available Civil Comments dataset [START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF], which already provides wholepost toxicity annotations. We followed the toxicity definition that was used in Civil Comments, i.e., we used 'toxic' as an umbrella term that covers abusive language phenomena, such as insults, hate speech, identity attack, or profanity. This definition of toxicity has been used extensively in previous work [START_REF] Hosseini | Deceiving google's perspective api built for detecting toxic comments[END_REF][START_REF] Van Aken | Challenges for toxic comment classification: An in-depth error analysis[END_REF][START_REF] Karan | Preemptive toxic language detection in Wikipedia comments using thread-level context[END_REF][START_REF] Han | Fortifying toxic speech detectors against veiled toxicity[END_REF][START_REF] Pavlopoulos | Toxicity detection: Does context really matter? In ACL[END_REF]. We asked crowd annotators to highlight the spans that constitute "anything that is rude, disrespectful, or unreasonable that would make someone want to leave a conversation". Besides toxicity, our annotators were also asked to select a subtype for each highlighted span, choosing between insult, threat, identity-based attack, profane/obscene, or other toxicity. Asking the annotators to also select a category was intended as a priming exercise to increase their engagement. Still, it may have also helped them align their notions of toxicity further, increasing inter-annotator agreement. For the purposes of our experiments, we collapsed all the subtypes into a single toxic class, and we did not study them further; but the subtypes are included in the new dataset we released.

Annotation From the original Civil Comments dataset (1.2M posts), we retained only posts that had been found toxic by at least half of the crowd-raters. This left approx. 30, 000 toxic posts. Toxic comments are rare, especially in fora that are not anonymous and where people have expectations that moderators will be watching and taking action.

We selected a random 11, 000 subset of the 30, 000 posts for toxic spans annotation. We used the crowd-annotation platform of Appen. 2 We employed three crowd-raters per post, all of whom were warned for explicit content. Raters were selected from the smallest group of the most experienced and accurate contributors. The raters were asked to mark each post's toxic word sequences (spans) of by highlighting each toxic span on their screen. For each post, the dataset includes the spans of all three raters. If the raters believed a post was not actually toxic, or that the entire post would have to be annotated, they were instructed to select appropriate tick-boxes in the interface without highlighting any span. The tick-boxes were separate, and the dataset shows when (if) any of the two were ticked.

Hence, when no toxic spans are provided (for a particular post by a particular rater), it is clear if the rater thought that the post was not actually toxic, or that the entire post would have to be annotated.

It is not possible to annotate toxic spans for every toxic post. For example, in some posts the core message being conveyed may be inherently toxic (e.g., a sarcastic post indirectly claiming that people of a particular origin are inferior) and; hence, it may be difficult to attribute the toxicity of those posts to particular spans. In such cases, the posts may end up having no toxic span annotations, according to the guidelines given to the annotators; see the last posts of Table 4.1 for examples. In other cases, however, it is easier to identify particular spans (possibly multiple per post) that make a post toxic, and these toxic spans often cover only a small part of the post (see Table 4.

for examples).

Agreement We measured inter-annotator agreement on 87 randomly selected posts of our dataset, using five crowd-annotators per post in this case. We calculated the mean pairwise (for a pair of annotators) Cohen's kappa per post, using character offsets as instances being classified as toxic (included in a toxic span) or non-toxic; we then averaged over the posts. Although our dataset contains only posts found toxic by at least half of the original crowd-raters, only 31 of the 87 posts were found toxic by all five of our annotators, and 51 were found toxic by the majority of our annotators; this is an indicator of the well-known subjectivity of toxicity detection. On the 31, 51, and 87 posts, the average kappa score was 65%, 55%, 48%, respectively, indicating that when the raters agree (at least by a majority) about the toxicity of the post, there is also reasonable agreement regarding the toxic spans.

Note that the toxic spans are typically short. This leads to class imbalance (most offsets are marked as non-toxic), increases agreement by chance (on the non-toxic offsets), and leads to low kappa scores (kappa adjusts for chance agreement). Another reason behind this modest (compared to other tasks) inter-annotator agreement is the inherent subjectivity of deciding if a post is toxic or not. Our kappa score is in fact, slightly higher than in previous work on toxicity detection, classifying posts as toxic or not [START_REF] Sap | Social bias frames: Reasoning about social and power implications of language[END_REF][START_REF] Pavlopoulos | Deep learning for user comment moderation[END_REF], and in that sense, our inter-annotator agreement can be seen as an improvement.

Ground truth

To obtain the ground truth of our dataset, we averaged the labels per character of the annotators per post. We used the following process: for each post t, first, we mapped each annotated span of each rater to its character offsets. We then assigned a toxicity score to each character offset of t, computed as the fraction of raters who annotated that character offset as toxic (included it in their toxic spans). We retained only character offsets with toxicity scores higher than 50%; i.e., at least two raters must have included each character offset in their spans.

Table 4.1 shows examples.

The dataset TOXICSPANS contains the 11, 035 posts we annotated for toxic spans. The unique posts are actually 11, 006 since a few were duplicates and were removed in subsequent experiments. A few other posts were used as quiz questions to candidate annotators' reliability and were also discarded in subsequent experiments. Although we instructed the crowd-raters to click the appropriate tick-box and not highlight any span when the whole post would have to be highlighted, the ground truth of 34 out of the 11, 000 posts covers the entire post. However, 14 out of the 34 posts are single-word texts, while the other posts are very short; it seems that in very short posts the raters sometimes did not realize they ended up highlighting the entire post. Furthermore, about 5, 000 of the 11, 000 posts have an empty ground truth set of toxic character offsets (as in the last posts of Table 4.1), even though all the posts of our dataset had been found toxic by the original raters. This is partly due to the fact that we include in the ground truth only character offsets that were included in the toxic spans of the majority of our annotators. It also confirms it is not always possible to attribute (at least not by consensus) the toxicity of a post to particular toxic spans. In almost all posts, the ground truth covers less than half of the post; and in the vast majority, less than 20% of the post. A dense toxic span of a post is a maximal sequence of contiguous toxic characters.

Exploratory analysis

There exist posts with more than one dense toxic span, but most posts include only one. Table 4.2 provides further statistics.

Evaluation framework for toxic spans

For the newly introduced toxic spans detection task, we evaluated systems in terms of F 1 score, as in the work of Da San Martino et al. [START_REF] Da San | Fine-grained analysis of propaganda in news article[END_REF]. Given the i th test post, let Ŷi be the predicted set of character offsets and Y i the ground truth character offsets. The per-post F 1 score is defined by Equation (4.1).

F i 1 = 2 • P i • R i P i + R i where P i = | Ŷi ∩ Y i | Ŷi and R i = | Ŷi ∩ Y i | Y i (4.1)
If Y i is empty (no gold spans are given), we set F i 1 = 1 if Ŷi is also empty, and F i 1 = 0 otherwise. The final test F 1 score is the average of the per-post F i 1 scores over all test posts. We used F 1 as the main evaluation measure in the experiments reported below.

Supervised sequence labelling

Toxic spans detection can be seen as sequence labeling (tagging words). As a baseline of this kind, we employed SPACY'S CNN (cf. Section 2.6.2), which is pretrained for tagging, parsing, entity recognition [START_REF] Honnibal | spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing[END_REF]. We called this model CNN-SEQ and fine-tuned it on dense toxic spans, treated as 'entities'. We also trained a bidirectional LSTM (BILSTM-SEQ, cf. Section 2.6.2), 3 and fine-tuned BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] (cf. Section 2.5.8) and SPAN-BERT [START_REF] Joshi | SpanBERT: Improving pre-training by representing and predicting spans[END_REF] for toxic spans (BERT-SEQ, SPAN-BERT-SEQ). These methods require training data manually annotated with toxic spans, so we refer to them as strongly supervised as opposed to weakly supervised methods.

Weakly supervised learning

We trained binary classifiers to predict the toxicity label of each post, and we employed attention as a rationale extraction mechanism at inference to obtain toxic spans, an approach Pavlopoulos et al. [START_REF] Pavlopoulos | Deep learning for user comment moderation[END_REF] found to work reasonably well in toxicity detection. 4 We experimented with two classifiers: a BILSTM with deep self-attention as in the work of Pavlopoulos et al. [START_REF] Pavlopoulos | Deep learning for user comment moderation[END_REF], but training with a regression objective and probabilistic labels following D'Sa et al. [START_REF] D'sa | Towards non-toxic landscapes: Automatic toxic comment detection using DNN[END_REF] and Wulczyn et al. [START_REF] Wulczyn | Ex machina: Personal attacks seen at scale[END_REF]; and BERT with a dense layer and sigmoid on the <CLS> embedding. To detect toxic spans, we used the attention scores of the BILSTM and the attention scores from the heads of BERT's last layer averaged over the heads, respectively. In both cases, we obtained a sequence of binary decisions (toxic, non-toxic)

for the tokens of the post (inherited by their character offsets) by using a probability threshold (tuned on development data) applied to the attention scores. We refer to these two attention-based rationale extraction methods as Further implementation details can be found in our code repository (cf. the chapter's abstract).

F 1 (%) P (%) R (%)

Experimental results for toxic spans

We used a 5-fold Monte Carlo cross-validation (5 random training/development/test splits) on the 11, 000 posts of TOXICSPANS. In each fold, we use 10% of the data for testing, 10% for development, and 80% for training. In AREbased methods, which rely on an underlying classifier to predict if a post is toxic or not, the classifier is trained on the training part of the fold (which contains only toxic posts, ignoring the toxic span annotations) and a randomly selected equal number of non-toxic posts from Civil Comments that are not included in our dataset. When measuring the (binary) classification performance of the underlying classifier, the classifier is evaluated on a new equally balanced test set of 3, 000 randomly sampled unseen posts from Civil Comments.

Both look-up methods (TRAIN-MATCH, HATE-MATCH) outperform the random baseline (Table 4.4). However, TRAIN-MATCH performs much better, which agrees with our hypothesis that toxicity detection is a broader problem than hate speech detection. Both look-up methods are outperformed by the sequence labeling models (-SEQ), especially SPAN-BERT-SEQ, which is pretrained to predict spans. These results show that the tokens of toxic spans are context-dependent, and their meaning is not captured well by context-unaware look-up lexicons. BERT+ARE performs worse than BILSTM+ARE, despite the fact that the underlying BERT classifier is much better (ROC AUC 96.1%) at separating toxic from non-toxic posts than the underlying BILSTM (90.9%). Interestingly, the BILSTM binary toxicity classifier with the attention-based toxic span detection mechanism [START_REF] Pavlopoulos | Deep learning for user comment moderation[END_REF] is close in performance with BILSTM-SEQ, despite the fact that the latter is directly trained on toxic span annotations, whereas the former is trained with binary post-level annotations only (toxic, non-toxic post).

You can stick your d**k up anyone's butt. Why have any laws at all? Of course they do. Stupid people really have to meet everyone else half way if they don't want to be called stupid, starting with not saying stupid things.

Table 4.5: Examples of posts where SPAN-BERT-SEQ incorrectly predicted no spans. Ground truth in red.

Play stupid games, win stupid prizes.

I always smile when I've been called stupid by a fool.

Error analysis of SPAN-BERT-SEQ

We performed an error analysis on our best toxic spans detector (SPAN-BERT-SEQ). We analyzed its predictions on the first fold of the Monte Carlo Cross-Validation, which comprises 10% of the dataset or 1, 001 posts. We identified three main types of errors. The first type of error, which is the most frequent one occurring in 235 out of 1, 001 posts (23.5%), comprises posts for which SPAN-BERT-SEQ failed to find all toxic spans. This type of error can be divided in two sub-types: the first sub-type comprises posts for which SPAN-BERT-SEQ predicted no spans at all (Table 4.5), while the second sub-type comprises posts for which SPAN-BERT-SEQ predicted some, but not all of the gold spans (Table 4.6). The first sub-type occurs more often, with 217 out of the 235 total occurrences of the first error type, while the second sub-type occurs only a few times (18 out of 235). The second type of error, which is the second most frequent one, occurred in 173 out of the 1, 001 posts (17.3%). It occurs when the ground truth of a post is empty, but SPAN-BERT-SEQ predicts at least one toxic span (Table 4.7).

The last type of error occurs rarely (only 10 out of 1, 001 posts) when the ground truth of a post is not empty, and SPAN-BERT-SEQ predicts more (or larger) toxic spans than it should (Table 4.8).

Improving attribution-based detection

Several large datasets with post-level toxicity annotations are publicly available [START_REF] Pavlopoulos | Convai at semeval-2019 task 6: Offensive language identification and categorization with perspective and bert[END_REF]. Therefore, attribution-based toxic span detectors, such as BILSTM+ARE, can, in principle, perform even better if the underlying binary classifier is trained on a larger existing dataset. To investigate this, we increased Not sure if "people are dumb" is the best descriptor, but you are correct that we tend to seek out and grasp at anything that supports our beliefs and hopes. Hence the proliferation of "fake news", which feeds those wants.

They can shuffle the cabinet seven ways from Sunday and it's still a cabal of losers.

This outlet should hire some editors. Nobody I've crossed paths with would green light this crap.

Actually, Seaton is a wealthy man and can do without his day job quite easily. If he would just get rid of that friggin' stupid cap....

In other word, blah, blah, blah, blah. It's bullshit. Deal with it. No proof=doesn't exist.

Or maybe we should place a tax on stupid ideas like yours People don't normally take it to heart when an idiot calls someone stupid.

$10B a GW avg compared to $2.5B a GW for a 2nd Candu nuke at LePreau. Stupid is as stupid does I guess.

All useless piles of crap.

oh no, this isn't even in the top 10 moronic statements by this babbling fool.

Toxic spans in toxic-to-civil transfer

As shown in Section 4.7.2, a toxic span detection method can be used to highlight toxic parts of a post, to assist, for instance, human moderators. The new TOXICSPANS dataset and toxic span detection methods, however, can assist in more ways. This section describes how we combined the new dataset and the best-performing toxic span detector (SPAN-BERT-SEQ) to show how they can be useful in toxic-to-civil text transfer (cf. Chapter 3). In the context of detoxifying comments to nudge users toward healthier conversations online, this task aims at suggesting civil rephrasings of toxic posts. More specifically, we studied the following research question:

RQ: Can TOXICSPANS data and toxic span detectors be used to assess the mitigation of explicit toxicity in toxicto-civil transfer?

To answer this question, we proceeded in two ways:

(i) Evaluating the transfer of toxic spans in system-detoxified posts (ii) Studying any remaining toxic spans in human-detoxified posts.

System-detoxified posts

We first compared the performance of two toxic-to-civil transfer models, CAE-T5 and SED-T5, both based on the T5 encoder-decoder transformer architecture [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF]; they both fine-tune the weights of the same pretrained model, namely T5-large. CAE-T5 [START_REF] Laugier | Civil rephrases of toxic texts with self-supervised transformers[END_REF] a Toxic posts Table 4.9: Statistics for the parallel (P) and non-parallel (NP) datasets, used to train the SED-T5 and CAE-T5 toxicto-civil models, respectively. Average lengths are reported by counting SentencePiece [START_REF] Kudo | SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing[END_REF] tokens.

Civil to toxic (P)

Input stop being ignorant and lazy and try reading a bit about it.

CAE-T5

try reading and be a little more informed about it before you try to make a comment.

SED-T5

stop being lazy and lazy and try reading a bit about it.

Input this is absolutely the most idiotic post i have ever read on all levels.

CAE-T5

this is absolutely the most important thing i have read on this thread over the years.

SED-T5

this is not the most useless post post I have ever read on all levels Input trump may be a moron, but clinton is a moron as well.

CAE-T5

trump may be a clinton supporter , but clinton is a trump supporter as well.

SED-T5

trump may be a badon, but clinton is a badon as well.

Input shoot me in the head if you didn't vote for trump.

CAE-T5

@ @ @ @ @ @ you're right if you didn't

vote for trump. h h h h h h h h h

i'm not sure i'd vote SED-T5
shoot me the head if you didn't vote for Trump Input 50% of teachers don't have any f*cks to give.

CAE-T5

50% of teachers don't have @ @ @ @ @ @ @ @ @ h h h h h h h h h a phd in anything.

SED-T5

50% of teachers don't have any tofls to give.

Table 4.10: Examples of automatically transferred sentences from the NP test set (toxic to civil), valid rewriting, and highlighted flaws failure in attribute transfer or fluency , h h h h h h h supererogation, @ @ @ @ @ @ @ @ position reversal, and @ @ @ @ @ @ h h h h h h hallucination.

been fine-tuned on a large non-parallel (NP) dataset based on pre-processed posts from the Civil Comments (CC) dataset, the dataset (with post-level annotations) that TOXICSPANS was also based on. SED-T5 is a Supervised Encoder-Decoder; we fine-tuned it on a smaller parallel (P) dataset created by Dementieva et al. [START_REF] Dementieva | Crowdsourcing of parallel corpora: the case of style transfer for detoxification[END_REF], consisting of pairs of comments: a toxic comment and a detoxified paraphrase written by a crowdworker. Table 4.9 summarizes statistics of the two datasets (P, NP) and highlights a trade-off between the level of supervision and number of samples: there is a 1 : 40 ratio between toxic comments in P (direct supervision, parallel data) and NP (indirect supervision, no parallel data). Table 4.11 shows our experimental results. We report accuracy (ACC), Perplexity (PPL), similarity (SIM), and the geometric mean (GM) of ACC, 1/PPL, SIM. The systems computing these metrics are the same as in Chapter 3. As a reminder, accuracy measures the rate of successful transfers from toxic to civil. It computes the fraction of posts whose civil version is classified as non-toxic by a BERT toxicity classifier. Perplexity is used here as a measure of fluency. Similarity measures content preservation between the original toxic text and its system-rephrased civil version (self-SIM) or the gold (human) civil rephrasing (ref-SIM, only for P).

As can be seen in Table 4.11, CAE-T5 has better aggregated results (higher GM) than SED-T5 in all three datasets, which are due to lower Perplexity and (in NP and TOXICSPANS) higher accuracy. However, SED-T5 learned to preserve content better (higher SIM in all three datasets) because of the parallel data (P, with gold rephrases) it was trained on. By contrast, CAE-T5 was trained without parallel data (NP) using a cycle-consistency loss, which leads to more frequent hallucinations of content that was not present in the original post [START_REF] Laugier | Civil rephrases of toxic texts with self-supervised transformers[END_REF]. These hallucinations may also help CAE-T5 obtain better Perplexity scores by generating fluent civil 'rephrases' that do not preserve, however, the original semantics. Examples shown in Table 4.10 illustrate the tendency of CAE-T5 to remove toxicity with hallucination and position reversal compared to SED-T5's rephrasings, often more faithful to the input comment (rare cases of hallucination), but at the cost of fluency and detoxification accuracy. Also, although the general trends are similar in all three datasets (SED-T5 preserves content better, CAE-T5 is better in Perplexity and GM), there are several differences too across the three datasets. For example, CAE-T5 is much better than SED-T5 in accuracy (posts detoxified) on NP and TOXICSPANS, but both systems have the same accuracy on P; and the scores of the systems vary a lot across the three datasets.

These considerations motivated us to seek ways to analyze the behavior of toxic-to-civil transfer models further.

TOXICSPANS and toxic span detectors are an opportunity to move towards this direction by studying how well transfer models cope with explicit toxicity, i.e., spans that can be explicitly pointed to as sources of toxicity. We leave for future work the flip side of this study, i.e., studying cases where transfer models rephrase spans not explicitly marked (by toxic span detectors or human annotators) as explicitly toxic.

Explicit Toxicity Removal Accuracy

Recall that the accuracy (ACC) scores of Table 4.11 measure the percentage of toxic posts that the transfer models (CAE-T5, SED-T5) rephrased to forms that a (BERT-based) toxicity classifier considered non-toxic. One could question, however, if it is possible (even for humans) to produce a civil rephrase of a toxic post when it is impossible to point to particular spans of the post that cause its toxicity (as in the last posts of Table 4.1). Detoxifying posts of this kind may constitute a mission impossible for most models (possibly even for humans); the only way to produce a non-toxic 'rephrase' may be to change the original post beyond recognition, which may be rewarding systems like CAE-T5 that often hallucinate in their rephrases, as already discussed.

Hence, it makes sense to focus on posts that contain explicit toxic spans, marked by human annotators (for TOXICSPANS) or our best toxic span detector (SPAN-BERT-SEQ). Using these toxic spans, we defined three additional variants of accuracy: ACC2 is the same as ACC, but ignores posts that do not contain at least one toxic span; ACC3 also considers (in its denominator) only posts that contained at least one toxic span, but computes the fraction of these posts that had all of their toxic spans rephrased (even partly) by the transfer model; ACC4 is a stricter version spans to the source posts and retained only the 1, 354 (out of 2, 778 in total) source-target pairs of posts with at least one toxic span in their source post. 8 In all but 6 of the 1, 354 posts, the humans have rephrased (in the gold target post they provided) all the toxic spans of the source post. The 6 posts were mainly cases where the human changed the context to mitigate toxicity, while retaining the original toxic span. For example, "he's not that stupid" became "he's not stupid" (original toxic span shown in bold); in this case removing the "that" from the context arguably makes the post less offensive. Overall, we conclude that humans did rephrase almost all cases of explicit toxicity in the toxic posts they were given.

We also applied SPAN-BERT-SEQ to the gold target (rephrased) posts that the humans provided to check if any explicit toxicity remained or was introduced by the rephrases. This flagged 93 gold target posts as comprising at least one toxic span. A manual inspection of the 93 posts revealed that they fall into two main categories. The first category comprises cases where a toxic span of the source post was rephrased, but the rephrase might not be considered totally civil; e.g., "how freaking narcissistic do you have to be?" became "how narcissistic do you have to be?", where SPAN-BERT-SEQ marked the "narcissistic" of the rephrase as a toxic span. The second category comprises cases where SPAN-BERT-SEQ produced false positives; e.g., the source post "most of the information is total garbage" became "most of the information is totally useless", but SPAN-BERT-SEQ marked (arguably incorrectly) "useless" as a toxic span.

Toxicity scores of posts with and without explicit toxicity

We also applied the BERT-based text toxicity classifier of Chapter 3 to the 2, 778 posts of the P dataset, dividing them into two sets: posts that comprised at least one toxic span detected by SPAN-BERT-SEQ (1, 354 posts with explicit toxicity) and the rest (implicit toxicity). The BERT-based toxicity classifier considered more toxic (higher average toxicity score) the 1, 354 posts of the first set compared to the second one, i.e., it was more confident that the posts of the first set (explicit toxicity) were toxic, as one might expect. By resampling 1, 000 subsets (of 50 posts each) from the two sets, we confirmed that this is a statistically significant difference (P = 0.001). The difference in the average predicted toxicity score between the two sets is 14% (from 0.94 down to 0.80).

Discussion

The posts we annotated for toxic spans were extracted from an already heavily studied public domain benchmark dataset (Civil Comments) that has been examined by thousands of teams in a Kaggle competition, 9 and that has been cited in over 50 academic publications. The Civil Comments dataset was filtered to remove any potential personally identifiable information before it was released. Our annotation cost was $21, 089 for 59, 486 judgements, paying $0.30 per item. All raters were warned for the explicit content of the job, and only high accuracy raters were selected (70 + %) based on performance on quiz questions. The most common countries of origin of our crowdannotators were Venezuela and USA (cf. Figure 4.8). In the contributor satisfaction survey, 51 participants gave an overall task rating of 3.6/5.0, with pay and test question fairness rated slightly higher than ease of job and clarity of instructions.

We note that it is more difficult and costly (approximately 3 times more) to manually annotate toxic spans, instead of just labeling entire posts as toxic or not. This is why we also explored adding rationale extraction components on top of toxicity classifiers trained on existing much larger datasets. We showed that BILSTM+ARE has the potential to reach the performance of BILSTM-SEQ, which is important for future work aiming to build toxic span detectors without any toxic span annotations in the training data. This may be particularly useful in low-resourced languages with limited resources for text toxicity [START_REF] Zampieri | Semeval-2020 task 12: Multilingual offensive language identification in social media (offenseval 2020)[END_REF].

Having two separate systems, one for toxicity detection and one for toxic spans identification, is more easily compatible with existing deployed toxicity detectors. One can simply add a component for toxic spans at the end of a pipeline for toxicity detection, and the new component would be invoked only when toxicity would be detected, leaving the rest of the existing pipeline unchanged. Since the vast majority of posts in real-world applications are non-toxic [START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF], this pipeline approach would only increase the computational load for the relatively few posts classified as toxic. Using only toxic posts in this study was also a way to simplify this first approach to toxic spans detection, assuming an oracle system achieved the first step (deciding which posts are toxic). However, we note that future work could study adding non-toxic posts to our dataset and require systems to first detect toxic posts, then extract toxic spans for toxic posts.

A direct comparison (in terms of size) of TOXICSPANS with other existing toxicity datasets is only possible if one focuses on the toxic class, typically the minority one, since our dataset contains only toxic posts. By adding non-toxic posts, much larger versions of our dataset can be compiled, of sizes similar to those of existing previous datasets this work focused on rating prediction systems that try to predict the rating a user will give an item. For example, when a user is trying to evaluate a given item, e.g., they heard about a movie and are looking it up before choosing whether to watch it or not. In this setting, search engines and film streaming sites like Netflix show information to help the user make a decision, such as the average rating, public reviews, and sometimes personalized ratings and explanations for why the user may enjoy it (so-called endorsements). This work focuses on providing high-quality personalized rating prediction leveraging review text in a way that supports explanations based on snippets of review text from other users.

Statistical and ML models are often used to compute personalized predictions based on the past interactions (e.g., ratings and reviews) r ui of a set of users u ∈ U on a set of items i ∈ I. The archetypal baseline for recommender systems is Collaborative Filtering (CF) [START_REF] Ricci | Introduction to Recommender Systems Handbook[END_REF]. CF-based approaches infer preference scores rui from the similarity between users who expressed common tastes. The computation of these commonalities varies from the data the system has access to, though the majority of traditional models rely on past ground-truth ratings. Even if they show acceptable quantitative results, rating-based recommender systems often suffer from a lack of clear and precise explanation for why the recommendation was made.

Srifi et al. [START_REF] Srifi | Recommender systems based on collaborative filtering using review texts-a survey[END_REF] recently reviewed attempts to integrate review text in recommender systems. They examined existing methods, classified by Chen et al. [START_REF] Chen | Recommender systems based on user reviews: The state of the art[END_REF], that extract information from reviews in order to enhance recommender systems. While many of these studies inferred high-level features such as sentiments or topics, only a few directly built semantic-aware recommender systems from the unstructured text found in reviews.

Memory-based and model-based systems are the main two families of methods seen in CF [3,[START_REF] Chen | Preference-based clustering reviews for augmenting e-commerce recommendation[END_REF]. Memorybased CF predicts user ratings by directly aggregating ratings from other users with similar preferences. Modelbased CF uses interaction data to train a model-typically a high dimensional vector-that can be used to generate ratings for future items. Even when memory-based CF shows worse absolute quantitative evaluation on standard metrics, it benefits from the simplicity of implementation and ease of creating explanations.

To the best of our knowledge, the work from Terzi et al. [START_REF] Terzi | Text-based user-knn: measuring user similarity based on text reviews[END_REF] (cf. Section 5.2) is the only effort to leverage text in memory-based CF by using word-based similarity measures between reviews to compute user similarities. However, recent progress in NLU enables far richer and more effective semantic representations of text [START_REF] Vaswani | Attention is all you need[END_REF][START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF]. In this chapter, we used modern large Pretrained Language Models [START_REF] Yang | Multilingual universal sentence encoder for semantic retrieval[END_REF] to embed sentences of reviews into a so-called semantic vector space (Section 5.5). We then employed a user k-NN model based on user similarity (Section 5.6).

The key insight is that user similarity can be defined in terms of a graph based on semantic similarity from text in user reviews. Specifically, we showed how to build sentence-based k-Nearast Neighbors (k-NN) graphs (Section 5.7) to compute user similarity. We denote our model Text-KNN1 (Figure 5.2).

We evaluated our model in the context of two large datasets containing both numerical and text reviews; the Amazon Review dataset [START_REF] He | Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering[END_REF] and the Yelp dataset [START_REF] Kaggle | Yelp dataset[END_REF]. Offline metrics are often used to evaluate recommender systems. Although top-N ranking metrics such as discounted cumulative gain [START_REF] Järvelin | Cumulated gain-based evaluation of ir techniques[END_REF] and hit rate evaluate item recommendation, rating prediction has traditionally been evaluated with error-based predictive accuracy metrics like the Root-Mean-Square Error (RMSE). However, it has been observed that RMSE is not effective at distinguishing the quality of algorithms-high performance can even be achieved by naive baselines [START_REF] Cremonesi | Performance of recommender algorithms on top-n recommendation tasks[END_REF]. Moreover, RMSE unfairly represents users with a smaller spread of scores (as error values will be smaller). In Sections 5.8, we explored arguments in favor of rating-prediction evaluation based on rankings, leading us to also report a metric inspired by the Fraction of Concordant Pairs [START_REF] Koren | Collaborative filtering on ordinal user feedback[END_REF].

. . USE σ ′′ 1 σ σ ′ σ 1 σ ′ 1 σ ′′ 1 σ 2 σ 3 v 1 u v 1 v 2 v 3 Sentence k-NN graph σ σ ′ σ 1 σ 2 σ 3 σ ′ 1 σ ′′ 1 0 . 9 0 . 7 0 . 9 0 . 8 User k-NN regressor u v 1 v 2 v 3 3 1 ruj = 3 4 • r v1j + 1 4 • r v2j s w

Related work

User k-NN is the main approach studied in memory-based CF. The cornerstone of k-NN is the definition of a similarity measure between two users [START_REF] Ning | A Comprehensive Survey of Neighborhood-Based Recommendation Methods[END_REF]. Several similarities can be computed from the vectors of the ratings of co-reviewed items, among which mean squared difference (MSD, [START_REF] Shardanand | Social information filtering: Algorithms for automating "word of mouth[END_REF][START_REF] Pazzani | A framework for collaborative, content-based and demographic filtering[END_REF], cosine similarity [START_REF] Linden | com recommendations: Item-to-item collaborative filtering[END_REF], Pearson correlation coefficient [START_REF] Herlocker | Evaluating collaborative filtering recommender systems[END_REF] and Jaccard coefficient [START_REF] Koutrika | Flexrecs: Expressing and combining flexible recommendations[END_REF].

Model-based CF rely on intrinsic models trained with various machine learning techniques [START_REF] Su | Collaborative filtering for multi-class data using belief nets algorithms[END_REF][START_REF] Xia | Support vector machines for collaborative filtering[END_REF][START_REF] Zhang | Deep learning based recommender system: A survey and new perspectives[END_REF] to directly predict user preferences. In particular, state-of-the-art latent factor models [START_REF] Koren | Matrix factorization techniques for recommender systems[END_REF] such as probabilistic matrix factorization [START_REF] Salakhutdinov | Probabilistic matrix factorization[END_REF], Singular Value Decomposition (SVD) [START_REF] Funk | Netflix update: Try this at home[END_REF] and SVD++ [START_REF] Koren | Factor in the neighbors: Scalable and accurate collaborative filtering[END_REF] achieve the most precise rating predictions.

Research has explored different ways of extending the rating-only-based systems with additional data ources to represent users and items. Non-text-review information include social information [START_REF] Su | Effective social content-based collaborative filtering for music recommendation[END_REF], tags [START_REF] Han | An extended-tag-induced matrix factorization technique for recommender systems[END_REF][START_REF] Balog | Transparent, scrutable and explainable user models for personalized recommendation[END_REF] and descriptions [START_REF] Alshammari | A switching multi-level method for the long tail recommendation problem[END_REF]. However, text found in reviews is the most promising type of information to augment rating-based datasets and models [START_REF] Zhang | urcf:user review enhanced collaborative filtering[END_REF][START_REF] Chen | Recommender systems based on user reviews: The state of the art[END_REF][START_REF] Hernández-Rubio | A comparative analysis of recommender systems based on item aspect opinions extracted from user reviews[END_REF]. Many works are based on that approach. Some strategies consist in extracting relevant information from the text review before integrating it into a recommender system. This information can narrow-down to specific content, such as topics [START_REF] Mcauley | Hidden factors and hidden topics: Understanding rating dimensions with review text[END_REF][START_REF] Bao | Topicmf: Simultaneously exploiting ratings and reviews for recommendation[END_REF][START_REF] Chin | Anr: Aspect-based neural recommender[END_REF] or sentiments [START_REF] Poirier | Reducing the Cold-Start Problem in Content Recommendation Through Opinion Classification[END_REF][START_REF] Zhang | Explicit factor models for explainable recommendation based on phrase-level sentiment analysis[END_REF][START_REF] Dau | Weighted aspect-based opinion mining using deep learning for recommender system[END_REF].

Further strategies rely directly on the words found in reviews without trying to focus on specific informative elements (like topics, overall sentiment, or aspect-based opinion). Considering integrating the information at the word level provides potentially richer compositionality power. The literature contains examples of approaches building latent representations of sentences with neural networks before integrating these into matrix factorization CF [START_REF] Kim | Convolutional matrix factorization for document context-aware recommendation[END_REF][START_REF] Zheng | Joint deep modeling of users and items using reviews for recommendation[END_REF][START_REF] Chen | Neural attentional rating regression with review-level explanations[END_REF][START_REF] Liu | Hybrid neural recommendation with joint deep representation learning of ratings and reviews[END_REF].

Even if a few works combine text information with review information in memory-based CF [START_REF] Leung | Integrating collaborative filtering and sentiment analysis: A rating inference approach[END_REF][START_REF] Levi | Finding a needle in a haystack of reviews: Cold start context-based hotel recommender system[END_REF][START_REF] Musat | Recommendation using textual opinions[END_REF], Terzi et al. [START_REF] Terzi | Text-based user-knn: measuring user similarity based on text reviews[END_REF] were the first to design user-based k-NN similarities directly from the words found in text reviews and therefore their work is the most relevant to us. They introduced six word-based similarity measures ranging from simple word overlap, to depth of two words [START_REF] Pedersen | Wordnet::similarity -measuring the relatedness of concepts[END_REF] in external lexical graphs2 , and information content [START_REF] Resnik | Using information content to evaluate semantic similarity in a taxonomy[END_REF][START_REF] Jiang | Semantic similarity based on corpus statistics and lexical taxonomy[END_REF][START_REF] Lin | An information-theoretic definition of similarity[END_REF]. Even though this is the closest approach to ours, we differed in that sentence similarity relies on continuous (sentence embedding) rather than discrete (co-occurrence of terms) semantic representations.

Evaluating recommender systems is a critical step in their development and remains an open research topic.

Many offline (automatic) evaluation metrics have been introduced as proxies to approximate online (human) evaluation. It has not been shown yet that a metric is universally better than others, though RMSE seems to be the dominant approach for its simplicity of implementation and understanding. Cremonesi et al. [START_REF] Cremonesi | Performance of recommender algorithms on top-n recommendation tasks[END_REF] justified why different metrics perform differently on different tasks (rating prediction versus item recommendation) and Herlocker et al. [START_REF] Herlocker | Evaluating collaborative filtering recommender systems[END_REF] summarized existing metrics targeting specific criteria of recommender systems. Chandar et al. [START_REF] Chandar | Beyond accuracy: Grounding evaluation metrics for human-machine learning systems[END_REF] argued for taking into account user behavior models when designing and selecting metrics. The Fraction of Concordant Pairs was first introduced by Koren and Sill [START_REF] Koren | Collaborative filtering on ordinal user feedback[END_REF] as an offline metric relevant for evaluating if predicted ratings were ranked the same way as the ground truth. It was re-used by Wu et al. [START_REF] Wu | Optimization matrix factorization recommendation algorithm based on rating centrality[END_REF], Beniwal et al. [START_REF] Beniwal | A comparison of similarity measures for neighbourhood based collaborative filtering recommender systems[END_REF].

Besides recommender systems, the field of information retrieval (IR) [START_REF] Buttcher | Information retrieval: Implementing and evaluating search engines[END_REF] is relevant to our study. There has been a large trend of work in IR, including snippet extraction [START_REF] Li | Personalized text snippet extraction using statistical language models[END_REF] and endorsement [START_REF] Li | Exploiting endorsement information and social influence for item recommendation[END_REF].

The more explicit the feedback, the richer the information

Our preferences are only partially reflected by numerical scores. The lowest-level information can be found in implicit feedback, where data only indicates whether a user has interacted with an item (e.g., movie watched or not, product bought or not). Numerical explicit feedback provides richer information: 5-points scale ratings enable systems to ignore the non-rated items, focusing on a linear scale of preferences. Explicit feedback is not limited to quantitative assessment, and the Web facilitated immediate exchanges of opinion on various topics in online fora ITEM j LIKED BY SOME USER u OTHER ITEM REVIEWED BY u, ALONG WITH A SENTENCE u WROTE

Sleeping beauty

Ratatouille -" Ratatouille " is a thoroughly entertaining movie that is perfect family fare for the summer .

The Shining

Haloween -This is a must movie for all true horror fans .

Groundhog day

The Truman Show -" The Truman Show " will make you laugh , and keep you on the edge of your seat , wondering if Truman will ever get out.

Table 5.1: Examples of sentences (right column) written by a user u on some item (in bold) they reviewed and useful to infer that u likes another item j (left column). An item is "liked" when the rating is above 4/5.

first. Then, social networks3 , e-commerce4 and streaming service5 platforms democratized it and nowadays users and customers get the opportunity to review services and products they had consumed. Two users u and v may have rated equally the same item r ui = r vi but for totally different reasons. For example, one can appreciate the Matrix franchise for its spectacular action scenes, groundbreaking special effects, dystopian sci-fi plot, cyberpunk characters, cult dialogues, actors, the philosophical/spiritual themes addressed, the references to films, literature, myths and religion, etc. A system trained on rating-only data, precise as it is, will never be able to capture those reasons; hence it will make biased predictions.

Contrary to tags or movie plot descriptions, text reviews are totally free-form and may contain many levels of preference information (cf. examples shown in Table 5.1). Text reviews include sentences useful to represent users' tastes or movie features. Some may even contain direct explicit recommendations from a human, which is gold information from the point of view of the task we address. This motivated integrating such information to design or explain recommender systems and assess to what extent they can benefit from it.

Datasets

We experimented with the 2014 Amazon Product Review dataset [START_REF] He | Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering[END_REF], made of reviews from users on items with 5-points scale ratings and text reviews. It has been used in related work [START_REF] Wang | Neural graph collaborative filtering[END_REF][START_REF] Wang | Kgat: Knowledge graph attention network for recommendation[END_REF][START_REF] Zhou | Deep interest network for click-through rate prediction[END_REF] for its important size. Specifically, we focused on the movie subset because common knowledge makes it easier to interpret movie reviews than reviews of other items. Following He et al. [START_REF] He | Neural collaborative filtering[END_REF], López et al. [START_REF] López | Augmenting the user-item graph with textual similarity models[END_REF] we restricted our study on the k-core subset, where k = 20. Additionally, we trained and evaluated our method on the k-core subset of the Yelp review dataset [START_REF] Kaggle | Yelp dataset[END_REF], made of reviews of businesses (like restaurants). Table 5.2 shows statistics for the datasets.

The datasets were split into training, validation, and test subsets. Various splitting strategies exist [START_REF] Meng | Exploring data splitting strategies for the evaluation of recommendation models[END_REF], showing benefits and limitations depending on the nature of the feedback (explicit or implicit), the task (rating prediction or item recommendation), and the model used. Following previous works [START_REF] Terzi | Text-based user-knn: measuring user similarity based on text reviews[END_REF][START_REF] López | Augmenting the user-item graph with textual similarity models[END_REF], we preserved time ordering [START_REF] Shani | Evaluating Recommendation Systems[END_REF] by employing the Leave One Last Item strategy.

Semantic matches

Semantic representations of reviews

The Universal Sentence Encoder (USE) is a sentence embedding model pretrained on a variety of NLP tasks with diverse degrees of supervision, namely Retrieval Question-Answering, Translation Ranking and Natural Language Inference [START_REF] Bowman | A large annotated corpus for learning natural language inference[END_REF]. It can be thought of as a paraphrase similarity model, and Yang et al. [START_REF] Yang | Multilingual universal sentence encoder for semantic retrieval[END_REF] showed such embeddings were useful for a variety of downstream NLP tasks. We encoded review sentences in a single high-dimensional space using the USE model. To illustrate the intuition for the relevancy of USE on review sentences, we applied it to three popular and distinct categories of items. We selected three items per category, according to the genre tag found on the Internet Movie Database 6 . The sentences found in the train reviews of the nine items were embedded in the USE model's 512 dimensional semantic space and we show in Figure 5.1 their 2D t-SNE [START_REF] Van Der Maaten | Visualizing data using t-sne[END_REF] projection. We note that individual items have clustered sentences. On top of that, we observe that the projections of categories are Y-shaped, indicating some clustering at the category level as well, though sentences at the origin seem to share a high similarity across categories. We note many of the latter sentences, with low semantic variability, are general sentences, not specific to any item, such as "I liked it".

Aggregation of matches per-item category

We computed the k-NN sentence graph G of all the sentences found in the 20-core Amazon movie review trainset, embedded with USE. It is directed since each edge leaves sentence σ 1 and enters their k nearest neighbors σ 2 .

Following the approach of Yang et al. [START_REF] Yang | Multilingual universal sentence encoder for semantic retrieval[END_REF] for transfer learning, we used the cosine distance (defined as 1 minus the cosine similarity) 7 . Doing so echoes the approach we introduced in Chapter 3 when we evaluated sentence similarity with SIM (cf. Section 3.9.2 for more details). We call a "semantic match" a pair of connected sentences in the graph. Figure 5.3 shows the heatmaps of matches in the sub-graph of the nine mentioned items, aggregated by items (left) and categories (right). Since the items have different numbers of train sentences, the match count matrices C are normalized with the square root of its row and column weights [START_REF] Hirschfeld | A connection between correlation and contingency[END_REF], i.e., the heatmaps correspond to the matrices computed in Equation (5.1).

diag (C1)

-1 2 • C • diag C ⊤ 1 -1 2 (5.1)
It appears that the relative number of matches is higher for pairs of sentences belonging to the same category of items.

Visualization and quantitative analysis of matches indicate that one can represent item reviews with USE and build some recommender system from it.

Text-based k-nearest neighbors recommenders

Text-KNN estimates the test rating as the following weighted sum:

rui = v∈N k ′ i (u) w (u, v) • r vi v∈N k ′ i (u) w (u, v) (5.2)
where w (u, v) is the weight of v for predicting u's rating. Among all the possibilities to compute w (u, v), we present below three alternatives based on counting the semantic matches. N k ′ i is the set of k ′ nearest neighbor users of user u who have rated item i.

A simple method, called "baseline" in our experimental setups and introduced by Koren [START_REF] Koren | Factor in the neighbors: Scalable and accurate collaborative filtering[END_REF], estimates ratings b ui as the overall average µ rating summed to user and item biases (respectively b u and b i ). We experimented with the baseline-aware user k-NN (Text-BKNN) variant:

rui = b ui + v∈N k ′ i (u) w (u, v) • (r vi -b vi ) v∈N k ′ i (u) w (u, v) (5.3)
We proposed a user similarity measure based on the representations of sentences seen in Section 5.5.1.

Notations Let G = (V, E) be the k-NN directed graph of sentences. Each directed edge (σ 1 , σ 2 ) ∈ E has an associated semantic similarity weight s (σ 1 , σ 2 ) ∈ [0, 1], detailed in Section 5.7. By extension, s (σ 1 , σ 2 ) = 0 if (σ 1 , σ 2 ) / ∈ E. The three approaches experimented with computing w (u, v), denoted "One-to-One", "Many-to-One"

and "Many-to-Many", rely on the fact that the set of sentences is partitioned by the set of users. For that reason, let S u be the set of sentences written by u. Many-to-Many matching Our third approach consisted in counting the number of sentence matches when considering all pairs of sentences written by u and v. In this case,

N (σ 1 ) = {v ∈ U : ∃σ 2 ∈ S ν , s (σ 1 , σ 2 ) > 0}
w (u, v) = max   (σ1,σ2)∈(Su×Sv) s (σ 1 , σ 2 ) , 0   (5.6)
As we shall see in the next section, s (σ 1 , σ 2 ) is not necessary non-negative, though Equations (5.2) and ( 5.3) require non-negative weights.

Sentence graph

The weight s (σ 1 , σ 2 ) introduced in section 5.6 corresponds to the semantic similarity of sentence σ 2 regarding σ 1 .

Again, we considered various ways of computing it: "binary", "continous" and "polarized".

Binary count

The basic sentence weight consisted in counting 1 if and only if the edge from σ 1 to σ 2 appears in the graph.

Continuous similarity scores

To include finer information about sentence similarity, we experimented with the edge weight to be the cosine similarity scaled between 0 and 1. We defined s by:

s (σ 1 , σ 2 ) = 1 + cosine similarity (σ 1 , σ 2 ) 2
Polarization Supplementary information at the sentence level may be relevant where semantic matching fails. For instance, the sentences "I love DiCaprio" and "I hate DiCaprio" have a cosine similarity of 0.94. We proposed to integrate information from a sentiment attribution mechanism α(σ) ∈ [-1, 1] with a polarization function p defined in equation (5.7). Then, the sentence weight becomes p • s.

p (σ 1 , σ 2 ) =        1, if |α(σ 1 ) -α(σ 2 )| ≤ 1 -1, otherwise (5.7)
We experimented with a rating-aware sentiment attribution defined in equation (5.8).

α(σ) =                1, if r (σ) ≥ 4 -1, if r (σ) ≤ 2 0, otherwise (5.8)
r(σ) being the rating of the review σ belongs to.

Item graphs Moreover, we experimented with the methods discussed above, considering graphs made of sentences of a single item, rather than a single "global" graph of sentences from all items. The final user weight w (u, v)

was the sum of the per-item user weights, though other aggregations may be considered.

Normalization Some users write more sentences than others, some items receive more sentences than others, and users may write a different number of sentences on various items. For all these reasons, bias may appear when the user weights are computed from counting semantic matches between sentences (or co-occurrence like in the ablation study in Section 5.10). To mitigate this potential source of bias, we integrated different ad hoc normalizations in the computation of the user weights.

On the one hand, we tried to normalize w (u, v) by the number of sentences |S v | written by v. In addition to this option, we considered normalizing the weights by the number of sentences |S i | written on i or the number of sentences |S ui | written by u on i when per-item graphs were considered. On the other hand, bias induced by an imbalance of sentence set sizes could also be mitigated by normalizing the match counting methods introduced in section 5.6. Concerning the One-to-One matching (Equation (5.4)), we tried to normalize by N (u). When considering Many-to-One matching, each term in Equation (5.5) was divided by 1 or N (σ 1 ). For the Many-to-Many matching, the terms in the Equation's (5.6) sum are divided either by 1, δ -= σ∈V |s (σ, σ 2 ) | or δ + = σ∈V |s (σ 1 , σ) |. The latter two options respectively correspond to extensions of in and out-degrees adapted to our weighted graph. Normalizing by degrees is unsupervised mitigation of semantic similarity between "common" sentences, which are irrelevant for the sake of representing user preferences (cf. Section 5.5.1).

Evaluation measures

Offline evaluation of rating prediction is popularly measured through RMSE. However, Cremonesi et al. [START_REF] Cremonesi | Performance of recommender algorithms on top-n recommendation tasks[END_REF] showed the limits of pure error-based metrics.

As an example to understand RMSE's limitations, consider the toy dataset in The baseline asymptotically predicts lim n→∞ ru2i0 (n) = 3 + and lim n→∞ ru2i1 (n) = 3 -. Therefore, it yields lim n→∞ RMSE(n) = 0.5, relatively good compared to the uniform random (E [RMSE] = 17 16 ). However, the baseline is non-personalized when it comes to compare rankings, and always predicts ru2i0 (n) > ru2i1 (n). This example illustrates RMSE's limitation in reflecting user preferences modeled by systems. This motivates a ranking-based metric measuring how well the ordering of items is preserved.

We argue that the Fraction of Concordant Pairs (FCP), measuring the proportion of well-ranked item pairs, is a suitable metric for three reasons. First, it directly and unambiguously measures preferences expressed by users, by definition. Second, it is grounded in statistics since concordant and discordant pairs have already proved relevant to comparing two measured quantities [START_REF] Kendall | A new measure of rank correlation[END_REF]. Third, it generalizes to non-binary ordered sets the ROC-AUC binary metric, known to test whether positive examples are ranked higher than negative examples by classifiers.

With TFCP, we handle equalities the same way as Hug [START_REF] Hug | Surprise, a Python library for recommender systems[END_REF], i.e. ((r ui , rui ) , (r uj , ruj )) is:

• Concordant (CP) iif r ui ̸ = r uj and sgn (r ui -r uj ) = sgn (r ui -ruj ),

• Discordant (DP) iif r ui ̸ = r uj and sgn (r ui -r uj ) ̸ = sgn (r ui -ruj ),

• Ignored if r ui = r uj In the toy example, the baseline has a FCP of 0, signaling its inability to model u 2 's preferences. Koren and Sill [243] randomly split the dataset into train and test data, enabling the computation of FCP on pairs of test items rated by the same user. Yet, modern time-based Leave One Last Item splitting strategies provide one single test item per user. We adapted FCP to consider, for each user, all the pairs made of the test item and a train item. We called it Time-based FCP (TFCP). Thereby, the evaluation is equivalent to assessing the frequency of the model to correctly rank the next rating compared to all past ratings for each user. Denoting respectively n c and n d the number of concordant and discordant pairs, we reported the per-user macro-averaged metric: TFCP = u∈U nc(u) nc(u)+n d (u) .

Results

Experimental setup

Discussion

Quantitative evaluation indicates that our approach is comparable to previous text-agnostic memory-based systems for both RMSE and TFCP. A general trend is that model-based systems give better RMSE but worse TFCP. Surprisingly, RMSE suggests the baseline's performance is better than all memory-based system, although being a naive approach. This echoes our analysis in Section 5.8 and validates the motivation behind the ranking-based evaluation. TFCP does indeed rank the non-"rankingwise personalized" baseline in penultimate place among non-random systems. Similarly, NARRE outperformed all other systems when evaluated with RMSE while being only slightly better than random predictions regarding TFCP. For further analysis of the correlation between RMSE and TFCP, Figure 5.6 shows Spearman's ρ and Kendall's τ correlation coefficients of the rankings produced by the metrics.

RMSE and TFCP produce decorrelated rankings.

After hyperparameter tuning on the Amazon dataset, our systems are ranked first on TFCP, while showing competitive RMSE scores. In particular, the metric ranks text-based systems as the best among memory-based. Even when trained and evaluated on the Yelp dataset, our models rank first and third if we do not tune the hyperparameters specifically for that dataset.

Furthermore, results show that one can tune our approach's hyperparameters to optimize each metric. Doing so enables text-based models to yield better evaluation scores than their text-agnostic versions (e.g., once optimized for a metric, text-BKNN has a better score than BKNN), except for KNN on the RMSE metric. There is no winner-take-all hyperparameter set, achieving best results on both RMSE and TFCP. We observed that three out of four of the tuned text-based models involved the Many-to-Many matching and polarization. The latter signals the 5.4's second row). According to this metric, the best way of computing user similarity with semantic matches is to use the fine-grained Many-to-Many match count, with continuous similarity and polarization information, when per-item graphs are considered. On the contrary, the best Text-based model for RMSE (Text-BKNN-R) was obtained with the coarse-grained One-to-One matching, using minimal information from the binary count. It is worth noting that in the latter case, the global graph is preferred. Plus, we also considered filtering out matches with sentence similarity below a certain threshold, but interestingly, early experiments showed no quantitative benefit.

Ablation study

We observed that our best text-based models involved per-item graphs instead of the global graph.

To assess the benefit of semantic similarity, we studied the performance of three naive implementations counting the co-occurrence of sentences written by a pair of users u and v on an item i. The weights w (u, v) was computed by summing user weights w i (u, v) over all items i. The implementations we tried were:

• w i (u, v) = I{|S ui | > 0 and |S vi | > 0}, i.e. the similarity between users u and v relative to item i is 1 if both users reviewed i, and 0 otherwise.

• w i (u, v) = |S ui | • I{|S vi | > 0}, i.
e. the similarity between users u and v relative to item i is proportional to the number of sentences written by u on i if v reviewed i, and 0 otherwise.

• w i (u, v) = |S ui | • |S vi |, i.
e. the similarity between users u and v relative to item i is proportional to the number of sentences written by u and v on i.

After selecting the best option on the Amazon validation set, we found the best test scores to be RMSE = 1.197 and TFCP = 0.787. Even though the evaluation scores indicate existing signals from this similarity-agnostic approach, we see significant improvement and interest of integrating semantic similarity and finer relationships between sentences.

Explainability Besides quantitative performances, our text-based approach has benefits regarding the explainability of automatic recommendation. Table 5.5, Table 5.6, Table 5.7 and Table 5.8 show pairs of sentences from the Amazon train reviews written by some user u and u's nearest user neighbor v. Both users u and v liked u's test item j, i.e. r uj ≥ 4 and r vj ≥ 4. We manually selected sentence pairs counted as semantic matches by the system and relevant for explaining why the system predicts that u's opinion on j (r uj ) should resemble v's opinion on j (r vj ), through a high user similarity w (u, v). For readers unaware of the items, we added relevant attributes and commonalities relative to items. Table 5.5 and Table 5.6 correspond to the case of per-item graphs while table 5.7 and table 5.8 gather matches between sentences from different items in the global graph. The latter tables differ from each other in whether tail sentences (written by v) review u's test item j or not, which is equivalent for the recommender system but may matter in human interpretation.

Qualitative analysis of random semantic matches

Table 5.9 provides random matches found in 5 random per-item graphs. The first two examples have lower semantic similarity than the last three examples. Both sentences of the first example are descriptive, and the association focuses on the "killing" aspect. The second match is even unclearer; the only commonality is a mention of some female character/actress. The third example is made of a long and detailed tail sentence. The explicit reference to the Madea cinematic universe and the emphasized funniness are expressed in both sentences. The fourth pair of sentences refers to both the item's name they review as well as the "family" audience. We remark that mentionning an item name or title in review sentences may mislead the whole system, as it can trigger irrelevant semantic matches, especially if names are long. The fifth example includes a direct recommendation for lovers of suspense in both sentences.

5 random matches found in the global graph are shown in Table 5.10. The first match shows a pair of sentences reviewing the same item in which the item's name does not appear. Even if the director's name is common to both sentences, references to the Battle of Mount Austen are phrased with different vocabulary. Indeed, the head sentence refers to "empathy and apathy" during "the conflict at Guadalcanal" while the tail sentence describes "lush beauty of the South Pacific" contrasting with "the destruction of war". Fever Pitch -Drew Barrymore , as always , is phenomenal .

Ice Age -While not perfect , it is full of laughs and beautiful computer animation .

Finding Nemo -Highlights : Spectacular computer animation ; hilarious , well -developed characters ; original plot .

Monsters, Inc. -Great for parents and kids ( or people without kids ) .

Toy Story 3 -Great for kids and adults alike .

Island in the Sky -The Duke a does a great job in his role as Dooley -the plane's captain .

The Alamo -The Duke turns out one of his best performances , as well as putting together this film .

I Am Legend -At the end of the movie you fail to realise it was just one man , Will Smith in most of the scences and yet the movie is neither boring , or lacking in elements that make for a great thriller .

Hancock -Will Smith adds a lot of flare to the movie , when it could ' ve been bland and cheesy .

War of the Worlds -The special effects were great and the acting was believable .

Munich -The acting , story line and special effects were great .

Red River -I think this is my favorite early John Wayne film ( it's not exactly early , but it was one of his earlier big hits ) .

The Searchers -This is John Wayne's favorite " John Wayne movie , " and his acting is superb .

Bubba Ho-Tep -Fans of Bruce Campbell will love this movie , but I don ' t know how fans of the King will take it .

My Name Is Bruce -First of all if your not a bruce campbell fan or dont enjoy his movies then you probably wont like this one as it is trademark campbell . . . . a A nickname for the American actor John Wayne Kilmer's character survives and even thrives in this morass because he is an unsentimental machine , the " Spartan " ideal of an all -male world where it's kill or be killed , and there's plenty of both -even a suicide . I would reccomend that anyone who likes mystery or suspense thrillers should go see this .

Table 5.9: Random examples of semantic matches when our system considers per-item graphs. The last column shows the head sentence first and the tail sentence then.

information when taken out of context. The last two matching pairs are triggered by common vocabulary or clauses but can't be considered matches in users' preferences.

Domain-specific sentence embeddings

In addition to the pretrained USE model, we experimented with domain-specific embeddings. First, we fine-tuned a pretrained T5 [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF] LM on an item prediction classification task using the 2018 version of the Amazon Product Review dataset. We then applied our method to the representations learned by the fine-tuned T5 model in the hope that these representations would be finer than the more general USE representations. Results of baselines and our best models (cf. The Thin Red Line -/ / Director Terrence Malick focuses on the conflict at Guadalcanal from ground up and showing empathy and apathy along the way .

The Thin Red Line -Mr . Malick captures the lush beauty of the South Pacific and uses it to perfectly contrast with the destruction of war .

1.00

Seabiscuit -This is one of my all time favorite movies .

The Jazz Singer -This is one of my all -time favorite movies .

0.86

Tropic Thunder -it stole the show .

My Week with Marilyn -steals the show .

0.75

Heist -It's a known fact that he uses a metronome in order to keep his dialogue to have a certain rhythm to it .

Black Dynamite -He draws attention to it by repeatedly glaring at the mic throughout the scene , but doesn ' t miss a beat of the dialogue .

0.88

Grown Ups -Now I will say from the get go this film is not for everybody and I have noticed that some people just don ' t get this film .

Barbarella -This film is definately not for everyone & I ' d honestly recommend that most people rent it before they but it .

Table 5.10: Random examples of semantic matches when our system considers the global graph. The second column shows the head sentence first and the tail sentence then, along with the respective item they review.

Limitations

Regarding scalability and time complexity, the first limiting phase is the embedding of sentences. Embedding sentences with an on-the-shelf pretrained USE took a few hours using Apache Beam9 on 1, 000 machines. Besides this step, the bottleneck characteristic of our method is the computing of sentence k-NN graphs. When per-item graphs are considered, we iterated Scikit-learn's 10-NN graph algorithm10 on an average of ∼ 700 train sentences per item. For reasonable amounts of items, this can be achieved quickly, though this becomes more demanding for large item sets. It took 3 minutes to compute all sentence graphs on the Amazon dataset and 23 minutes on the Yelp dataset. The global graph, though, is much heavier to compute. We used the ScaNN similarity search method [START_REF] Guo | Accelerating large-scale inference with anisotropic vector quantization[END_REF] to compute the 10-NN global graph at scale on ∼ 2.9 million sentences for the Amazon dataset and ∼ 20 million sentences for the Yelp dataset. The computations were enabled by several hours of computing time on a 1TB RAM server. By comparison, the NARRE Deep Learning model was trained during 10 epochs with respective average epoch runtime of 17 minutes and 168 minutes for the Amazon and Yelp datasets, using an NVIDIA A100 GPU.

A second limitation to this work that we reserved for future work is a missing stronger theoretical background on FCP's relevance compared to RMSE, especially for contexts where the rating distribution is skewed (as is the case for the Amazon dataset).

Chapter 5 conclusion

This study explored, for the first time to our knowledge, the direct integration of semantic similarities from review text in recommender systems, through the representation power of modern pretrained NLP models.

Our experiments quantitatively showed the benefits of this approach relative to memory-based models. Further, preference representation from text is a step toward better explainability of machine learning systems.

Lastly, we argued in favor of ranking-based metrics to evaluate rating-prediction systems.

Future work includes evaluating explanations from our approach as well as building better representations of users and items, for example, by using more recent advances in language modeling or by joint tuning of a LM with the recommendation task. Unsupervised techniques could also help filter out irrelevant or redundant text. For instance, rationale extraction systems such as WT5 [START_REF] Narang | Wt5?! training text-to-text models to explain their predictions[END_REF] could be employed as a preprocessing step to select sentences more likely to represent meaningful preferences. Another promising advance at the intersection of text and recommenders that could be built on is entity prediction, following the work of Zemlyanskiy et al. [START_REF] Zemlyanskiy | Docent: Learning self-supervised entity representations from large document collections[END_REF]. More generally, our work highlights an exciting and emerging intersection between the fields of Natural Language Understanding and recommender systems.

Chapter 6

General conclusion and perspectives

Industrial revolutions transform societies by relying on emerging technologies. Scientific breakthroughs coupled with innovative engineering bring humans to design and implement novel machines, as well as usages impacting people's lives. After the digital revolution democratized computers and made Internet users globally connected, the Fourth Industrial Revolution, driven by Artificial Intelligence, seeks to blur the boundaries between the physical and digital worlds. In this game, Natural Language Processing plays a key role. Indeed, machines will be incredibly powerful and/or useful when they become able to hold complex reasoning automatically, and perhaps someday even think. Even though "the ability to speak does not make you intelligent."1 , language has long been thought of as a crucial part of the reasoning process. In The Theaetetus, Plato defined "thinking" with the Ancient Greek word λόγος2 ("reason", "speech", "computation").

"So then, I, for my part, refer to thinking as speaking, and opinion as speech."

In Chapter 2, we saw that computer scientists, together with mathematicians, linguists, and neuroscientists, have proposed algorithms to bridge the gap between machines and humans through automatic natural language understanding and generation. Recent progress in Machine Learning, enabled by weakly supervised Deep Learning architectures, software and hardware advances as well as increasingly larger datasets, have made NLP ripe for hitherto unseen applications. This thesis explored a subset of the possibilities offered by pretrained NLP models, focusing on assisting humans in their online interactions.

Chapter 3 introduced an original method (CAE-T5) to address toxic-to-civil transfer with pretrained Language Models, in the hope that it will nudge healthier conversations online. Specifically, we tackled the problem in a selfsupervised manner, where the dataset we had access to was made of comments annotated in toxicity only. Given

these unpaired examples, we were able to fine-tune a pretrained T5 model with an end-to-end denoising objective function requiring no previous sentence alignment. Even though human and automatic evaluations show that our approach significantly outperforms baselines, issues regarding hallucination remain to be solved before the system can be implemented in real use cases. At the time of our project, no parallel dataset existed, but posterior works released small paired datasets, which we then used to train a strongly supervised sequence-to-sequence model (SED-T5). Chapter 4 revealed that this latter approach shows less hallucination but at the cost of lower fluency and detoxification accuracy. Future work should investigate ensembling CAE-T5 and SED-T5, or even training a single incompletely supervised model leveraging both ideas (cf. Section 2.4.2).

Besides, a new task was presented in Chapter 4. We released the first dataset annotated in toxicity at the span level in the hope that it will help future developments in this direction. We described the setup along with an evaluation metric suited to the task. We also experimented with a few baselines that were released too. In particular, we showed that weakly supervised models trained on potentially larger datasets annotated at the post level only could almost reach the performance of strongly supervised models. Furthermore, the TOXICSPANS dataset proved useful for assessing the mitigation of explicit toxicity in toxic-to-civil transfer. The takeaways were twofold: first, models often successfully detect toxic spans and try to rephrase them, and then humans did rephrase almost all cases of explicit toxicity in the toxic posts they were given. The outcome of the online competition we organized using TOXICSPANS in a new shared task (cf. Appendix A) has been satisfying. The strong participation both indicated the community's high interest and helped us identify avenues worth exploring. First, low performance from both human crowdworkers and Machine Learning systems on highly-context-dependent posts could be targeted with automatic context sensitivity estimation, as described in Xenos et al. [START_REF] Xenos | Toxicity detection sensitive to conversational context[END_REF]. Then, as systems struggle to predict empty toxic span when the post is either non-toxic, implicitly toxic, or toxic in its entirety, span detection models could benefit from a pre-filtering process detecting if a post falls in one of these categories.

Finally, Chapter 5 describes an unprecedented way of predicting items' ratings from users according to their reviews of past items. We demonstrated that our method may not only outperforms rating-only-based recommender systems but also provide explanations for the predictions. Additionally, we investigated a concrete application leveraging the metric space of semantic embeddings produced by a pretrained sentence encoder. Our approach based on k-Nearast Neighbors illustrates a substantial procedure using graph techniques and deep semantic representations. In the future, we wish to find efficient unsupervised strategies to select sentences written by users best representing their tastes. Our node ranking experiments were inconclusive but we think that a clever combination of PageRank and Language Models may lead to effective solutions.

All in all, this thesis was a journey in the hot field of Natural Language Processing. This field and our contributions provide evidence that Artificial Intelligence can assist people in their interactions online by improving social exchanges, moderating conversations, and offering explainable recommendations based on what Internet users express. There remain ethical concerns that should be addressed in a multidisciplinary approach, such as potential bias in the definition of toxicity, risks of automated moderation being used for political censorship, or energy-intensive computing to run our models.

The main challenges encountered throughout this thesis outlined the following general perspectives regarding evaluation, datasets, and modeling. First, evaluation metrics, especially when no or few paired examples are available, remain an open topic and raise central issues to the development of weakly supervised Machine Learning algorithms. Then, while we focused here on the English language, we are excited by the increasing number of multilingual resources (regarding data and pretrained models) currently available and wish to see our works transferred to more languages. Lastly, we would be happy to analyze the few-shot performances of large pretrained Language Models on the three tasks studied here: style transfer, TOXICSPANS prediction, and rating prediction.

A.2 Participation overview

We received 479 individual participation requests, 92 team formations, and 1, 449 submissions. 91 teams submitted valid predictions (1, 385 valid submissions in total) and were scored; out of these, only 36 submitted system descriptions.

A.2.1 The HITSZ-HLT submission

The best performing team (HITSZ-HLT) formulated the problem as a combination of token labeling and span extraction [START_REF] Zhu | HITSZ-HLT at SemEval-2021 Task 5: Span-based ensemble model with toxic lexicon[END_REF].

For their token labeling approach, the team used two systems based on BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF]. Both systems had a Conditional Random Field (CRF) layer [START_REF] Sutton | An Introduction to Conditional Random Fields for relational learning[END_REF] on top, but one of the two also had an LSTM layer [START_REF] Hochreiter | Long short-term memory[END_REF] between BERT and the CRF layer. In both approaches, word-level BIO tags were used, i.e., words were labeled as B (beginning word of a toxic span), I (inside word of a toxic span), or O (outside of any toxic span).

For their span extraction approach, the team also used BERT. Roughly speaking, in this case, BERT produces probabilities indicating how likely it is for each token to be the beginning or end of a toxic span. Then a heuristic search algorithm, originally developed for target extraction in sentiment analysis by Hu et al. [START_REF] Hu | Open-domain targeted sentiment analysis via span-based extraction and classification[END_REF], selects the best combinations of candidate begin and end tokens, aiming to output the most likely set of toxic spans per post.

The character predictions of the three systems described above were combined with majority voting per character. That is, if any two systems considered a character to be part of a toxic span, then the ensemble classified the character as toxic; otherwise, the ensemble classified it as non-toxic.

A.2.2 The S-NLP submission

The team with the second best performing system (S-NLP) consists of individual participants who grouped and submitted an ensemble of their systems [START_REF] Nguyen | S-NLP at semeval-2021 task 5: Toxic spans detection[END_REF]. The ensemble combines two approaches, both of which are based on a RoBERTa model [START_REF] Liu | A robustly optimized bert pretraining approach[END_REF]. The latter is first fine-tuned to classify posts as toxic or non-toxic, using three Kaggle toxicity datasets2 . For toxic span detection, RoBERTa's subword representations from three different layers [START_REF] Abid | Concrete autoencoders for differentiable feature selection and reconstruction[END_REF][START_REF] Alain | What regularized auto-encoders learn from the data-generating distribution[END_REF][START_REF] Aßenmacher | On the comparability of pre-trained language models[END_REF] are summed to produce the corresponding word embeddings. A binary classifier on top of RoBERTa, operating on the word embeddings, predicts whether a word belongs to a toxic span or not.

For the first component of the ensemble, the word embeddings obtained from RoBERTa's subword representations are concatenated with FLAIR [5] and FastText [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF] embeddings. 3 The resulting embeddings are passed on to a two-layer stacked BiLSTM with a CRF layer on top to generate a BIO tag per word.

The second component of the ensemble used the RoBERTa model as a teacher to produce silver toxic spans for 30, 000 unlabelled toxic posts [START_REF] Borkan | Nuanced metrics for measuring unintended bias with real data for text classification[END_REF]. RoBERTa was then retrained as a student on the augmented dataset (30, 000 posts with silver labels and the training posts we provided) to predict toxic offsets.

The ensemble returns the intersection of the toxic spans identified by the two components.

A.2.3 Additional interesting approaches

We now discuss some of the most interesting alternative approaches tried by the participants, even if they did not lead to high scores.

Rationales Some participants experimented with training toxicity classifiers on external datasets containing posts labeled as toxic or non-toxic; and then employing model-specific or model-agnostic rationale extraction mechanisms to produce toxic spans as explanations of the decisions of the classifier. The model-specific rationale mechanism of Rusert [START_REF] Rusert | NLP_UIOWA at Semeval-2021 Task 5: Transferring toxic sets to tag toxic spans[END_REF] used the attention scores of an LSTM toxicity classifier to detect the toxic spans. Pluci ński and Klimczak [START_REF] Pluci | GHOST at SemEval-2021 Task 5: Is explanation all you need?[END_REF] used the same approach, but also employed an orthogonalisation technique [START_REF] Mohankumar | Towards transparent and explainable attention models[END_REF]. The modelagnostic rationale mechanism of Rusert [START_REF] Rusert | NLP_UIOWA at Semeval-2021 Task 5: Transferring toxic sets to tag toxic spans[END_REF] combined an LSTM classifier with a token-masking approach that we call Input Erasure (IE), due to its similarities to the method of Li et al. [START_REF] Li | Understanding neural networks through representation erasure[END_REF]. The model-agnostic approach of Pluci ński and Klimczak [START_REF] Pluci | GHOST at SemEval-2021 Task 5: Is explanation all you need?[END_REF] combined SHAP [START_REF] Lundberg | A unified approach to interpreting model predictions[END_REF] with a fine-tuned BERT model. Ding and Jurgens [START_REF] Ding | HamiltonDinggg at SemEval-2021 Task 5: Investigating toxic span detection using RoBERTa pre-training[END_REF] and Benlahbib et al. [START_REF] Benlahbib | LISAC FSDM USMBA at SemEval 2021 Task 5: Tackling toxic spans detection challenge with supervised spanBERT-based model and unsupervised LIME-based model[END_REF] also experimented with model-agnostic approaches, but they combined LIME [START_REF] Ribeiro | Why Should I Trust You?" Explaining the predictions of any classifier[END_REF] with a Logistic Regression (LR) or with a linear Support Vector Machine (SVM) toxicity classifier. All the above-mentioned approaches used a threshold to turn the explanation scores (e.g., attention or LIME scores) of the words into binary decisions (toxic/non-toxic words).

Lexicon-based

No team relied on a purely lexicon-based approach, but few experimented with lexicon-based baselines [START_REF] Zhu | HITSZ-HLT at SemEval-2021 Task 5: Span-based ensemble model with toxic lexicon[END_REF][START_REF] Palomino | An ensemble approach to identify toxicity in text[END_REF] or used such components in ensembles [START_REF] Ranasinghe | WLV-RIT at SemEval-2021 Task 5: A neural transformer framework for detecting toxic spans[END_REF]. Three kinds of lexicon-based methods were used.

First, the lexicon was handcrafted by domain experts [START_REF] Smedt | Profanity & offensive words (POW)[END_REF], and it was simply employed as a list of toxic words for lookup operations [START_REF] Palomino | An ensemble approach to identify toxicity in text[END_REF]. Second, the lexicon was compiled using the set of tokens labeled as toxic in our spanannotated training set, and it was used as a lookup table [START_REF] Burtenshaw | UAntwerp at SemEval-2021 Task 5: Spans are spans, stacking a binary word level approach to toxic span detection[END_REF], possibly also storing the frequency of each lexicon token in the training set [START_REF] Zhu | HITSZ-HLT at SemEval-2021 Task 5: Span-based ensemble model with toxic lexicon[END_REF]. The former two were also combined [START_REF] Ranasinghe | WLV-RIT at SemEval-2021 Task 5: A neural transformer framework for detecting toxic spans[END_REF]. Third, the least supervised lexicons were built with statistical analysis on the occurrences of tokens in a training set solely annotated at the comment level (toxic/non-toxic post) [START_REF] Rusert | NLP_UIOWA at Semeval-2021 Task 5: Transferring toxic sets to tag toxic spans[END_REF]. An added value of these approaches is that easy-to-use resources (toxicity lexicons) are built and shared publicly, such as the one suggested by Pluci ński and Klimczak [START_REF] Pluci | GHOST at SemEval-2021 Task 5: Is explanation all you need?[END_REF]. 4Custom losses Zhen Wang and Liu [START_REF] Zhen Wang | MedAI at SemEval-2021 Task 5: Start-to-end tagging framework for toxic spans detection[END_REF] experimented with a new custom loss, which weighted false toxicity predictions based on their location in the text. If a false prediction was located near a ground truth toxic span, then it would contribute less to the overall loss for that post compared to one located further away. The loss function used by Kuyumcu et al. [START_REF] Kuyumcu | Sefamerve_arge at SemEval-2021 Task 5: Toxic span detection using segmentation based 1-d convolutional neural network model[END_REF] to train their system is the Tversky Similarity Index [START_REF] Tversky | Features of similarity[END_REF], a generalization of the Sørensen-Dice coefficient and the Jaccard index, which the authors adjusted to weigh up false negatives.

Data augmentation

The vast majority of the participating teams employed additional training data annotated at the post level. That is, either to build lexicons [START_REF] Rusert | NLP_UIOWA at Semeval-2021 Task 5: Transferring toxic sets to tag toxic spans[END_REF], to leverage unsupervised rationale extraction methods [START_REF] Rusert | NLP_UIOWA at Semeval-2021 Task 5: Transferring toxic sets to tag toxic spans[END_REF][START_REF] Pluci | GHOST at SemEval-2021 Task 5: Is explanation all you need?[END_REF][START_REF] Ding | HamiltonDinggg at SemEval-2021 Task 5: Investigating toxic span detection using RoBERTa pre-training[END_REF][START_REF] Benlahbib | LISAC FSDM USMBA at SemEval 2021 Task 5: Tackling toxic spans detection challenge with supervised spanBERT-based model and unsupervised LIME-based model[END_REF], or to filter posts [START_REF] Luu | UIT-ISE-NLP at SemEval-2021 Task 5: Toxic span detection with BiLSTM -CRF and toxic BERT comment classification[END_REF] that were not labeled as toxic by a toxicity classifier. Suman and Jain [START_REF] Suman | AStarTwice at SemEval-2021 Task 5: Toxic span detection using RoBERTa-CRF, domain specific pre-training and self-training[END_REF] astutely produced silver data from external sources to augment the initial golden annotated dataset, training their model iteratively in a semi-supervised manner.

A.3 Results

Table A.1 shows the scores and ranks of all participating teams that described their approach, i.e., 36 out of 91 teams that participated. HITSZ-HLT (Section A.2.1) was ranked first, followed by S-NLP (Section A.2.2) that scored 0.06% lower. The rest of the teams followed with scores lower than 70%.

The score of the median system is 67.58%, which is not far below the top scored team (-3.22 percent units), while it is far above the last two (+17.52 percent units). The standard deviation of system scores above the median is much lower (0.94) than that of the systems below the median (4.12). Most teams that were excluded from the table (because they did not describe their methods) scored lower than the median. However, there were also top-scoring teams among those that were excluded, such as a team with a RoBERTa-based token-level ensemble that was ranked 4 th .5

A.4 Analysis and discussion

Overall we were happy to see the degree of involvement in this shared task and the resulting diversity of approaches to this problem. We include some of our observations regarding the evaluation administration and what we have learned from the results.

A.4.1 Participation

We reached out to teams that decided not to submit a description paper, and the vast majority were students who were time-limited. The fact that students participated in the task is promising, and we plan to consider more ways to introduce SemEval tasks in classrooms. On the other hand, 60% of the participants chose not to describe their approach, which is problematic and should be addressed. A team could take advantage of such an option to create It is also worth mentioning that the extended timeline allowed participants to join forces. For instance, a number of participants decided to combine their systems and form the 2 nd ranked S-NLP. Their ensemble scored higher than all their standalone systems, though their best standalone system would still be ranked 2 nd . In any case, we welcome the collaboration between participants, which may provide further insights regarding effective combinations of architectures.

A.4.2 General remarks on the approaches

Except for lexicon-based baselines, we observed that the vast majority of systems adopted the recent paradigm in NLP: fine-tuning large off-the-shelf transformers [START_REF] Vaswani | Attention is all you need[END_REF] pretrained on massive corpora. Non-transformer-based approaches, mostly LSTMs with pretrained word embeddings, were also used. The nature of the task, similar to the well-studied Named Entity Recognition task, led many competitors to use a CRF layer on top of the model (e.g., transformers or LSTMs) of their choice.

A.4.3 Performance

The winning team (HITSZ-HLT) combined BERT with two approaches for their ensemble: a token labeling approach (two versions, with/without an LSTM between BERT and the CRF) and a span extraction approach (Section A.2.1).

The comparison of the two showed that span extraction is slightly better on posts with a single span, but token labeling is clearly better on multi-span posts [START_REF] Zhu | HITSZ-HLT at SemEval-2021 Task 5: Span-based ensemble model with toxic lexicon[END_REF]. The complementary nature of the two approaches is probably what makes even a simple majority voting ensemble better than its competitors.

The system ranked second (S-NLP) also employed an ensemble, using a RoBERTa model initially fine-tuned to classify posts as toxic or non-toxic as the starting point [START_REF] Nguyen | S-NLP at semeval-2021 task 5: Toxic spans detection[END_REF]. The ensemble combined (i) the resulting RoBERTa model, now fine-tuned to predict toxic spans, with additional FLAIR and FastText embeddings, and (ii) a RoBERTa model retrained as a student to predict toxic spans (Section A.2.2). Although the two standalone models achieved higher scores than the standalone models of the top-ranked team (HITSZ-HLT), the ensemble did not yield significant improvements. This may be due to the student's decisions not being that complementary to the teacher's, as the team notes [START_REF] Nguyen | S-NLP at semeval-2021 task 5: Toxic spans detection[END_REF].

Teams that experimented with rationale extraction mechanisms (Section A.2.3) did not find this approach advantageous compared to supervised sequence labeling in terms of F 1 scores. However, the reported results of the rationale-based systems show that this approach is promising, especially because it does not require any data annotated at the span level. Hence, we explored this direction in Section 4.7. used were LSTM, Logistic Regression (LR), Support Vector Machines (SVM), and BERT. The attention scores of an LSTM were used with [START_REF] Pluci | GHOST at SemEval-2021 Task 5: Is explanation all you need?[END_REF] and without an orthogonality method [START_REF] Rusert | NLP_UIOWA at Semeval-2021 Task 5: Transferring toxic sets to tag toxic spans[END_REF], with the latter being slightly better; these are model-specific rational extraction methods (Section A.2.3). Model-agnostic approaches (Input Erasure, LIME, SHAP) were better than the model-specific ones. The best rationale-based method employed a BERT model, fine-tuned for toxic post classification and SHAP.

Lexicon-based approaches were only used as baselines or components in ensembles, as already noted. In principle, all lexicon-based systems are extremely efficient and interpretable. Table A.3 shows they can also achieve surprisingly high scores.

A.4.4 Error analysis

A common theme across many competitor reports was the serious challenge posed by comments with no toxic spans. It is not readily evident why this is a common occurrence in the task, and certainly, the way that annotation consensus is used to combine annotations can be a contributing factor. However, many systems seemed determined to tag some spans and many authors noted that performance on posts with no tagged span was extremely poor compared to performance on posts with tagged spans.

Many systems were also reluctant to tag function words like "of" and "and", which can be included in multi-word

4. 7 96 4. 8 1 Introduction 1 . 1

 7968111 Figure 5.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7 Bar chart of the models' test TFCP scores on the 2018 version of the Amazon dataset, when we considered domain-specific embeddings rather than Universal Sentence Encoder's embeddings . . .

•

  L. Laugier, J. Pavlopoulos, J. Sorensen and L. Dixon: Civil Rephrases Of Toxic Texts With Self-Supervised Transformers. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2021) [267] • J. Pavlopoulos, L. Laugier, J. Sorensen and I. Androutsopoulos: Semeval-2021 task 5: Toxic spans detection. Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval 2021) [381] • J. Pavlopoulos, L. Laugier, A. Xenos, J. Sorensen and I. Androutsopoulos: From the Detection of Toxic Spans in Online Discussions to the Analysis of Toxic-to-Civil Transfer. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022) [382] • A. Xenos, J. Pavlopoulos, I. Androutsopoulos, L. Dixon, J. Sorensen, L. Laugier: Toxicity Detection can be Sensitive to the Conversational Context. First Monday, 27(5). [544] • L. Laugier, T. Bonald and L. Dixon: Semantic Encoding of Review Sentences for Memory-Based Recommenders . Preprint Chapter 2

Figure 2 . 1 :

 21 Figure 2.1: Venn diagram of the scientific topics explored in the present research. Chapter 4 and Chapter 5 lie in the blue-green (large) colorized area while Chapter 3 can be categorized in the brown (small) colorized area. Figure best viewed in color.

Figure 2 . 5 :

 25 Figure 2.5: Structures of neural models found in the literature. h i is the contextualized representation corresponding to the i th input token.

H 0

 0 = ∅ H t = arg max H⊂ Ht: |H|=B ŝ1:t ∈H P S 1:t ∼H S 1:t = ŝ1:t |θ where Ht = (ŝ 1:t-1 ,w) ∈Ht-1×V ŝ1:t-1 ⌢ (w) Finally ŝ = arg max ŝ1:T ∈H T P S 1:T ∼H S 1:T = ŝ1:T |θ (2.4) Beam search has a linear run time in Θ (B • |V | • T ) with a constant factor equal to B. Beam search is widely used in NLG, especially in machine translation, and plays a role in increasing the diversity of generated text since it yields B alternative generated text. The beam width B gives direct control over the quality-complexity tradeoff: B = 1 is greedy search, while incrementing B exponentially from V to V T at each time step corresponds to exhaustive search.

  but only different oppositions to what precedes and what follows. [. . .]

Figure 2 . 6 :

 26 Figure 2.6: Computations made by a single artificial neural unit. Figure best viewed in color.

Figure 2 . 7 :

 27 Figure 2.7: Computations made by a single feedforward neural layer. Figure best viewed in color.

Figure 2 . 8 :

 28 Figure 2.8: Computations made by a fully-connected feedforward two-layer neural network. Figure best viewed in color.

Figure 2 . 9 :

 29 Figure 2.9: Standard structure of a deep Feedforward Neural Network. Figure best viewed in color.

Figure 2 . 11 :

 211 Figure 2.11: Illustration of the transformer architecture. θ indicates that the layer (in blue) is parametric. Left: Unitransformer stack. Middle: A detailed transformer block. Right: Matrix representation of possible alternative attention masking patterns. The input token at row i is allowed to attend the input token at column j if and only if the cell at (i, j) is shaded. Figure best viewed in color.

Figure 3 . 1 :

 31 Figure 3.1: Examples of image-to-image transfer with CycleGAN. Figure from Zhu et al. [594].

8 )

 8 Figure 3.2.

Figure 3 . 2 :

 32 Figure 3.2: Illustration of the training (a and b) and inference (c) procedures. (a) DAE:The bitransformer encodes the corrupted input text x in a latent variable z that is then decoded conditioned on the source attribute α(x) with the objective of minimizing the cross entropy between x and the generated text x. Here, generation is not autoregressive since the DAE is trained with teacher forcing. (b) CC: The input x is pseudo-transferred with attribute ᾱ(x) via auto-regressive decoding because we do not know the ground-truth y. The generated output ŷ is then backtransferred to the original space of sentences with attribute α(x). Back-transfer generation is not auto-regressive because we use teacher-forcing here. Thus, we can trivially back-propagate the gradients through f θ (back-transfer) but not through f θ (pseudo-transfer). (c) Model overview at prediction time. x and ŷ are the input and generated text, a is the destination attribute, z is the latent input representation and c(a) is the control code.

  Following Li et al.[START_REF] Li | Delete, retrieve, generate: a simple approach to sentiment and style transfer[END_REF], Zhang et al.[START_REF] Zhang | Learning sentiment memories for sentiment modification without parallel data[END_REF][START_REF] Zhang | Style transfer as unsupervised machine translation[END_REF], Wu et al.[START_REF] Wu | A hierarchical reinforced sequence operation method for unsupervised text style transfer[END_REF][START_REF] Wu | Mask and infill: Applying masked language model for sentiment transfer[END_REF], Wang et al.[START_REF] Wang | Controllable unsupervised text attribute transfer via editing entangled latent representation[END_REF], John et al.[START_REF] John | Disentangled representation learning for non-parallel text style transfer[END_REF], Liu et al.[START_REF] Liu | Revision in continuous space: Unsupervised text style transfer without adversarial learning[END_REF], Luo et al.[START_REF] Luo | A dual reinforcement learning framework for unsupervised text style transfer[END_REF], Jin et al.[START_REF] Jin | Imat: Unsupervised text attribute transfer via iterative matching and translation[END_REF] and to further confirm the performance of CAE-T5, we hired human annotators on Appen to rate in a blind fashion different models' civil rephrasings of 100 randomly selected test toxic comments, in terms of attribute transfer, fluency, content preservation and overall quality on a Likert scale from 1 to 5. Each rephrasing was annotated by 5 different crowd-workers whose annotation quality is controlled by test questions. If a rephrasing is rated 4 or 5 on attribute transfer, fluency, and content preservation, then it is "successful".

Figure 3 .

 3 Figure 3.3 and Figure 3.4 detail the guidelines we wrote on the crowdsourcing website Appen 5 , when we asked human crowd-workers to rate automatic rephrasings and to rephrase toxic comments. The contributor level was set to level 3, which corresponds to the highest quality standard.

  biased leftist trash article. CAE-T5 biased leftist article. Input dumb people vote for trump. CAE-T5 most people vote for trump. Input what hypocrites your sources are! CAE-T5 what are your sources of information? ! ! ! ! Input bill o'reilly is a narcissistic sociopath. CAE-T5 bill o'reilly is a neo-liberal.............

  The tasks used to evaluate the metrics were Machine Translation (WMT 2019 323), Data-to-Text (WebNLG-eval 469), Text Simplification (Asset-eval 8 and MUSS-eval 332), Image Captioning (PASCAL50S 516 and Flickr8k 194), Summarization (REALSumm 34 and SummEval 122) and Question Answering (Efficient QA 347 and OKVQA-Eval 328).

Chapter 4 From

 4 the Detection of Toxic Spans in Online Discussions to the Analysis of Toxic-to-Civil Transfer This chapter presents research conducted with John Pavlopoulos, Alexandros Xenos, Jeffrey Sorensen, and Ion Androutsopoulos. It led to a task description published in the Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval 2021) as well as an article published in the Proceedings of the 60th Annual Meeting

Table 4 . 1 :

 41 Examples of toxic posts and their ground truth toxic spans (also shown in bold red). In the left column, toxic spans are shown as sets of character offsets. No toxic spans are included in the ground truth of the last posts.

Figure 4 . 1 :

 41 Figure 4.1: Distribution of the percentage of characters of each post that are covered by the ground truth spans.

Figure 4 .

 4 [START_REF] Abid | Concrete autoencoders for differentiable feature selection and reconstruction[END_REF] shows the distribution of the percentage of character offsets of each post that are included in toxic spans.

Figure 4 .

 4 2 illustrates the distribution of dense toxic spans per post.

Figure 4 . 3

 43 shows the most frequent toxic spans in the dataset (after lower-casing each post) and their frequencies.

Figure 4 . 4

 44 shows the most frequent multi-word toxic spans (again after lower-casing).

Figure 4 .

 4 5 illustrates the distribution of the size (in words) of those posts whose ground truth covers the whole post.

Figure 4 . 2 :

 42 Figure 4.2: Distribution of the number of dense ground truth toxic spans per post in TOXICSPANS.

Figure 4 . 3 :

 43 Figure 4.3: Most frequent toxic spans in TOXICSPANS.

Figure 4 . 4 :

 44 Figure 4.4: Most frequent multi-word toxic spans.

Figure 4 . 5 :

 45 Figure 4.5: Distribution of size (in words) of posts whose ground truth covers the whole post.

Table 4 . 8 :

 48 Examples of posts where the ground truth was not empty, and SPAN-BERT-SEQ incorrectly predicted more (or larger) toxic spans. Ground truth in red. Predictions of SPAN-BERT-SEQ in bold.

Figure 4 . 6 :

 46 Figure 4.6: Improvement in the F1 of BILSTM+ARE when increasing the training set of its underlying BILSTM with posts tagged at the post-level (toxic/non-toxic, no toxic spans). Standard error of mean shown as error bars.

Figure 4 .

 4 Figure 4.6 shows the improvement in the F 1 score of BILSTM+ARE when increasing the training set of the underlying BILSTM with 5000, 10000, 20000, 40000, 80000 more posts (always balanced toxic/non-toxic) with post-level annotations only (no toxic span annotations). The dashed lines represent the sequence labeling methods, which cannot benefit directly from training data without toxic span annotations. Similarly, Figure 4.7 shows the corresponding improvement in the ROC AUC score of the underlying BILSTM in the toxic/non-toxic text classification task.

Figure 4 . 7 :

 47 Figure 4.7: Improvement in the ROC AUC of BILSTM+ARE in the toxic spans detection task, when increasing the training set of its underlying BILSTM with posts tagged at the post-level (no toxic spans).

Figure 4 . 8 :

 48 Figure 4.8: Frequency of annotations based on the country of origin of the crowd-annotators.

Figure 5 . 1 :

 51 Figure 5.1: 2D t-SNE projection of sentence embeddings. Red, green, and blue colors represent sentences from horror, comedy, and animation movies, respectively. Movie titles are displayed at the barycenter of their review sentences (with adjustment for readability purposes). Figure best viewed in color.

.

Figure 5 . 2 :

 52 Figure 5.2: Text-KNN pipeline. σ n , σ ′ n , σ ′′ n , . . . represent the embeddings of sentences written by user v n . k-NN graphs are built from the sentence similarity s, and are used for computing the weights w of the user k-NN regressor. Figure best viewed in color.

Figure 5 . 3 :

 53 Figure 5.3: Heatmaps of k-NN sentence embedding matches where k = 10. Rows and columns respectively represent head and tail vertices. Same-item matches are discarded. Left: matches aggregated per movie. Right: matches aggregated per category.

  is the set of users v whose sentences appear at least once in the neighborhood of sentence σ 1 .N (u) = σ1∈Su N (σ 1 )is the set of users v whose sentences appear at least once in the neighborhood of sentences written by user u. Indicators are denoted by I.One-to-One matching First, we considered u and v to have expressed similar preferences if v wrote at least one sentence in the semantic neighborhood of at least one sentence written by u, i.e.w (u, v) = I{v ∈ N (u)}(5.4)Many-to-One matching Alternatively, we computed the occurrence of v's sentences in the neighborhood of u's sentences, i.e.w (u, v) = σ1∈Su I{v ∈ N (σ 1 )} (5.5)

Figure 5 . 4 :Figure 5 . 5 :

 5455 Figure 5.4: Bar charts of the models' test scores (average and standard deviation) on the Amazon dataset.

Figure 5 . 6 :

 56 Figure 5.6: Correlation coefficients of the rankings produced by the metrics when considering the systems in Figure 5.4.

Figure 5 . 7 )

 57 show that there is no clear advantage to using domain-specific sentence representations over general embeddings.

Figure 5 . 7 :

 57 Figure 5.7: Bar chart of the models' test TFCP scores (average and standard deviation) on the 2018 version of the Amazon dataset, when we considered domain-specific embeddings rather than USE's embeddings.

Figure A. 1 :

 1 Figure A.1: Number of submissions per evaluation day.

Figure A. 2 :

 2 Figure A.2: The evaluation score (character F 1 ) of the best submission per day during the evaluation period.

Titre:

  Analyse et Contrôle des Interactions en ligne avec des Réseaux Neuronaux Artificiels de Traitement Automatique du Langage Naturel Mots clés: Apprentissage automatique, Intelligence artificielle, Traitement automatique du langage naturel, Informatique, Réseaux de neurones artificiels, Systèmes de recommandation Résumé: Le traitement automatique du langage naturel est motivé par des applications où les ordinateurs doivent acquérir une compréhension sémantique et syntaxique du langage humain. Récemment, le domaine a été impacté par un changement de paradigme. Les architectures d'apprentissage profond couplées à des techniques d'apprentissage auto-supervisé sont devenues le coeur des modèles correspondant à l'état de l'art en compréhension et génération du langage naturel. Parfois considérés comme des "foundation models", ces systèmes ouvrent la voie à de nouveaux cas d'utilisation. Née d'un partenariat académique et industriel entre l'Institut Polytechnique de Paris et Google AI Research, la présente recherche s'est concentrée sur l'étude de la façon dont les modèles neuronaux de traitement du langage naturel pré-entraînés pouvaient être utilisés pour améliorer les interactions en ligne. Cette thèse a d'abord exploré comment le transfert de style auto-supervisé pouvait être appliqué à la reformulation non-toxique de commentaires offensants dans les conversations en ligne. Dans le contexte de la modération de contenu toxique en ligne, nous avons proposé une méthode de réglage fin d'un modèle texte-à-texte pré-entraîné (T5) avec une fonctionobjectif consistant en un auto-encodeur débruiteur cyclique. Ensuite, les travaux de recherche ont porté sur l'étude de l'annotation humaine et la détection automatique des sous-ensembles de mots toxiques dans les conversations en ligne. Nous avons publié un nouveau jeu de données annoté pour entraîner et évaluer les systèmes automatiques, ce qui a conduit à une tâche partagée lors du 15e International Workshop on Semantic Evaluation. Enfin, nous avons développé un système de recommandation basé sur des avis en ligne, s'inscrivant dans l'explicabilité des préférences prises en compte par les recommandations prédites. La méthode utilise des modèles basés sur la similarité sémantique textuelle pour représenter les préférences d'un utilisateur sous la forme d'un graphe de fragments de texte, où les arrêtes sont définies par la similarité sémantique. Title: Analysis and Control of Online Interactions through Neural Natural Language Processing Keywords: Machine learning, Artificial intelligence, Natural language processing, Computer Science, Artificial neural networks, Recommender systems Abstract: Natural Language Processing is motivated by applications where computers should gain a semantic and syntactic understanding of human language. Recently, the field has been impacted by a paradigm shift. Deep learning architectures coupled with self-supervised training have become the core of state-of-the-art models used in Natural Language Understanding and Natural Language Generation. Sometimes considered as foundation models, these systems pave the way for novel use cases.Driven by an academic-industrial partnership between the Institut Polytechnique de Paris and Google AI Research, the present research has focused on investigating how pretrained neural Natural Language Processing models could be leveraged to improve online interactions. This thesis first explored how self-supervised style transfer could be applied to the toxic-to-civil rephras-ing of offensive comments found in online conversations. In the context of toxic content moderation online, we proposed to fine-tune a pretrained text-to-text model (T5) with a denoising and cyclic auto-encoder loss. Then, a subsequent work investigated the human labeling and automatic detection of toxic spans in online conversations. We released a new labeled dataset to train and evaluate systems, which led to a shared task at the 15th International Workshop on Semantic Evaluation. Finally, we developed a recommender system based on online reviews of items, taking part in the topic of explaining users' tastes considered by the predicted recommendations. The method uses textual semantic similarity models to represent a user's preferences as a graph of textual snippets, where the edges are defined by semantic similarity.Institut Polytechnique de Paris 91120 Palaiseau, France

⌢

  Concatenation operator: (a 1 , a 2 , . . . , a n ) ⌢ (b 1 , b 2 , . . . , b m ) = (a 1 , a 2 , . . . , a n , b 1 , b 2 , . . . , b n ):Slicing operator:a 1:n = (a 1 , a 2 , . . . , a n ) GLUE General Language Understanding Evaluation. 19, 22, 40, 46 GPT Generative Pre-trained Transformer. 23, 31, 38, 43-46, 66, 71, 73 GPU Graphics Processing Unit. 45, 127 HPC High-Performance Computing. 51, 54, 56 LM Language Model. 11, 15, 23, 34-38, 40, 42-46, 54, 62-64, 71, 73, 80, 125, 127, 129-131 LSTM Long Short-Term Memory. 23, 52, 135, 136, 139, 140 ML Machine Learning. 14, 15, 17, 19, 24, 26-28, 34, 35, 37, 43, 45-47, 50, 54, 56-58, 82, 106, 129-131 MLM Masked Language Model. 32, 40 NLG Natural Language Generation. 19-22, 28, 31, 35, 37, 38, 42, 45, 52, 58, 61, 62, 82 NLP Natural Language Processing. 9, 11, 14, 15, 17-24, 26, 29-36, 38, 42-46, 48, 50-52, 54, 56, 58, 111, 127,

	T5 Text-To-Text Transfer Transformer. 23, 31, 43-46, 64, 65, 125
	TFCP Time-based Fraction of Concordant Pairs. 10, 116-120, 126
	TPU Tensor Processing Unit. 45, 71

129, 130

NLU Natural Language Understanding.

19-22, 28, 31, 32, 35, 38-40, 42, 43

, 50, 52, 58, 82, 105, 106, 127 PLM Pretrained Language Model. 23, 45, 106 PPL Perplexity. 20, 66, 68, 71-73, 79, 80, 97-99 RMSE Root-Mean-Square Error. 108, 109, 115-119 RNN Recurrent Neural Network. 23, 52-54, 62 ROC-AUC Area Under the Receiver Operating Characteristic Curve. 66 SuperGLUE Super General Language Understanding Evaluation. 19, 22, 46 SVD Singular Value Decomposition. 108, 117 t-SNE t-distributed Stochastic Neighbor Embedding. 10, 63, 107, 111 UNLG Unconditional Natural Language Generation. 20, 21, 27, 31, 38, 42, 43 USE Universal Sentence Encoder. 10, 51, 67, 71, 80, 111, 112, 125-127

Table 2 .

 2 1: Common NLP downstream tasks and evaluation datasets

	Learning
	strategy
	Adaptation
	Input tuning
	Automatic
	Continuous	Prompt Tuning [280]

  LMs can be auto-regressive (cf. Section2.5.4). At each generation step t,

	the model has access to representations of the previously generated token(s) ŝ1:t-1 = ŝ1 , ŝ2 , . . . , ŝt-1 . Auto-
	regressive LMs are computed with the chain rule (cf. Equation (2.2)).
			m
	P (s|θ) = P s 1 , s 2 , . . . , s m |θ =	P s t |ŝ 1 , ŝ2 , . . . , ŝt-1 ; θ	(2.2)
			t=1
	Search techniques for Generation There are various strategies to generate tokens from an auto-regressive LM
	at inference time. They can be grasped by their quality-complexity trade-off. Let T be the maximum size of a
	generated sequence. In theory, T can be infinite, but in practice, T ∈ N.
	Exhaustive search Under the assumption of a good auto-regressive LM, searching the sequence with maximum
	likelihood requires a brute-force approach. Exhaustive search consists in exploring the entire distribution of all
	possible sequences made of tokens from the vocabulary V . Equation (2.3) formalizes the exhaustive search.
	ŝ = arg max	P (s|θ)	(2.3)
	T
	s∈	V k
	k=1
	Exhaustive search has exponential complexity as its running time is in Θ |V | T . In NLP, |V | ∼ 10 4 so generating
	long and/or many sequences quickly becomes intractable.

  1 , s 2 , . . . , s m |θ requires to compute the sequence P s t |ŝ 1 , ŝ2 , . . . , ŝt-1 ; θ t∈[1..T ] one generation step at a time while P s t |s 1 , s 2 , . . . , s t-1 ; θ t∈[1.

now have to defend this guy from his russian ties........

  OFFENSIVE COMMENT you now have to defend this clown along with his russian corruption. GENERATED CIVIL COMMENT you INPUT OFFENSIVE COMMENT blaming trudeau and the government is just stupid. GENERATED CIVIL COMMENT blaming

	This work focuses
	on models that can help suggest rephrasings of toxic comments in a more civil manner. Inspired by recent
	progress in unpaired sequence-to-sequence tasks, a self-supervised learning model is introduced, called
	CAE-T5 a . CAE-T5 employs a pretrained text-to-text transformer, which is fine-tuned with a denoising and
	cyclic Auto-Encoder loss. Experimenting with the largest toxicity detection dataset to date (Civil Comments)
	our model generates sentences that are more fluent and better at preserving the initial content compared
	to earlier text style transfer systems which we compare with using several scoring systems and human
	evaluation.

a The code is publicly available at github.com/LeoLaugier/conditional-auto-encoder-text-to-text-transfer-transformer with an Apache License 2.0.

INPUT

trudeau and the liberal government is just wrong.

Table 3 .

 3 2: Statistics for the Yelp dataset and the processed version of the Civil Comments dataset. Average lengths are the average numbers of SentencePiece tokens.

				Polarized Civil Comments
		Positive	Negative	Toxic	Civil
	Train	266,041	177,218	90,293	5,653,785
	Dev	2,000	2,000	4,825	308,130
	Test	500	500	4,878	305,267
	Average length	11.0	13.0	19.4	21.9

  The test ROC-AUC of the toxicity classifier was 0.98 with a precision of 0.95 and a recall of 0.38. Even with this low recall |X T | was large enough (approximately 90, 000, see Table 3.2).

, supporting the israelis doesn't help things a bit.

	TEXT	BLEU	SIM
	Original Human rephrasing alsoOriginal furthermore, kissing israeli ass doesn't help things a bit just like the rest of the marxist idiots. Human rephrasing it	57.6 3.4	70.6 % 65.3 %

is the same thing with people who follow Karl Marx doctrine

	Original Human rephrasing you you will go down as being the most incompetent buffoon ever elected, congrats!	2.3	16.2 %

could find out more about it.

Table 3 .

 3 3: Evaluation with BLEU and SIM of examples rephrased by human crowdworkers. 3. Content preservation: Content preservation is the most difficult aspect to measure. Unsupervised Neural Machine Translation

Table 3 .

 3 4 shows quantitative results on the Civil Comments dataset. Surprisingly, the Perplexity (capturing fluency) of text generated by our model was lower than the Perplexity computed on human comments. This can be explained by social media authors of comments expressing an important variability in language formal rules that was only

	Model	ACC↑ PPL↓ self-SIM↑ GM↑
	Copy input	0.0% 6.8	100.0%	0.005
	Random civil	100.0% 6.6	20.0%	0.311
	Human	82.0% 9.2	73.8%	0.404
	CrossAlignment [466]	94.0% 11.8	38.4%	0.313
	IE (BERT)	86.8% 7.5	55.6%	0.401
	StyleTransformer (Conditional) [96]	97.8% 47.2	68.3%	0.242
	StyleTransformer (Multi-Class) [96]	98.8% 64.0	67.9%	0.219
	CAE-T5	75.0% 5.2	70.0% 0.466

Table 3 .

 3 4: Automatic evaluation scores of different models trained and evaluated on the processed Civil Comments dataset. The scores are computed on the toxic test set. "Human" corresponds to 427 human rewritings of randomly sampled toxic comments from the train set. "Random civil" means we randomly sampled 4,878 comments from the civil test set.

partially replicated by CAE-T5. Other approaches such as StyleTransformer (ST) and CrossAlignment (CA) have

Table 3 .

 3 [START_REF] Alshammari | A switching multi-level method for the long tail recommendation problem[END_REF] shows examples of rephrases of toxic comments automatically generated by our system. The top first two examples emphasize the ability of the model to perform fluent control generation conditioned on both the input sentence and the destination attribute. We present more results showing that we can effectively suggest fluent civil rephrases of toxic comments in Table3.8. However we observed more failures for the civil rephrasing task (cf.

Table 3 .

 3

	Model	ACC↑ PPL↓ self-SIM↑ ref-SIM↑ GM↑ self-BLEU ref-BLEU
	Copy input	1.3% 11.1	100.0%	80.2% 0.105	100.0	32.5
	Human references	79.4% 14.0	80.2%	100.0% 0.357	32.7	100.0
	CrossAlignment [466]	73.5% 54.4	61.0%	59.0% 0.202	21.5	9.6
	[289]
	RetrieveOnly	99.9% 4.9	47.1%	48.0% 0.213	2.7	1.8
	TemplateBased	84.1% 46.0	76.0%	68.2% 0.240	57.0	23.2
	DeleteOnly	85.2% 48.7	72.6%	67.7% 0.233	33.9	15.2
	D&R	89.8% 35.8	72.0%	67.6% 0.262	36.9	16.9
	[136]
	StyleEmbedding	8.1% 29.8	83.9%	69.8% 0.132	67.5	21.9
	MultiDecoder	47.2% 74.2	67.7%	61.4% 0.163	40.4	15.2
	DualRL [320]	88.1% 20.5	83.6%	77.2% 0.330	58.7	29.0
	[96]
	StyleTransformer (Conditional)	91.7% 44.8	80.3%	74.2% 0.254	53.2	25.6
	StyleTransformer (Multi-Class)	85.9% 29.1	84.2%	77.1% 0.292	62.8	29.2
	CAE-T5	84.9% 22.9	67.7%	64.4% 0.293	27.3	14.0

[START_REF] Ando | A high-performance semi-supervised learning method for text chunking[END_REF]

than in the sentiment transfer task (cf. Table

3

.10 and Table

3

.11). We identified three natures of failure: Supererogation generation does not stop early enough and produces fluent, transferred related but unnecessary content.

Table 3 .

 3

	Model	Attribute transfer↑ Fluency↑ Content Preservation↑ Success rate↑ Overall↑
	CA	2.98	2.32	1.89	6%	1.81
	IE (BERT)	2.77	2.39	2.20	6%	1.89
	ST (Cond)	2.91	2.36	2.08	5%	1.87
	ST (M-C)	2.93	2.42	2.10	5%	1.93
	CAE-T5	2.72	3.06	2.63	13%	2.52

5: Automatic evaluation scores of different models trained and evaluated on the Yelp dataset. Accuracy is computed by a BERT classifier fine-tuned on the Yelp train set (accurate at 98.7% on the test set). Perplexity is measured by a GPT-2 LM fine-tuned on the Yelp train set. "self-" refers to a comparison to the input and "ref-" to a human reference.

Table 3 .

 3 6: Human evaluation of different models trained and evaluated on the Civil Comments dataset.

	INPUT

Table 3 .

 3 8: Examples of automatically transferred toxic test sentences by our system and valid rewriting.

	Toxic to Civil

repeating the same unproven, ill- informed propaganda that you're

  portions are very generous and food is fantastically flavorful . DualRL portions are very thin and food is confusing . ST (Multi) portions are very poorly and food is springs flavorless .

	Input	staff : very cute and friendly .
	DualRL	staff : very awful and rude .
	ST (Multi) staff : very nightmare and poor .
	CAE-T5

CAE-T5 portions are very small and food is awfully greasy for the price . Human portions are very small and food is not flavorful . staff : very rude and pushy . Human staff : very ugly and mean .

  the store is dumpy looking and management needs to change . DualRL the store is perfect looking and management speaks to change perfectly . ST (Multi) the store is dumpy looking and management moved to change .

	Input	i emailed to let them know but they apparently dont care .
	DualRL	i loved them know them know but they dont care .
	ST (Multi) i emailed to let them know but they honestly played their .
	CAE-T5
	Input	do not go here if you are interested in eating good food .
	DualRL	definitely go here if you are interested in eating good food .
	ST (Multi) do not go here if you are interested in eating good food .
	CAE-T5	definitely recommend this place if you are looking for good food	h h h h h h h at a good price .

Ours the store is neatly organized and clean and staff is on top of it . Human managment is top notch , the place looks great . i emailed to let them know and they happily responded right away . a great service Human i emailed to let them know they really do care . Input this place is dirty and run down and the service stinks ! DualRL this place is clean and run perfect and the service helped ! ST (Multi) this place is quick and run down and the service stunning ! CAE-T5 this place is clean and well maintained and the service is great ! ! ! Human this place is clean , not run down , and the service was great . Human do not go here if you are interested in eating bad food .

Table 3 .

 3

13: Pearson correlation coefficients between quantitative human judgments and reference-based metric scores. #Ref indicates whether all or a single human reference was considered by the metric. WMT: WMT-2019. WNLG: WebNLG-eval. ASV: Asset-eval. MUSS: MUSS-eval. PSCL: PASCAL50S. FLCKR: Flickr8k. RLS: REAL-Summ. SE: SummEval. EQA: Efficient QA. VQA: OKVQA-Eval. AM: Arithmetical mean.

  , Survival of the fittest would not have produced you. You are alive because your weak blood is supported by welfare and food stamps. Please don't reference Darwin in your icon. Loser.

	Gold Spans (set of character offsets)	Post
	{55, 56, 57, 58, 59, 60}	What if his opinion is that most other commenters are idiots? :-)
	{80, 81, 82, 83, 84, 85, 86, 87, 88, 89,
	176, 177, 178, 179, 180}
	{55, 56, 57, 58}	Time to give the opposition weapons and overthrow this jerk.
	{31, 32, 33, 34, 35, 36, 37, 38, 39}	you are a paid liberal hack of stupidity shore living the liberal lie
		every day
	{9, 10, 11, 12, 13}	Youre an idiot.
	{14, 15, 16, 17, 18, 19, 20, 29, 30, 31,
	32, 33, 34, 35}

{7, 8, 9, 10, 11, 14, 15, 16, 17, 18} what a dummy, dummy has no regard for the law. you cannot brutalize a suspect. he has complete lack of respect for any law and is acting like a dictator. he is trying to emulate putin. {12, 13, 14, 15, 16, 17, 94, 95, 96, 97,

98, 99, 100, 101, 102}

People make stupid decisions and then expect the gov't to bail them out. There is no cure for stupidity.

Table 4 .

 4 2: TOXICSPANS statistics. Lengths in characters.

		Mean Min Max
	Post length	208.14	4	1,000
	Dense toxic span length	7.01	3	87
	# Dense toxic spans	0.58	0	8

4.

6 Methods for toxic spans detection 4.6.1 Simplistic baselines TRAIN

  -MATCH, is a simple lookup-based model that classifies as toxic any tokens encountered inside toxic spans of the training data. HATE-MATCH operates similarly, but the lookup is within the hateful/offensive spans of the data of Mathew et al.[START_REF] Mathew | Hatexplain: A benchmark dataset for explainable hate speech detection[END_REF]. A naive baseline, RAND-SEQ, randomly classifies tokens as toxic or not.

Table 4 .

 4 4: F 1 , Precision (P ), Recall (R) of sequence labeling (1st zone), attentional (2nd), and look-up methods (3rd) in toxic spans detection. Average scores of a 5-fold Monte Carlo cross-validation are shown. The standard error of the mean is always lower than a percentage point. The ROC AUC scores of BILSTM and BERT (the attention-based rationale extraction methods) in toxic/non-toxic text classification were 90.9% and 96.1%, respectively. toxic) was fine-tuned on the development set of each Monte Carlo cross-validation fold.

		RAND	7.3	5.3	25.4
	Baselines	TRAIN-MATCH	41.0	39.1	48.7
		HATE-MATCH	10.6	7.1	43.7
		BILSTM-SEQ	58.9	59.8	58.9
	Strong supervision	CNN-SEQ BERT-SEQ	59.3 59.7	60.7 60.7	59.0 60.0
		SPAN-BERT-SEQ	63.0	63.8	62.8
	Weak supervision	BILSTM+ARE BERT+ARE	57.7 49.1	58.4 49.4	57.3 49.5

Table 4 .

 4 6: Examples of posts where SPAN-BERT-SEQ predicted some, but not all of the gold spans. Ground truth in red. Predictions of SPAN-BERT-SEQ in bold.

Table 4 .

 4

7:

Examples of posts where the ground truth was empty, but SPAN-BERT-SEQ incorrectly predicted at least one span. Predictions of SPAN-BERT-SEQ in bold.

  is the self-supervised Conditional Auto-Encoder presented in Chapter 3. It had

		Parallel (P)	Non-Parallel (NP)
		Toxic-to-Civil pairs	Toxic	Civil
	Train	2,222	90,293	5,653,785
	Dev	278	4,825	308,130
	Test	278	4,878	305,267
	Average length	19.8 a	19.4	21.9

Table 5 .

 5 The test and validation sets were respectively made of all users' 2: Statistics for the 2014 Amazon Movie Review dataset and the Yelp Review dataset. Following Yang et al.[START_REF] Yang | Multilingual universal sentence encoder for semantic retrieval[END_REF], sentences are tokenized with SentencePiece[START_REF] Kudo | SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing[END_REF]. last and penultimate interactions; the training set consisted of the remaining interactions. In the next section, we describe how a Universal Sentence Encoder can be used to build representations of sentences found in reviews.

	Dataset	Amazon 20-core Yelp 20-core
	Users	3,728	38,595
	Items	3,911	27,823
	Train reviews	205,158	1,929,332
	Validation reviews	3,728	38,595
	Test reviews	3,728	38,595
	Average sentence # per review	14.1	9.8
	Average token # per sentence	46.2	30.5

Table 5 .

 5 3. Neighborhood-basedCF clearly indicates that u 2 behaves like u 0 and the ground truth tells that u 2 prefers i 1 over i 0 .

	User	Item	i0	i1	i2 i3
	u0		2	4	1	5
	u1		5	1	5	1
	u2		2.5 3.5 1	5
	u3 . . . un	5	5	5	5

Table 5 .

 5 3: Toy dataset of ratings. Bold values are in the test set, and the remaining values are in the train set.

Table 5 .

 5

		TRAIN ITEM i	COMMONALITY	MATCHING SENTENCES IN THE TRAIN SET
		CO-REVIEWED	SHARED BY BOTH i
		BY u AND v	AND j
				You simply must check out this awesome super-
	Justice League: The Flashpoint Paradox	Green Lantern: First Flight	Animated superhero film	hero epic . Personally , it's the finest superhero animated ef-fort ever brought to screen .
	Fort Apache	The Searchers	Western film directed by John Ford and starring John Wayne	That aside , and at the risk of repeating myself it's vintage John Ford and John Wayne with some magnificent scenery . This is one of John Ford's and one of John Wayne's best movies .
				This is a movie that I ' d suggest any family to
	Peter Pan	Monsters, Inc.	Animated film	pop in and have a family movie night .
				a great family movie .
				This movie is Bloody , gory , violent , emotional ,
	Killing Them Softly	Django Unchained	Violence & Gore	serious and hilarious . All in all . . . . a really good movie , but a bit
				bloody .
				It is almost a must see for historians and those
	1776	Downfall	Historical drama film	with an interest in history . Should be required viewing for high school his-
				tory students .
				It can be immensely dark and scary at times -
			Dark drama film	even for adults !
	Black Swan	Coraline	with frightening &	The story and the situations became spooky . .
			intense scenes	. ( button eyes ) , and may be a little intense for
				younger children .
	Shallow Hal	Bridget Jones's Diary	Romantic comedy	It was wonderfully funny and romantic . This is a totally winning romantic comedy !
				Great acting by all parties : Monroe , Cotten ,
			'50s film noir	Peters et .
	Clash by Night	Niagara	starring Marilyn	Monroe aside , this movie is definitely a Joseph
			Monroe	Cotten vehicle and may be his finest acting job
				from a long career .
				Sylvester Stallone delivered a movie with great
				screenplay , great characters , a great plot and
	Rocky III	Rocky II	Installment in Rocky franchise	all together a wonderful movie everyone will re-member . Stallone is good in all his roles , the playing of
				Rocky as a working class bum made good is
				bang on .

The second and third examples are short sentences with almost identical head and tail sentences. These matches convey limited u'S TEST ITEM j 5: Examples of semantic matches when our system considers per-item graphs. The last column shows the head sentence first (written by u) and the tail sentence then (written by v). Additional examples from the same setup are shown in Table

5

.6.

Table 5 .

 5 6: Examples of semantic matches when our system considers per-item graphs. The last column shows the head sentence first (written by u) and the tail sentence then (written by v). Additional examples from the same setup are shown in Table 5.5. HEAD SENTENCE σ 1 WRITTEN BY u ON SOME ITEM i TAIL SENTENCE σ 2 WRITTEN BY v ON u'S TEST ITEM j AND MATCHING σ 1

	Cemetery Man -This highly entertaining little zombie	The Horde -one of the best horror zombie movies of
	movie from Italy has all the elements that make it a won-	all the times , this movie is equal than 28 days later , in
	derfully dark horror -comedy in the same vein asEvil	these days European horror movies are the best of the
	Dead 2 : Dead by DawnandAn American Werewolf in	best . . good for Friday at night
	London .
	Charlie's Angels -I like drew barrymore , she is the
	best angel out of the three .

Table 5 .

 5 8: Examples of semantic matches when our system considers the global graph. Sentences are preceded by the item they review. v represents u's nearest user neighbor. Here, tail sentences review j, though tail sentences can just as well review other items and be counted as semantic matches. as in Table5.7. Spoiler alert / / Our hero , of course , is a loner --which is a sure tip -off that anyone he sort of becomes close to is going to get killed .

	ITEM	SEMANTIC SIMILARITY	MATCHING SENTENCES IN THE TRAIN SET
		/ /
	Spartan	0.69

  This movie is a very funny movie and Madea is trying to be a private Detective and she finally gets her man This Madea was the first time in the series to be released in the summer and did pretty well as the second highest grossing Madea film ever behind Madea Goes to Jail and this one actually crossed over with other audiences and other demographics besides his usual african -american audience and this one was the funniest film of the series and Tyler Perry's best film yet .

	Shopgirl	0.66	I have a feeling there is probably more of her left on the cutting room floor .
			It's no surprise that she's on medication for depression .
	Madea's
	Witness	0.87
	Protection
			' Radio ' is a wonderful film that the whole family can enjoy and
			learn from together !
	Radio	0.82	Radio is good , touching and very sad at times but it has a lot to
			learn about standing on its own as a family classic .
			Highly recommended for suspense fans .
	The Gift	0.75

  TableA.1: Official rank and F 1 score (%) of the 36 participating teams that submitted system description papers. (There were 91 teams with submissions in total.) The median is shown in blue. duplicate submissions and bypass any submission limits. More importantly, potentially interesting approaches are not discussed and properly compared to others.

	RANK	TEAM	F 1 SCORE (%)
	1	HITSZ-HLT	70.83
	2	S-NLP	70.77
	3	hitmi&t	69.85
	5	YNU-HPCC	69.63
	7	Cisco	69.22
	8	MedAI	69.03
	9	IITKDetox	68.95
	13	GHOST	68.59
	14	HLE-UPC	68.54
	15	UTNLP	68.44
	16	YoungSheldon	68.42
	17	Lone Pine	68.38
	18	sk	68.32
	20	WLV-RIT	68.01
	21	CSECUDSG	67.95
	22	LISAC FSDM USMBA	67.84
	23	UoT-UWF-PartAI	67.70
	25	uob	67.61
	MEDIAN	The median score	67.58
	26	UAntwerp	67.55
	27	MIPT-NSU-UTMN	67.55
	28	NLRG	67.53
	30	HamiltonDinggg	67.15
	33	lz1904	67.00
	34	UIT-E10dot3	66.99
	36	UniParma	66.72
	37	hub	66.40
	38	GoldenWindPlymouth	66.37
	41	AStarTwice	66.16
	44	sefamerve_arge	66.01
	46	UPB	65.73
	49	Entity	65.61
	57	BennettNLP (Fuchsia)	64.53
	58	TeamGriek	64.31
	63	UIT-ISE-NLP	62.23
	75	NLP_UIowa	50.09
	90	macech	7.33

Table A .

 A 2. TableA.2 shows the F 1 scores of all the rationale-based systems that were reported by participants. The binary toxic post classifiers that were 2: F 1 on the evaluation set for systems employing rationale extraction (RE) mechanisms combined with post-level toxicity binary classifiers (TBC). Rationales are obtained via Input Erasure (IE), Attention (ATT), LIME, or SHAP. The binary classifier is an LSTM, Logistic Regression (LR), SVM, or BERT.

	TBC	RE	F 1 SCORE (%)	Report
	LSTM	IE	38.29	Rusert [434]
	LSTM	ATT	49.70	Pluci ński and Klimczak [392]
	LSTM	ATT	50.07	Rusert [434]
	LR	LIME	58.88	Benlahbib et al. [31]
	SVM	LIME	59.21	Benlahbib et al. [31]
	BERT	SHAP	59.87	Pluci ński and Klimczak [392]
	Lexicon Name F 1 SCORE (%)	Report
	WIEGAND 1 †	33.07	Zhu et al. [595]
	WORD-MATCH	40.86	Ranasinghe et al. [413]
	FREQ-RATIO †	41.55	Rusert [434]
	LOOKUP ‡		41.61	Burtenshaw and Kestemont [57]
	WIEGAND 2 †	50.98	Zhu et al. [595]
	ORTHRUS		61.07	Palomino et al. [369]
	HITSZ-HLT ‡	64.98	Zhu et al. [595]
	+WORDNET	64.09	Zhu et al. [595]
	+GLOVE		64.19	Zhu et al. [595]

Table A .

 A 3: F 1 on the evaluation set for lexicon-based systems. Systems that are followed by † and ‡ use exclusively external and internal resources, respectively.

A.k.a. quae pro quibus in Latin languages.

This phenomenon is not exclusive to linguistics, as similar needs have existed in music transcription for instance.

https://gluebenchmark.com/

https://super.gluebenchmark.com/

https://rajpurkar.github.io/SQuAD-explorer/

Semi-supervised learning is also employed in the literature but often denotes a subcategory of weak supervision introduced later: incomplete supervision.

https://appen.com/

https://www.mturk.com/

Term referring to American football and meaning "someone who criticizes from hindsight."[START_REF]Monday-morning quarterback -wiktionary, the free dictionary[END_REF]

Term referring the American Civil War and meaning "one who comes to a place or organization with which they have no previous connection with the sole or primary aim of personal gain, especially political or financial gain."[START_REF]Wiktionary. carpetbagger -wiktionary, the free dictionary[END_REF]

https://twitter.com/yoavgo/status/1489364641000181774

Note that strong supervision and self-supervision are equivalent from the point of view of UNLG.

https://en.wikipedia.org/

https://commoncrawl.org/

https://twitter.com/

https://www.reddit.com/

Compared to the pretraining dataset size

Latin for "know thyself"

n-grams are particular skip-grams, seen in Section 2.5.3

On the nature versus nurture debate, see Coll et al.[START_REF] Coll | Nature and nurture: The complex interplay of genetic and environmental influences on human behavior and development[END_REF], Goldhaber[START_REF] Goldhaber | The nature-nurture debates: Bridging the gap[END_REF] and Keller[START_REF] Keller | The mirage of a space between nature and nurture[END_REF]

Quote attributed to Voltaire

Sometimes called "BERTology" when investigating modern systems

The Invisible Gorilla Test[START_REF] Chabris | The invisible gorilla: And other ways our intuitions deceive us[END_REF] is a well-known study on intentional blindness

https://www.tensorflow.org/datasets/catalog/civil_comments

https://www.nltk.org/api/nltk.tokenize.html

https://github.com/lijuncen/Sentiment-and-Style-Transfer/tree/master/data/yelp

The geometric mean is not sensitive to the scale of the individual metrics.

https://appen.com

gs://t5-data/vocabs/cc_all.32000/sentencepiece.model

https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/sacrebleu.py

https://tfhub.dev/google/universal-sentence-encoder/2

Toxic-to-civil style transfer could be seen as equivalent to "reducing toxicity", but it cannot be measured with character-based metrics such as the length or the Levenshtein distance[START_REF] Levenshtein | Binary codes capable of correcting deletions, insertions and reversals[END_REF], used in unsupervised simplification.

See Section 2.5.9 for details on recent promises offered by prefix prompts in few-shot learning

Similarly, when we needed a scalable nearest neighbor searcher on sentence embeddings (cf. Chapter 5), we opted for Google's ScaNN[START_REF] Guo | Accelerating large-scale inference with anisotropic vector quantization[END_REF] while MUSS used Meta's Faiss[START_REF] Johnson | Billion-scale similarity search with gpus[END_REF].

www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification

https://appen.com/

We used the probabilistic ground truth for training and mean squared error as the loss function of BILSTM-SEQ, which yielded best results in preliminary experiments.

Similar attention-based rationale-extraction methods have been used, e.g., by DeYoung et al.[START_REF] Deyoung | ERASER: A benchmark to evaluate rationalized NLP models[END_REF] and Jain et al.[START_REF] Jain | Learning to faithfully rationalize by construction[END_REF], but not in toxicity detection. See also Wiegreffe and Pinter[START_REF] Wiegreffe | Attention is not not explanation[END_REF], Kobayashi et al.[START_REF] Kobayashi | Attention is not only a weight: Analyzing transformers with vector norms[END_REF], Ferrando and Costa-jussà[START_REF] Ferrando | Attention weights in transformer NMT fail aligning words between sequences but largely explain model predictions[END_REF] for a broader discussion of attention as an explainability mechanism.

https://keras.io/

https://huggingface.co/transformers/

The most frequent spans were "sh*t", "st*p*d", "f*ck".

shorturl.at/hqEJ3

Our code is available at https://bit.ly/3NlL6Qa

WordNet was employed[START_REF] Miller | WordNet: An electronic lexical database[END_REF].

https://www.facebook.com/, https://twitter.com/

https://www.amazon.com/, https://www.ebay.com/

https://www.netflix.com/

https://www.imdb.com/

USE's embedding are normalized on the unit sphere.

Relative to error-based metrics and rating prediction tasks.

https://beam.apache.org/

https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.kneighbors_graph.html

Quote from Star Wars: Episode I -The Phantom Menace

lógos

https://github.com/unitaryai/detoxify

In the latter case, in-vocabulary word embeddings were imported to Word2vec for efficiency, and out of vocabulary words were handled with BPEs[START_REF] Sennrich | Neural machine translation of rare words with subword units[END_REF].

https://github.com/Orthrus-Lexicon/Toxic

We asked for details from participants that did not submit description paper, but not all of them replied.

Acknowledgements

The classification threshold was set to 0.5. CNN-SEQ was trained for 30 epochs; we used 0.5 recurrent dropout;

progressively increasing batch size from 4 to 32 with step 1. All the other hyper-parameters were set to their default values. BERT-SEQ was implemented using the Huggingface Transformers library. 6 We used the BERT-base-cased model, binary cross entropy loss; the Adam optimiser; learning rate 2 • 10 -5 ; maxlen 128; batch size 32; training for max. 100 epochs; early stopping with 5 epoch patience, monitoring validation loss. The classification threshold was 0.5.

SPAN-BERT base (cased) was fine-tuned in the same way that Joshi et al. [START_REF] Joshi | SpanBERT: Improving pre-training by representing and predicting spans[END_REF] fine-tuned it on SQUAD 2.0 [START_REF] Rajpurkar | Know what you don't know: Unanswerable questions for SQuAD[END_REF] with the format mapping presented in of ACC3 that requires the posts to also be judged non-toxic by the (BERT-based) toxicity classifier.

Table 4.11 shows that restricting ACC to consider only posts with at least one toxic post (ACC2) substantially improved the performance of both models on the NP dataset, indicating that it contains many 'mission impossible' instances (posts with no toxic spans) that the original ACC considers. By contrast, switching from ACC to ACC2 led to mostly negligible changes on the P and TOXICSPANS datasets, which is in accordance with the fact that they contain fewer posts with no toxic spans (11.5% and 48.7%, respectively, compared to 67.4% for NP). Another interesting observation is that ACC4 was always substantially lower than ACC3 (for both systems, on all three datasets), indicating that the models often successfully detected toxic spans and try to rephrase them, but the rephrases were still toxic, at least according to the toxicity classifier.

Human-detoxified posts

In this experiment, we wished to study the extent to which humans rephrase known toxic spans when asked to produce civil rephrases of toxic posts. We used the P dataset, the only one of the three considered that contains human rephrases. 7 Since P does not contain gold toxic spans, we again employed SPAN-BERT-SEQ to add toxic 7 We used all the P data since no training was involved.

(that provide post-level annotations only). Hence, our TOXICSPANS dataset is accessible with the following versions:

First, only toxic posts are included (11, 006 posts), which is the version we discuss in this work. Second, the previous version will be augmented with the same number of randomly selected non-toxic Civil Comments posts. Third, a version similar to the previous one, but where the ratio of toxic to non-toxic posts will be 1 : 40 to be closer to that of real-world datasets (325, 499 posts).

As shown in Section 4.8, the TOXICSPANS dataset and toxic span detectors can also help study and evaluate explicit toxicity removal when rephrasing toxic posts to be civil. In this case, toxic spans can be used to get a better understanding of how toxic-to-civil models operate by showing the toxic spans and their context, along with their rephrases.

For more details on the shared task we organized at SemEval-2021, please see the competition description in Appendix A.

Intended use and misuse potential

The toxic span detection systems we considered are trained (the sequence-labeling ones) and tested (all systems) on posts with binary ground-truth character offset labels (toxic or not), reflecting the majority opinion of the annotators (Section 4.4). This runs the risk of ignoring the opinions of minorities, who may also be minorities among crowd-annotators. To address this issue, we also released the toxic spans of all the annotators and the pseudonymous rater identities, not just the spans that reflect the majority opinion, to allow different label binarisation strategies and further studies.

Toxic span detection systems are intended to assist the decision-making of moderators, not to replace moderators. When they operate correctly, systems of this kind are expected to ease decision-making (reject/accept a post). Incorrect results could be of two types; toxic spans that were not highlighted and non-toxic spans that were highlighted. Mistakes of both types, especially the first one, may mislead a moderator working under time pressure.

As with other content filtering systems (e.g., spam filters, phishing detectors), toxic span detectors may trigger an adversarial reaction of malicious users, who may study which types of toxic expressions evade the detectors (esp.

publicly available ones) and may gradually start using more implicit toxic language (e.g., irony, false claims), which may be more difficult to detect. However, this is a danger that concerns any toxicity detection system, including systems that classify user content at the post level (without detecting toxic spans).

Subsequent related works

Following our work on English toxic spans, Ravikiran et al. [START_REF] Ravikiran | Findings of the shared task on offensive span identification fromCode-mixed Tamil-English comments[END_REF] published a similar approach adapted to Code-Mixed Tamil-English comments. We hope our project initiated a larger trend in multilingual toxic span detection as non-English-only platforms may benefit from automated online moderation at the span level as well.

Chapter 4 conclusion

We studied toxicity detection, which aims to identify the spans of a user post that make it toxic. Our work is the first of this kind in general toxicity detection. We constructed and released a dataset for the new task, along with baselines and models. Fine-tuning the SPAN-BERT sequence labelling model of Joshi et al. [START_REF] Joshi | SpanBERT: Improving pre-training by representing and predicting spans[END_REF],

yielded the best results. A post-level BILSTM toxicity classifier that was combined with an attention-based attribution method, not trained on annotations at the span level, performed well for the task. By leveraging the dataset of posts annotated as toxic or non-toxic (without spans), we showed that this method can reach the performance of a BILSTM sequence labeling approach that was trained on the more costly toxic spans annotations. This result is particularly interesting for future work aiming to perform toxic spans detection by using only datasets with whole-post toxicity annotations. In a final experiment, we examined toxic-to- and not reviewed yet.

Chapter 5 abstract

This chapter explores a novel use of review text to represent user-preferences for rating prediction. The approach leveraged textual semantic similarity models to represent a user's preferences as a graph of textual snippets, where the edges were defined by semantic similarity. This textual, memory-based approach to rating prediction offers the promise of improved explanations for recommendations. The method was evaluated quantitatively, highlighting that leveraging text in this way can out-perform both memory-based and model-based collaborative filtering baselines.

Introduction

With the democratization of online shopping and content delivery platforms, internet users are accustomed to writing textual reviews of products and services. While recommender systems initially relied only on quantitative preference scores, newer datasets with textual reviews, combined with advances in Natural Language Understanding, suggest that both the performance of recommenders and their explainability might benefit from leveraging textual representations.

While item recommendation systems aim to predict the next interaction corresponding to the users' preferences,

ATTRIBUTE OF j MATCHING SENTENCES IN THE TRAIN SET

Hancock Action film

To Live -Usually I ' m bouncing off the walls and watching a crazy action flick .

Gladiator -Sometimes I just fast forward straight to the epic battle scenes .

Non-Stop Action thriller film

The Debt -Plot is very slow developing , so slow and uninteresting , did not even finish watching .

Whiteout -Plot was so S-L-O-W and dull .

Dune Action-adventure science-ficton film

Oblivion -Good action and story line .

Serenity -great action and story line .

Evil Dead Horror film

Suspiria -A legend in the horror movie genre .

Prince of Darkness -A horror movie classic .

Rocky II

Written by and starring Sylvester Stallone

Rambo III -His range may be limited , as we saw in his ' comedy ' films , but when he sticks to his forte , playing great heroes , Stallone is the greatest ever .

Rocky -But Stallone gives just about the best performance of his career here .

The Living Daylights

Entry in the James Bond (a.k.a. 007 ) series Never Say Never Again -" Never Again " ultimately retains a very watchable magic featuring the original Agent 007 one last time .

Goldfinger -After the first two 007 films , this third Bond adventure cemented forever the style and fun of the series .

My Neighbor Totoro Animated film

Popeye -Hey parents , want a good , clean , wholesome movie for your kids ?

Kiki's Delivery Service -An excellent movie for kids that parents don ' t have to worry about .

Table 5.7: Examples of semantic matches when our system considers the global graph. The last column shows the head sentence first (written by u) and the tail sentence then (written by v), along with the respective item's review they belong to. Here, tail sentences do not review j but express attributes also present in j and indicated in the second column. The Toxic Spans Detection task of SemEval-2021 required participants to predict the spans of toxic posts that were responsible for the toxic label of the posts, when detecting such spans is possible 1 . Systems had to extract a list of toxic spans, or an empty list, per post. The task could be addressed as supervised sequence labeling, using training data with gold toxic spans provided by us. It could also be treated as rationale extraction, using classifiers trained on potentially larger external datasets of posts manually annotated as toxic or not, without toxic span annotations.

Participants submitted their predicted spans for a held-out test set, and were scored using character-based F 1 (cf.

Section 4.5).

The evaluation period started on January 10, 2021, and finished on January 31, 2021. In the first week, 10 submissions were allowed per day per team. In the second week, this number was reduced to 5, and further to 1 during the final week. We chose to allow an extended evaluation period combined with multiple team submissions to promote the competition. However, we also dicided on a decreasing submission limit to make it harder for participants to overfit the test set. As shown in 1 Although we defined the task at the word level, gold labels were provided at the character level, counting from zero.

Type Description

INCONSISTENCIES Not all the occurrences of the same toxic span are annotated in the same post. FALSE NEGATIVES Toxic words missed. FALSE POSITIVES Non-toxic words labeled.

Table A.4: The types and descriptions of the annotation mistakes that were detected by some of the participants.

spans (e.g., "piece of crap'), leading to a decline in performance as measured by the chosen F 1 measure. The overwhelming presence of single-word gold spans in the training set favors short spans. But the majority of the short spans comprise common cuss, or clearly abusive words, which can be directly classified as toxic [START_REF] Ghosh | Cisco at SemEval-2021 Task 5: What's toxic?: Leveraging transformers for multiple toxic span extraction from online comments[END_REF]; by contrast, the infrequent longer spans are rather context-dependent and more challenging to detect. This probably also contributed to the performance of the best system (HITSZ-HLT), since one of the two components of that ensemble handled better long spans, as already discussed in Section A.4.3.

Annotation mistakes reported are summarized in Table A.4.

Participants that were notable for their effort in error analysis include Bansal et al. [START_REF] Bansal | IITK@Detox at SemEval-2021 Task 5: Semi-supervised learning and dice loss for toxic spans detection[END_REF], Hoang and Nguyen [START_REF] Hoang | UIT-E10dot3 at SemEval 2021 Task 5: Toxic spans detection with roberta and spacy's library base systems[END_REF],

Ding and Jurgens [START_REF] Ding | HamiltonDinggg at SemEval-2021 Task 5: Investigating toxic span detection using RoBERTa pre-training[END_REF], and Ghosh and Kumar [START_REF] Ghosh | Cisco at SemEval-2021 Task 5: What's toxic?: Leveraging transformers for multiple toxic span extraction from online comments[END_REF], where an additional effort was made to examine their model's ability to correctly tag words in toxic and non-toxic contexts. Interestingly Sans and Farràs [START_REF] Sans | HLE-UPC at SemEval-2021 Task 5: Multi-Depth DistilBERT for toxic spans detection[END_REF] also noted in their analysis that racial and ethnic terms are labeled in biased ways that reflect patterns not only in the training toxic spans but also in external data used to pretrain underlying transformer models.