Michele Corazza
email: michele.corazza2@unibo.itmenini

Stefano Menini

Elena Cabrio

Sara Tonelli
email: satonelli@fbk.euelena.cabrio

Serena Villata
email: serena.villata@univ-cotedazur.fr

Hybrid Emoji-Based Masked Language Models for Zero-Shot Abusive Language Detection

Recent studies have demonstrated the effectiveness of cross-lingual language model pretraining on different NLP tasks, such as natural language inference and machine translation. In our work, we test this approach on social media data, which are particularly challenging to process within this framework, since the limited length of the textual messages and the irregularity of the language make it harder to learn meaningful encodings. More specifically, we propose a hybrid emoji-based Masked Language Model (MLM) to leverage the common information conveyed by emojis across different languages and improve the learned cross-lingual representation of short text messages, with the goal to perform zeroshot abusive language detection. We compare the results obtained with the original MLM to the ones obtained by our method, showing improved performance on German, Italian and Spanish.

Introduction

The extensive use of large-scale self-supervised pretraining has greatly contributed to recent progress in many Natural Language Processing (NLP) tasks [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Liu | Roberta: A robustly optimized BERT pretraining approach[END_REF][START_REF] Conneau | Crosslingual language model pretraining[END_REF]. In this context, masked language modelling objectives represent one of the main novelties of these approaches, where some tokens of an input sequence are randomly masked, and the objective is to predict these masked positions taking the corrupted sequence as input. Still, little attention has been devoted to the adaptation of these techniques to tasks dealing with social media data, probably because they are characterized by a very domain-specific language, with high variability and instability. Nevertheless, all these challenges make social media data an interesting testbed for novel deep-learning architectures, around the research question: how could the masking mechanism be adapted to target social media language?

In this paper, we address the above issue by adapting a novel architecture for cross-lingual models called XLM [START_REF] Conneau | Crosslingual language model pretraining[END_REF] to zero-shot abusive language detection, a task that has gained increasing importance given the recent surge in abusive online behavior and the need to develop reliable and efficient methods to detect it. In particular, we evaluate two methods to pre-train bilingual language models, one similar to the original XLM masked model, and the other based on a novel hybrid emoji-based masked model. We then evaluate them on zero-shot abusive language detection for Italian, German and Spanish, showing that, although our results are below the state-of-the-art in a monolingual setting, the proposed solutions to adapt XLM to social media data are beneficial and can be effectively extended to other languages.

In the following, Section 2 discusses the related work. Section 3 describes our approach to train cross-lingual models for social media data classification, while Section 4 presents the experimental setup. Section 5 reports on the evaluation results, while Section 6 summarizes our findings.

Related work

The focus of this paper is the abusive language detection task, which has been widely explored in the last years thanks to numerous datasets, approaches and shared tasks [START_REF] Waseem | Proceedings of the First Workshop on Abusive Language Online[END_REF]Fišer et al., 2018;[START_REF] Ángel | Overview of MEX-A3T at IberEval 2018: Authorship and Aggressiveness Analysis in Mexican Spanish Tweets[END_REF][START_REF] Wiegand | Overview of the germeval 2018 shared task on the identification of offensive language[END_REF][START_REF] Bosco | Overview of the EVALITA 2018 hate speech detection task[END_REF]Zampieri et al., 2019b;Roberts et al., 2019) covering different languages. An increasing number of approaches has been proposed to detect this kind of messages (for a survey on the task, see [START_REF] Schmidt | A survey on hate speech detection using natural language processing[END_REF] and [START_REF] Fortuna | A survey on automatic detection of hate speech in text[END_REF]).

Abusive language detection is usually framed as a supervised learning problem, built using a combination of manually crafted features such as n-grams [START_REF] Wulczyn | Ex machina: Personal attacks seen at scale[END_REF], syntactic features [START_REF] Nobata | Abusive language detection in online user content[END_REF], and linguistic features [START_REF] Dawei Yin | Detection of harassment on web 2.0[END_REF], to more recent neural networks [START_REF] Ho | One-step and twostep classification for abusive language detection on twitter[END_REF][START_REF] Zhang | Detecting hate speech on twitter using a convolution-gru based deep neural network[END_REF][START_REF] Agrawal | Deep learning for detecting cyberbullying across multiple social media platforms[END_REF][START_REF] Corazza | Comparing different supervised approaches to hate speech detection[END_REF]. [START_REF] Lee | Comparative studies of detecting abusive language on twitter[END_REF] address a comparative study of various learning models on the Hate and Abusive Speech on Twitter dataset [START_REF] Founta | Large scale crowdsourcing and characterization of twitter abusive behavior[END_REF], while (Zampieri et al., 2019a) build the Offensive Language Identification Dataset and experiment with SVMs, BiLSTM and CNN both on the binary abusive language classification and on a more fine-grained categorization. Our work deals with the same task, addressed from a cross-lingual perspective.

In recent years, some proposals have been made to tackle abusive language detection in a cross-lingual framework [START_REF] Sohn | MC-BERT4HATE: hate speech detection using multichannel BERT for different languages and translations[END_REF][START_REF] Wahyu | Cross-domain and cross-lingual abusive language detection: A hybrid approach with deep learning and a multilingual lexicon[END_REF][START_REF] Casula | FBK-DH at SemEval-2020 Task 12: Using Multi-channel BERT for Multilingual Offensive Language Detection[END_REF], with some attempts at zero-shot learning [START_REF] Stappen | Cross-lingual zero-and few-shot hate speech detection utilising frozen transformer language models and AXEL[END_REF]. Most systems, however, rely on pretrained models and do not investigate the potential of indomain data for pretraining. Additionally, as regards masked language models, we are not aware of any work in the literature modifying masking mechanisms for this task.

3 Cross-Lingual Language Models

MLM and HE-MLM training objectives

Our basic architecture relies on the XLM approach described in [START_REF] Conneau | Crosslingual language model pretraining[END_REF], specifically developed to learn joint multilingual representations enabling knowledge transfer across languages. In particular, we borrow from XLM the method developed for unsupervised machine translation, that relies on the Masked Language Model (MLM) objective [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] applied to multiple monolingual datasets as pretraining. We choose to adopt the unsupervised approach because the alternative (i.e., the supervised one based on the Translation Language Modeling) would need to be trained on parallel data, which are not available at scale for social media. As in XLM, we use Byte Pair Encoding (BPE) [START_REF] Sennrich | Neural machine translation of rare words with subword units[END_REF] to learn a shared vocabulary of common subwords between the languages. This technique has proven beneficial to the alignment of embeddings from different languages, when used on languages that share some common traits, such as alphabet and digits.

Following the original approach to MLM, 15% of the tokens in a sentence get selected, which get masked 80% of the times, replaced by a random token in 10% of the cases and kept unchanged 10% of the times. In order to reduce the impact of relatively frequent words on the model, tokens are sampled according to a multinomial distribution that is proportional to the square root of their inverted frequency. While the original XLM operates on streams of text, split by sentence separators, we split the stream of tweets, so that each example contains only one tweet.

Since using a standard pre-trained language model to classify irregular data obtained from social networks would prove very challenging, we try to adapt our cross-lingual model to social media data as much as possible. Specifically, we rely on two main intuitions: emojis are linked to emotion expressions, correlated in turn with various forms of online harassment [START_REF] Arslan | Overwhelmed by Negative Emotions? Maybe You Are Being Cyber-bullied[END_REF]. Besides, emojis could be seen as common traits that are present in tweets across different languages, maintaining a similar meaning at least when comparing Indo-European languages [START_REF] Lu | Learning from the ubiquitous language: An empirical analysis of emoji usage of smartphone users[END_REF]. If we consider the data used in this paper, we can find a good coverage of emojis, with 16.82% of the tweets containing at least one emoji for English, 16.15% for German, 7.68% for Italian, and 18.39% for Spanish. Furthermore, in these datasets the most frequent emojis are shared among all the four languages, with 'red heart', 'face with tears of joy', 'thinking face' and 'smiling face with heart-eyes' among the top ten emojis in each dataset. We therefore compare a standard masked language model with one that targets emoji prediction instead of the cloze task [START_REF] Taylor | Cloze Procedure: A New Tool for Measuring Readability[END_REF]. However, since emojis are not always present in each tweet, we adopt a hybrid approach: when emojis are not present, the previously described MLM objective is trained. When emojis are found, we select them as candidates to be masked 80% of the time, replaced by a random token 10% of the time or kept unchanged 10% of the time as in MLM. With this technique, which we call Hybrid Emoji-based Masked Language Model (HE-MLM) we can use all the available data, while also leveraging the common information conveyed by emojis.

We test also a variant of MLM and HE-MLM, in which we put special tokens "<emoji>" and "</emoji>" around all emojis in the dataset, given that we are effectively performing two different tasks with the same model. This approach allows the model to distinguish between normal words and emojis in the text while training masked language models.

Fine-tuning for abusive language detection

In order to assess how invariant our tweet embeddings are with respect to the language provided as input to the encoder, we create a zero-shot framework, where the system is only trained on English tweets and is evaluated on multiple languages. In particular, we first load the pretrained transformer and attach to it a single feed-forward layer on top of the encoder with a single, sigmoid activated output neuron. The entire model is then fine-tuned on the English hate speech detection dataset using a binary cross-entropy loss function. The system uses early stopping with the minimum F1 score between the two classes as a stopping criterion, relying on a balanced dataset that contains all languages as validation set. Finally, the performance is evaluated on the German, Italian and Spanish test sets to assess how our classifier performs on the different languages using the bilingual models.

4 Experimental setting

Datasets

Since we run our classification in a zero-shot scenario, we use English data for training, and tweets in German, Italian and Spanish for validation and test. The datasets we used and the related number of tweets are reported in Table 1. To guarantee a comparable setting for our experiments, we carefully investigated data samples and the annotation schemes adopted for the different languages, concluding that the tweet content as well as the binary annotation tagsets (hate-speech/offensive and other) of the datasets are similar enough to use them in the same classification framework. Also the class distribution is similar, with the abusive class covering around 30% of the tweets in each dataset.

To pre-train our cross-lingual language models with in-domain data, we gather 5 million tweets for each of the targeted languages (i.e., English, German, Italian and Spanish). ish). For each classification language, the validation set comprises the same amount of language-specific and English tweets.

Streaming API using the stopwords of the target language as filter to query the API, as in [START_REF] Scheffler | A german twitter snapshot[END_REF].

Data splitting

Concerning the dataset splits into training and test instances, for the English dataset -since no standardized split is provided -we randomly selected 60% of the dataset for training, 20% for validation and 20% for testing. For the German and Italian datasets, we use the training and test split provided by the Germeval and Evalita task organisers, respectively. In both cases, we use 20% of the training set as validation set. Whenever we split the datasets, we use the train test split function from scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF], using 42 as a seed value. Finally, for Spanish, we use the development, test and training set provided by the HatEval task organisers.

For each combination of languages tested in our experiments (i.e., English-German, English-Italian and English-Spanish), the validation test is obtained by keeping the language-specific validation set as is and undersampling the English one to the same size, so that each language has the same weight during the early stopping phase.

Before classification, the text is first lowercased, all accents are removed, then it is tokenized with [START_REF] Koehn | Moses: Open source toolkit for statistical machine translation[END_REF] fastBPE implementation1 . We evaluate the classifier performance over a maximum of 100 training epochs, and use an early stopping mechanism with a patience of 5. The selected model is then used to evaluate performance on the test set.

Pretraining methods

Since we want to assess the impact of emojis on the pretraining results, we train four different configurations:

• Using the base MLM training objective;

• Using the base MLM training objective and <emoji> tokens;

• Using the HE-MLM training objective;

• Using the HE-MLM training objective and <emoji> tokens.

For each configuration, we pretrain two models in order to reduce the impact of random initialization on the final results and we fine-tune each model 10 times (20 total). The final results are obtained by averaging the results of these 20 runs.

Evaluation

We report the experiment results for each language in Tables 2, 3, 4. For all languages, training is performed using only English data.

Results for German (Table 2) show that using in-domain unlabeled data from Twitter instead of pre-trained models yields an improvement in performance on English, while on German the model is not able to outperform the pre-trained model. In this case, however, the pretrained model is only learning the non-hate class, while the other three models all achieve non zero recall on both classes. Beside the baseline, the HE-MLM model with <emoji> is the best performing one on the German data, while on English the best performance is achieved by using the vanilla MLM model.

We evaluate MLM and HE-MLM also for zeroshot Italian hate speech classification, comparing the configurations with and without <emoji> tokens like in the previous experiments (Table 3). For English, the best performing model is HE-MLM with emoji tokens, while on Italian the HE-MLM model with no tokens is better in terms of macro averaged F1. When comparing configurations, we observe that the MLM model with emoji tokens has better F1 score than the MLM one in the non hate speech class, while the MLM model has improved performance on the hate class. This results in the MLM model having better macro average F1 for Italian, while the MLM model with emoji tokens shows higher average F1 on English. When considering the hybrid emoji-based models, HE-MLM achieves a higher F1 for the hate speech class in English and for the non hate class in Italian. This results in the HE-MLM model having a higher macro averaged F1 in the Italian language, while the HE-MLM model with emoji tokens is better on English.

As a final test, we evaluate the performance of the model trained on English and Spanish (Table 4). Our English-Spanish models show a similar behaviour to the one observed for the English-Italian pair. In terms of macro averages, the HE-MLM model with emoji tokens has a higher average F1 for English, while the HE-MLM model has higher macro F1 for Spanish.

On all the runs, the classifier achieves a lower performance on German than on the other two languages, while the results on Italian and Spanish are comparable. This confirms the findings in [START_REF] Corazza | A multilingual evaluation for online hate speech detection[END_REF] suggesting that, even when using the same classification framework, experimental setting and amount of training data, offensive speech detection on German achieves lower performance than on other languages. This may have two possible reasons: on the one hand, German may have inherent characteristics that make it more challenging to classify for abusive language detection, for example the presence of compound words makes hashtag splitting more error-prone. On the other hand, the Germeval dataset was built by sampling data from specific users and avoiding keyword-based queries, so to obtain the highest possible variability in the offensive language. This led to the creation of a very challenging dataset, where lexical overlap between training and test data is limited and where hate speech is not associated with specific topics or keywords, as suggested in [START_REF] Wiegand | Detection of Abusive Language: the Problem of Biased Datasets[END_REF].

Conclusions

In this paper, we present a novel zero-shot framework for multilingual abusive language detection. We compare two cross-lingual language models, i.e., standard MLM and a hybrid version of MLM based on emojis (HE-MLM), highlighting that the latter shows some advantages over the MLM model when used on social media data: first of all, when using emojis, the pre-training step is aimed at predicting tokens that are inherently more relevant for the final abusive language detection task whenever possible, as opposed to random tokens. Secondly, emojis convey similar meaning in the languages that we consider, serving as a common trait between languages during pre-training. We also use <emoji> tokens around emojis to help the system discriminate between the two training objectives when using HE-MLM.

The proposed methods represent a novel contribution with respect to social media data processing and abusive language detection. Our aim is not to create a system comparable with monolingual state-of-the-art solutions, but to investigate the possibility to use an unsupervised approach for zeroshot cross lingual abusive language detection. As a first step in this direction, we focused on four European languages, for which similar data were available. The only existing work dealing with zero-shot abusive language detection, presented in [START_REF] Stappen | Cross-lingual zero-and few-shot hate speech detection utilising frozen transformer language models and AXEL[END_REF], only focuses on a language pair and, while obtaining promising results, relies on the English and Spanish corpora annotated for HatEval 2019 following the same guidelines and focusing on hate against immigrants and women. Our approach aims to be more robust, comparing datasets annotated for different shared tasks which may adopt slightly different guidelines.

In the near future, we plan to further extend the social media-specific datasets we are collecting to pre-train HE-MLM, since 5 million tweets we used for each language correspond to a small-sized corpus compared to standard pre-trained language models. Then, to investigate whether our results can be generalised also when dealing with typologically different languages, we will test our approach on additional abusive language datasets covering other languages [START_REF] Ousidhoum | Multilingual and multi-aspect hate speech analysis[END_REF][START_REF] Zampieri | SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (Offen-sEval 2020)[END_REF].

Table 1 :

 1 Such tweets have been collected in different time periods spanning from March to August 2019 through the Twitter Number of tweets used for fine-tuning (English), validation and testing (German, Italian, Span-

	ENGLISH (Waseem and Hovy, 2016)
	Train	Validation	Test
	9,534	-
	GERMAN (Wiegand et al., 2018)
	Train	Validation	Test
	-	1,002 (+ 1,002 EN)	3,532
		ITALIAN (Bosco et al., 2018)
	Train	Validation	Test
	-	600 (+ 600 EN)	1,000
		SPANISH (Basile et al., 2019)
	Train	Validation	Test
	-	500 (+ 500 EN)	1,600

Table 2 :

 2 's system. Finally, Byte Pair Encoding is applied to all datasets by using the Average performance (10 runs) on English and German, comparing the En-De model from[START_REF] Conneau | Crosslingual language model pretraining[END_REF] pre-trained on Wikipedia, our MLM re-trained on English and German tweets, and our Hybrid Emoji-based MLM (HE-MLM). MLM and HE-MLM are evaluated with and without the use of <emoji> tokens.

				pre-trained model		MLM		MLM with <emoji>	HE-MLM		HE-MLM with <emoji>
	Lang. Category	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
		Non-hate speech 0.682 0.993 0.809 0.700 0.736 0.688 0.698 0.387 0.465 0.690 0.830 0.738 0.685 0.907	0.773
	EN	Hate speech	0.423 0.013 0.023 0.340 0.293 0.257 0.320 0.625 0.412 0.304 0.185 0.175 0.248 0.099	0.101
		macro avg	0.553 0.503 0.417 0.520 0.515 0.473 0.509 0.506 0.439 0.497 0.507 0.456 0.466 0.503	0.437
		Non-hate speech 0.660 0.998 0.795 0.626 0.371 0.359 0.477 0.114 0.141 0.575 0.319 0.283 0.656 0.821	0.667
	DE	Hate speech	0.142 0.002 0.005 0.294 0.637 0.379 0.342 0.890 0.487 0.286 0.676 0.375 0.112 0.180	0.109
		macro avg	0.401 0.500 0.400 0.460 0.504 0.369 0.409 0.502 0.314 0.430 0.497 0.329 0.384 0.500	0.388
						MLM		MLM with <emoji>			HE-MLM		HE-MLM with <emoji>
		Lang. Category		P	R	F1	P	R	F1		P	R	F1	P	R	F1
			Non-hate speech 0.473 0.181 0.220 0.699 0.712 0.610 0.449 0.317 0.321 0.616 0.732	0.635
		EN	Hate speech		0.326 0.837 0.458 0.113 0.293 0.162 0.273 0.689 0.374 0.170 0.270	0.179
			macro avg		0.400 0.509 0.339 0.406 0.503 0.386 0.361 0.503 0.347 0.393 0.501	0.407
			Non-hate speech 0.688 0.718 0.679 0.680 0.891 0.765 0.698 0.740 0.713 0.664 0.446	0.452
		IT	Hate speech		0.352 0.301 0.262 0.221 0.122 0.137 0.381 0.326 0.326 0.296 0.587	0.349
			macro avg		0.520 0.510 0.470 0.451 0.507 0.451 0.539 0.533 0.519 0.480 0.517	0.401

Table 3 :

 3 Average performance (10 runs) on English and Italian after re-training the Masked Language Model (MLM) on tweets and using Hybrid Emoji-based MLM (HE-MLM).

Table 4 :

 4 Average performance (10 runs) on English and Spanish after re-training the Masked Language Model (MLM) on tweets and using Hybrid Emoji-based MLM (HE-MLM).

				MLM		MLM with <emoji>		HE-MLM		HE-MLM with <emoji>
	Lang. Category	P	R	F1	P	R	F1	P	R	F1	P	R	F1
		Non-hate speech 0.667 0.927 0.762 0.692 0.847 0.706 0.752 0.308 0.305 0.699 0.722	0.676
	EN	Hate speech	0.072 0.072 0.048 0.118 0.146 0.090 0.316 0.698 0.370 0.296 0.307	0.234
		macro avg	0.369 0.499 0.405 0.405 0.497 0.398 0.534 0.503 0.337 0.498 0.515	0.455
		Non-hate speech 0.599 0.760 0.655 0.577 0.679 0.599 0.598 0.725 0.648 0.595 0.740	0.643
	ES	Hate speech	0.365 0.267 0.275 0.407 0.307 0.298 0.438 0.303 0.332 0.451 0.275	0.280
		macro avg	0.482 0.513 0.465 0.492 0.493 0.449 0.518 0.514 0.490 0.523 0.507	0.461

https://github.com/glample/fastBPE


P Parise

T Garcia

A Helfer

A Miard

M Durif

G Richou

Antou

J B Parise

P Garcia

T Helfer

A Durif

M Richou

G Antou

High temperature creep study of recrystallized Tungsten: 3-point bending experiments and finite element simulation

Introduction / Objectives

Tungsten is a candidate as plasma-facing material for fusion reactors. It will operate at high temperature, under irradiation and under high thermal stresses over long periods of time. These operating conditions will cause it to creep and possibly fail as a result [START_REF] Rieth | Tungsten as a Structural Divertor Material[END_REF] . Understanding its mechanical behavior under such conditions is therefore essential. The ultimate aim of this study is to establish a constitutive law describing the creep behaviour of recrystallized Tungsten, based on a range of mechanical tests which include high temperature 3-point bending in association with FEM. We report here preliminary tests and results of simulations carried out using the Frederick-Armstrong law describing kinematic hardening.

Materials Equipment

Finite Element Simulation

Results

Tungsten block Iter Grade

Made by rolling • Two domains : low deflection rate (20 to 30 µm/min) and high deflection rate (more than 70 µm/min)

•
• At high deflection rates at least, small crosshead speed influence upon material behaviour: similar flow stress for tests between 70 to 200 µm/min. However, the steady-state creep is consistent with the strain rate (i.e. 𝜎 increasing function of ሶ 𝜀)

• Some inconsistencies during the primary creep: load applied is lower at 200µm/min than at 100µm/min

• Strong effect of environment

B. Relaxation test C. Atmosphere influence

Frederick-Armstrong law [START_REF] Lemaitre | Mécanique des matériaux solides[END_REF] viscoplastic constitutive law with kinematic hardening describing primary creep No particular effect on the behaviour licos@Cast3m [START_REF] Helfer | [END_REF] code used for the numerical calculations

Mfront [4] code used for the implementation of the constitutive law

Finite logarithmic strain approximation

  for 3h : batch A 1700°C for 2h : batch B Under Argon + 5% Hydrogen Three-point bending machine equipped with furnace • Tungsten furnace can heat up to 1800°C • Upper column and central point displacement monitored with two LVDTs. Displacement measurements are within a micron. Equipment usually used on ceramics. Many pieces of the device are made of Tungsten which spend many hours at high temperature. 2D and 3D FEM model Constant crosshead speed test: from 20 µm/min ( ሶ 𝜀 = 3,2. 10 -6 𝑠 -1 ) to 200 µm/min ሶ 𝜀 = 3,2. 10 -5 𝑠 -1 Conditions 1500°C & flowing mixture of Argon + 5% Hydrogen Except one test accidentally carried out under a static atmosphere • Hardening phenomenon (load increases with the deflection)

  total flow) Two similar creep tests: • Imposed deflection rate of 30 µm/min • At 1500°C But: • Sudden increase of the oxygen partial pressure during the second test Starting pO 2 = 1.10 -12 atm Ending pO 2 = 2.10 -11 atm

• Tests are modelled using Frederick-Armstrong behaviour law

• Some questions remain before we can establish a proper constitutive law:

1. Two distinct group of tests were observed at low and high deflection rates.

2. Inconsistencies in primary creep of high deflection rates.

• We feel that these issues may only be resolved through characterisations of microstructure (raw, after recrystallization and after deformation). • Dynamic recrystallization effects [5] (*) might explain the behaviour at early stages of the material


Michele Corazza

Stefano Menini

Elena Cabrio

A Multilingual Evaluation for Online Hate Speech Detection

The increasing popularity of social media platforms like Twitter and Facebook has led to a rise in the presence of hate and aggressive speech on these platforms. Despite the number of approaches recently proposed in the Natural Language Processing research area for detecting these forms of abusive language, the issue of identifying hate speech at scale is still an unsolved problem. In this paper, we propose a robust neural architecture which is shown to perform in a satisfactory way across different languages, namely English, Italian and German. We address an extensive analysis of the obtained experimental results over the three languages to gain a better understanding of the contribution of the different components employed in the system, both from the architecture point of view (i.e., Long Short Term Memory, Gated Recurrent Unit, and bidirectional Long Short Term Memory) and from the feature selection point of view (i.e., ngrams, social network specific features, emotion lexica, emojis, word embeddings). To address such in-depth analysis, we use three freely available datasets for hate speech detection on social media on English, Italian and German.

Introduction

The use of social media platforms such as Twitter, Facebook and Instagram has enormously increased the number of online social interactions, connecting billions of users, favouring the exchange of opinions and giving visibility to ideas that would otherwise be ignored by traditional media. However, this has led also to an increase of attacks targeting specific groups of users based on their religion, ethnicity or social status, and individuals often struggle to deal with the consequences of such offenses. This problem affects not only the victims of online abuse, but also stakeholders such as governments and social media platforms. For example, Facebook, Twitter, YouTube and Microsoft have recently signed a code of conduct 1 , 1 http://ec.europa.eu/justice/fundamental-rights/files/hate_speech_code_of_ 1 proposed by the European Union, pledging to review the majority of valid notifications for removal of illegal hate speech in less than 24 hours.

Within the Natural Language Processing (NLP) community, there have been several efforts to deal with the problem of online hate speech detection, since the computational analysis of language can be used to quickly identify offenses and ease the removal of abusive messages. Several workshops [START_REF] Waseem | Proceedings of the first workshop on abusive language online[END_REF][START_REF] Fišer | Proceedings of the 2nd workshop on abusive language online (alw2)[END_REF] and evaluation campaigns [START_REF] Fersini | Overview of the task on automatic misogyny identification at ibereval 2018[END_REF][START_REF] Bosco | Overview of the EVALITA 2018 hate speech detection task[END_REF][START_REF] Wiegand | Overview of the germeval 2018 shared task on the identification of offensive language[END_REF][START_REF] Valerio Basile | SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter[END_REF][START_REF] Zampieri | SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval)[END_REF] have been recently organised to discuss existing approaches to hate speech detection, propose shared tasks and foster the development of benchmarks for system evaluation. These have led to the creation of a number of datasets for hate speech detection in different languages, that have been shared within the NLP research community. Recent advances in deep learning approaches to text classification have then been applied also to deal with this task, achieving for some languages state-of-the-art results [START_REF] Cimino | Multi-task learning in deep neural networks at EVALITA[END_REF][START_REF] Founta | A unified deep learning architecture for abuse detection[END_REF][START_REF] Gambäck | Using convolutional neural networks to classify hate-speech[END_REF]. These systems are usually tailored to deal with social media texts by applying pre-processing, using domain-specific embeddings, adding textual features, etc. Given the number of configurations and external resources that have been used by systems for hate speech detection, it is rather difficult to understand what makes a classifier robust for the task, and to identify recommendations on how to pre-process data, what kind of embeddings should be used, etc. This is indeed the main contribution of the current paper: after identifying a deep learning architecture that is rather stable and well-performing across different languages, we evaluate the endowments of several components that are usually employed in the task, namely the type of embeddings, the use of additional features (text-based or emotion-based), the role of hashtag normalisation and that of emojis. We perform our comparative evaluation on English, Italian and German, focusing on freely available Twitter datasets for hate speech detection. Our goal is to identify a set of recommendations to develop hate speech detection systems, possibly going beyond language-specific differences.

The article is organised as follows: in Section 2 we present past work related to hate speech detection. In Section 3, we describe the neural architecture adopted in our experiments, while in Section 4 we present both the datasets used to train and test our classifier, and the external resources fed to the system. In Section 5, the experimental setup is presented, with details on the pre-processing step and the selection of hyperparameters. Finally, Section 6 reports on the evaluation results and discusses suggestions for the development of robust hate speech detection systems. In Section 7, we summarise our findings. NOTE : This paper contains examples of language which may be offensive to some readers. They do not represent the views of the authors.

Related work 2.1 Hate speech detection on English data

Given the well-acknowledged rise in the presence of toxic and abusive speech on social media platforms like Twitter and Facebook, an increasing number of approaches has been proposed to detect such a kind of messages in English. Automated systems for the detection of abusive language range from supervised machine learning models built using a combination of manually crafted features such as n-grams [START_REF] Wulczyn | Ex machina: Personal attacks seen at scale[END_REF], syntactic features [START_REF] Nobata | Abusive language detection in online user content[END_REF], and linguistic features [START_REF] Hong | Detection of harassment on web 2.0[END_REF], to more recent neural networks that take word or character sequences from comments and learn abusive patterns without the need for explicit feature engineering. The recent trend of using neural network-based approaches has been particularly evident for English, since several training datasets are available for this language, enabling more data-hungry approaches. Indeed, organisers of the 2019 Semeval task on Offensive Language Identification [START_REF] Zampieri | SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval)[END_REF] report that 70% of the participants adopt a deep learning approach. However, also simpler classification systems using logistic regression have been successfully applied to the task [START_REF] Waseem | Hateful symbols or hateful people? predictive features for hate speech detection on twitter[END_REF][START_REF] Davidson | Automated hate speech detection and the problem of offensive language[END_REF]. Among the neural network-based approaches, different algorithms have been presented, such as Convolutional Neural Network using pre-trained word2vec embeddings [START_REF] Zhang | Detecting hate speech on twitter using a convolution-gru based deep neural network[END_REF], bi-LSTM with attention mechanism [START_REF] Agrawal | Deep learning for detecting cyberbullying across multiple social media platforms[END_REF] and bidirectional Gated Recurrent Unit network [START_REF] Lee | Comparative studies of detecting abusive language on twitter[END_REF]. More recently, also the combination of different neural newtorks, capturing both the message content and the Twitter account metadata, has been proposed [START_REF] Founta | A unified deep learning architecture for abuse detection[END_REF]. In a comparative study of various learning models on the Hate and Abusive Speech on Twitter dataset built by Founta et al. [START_REF] Founta | Large scale crowdsourcing and characterization of twitter abusive behavior[END_REF], Lee et al. [START_REF] Lee | Comparative studies of detecting abusive language on twitter[END_REF] show that, in the classification of tweets as "normal", "spam", "hateful" and "abusive" a bidirectional Gated Recurrent Unit network trained on word-level features is the most accurate model. Instead, in the binary task of offensive language detection, Liu et al. [START_REF] Liu | NULI at SemEval-2019 task 6: Transfer learning for offensive language detection using bidirectional transformers[END_REF] achieve the best performance at Semeval 2019 by fine-tuning a bidirectional encoder representation from transformer [START_REF] Devlin | BERT: pre-training of deep bidirectional transformers for language understanding[END_REF].

In this paper, we propose a robust neural classifier for the hate speech binary classification task which is performing well across different languages (English, Italian and German), and we study the impact of each feature and component on the results across these languages. Our recurrent neural architecture shares some elements with the above approaches, namely the use of a Long Short Term Memory and a Gated Recurrent Unit. Embeddings, textual and social network specific features are employed. As in [START_REF] Kshirsagar | Predictive embeddings for hate speech detection on twitter[END_REF], we do not use metadata related to the social media accounts. The obtained results are compared in a more detailed way in Section 6.

Hate speech detection on languages different from English

While most approaches to hate speech detection have been proposed for English, other systems have been developed to deal with the task in German, Italian and Spanish, thanks to recent shared tasks. The 2018 GermEval Shared Task on the Identification of Offensive Language2 deals with the detection of offensive comments from a set of German tweets. The tweets have to be classified into the two classes offense and other, where the offense class covers abusive language, insults, as well as profane statements. Different classifiers are used by the participants, ranging from traditional feature-based supervised learning (i.e., SVMs for the top performing system TUWienKBS [START_REF] Padilla | Tuwienkbs at germeval 2018: German abusive tweet detection[END_REF]) to the more recent deep learning methods. Most top performing systems in both shared tasks employed deep learning (e.g., spMMMP [START_REF] Dirk Von Grunigen | spmmmp at germeval 2018 shared task: Classification of offensive content in tweets using convolutional neural networks and gated recurrent units[END_REF], uhhLT [START_REF] Wiedeman | Transfer learning from lda to bilstm-cnn for offensive language detection in twitter[END_REF], SaarOffDe [START_REF] Stadnikova | Offensive language detection with neural networks for germeval task[END_REF], Inri-aFBK [START_REF] Corazza | Inriafbk at germeval 2018: Identifying offensive tweets using recurrent neural networks[END_REF]). For example, SaarOffDe employs Recurrent Neural Networks and Convolutional Neural Networks produced top scores, while other systems (e.g., spMMMP, uhhLT) employ transfer learning. The usage of ensemble classification seems to often improve the classification approaches (e.g., Potsdam [START_REF] Scheffler | Feature explorations for hate speech classification[END_REF], RuG [START_REF] Bai | Rug at germeval: Detecting offensive speech in german social media[END_REF], SaarOffDe, TUWienKBS, UdSW [START_REF] Wiegand | Saarland university's participation in the germeval task 2018 (udsw) -examining different types of classifiers and features[END_REF]). Concerning the features, several systems include a combination of word embeddings, character n-grams and some forms of (task-specific) lexicon. Both the HaUA and the UdSW systems report that high performance scores can be achieved with a classifier solely relying on a lexicon. In 2018, the first Hate Speech Detection (HaSpeeDe) task for Italian has been organized at EVALITA-2018 3 . The task consists in automatically annotating messages from Twitter and Facebook, with a boolean value indicating the presence (or not) of hate speech. Similar to Germeval 2018 submissions, also in this case the participating systems adopt a wide range of approaches, including bi-LSTM [START_REF] Liz De La Peña Sarracén | Hate speech detection using attention-based LSTM[END_REF], SVM [START_REF] Santucci | Detecting hate speech for italian language in social media[END_REF], ensemble classifiers [START_REF] Polignano | Hansel: Italian hate speech detection through ensemble learning and deep neural networks[END_REF][START_REF] Bai | Rug @ EVALITA 2018: Hate speech detection in italian social media[END_REF], RNN [START_REF] Fortuna | Merging datasets for hate speech classification in italian[END_REF], CNN and GRU [START_REF] Dirk Von Grunigen | spmmmp at germeval 2018 shared task: Classification of offensive content in tweets using convolutional neural networks and gated recurrent units[END_REF]. The authors of the best-performing system, ItaliaNLP [START_REF] Cimino | Multi-task learning in deep neural networks at EVALITA[END_REF], experiment with three different classification models: one based on linear SVM, another one based on a 1-layer BiLSTM and a newly-introduced one based on a 2-layer BiLSTM which exploits multi-task learning with additional data from the 2016 SENTIPOLC task 4 .

Concerning Spanish, the IberEval 2018 edition5 has proposed the Aggressiveness Detection task [START_REF] Ángel | Overview of MEX-A3T at ibereval 2018: Authorship and aggressiveness analysis in mexican spanish tweets[END_REF] applied to Mexican Spanish, aiming at providing a classification of aggressive / non-aggressive tweets. A variety of systems is proposed, exploiting content-based (bag of words, word n-grams, term vectors, dictionary words, slang words) and stylistic-based features (frequencies, punctuation, POS, Twitter specific elements). Most of the systems rely on neural networks (CNN, LSTM and others). The top ranked team was INGEOTEC [START_REF] Graff | INGEOTEC at MEX-A3T: author profiling and aggressiveness analysis in twitter using µtc and evomsa[END_REF]: the system is based on MicroT, a text classification approach supported by a lexicon-based model that takes into account the presence of aggressive and affective words, and a model based on the Fasttext representation of texts. More recently, a task for the detection of hate speech against immigrants and women on Twitter has been organised at Semeval 2019 [START_REF] Valerio Basile | SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter[END_REF], providing an English and

x 1 ⊕ x n ⊕ . . . RNN x i s i s i-1 y i x e
Figure 1: The modular neural architecture Spanish dataset annotated according to the same guidelines. While for both languages a number of neural network approaches has been proposed, the best systems for hateful content detection still rely on SVM and embedding-based features [START_REF] Indurthi | FERMI at SemEval-2019 task 5: Using sentence embeddings to identify hate speech against immigrants and women in twitter[END_REF][START_REF] Manuel Pérez | Atalaya at SemEval 2019 task 5: Robust embeddings for tweet classification[END_REF][START_REF] Enrique | MineriaUNAM at SemEval-2019 task 5: Detecting hate speech in twitter using multiple features in a combinatorial framework[END_REF].

Looking at the descriptions of the systems participating in the above tasks, as well as at most recent hate speech detection classifiers for English, we observe that deep learning approaches usually share a number of features, such as word embeddings, the use of emotion or sentiment lexica, as well as specific pre-processing steps. Many exploit also other features related to the tweets (e.g. message length, punctuation marks, etc.). Nevertheless, more emphasis is usually put on the architecture, and no insight is given into the role played by variants of the above features and by the selected pre-processing strategy. Also, no attempt to understand differences across different languages has been made. This motivates the experiments presented in the remainder of this paper.

Classification framework

Since our goal is to compare the effect of various features, word embeddings and pre-processing techniques on hate speech detection, we use a modular neural architecture for binary classification that is able to support both word-level and message-level features. The components are chosen to support the processing of social-media specific language. The neural architecture and the features are detailed in the following subsections.

Modular Neural Architecture

We use a modular neural architecture (see Figure 1) in Keras [START_REF] Chollet | [END_REF]. The architecture that constitutes the base for all the different models uses a single feed-forward hidden layer of 100 neurons, with a ReLu activation and a single output with a sigmoid activation. The loss used to train the model is binary cross-entropy. We choose this particular architecture because we used it to participate to two shared tasks for hate speech detection, EVALITA HaSpeeDe 2018 [START_REF] Corazza | Comparing different supervised approaches to hate speech detection[END_REF] for Italian and Germeval 2018 [START_REF] Corazza | Inriafbk at germeval 2018: Identifying offensive tweets using recurrent neural networks[END_REF] for German, and it proved to be effective and robust for both languages, also across different social media platforms [START_REF] Corazza | Cross-platform evaluation for italian hate speech detection[END_REF]. In particular, in our original submissions the same architecture was ranked fourth in the Twitter EVALITA subtask (-1.56 F1 compared to the first ranked) and seventh in the Germeval coarse-grained classification task (-2.52 F1 from the top-ranked one).

The architecture is built to support both word-level (i.e. embeddings) and tweet-level features. In particular, we use a recurrent layer to learn an encoding (x n in the Figure ) derived from word embeddings, obtained as the output of the recurrent layer at the last timestep. This encoding gets then concatenated with the other selected features, obtaining a vector of tweet-level features.

Since the models derived from using different features are different both in terms of number of parameters and in terms of layers, we decided to keep the size of the hidden layer fixed. This allows us to compare different features, as the latent representation learned by the hidden layer that is ultimately used to classify the tweets has the same size regardless of the number and kind of features.

More formally, given an input represented as the set of features X = {x j |x j ∈ X m }, where X M is the set of all features supported by a model M (see Section 3.2) and s is the sum of the dimensions of all the features, we compute a function:

M (X) = s(W o H(X) + b o ) W o ∈ R 1×100 H(X) ∈ R 100 b o ∈ R 1 s(X) = (σ(x 1 ), . . . , σ(x n )) x ∈ R n σ(x) = 1 1 + e -x
where W o and b o are the learned weights for the output layer and σ(x) is the sigmoid activation function (note that in all the models we used n = 1 as we only have one binary output), and:

H(X) = g(W h C(X) + b h ) W h ∈ R 100×s C(X) ∈ R s b h ∈ R 100 g(X) = (f (x 1 ), . . . , f (x n )) x ∈ R n f (x) = max(0, x) x ∈ R
where H(X) represents the application of a hidden layer of size 100 and learned weights W h and b h and g(x) is the ReLU activation function. Additionally:

C(X) = xi∈X R(x i )
where denotes the concatenation of all vectors along their axes. For example, if we have a set of vectors X = [x 1 , x 2 , x 3 ], then:

xi∈X x i ∈ R a+b+c x 1 ∈ R a , x 2 ∈ R b , x 3 ∈ R c Finally: R(x) = x if x is a tweet-level feature RN N (x) if x is a word-level feature
where RNN is the function returning the output by a recurrent layer at the last timestep.

Features

In our experiments, we use the following features, with the goal of evaluating their impact on a hate speech detection model:

• Word Embeddings (x e in Figure 1): multiple word embeddings from various sources have been tested (for a full description of the different embeddings see Section 4.2). We evaluate in particular the contribution of word embeddings extracted from social media data, therefore belonging to the specific domain of our classification task, compared with the performance obtained using generic embedding spaces, like Fasttext [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF], which are widely used across different NLP tasks because of their good coverage.

• Emoji embeddings: emojis are a peculiar element of social media texts. They are often used to emphasize or reverse the literal meaning of a short message, for example in ironic or sarcastic tweets [START_REF] Hu | Spice up your chat: The intentions and sentiment effects of using emojis[END_REF]. It is therefore very important for hate speech detection to understand which is the best way to represent them and to include them in the embedding space. We compare different ways to embed emoji information in our classifier: i) we use embedding spaces created from social media data, where each emoji is also represented through a word embedding, or ii) in case of generic embedding spaces, where emojis are not present, we include emoji embeddings through the alignment of different spaces following the approach presented in [START_REF] Samuel | Offline bilingual word vectors, orthogonal transformations and the inverted softmax[END_REF], or iii) in order to cope with the low coverage of emojis, they are replaced by their description in plain text as suggested in [START_REF] Singh | Incorporating emoji descriptions improves tweet classification[END_REF].

• Ngrams: unigrams (x 1 in Figure 1) and bigrams derived from the tweets are also included as features. We first tokenize and lemmatize the tweets by using Spacy [START_REF] Honnibal | spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing[END_REF], then normalize the tweet-level ngram occurrence vector by using tf-idf. Our intuition is that these features should capture lexical similarities between training and test data, therefore they should be predictive when training and test set deal with the same type of offenses. Higher-level ngrams are not considered, as we expect them to be very sparse especially in social media, where tweets do not follow standard writing conventions.

• Social-network specific features: The character limit imposed by some social media platforms like Twitter affects the style in which messages are written: function words tend to be skipped, texts are very concise while punctuation and uppercase words are used to convey effective messages despite their brevity. Therefore, all these linguistic indicators can be used to identify the presence of hateful messages. We consider in particular the number of hashtags and mentions, the number of exclamation and question marks, the number of emojis, the number of words that are written in uppercase at the tweet-level. These features are then normalized by subtracting their mean and dividing them by their standard deviation.

• Emotion lexica: several emotion lexica have been (manually or automatically) created and used in classification tasks to represent the emotional content of a message [START_REF] Saif | Crowdsourcing a word-emotion association lexicon[END_REF][START_REF] Saif | Emotions evoked by common words and phrases: Using mechanical turk to create an emotion lexicon[END_REF][START_REF] Bassignana | Hurtlex: A multilingual lexicon of words to hurt[END_REF][START_REF] Staiano | Depeche mood: a lexicon for emotion analysis from crowd annotated news[END_REF]. While the importance of emotion information to hate speech detection may seem evident [START_REF] Arslan | Overwhelmed by Negative Emotions? Maybe You Are Being Cyber-bullied![END_REF], it is also true that an embedding space which is large and representative enough of the domain may make additional emotion features redundant. We therefore evaluate the contribution of emotion information using two freely available, multilingual emotion lexica, namely EmoLex and Hurtlex. Emolex [START_REF] Saif | Crowdsourcing a word-emotion association lexicon[END_REF][START_REF] Saif | Emotions evoked by common words and phrases: Using mechanical turk to create an emotion lexicon[END_REF] is a large list of English words and their associations with eight basic emotions (anger, fear, anticipation, trust, surprise, sadness, joy, and disgust) and two sentiments (negative and positive), manually annotated with Amazon's Mechanical Turk. The creators of the lexicon have additionally made available a multilingual version of the resource, that was created by translating each word with Google translate in 2017. We therefore use the German and Italian translations as well as the English one. Using EmoLex, we extract two sentiment-related features and eight emotionrelated features for each tweet by summing all the sentiment and emotion scores assigned to the words in a tweet and normalizing them by using tf-idf. The second resource, i.e., Hurtlex [START_REF] Bassignana | Hurtlex: A multilingual lexicon of words to hurt[END_REF], is a multilingual lexicon of hate words created starting from the Italian hate lexicon developed by the linguist Tullio De Mauro, organized in 17 categories. It has been expanded through the link to available synset-based computational lexical resources such as MultiWordNet [START_REF] Pianta | Multiwordnet: developing an aligned multilingual database[END_REF] and Babelnet [START_REF] Navigli | Babelnet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network[END_REF], and evolved in a multilingual perspective by semi-automatic translation and expert annotation. Since Hurtlex may contain the same word multiple times with different Part-of-Speech tags, we performed a union operation over the categories in order to represent all the categories that a word can belong to, independently of the POS. Using HurtLex, we assign with the same strategy a score for negative stereotypes, one for hate words and slurs and one for other insults to each tweet.

Data and linguistic resources

In the following, we present the datasets used to train and test our system for English, Italian and German and their annotations (Section 4.1). Then, we describe the word embeddings (Section 4.2) we have used in our experiments.

Datasets

English We use the dataset described in [START_REF] Waseem | Hateful symbols or hateful people? predictive features for hate speech detection on twitter[END_REF], containing 16k English tweets manually annotated for hate speech. More precisely, 1,924 are annotated as containing racism, 3,082 as containing sexism, while 10,884 tweets are annotated as not containing offensive language. We merge the sexist and racist tweets in a single class, so that 5,006 tweets are considered as positive instances of hate speech, as in Example 1.

1. Annotation: hateful. Since 1/3 of all #Islam believes that people who leave the religion should be murdered where are the moderate Muslim.

Italian We use the Twitter dataset released for the HaSpeeDe (Hate Speech Detection) shared task organized at Evalita 2018, the evaluation campaign for NLP and speech processing tools for Italian 6 . This dataset includes a total amount of 4,000 tweets [START_REF] Bosco | Overview of the EVALITA 2018 hate speech detection task[END_REF], comprising for each tweet the respective annotation, as can be seen in Example 2. The two classes considered in the annotation are "hateful post" or "not".

2. Annotation: hateful. altro che profughi? sono zavorre e tutti uomini (EN: Are they really refugees? they are ballast and all men).

German We use the dataset distributed for the shared task on the Identification of Offensive Language organized at Germeval 2018, a workshop in a series of shared tasks on German processing 7 . The dataset provided for task 1, where offensive comments are to be detected from a set of German tweets (binary classification), consists of 5,009 German tweets manually annotated at the message level [START_REF] Wiegand | Overview of the germeval 2018 shared task on the identification of offensive language[END_REF] with the labels "offense" (abusive language, insults, and profane statements) and "other" (i.e. not offensive). More specifically, 1,688 messages are tagged as "offense" (see Example 3), while 3,321 messages as "other". Table 1 summarizes the main statistics on the datasets. The reported values show that, although the datasets have different sizes, the distribution between positive and negative examples is similar. We also manually investigated data samples and the annotation schemes of the English, German and Italian datasets. Although the developers of the English and the Italian corpus focus on hate speech, while the Germeval organisers claim to target offensive language, the kind of messages they annotate as belonging to their respective 'positive' class largely overlap. The targets are different, i.e. the Italian messages focus on immigrants, Muslim and Roma, the English ones on sexist and racial offenses, while the German one has no specific targets, and includes both offensive messags towards groups and towards individuals. However, the types of offenses,

Word Embeddings

In our experiments we test several embeddings, with the goal to compare generic with social media-specific ones. In order to have a high coverage of emojis, we also experiment with aligned embedding spaces obtained by aligning the English, Italian and German ones. Another element we take into account is the access to the binary Fasttext model that originates the embedding space. When using that binary model, it is possible to greatly mitigate the problem of out-ofvocabulary words, since the system is able to provide an embedding for unknown words by using subword unit information [START_REF] Mikolov | Advances in pre-training distributed word representations[END_REF]. The binary model is often made available together with the standard model when pre-trained embeddings are released. When available, we always use this version. The tested embeddings, summarised in Table 2, are the following:

• Fasttext embeddings for German and Italian: we use embedding spaces obtained directly from the Fasttext website 8 for German and Italian. In particular, we use the Italian and German embeddings trained on Common Crawl and Wikipedia [START_REF] Grave | Learning word vectors for 157 languages[END_REF] with size 300. A binary Fasttext model is also available and was therefore used;

• English Fasttext Crawl embeddings: English embeddings trained by Fasttext9 on Common Crawl, with an embedding size of 300. A binary Fasttext model is provided;

• English Fasttext News embeddings: English embeddings trained on Wikipedia 2017 using subword information, UMBC web base corpus and statmt.org and released by Fasttext10 , with an embedding size of 300.

The available binary Fasttext model was used;

• Italian Twitter embeddings: we trained Fasttext embeddings from a sample of Italian tweets [START_REF] Basile | Sentiment analysis on italian tweets[END_REF], with embedding size of 300. We used the binary version of the model;

• German Twitter embeddings: trained by Spinning Bytes11 from a sample of German tweets [START_REF] Cieliebak | A twitter corpus and benchmark resources for german sentiment analysis[END_REF]. We used the model with embeddings of size 300. A binary Fasttext model was not provided, we therefore used the word-based version;

• English Twitter embeddings: English Fasttext embeddings from Spinning Bytes 12 , trained on an English Twitter sample [START_REF] Cieliebak | A twitter corpus and benchmark resources for german sentiment analysis[END_REF] with an embedding size of 200. Since a binary Fasttext model was not provided, we used the word-based version;

• Aligned embeddings: since Fasttext embeddings for Italian and German do not contain emojis, we extend them by aligning them with an English embedding space containing emojis [START_REF] Barbieri | What does this emoji mean? a vector space skip-gram model for twitter emojis[END_REF], following the alignment approach presented in [START_REF] Samuel | Offline bilingual word vectors, orthogonal transformations and the inverted softmax[END_REF]. All embeddings and the resulting aligned spaces have a size of 300. In summary, we were able to use a binary model for all the official Fasttext monolingual datasets and the Italian Twitter embeddings that we trained. For the remaining embedding spaces, we only had access to a dictionary-like structure, that contains the embedding for each word in the vocabulary.

Experiments

In this section, we detail the setup of our experiments, including the preprocessing step, the selection of hyperparameters and the combination of features and configurations tested for each language.

Preprocessing

Since hashtags, user mentions, links to external media and emojis are common in social media interactions, it is necessary to carefully preprocess the data, in order to normalize the text as much as possible while retaining all relevant semantic information. For this reason, we first replace URLs with the word "url" and "@" user mentions with "username" by using regular expressions. Since hashtags often provide important semantic content, we wanted to test how splitting them into single words would impact on the performance of the classifier. To this end, we use the Ekphrasis tool [START_REF] Baziotis | DataStories at SemEval-2017 Task 4: Deep LSTM with Attention for Message-level and Topic-based Sentiment Analysis[END_REF] to do hashtag splitting and evaluate the classifier performance with and without splitting. Since the aforementioned tool only supports English, it has been adapted to Italian and German by using language-specific Google ngrams. 13Another pre-processing step we evaluate in our experiments is the description of emojis in plain text, that proved to benefit tweet classification [START_REF] Singh | Incorporating emoji descriptions improves tweet classification[END_REF] but was evaluated so far only on English. In order to map each emoji with a description, we first retrieve an emoji list using the dedicated Python library 14 and replace each emoji with its English description according to the website of the Unicode consortium 15 . We then translate the descriptions using Google Translate and fix any mistakes by hand. In this way we create a list of emojis with the corresponding transcription in three languages (available at https://github. com/dhfbk/emoji-transcriptions).

Hyperparameters

In order to keep our setting robust across languages, we base our model on a configuration that performed consistently well on all subtasks of Evalita hate speech detection [START_REF] Corazza | Comparing different supervised approaches to hate speech detection[END_REF], both on Facebook and on Twitter data, even if it was not the best performing configuration on the single tasks. In particular, our model uses no dropout and no batch normalization on the outputs of the hidden layer. Instead, a dropout on the recurrent units of the recurrent layers is used. We select a batch size of 32 for training and a size of 200 for the output (and hidden states) of the recurrent layers. We also test the impact of different recurrent layers, namely long short-memory (LSTM) [START_REF] Hochreiter | Long short-term memory[END_REF], gated recurrent unit (GRU) [START_REF] Cho | Learning phrase representations using rnn encoder-decoder for statistical machine translation[END_REF] and bidirectional LSTM (BiLSTM) [START_REF] Schuster | Bidirectional recurrent neural networks[END_REF].

Settings

In our experiments, we perform a series of tests on the aforementioned modular neural model, concerning the following aspects:

• For each language, we test the corresponding embeddings presented in Section 4.2.

• We test all possible combination of features: embeddings, unigrams, bigrams, social features, EmoLex and Hurtlex.

• We test three possible recurrent layers, namely LSTM, GRU and Bidirectional LSTM.

• We train models with and without hashtag splitting.

• We test models that replace emojis with their description and models that do not. For Italian and German Fasttext embeddings that do not contain emojis, we also test the model performance after using emoji embeddings resulting from alignment with an English embedding space (see details in Section 4.2)

Overall, we compare 1,800 possible configurations for English, 1,080 for Italian and 1,224 for German. The difference is due to the availability of more embedding spaces for English, which increase the amount of possible settings and feature combinations to be tested.

Concerning the dataset splits into training and test instances, for the English dataset -since no standardized split is provided -we randomly selected 60% of the dataset for training, 20% for validation and 20% for testing. Since we want our experiments to be reproducible, we use the train test split function from scikit-learn [START_REF] Pedregosa | Scikitlearn: Machine learning in Python[END_REF] to shuffle and split the dataset 60%/40%. The remaining 40% was then split in half to obtain the validation and test set, respectively. We use 42 as a seed value for the random number generator used for shuffling.

The German dataset was already randomly split by the GermEval task organizers into training and test set, containing 5,009 and 3,532 messages respectively. For our experiments we keep the same split as proposed in the challenge, but we use 20% of the training set as validation set, obtained by invoking train test split from scikit learn with 42 as seed. Similarly, the Italian dataset was randomly split by the HaSpeeDe task organizers into training and test set of 3,000 and 1,000 messages respectively. Again, in our experiments we keep the same split as proposed in the challenge, but we used 20% of the training set as validation set applying the same function as for the German dataset.

For each language, the validation test is used to evaluate the classifier performance over 20 training epochs and select the best performing model in terms of macro averaged F1 score. The selected model is then used to evaluate performance on the test set.

Evaluation

In this section, we report a selection of the most relevant results from the pool of settings described in Section 5.3. In particular, the first row of Table 3, 4 and 5 reports the best run over all the configurations tested for English, Italian and German respectively, while the other rows show how the best performance changes when modifying one parameter at a time. We also provide an evaluation of the effectiveness of different configurations by comparing the three languages after downsampling the training sets. As comparison we provide a baseline obtained running a SVM (linear kernel) with a bag of word approach using tf-idf as weight.

Multilingual evaluation on the complete datasets

For English, the best result (0.823 F1) is obtained using an LSTM network and the Fasttext embeddings trained on Common Crawl. Table 3 shows how adding or removing single features from the best configuration affects the result: adding unigram and bigram-based features to the classifier leads to the largest drop in performance, while changing other features the impact is lower. This confirms the findings in [START_REF] Waseem | Hateful symbols or hateful people? predictive features for hate speech detection on twitter[END_REF], in which character n-grams outperform word ngrams in the classification of racial, sexist and not-offensive tweets. Overall we find that, although the best result is obtained using an LSTM network, replacing LSTM with Bi-LSTM keeping the same features achieves similar results, with a difference of F1 of 0.1-0.2% F1. This shows that having both forward and backward information when dealing with tweets is probably not needed because of the limited length of the messages. The use of hashtag normalization to split the hashtags into words improves the system performance in every configuration, increasing the coverage of the embeddings. Overall, the coverage of Fasttext embeddings trained on CommonCrawl is sufficient to deal with Twitter data, therefore adding specific embeddings or pre-processing them is not necessary. Also, the SVM baseline suffers from lower recall compared to the best neural configuration, especially when dealing with the hate category, that has less training instances. For Italian, the best result (0.805 F1) is obtained with a configuration using a LSTM network and the word embeddings we trained on a large corpus of Italian tweets. In Table 4 we show to what extent the different features affect the performance obtained with the best configuration. On Italian, differently from English and German, the use of unigrams in addition to word embeddings is beneficial to the classifier performance. The best result is obtained using the emoji transcription, but their impact is not significant (0.805 F1 using them vs. 0.804 not using them). The same trend can be found also with different configurations not reported in the table. Considering all runs with all configurations, the use of embeddings trained on the same domain of the dataset (Italian Tweets) always leads to better results compared with the use of more generic embeddings as the ones from Fasttext (trained on Common Crawl and Wikipedia). Almost all the best performing configurations take advantage of the use of hashtag splitting. BiLSTM performs generally worse than LSTM. Like in the English evaluation, the SVM baseline achieves a remarkably lower performance on the hate class, and shows recall issues. Table 5 reports the results obtained on German data. The best result is achieved with a GRU network, using the standard Fasttext embeddings (trained on Common Crawl and Wikipedia). Similar to English, adopting unigrams and bigrams as feature leads to a decrease in performance (0.05 points F1). Considering all the experiments run on German data, the results confirm that also for this language emoji transcriptions perform better than the emoji vectors obtained through multilingual alignment, but for the best configuration no specific emoji processing is needed. Hashtag splitting, which is included in the best performing configuration for English and Italian, is instead not beneficial to German tweet classification. Our intuition is that, since German is rich in compound words, Ekphrasis hashtag normalization approach based on Google n-grams tends to split terms also when it is not needed. Although social and emotion features are not used in the best output, they appear to help in most of the other configurations. Also for this language, the SVM baseline achieves a lower recall and less accurate classification on the hate speech class than the neural model. Table 5: Best performing configuration on German data (Macro AVG).

Beside the aforementioned experiments, we perform an additional evaluation using a character-based RNN. Indeed, character-based representations have been recently used in several NLP tasks including abusive language detection [START_REF] Mishra | Neural character-based composition models for abuse detection[END_REF] with promising results thanks to their ability to effectively handle rare and unseen words. We use the best performing systems for the three languages, replacing word-based RNN with a character-based one. In order to learn a dense representation for characters, we used a learned embedding layer with size 10. The results of this set of experiments are reported in Table 6, and show that using a character-based RNN the performance of the system drops significantly in all three languages compared to word-based RNNs, probably because Fasttext embeddings already account for subword information. We therefore decided not to perform further tests with this configuration.

Contribution of social and emotion information

In order to better understand the contribution of specific features or pre-processing steps on all the system runs, we present a comparative evaluation of the classifier performance with or without emoji transcription (in Figure 2) and with or without social and emotion features (Figure 3). This analysis is done with the goal of focusing not only on the best performing configuration, but also on general trends that could not be included in the previous tables. In particular, we plot the distribution of runs achieving different macro average F1 scores. Figure 2 shows that transcribing emojis yields the best performance for English but not for the two other languages. Nevertheless, this distinction is not clear-cut, since no clear trend can be associated with this feature. More details on the different configurations are shown in Table 7, confirming the above findings. Figure 3 analyses in a similar way the contribution of social network specific features (i.e. tweet length, punctuation marks, uppercase, etc.) and emotion features (i.e. based on EmoLex and Hurtlex). It shows that, while for English and Italian the best results are obtained without these two groups of features, other runs achieving on average a slightly lower performance make use of this information. For German, the improvement due to social and emotion features appears to be more consistent, even if it does not apply to all runs. Also, the averaged results summarised in Table 8 confirm that, like for emojis, the differences are not clear-cut. Comparing the results across the three languages, we summarize the main findings from the evaluation as follows:

• Using subword information has a positive impact on our task, since it can deal with the high language variability and creativity in the social media domain as well as with typos.

• Creating specific embeddings that cover well the domain of interest is beneficial to the task performance. If possible, a large amount of Twitter data should be collected to create embeddings when dealing with online hate speech classification. If not, pretrained Fasttext embeddings trained on CommonCrawl or similar are recommended, provided that it is possible to access the binary model

• If the above domain-specific embeddings are available, where emojis are also present, our experiments show that it is not needed to pre-process emojis in specific ways (e.g. transcribe, add emoji embeddings through alignment)

• Hashtag normalization is useful to classify hate speech in English and Italian, but current approaches to hashtag splitting may not perform well on languages that are rich in compounds like German, which in turn may affect classification

• Using domain-specific embeddings with a good coverage make emotion lexica redundant in our experiments. The fact that such lexica may be manually or semi-automatically created does not play a major role in classification performance

• Given the limited length of tweets, LSTM yielded better results than BiL-STM

Multilingual evaluation on downsampled datasets

We perform an additional set of experiments to investigate to what extent the size of the dataset affects the results. Therefore, we downsample both the German and the English datasets to match the size of the Italian Twitter dataset, the smallest one. In order to improve our ability to compare the results, we use the same distribution of labels (hate speech, non hate speech) as the Italian dataset for the two downsampled ones. We then replicate some of the best performing configurations presented in the previous tables, and report the results in Table 9. As expected, reducing the training data both for English and for German leads to a drop in performance (from 0.823 F1 to 0.782 for English, from 0.758 F1 to 0.713 for German). On all the runs, the classifier achieves a lower performance on German than on the other two languages, while the results on Italian and English are comparable. Our experiments suggest that German is more challenging to classify, partly because of inherent characteristics of the language (for example the presence of compound words that makes hashtag splitting ineffective), partly because of the way in which the Germeval dataset was built. Namely, the organisers report that they sampled the data starting from specific users and avoiding keyword-based queries, so to obtain the highest possible variability in the offensive language. They also manually checked and enriched the data so to cover all the political spectrum in their offenses, and avoid user overlaps between training and test data. While our main goal is not to develop a system achieving state-of-the-art results, it is interesting to compare our performance with the best systems dealing with hate speech detection. For Italian and German our approach can be easily compared to other existing classifiers using the same training and test split, since we relied on the official data released in two shared tasks. These results, however, were obtained in the context of the shared task, therefore the authors could not use information about the test set performance as we did. The comparison is still interesting, but it should be noted that we are reporting the best results on the test set, not on the development set.

On Italian, we observe that our best system configuration achieves state-ofthe-art results (F1 0.805). The best performing system in the EVALITA shared task [START_REF] Cimino | Multi-task learning in deep neural networks at EVALITA[END_REF] reached 0.800 F1 on the development set using an SVM-based classifier with rich linguistic features, while the best score obtained on the test set (0.799 F1) was yielded by a two-layer BiLSTM in a multi-task learning setting. Similar to our best setting, they also use embeddings extracted from social media data, and observe that using sentiment-based lexica does not increase system performance.

On German, the best performing system participating in Germeval [66] achieved 0.768 F1 [START_REF] Padilla | Tuwienkbs at germeval 2018: German abusive tweet detection[END_REF] and was a stacked ensemble system that combined maximum entropy and random forest classifiers and relied on five groups of features. However, the system performance in 10-fold cross-validation using only the training set reached 0.817 F1. Our best configuration on the task test set yields 0.758 F1 with a much simpler architecture, using only Fasttext and no other features except for word embeddings.

As for English, it is more difficult to draw a similar comparison because the dataset we use [START_REF] Waseem | Hateful symbols or hateful people? predictive features for hate speech detection on twitter[END_REF] was originally annotated with three classes (i.e. racism, sexism and none), thus most systems using the same data perform multiclass classification. Besides, they are run using ten-fold cross-validation like in the original paper [START_REF] Waseem | Hateful symbols or hateful people? predictive features for hate speech detection on twitter[END_REF]. One of the few attempts to distinguish between hate and non-hate speech on the same English data is described in [START_REF] Kshirsagar | Predictive embeddings for hate speech detection on twitter[END_REF], where the authors present a classifier combining word-based CNN and character-based CNN. They report 0.734 F1 on the binary task in ten-fold cross-validation. Other works using the same data set for three-class classification report much higher results (0.783 F1 in [START_REF] Gambäck | Using convolutional neural networks to classify hate-speech[END_REF] using CNN, 0.86 F1 in [START_REF] Kshirsagar | Predictive embeddings for hate speech detection on twitter[END_REF] using a multi-layer perceptron). Interestingly, as shown in [START_REF] Kshirsagar | Predictive embeddings for hate speech detection on twitter[END_REF], multi-class classification seems generally easier than the binary one on this specific data set, since sexist and racist tweets present lexical-based discriminating features that are easy to capture.

Qualitative evaluation

In our experiments we tested more that 1,000 configurations for each language, and it is therefore difficult to manually evaluate and compare the results, since each configuration may make specific mistakes and the distribution of false positives and negatives on the test split would change. In order to gain some insights into the specificity of each language and dataset, however, we focus on the output of the best performing configuration for each language, and we manually check the wrongly classified instances. In most of the cases, it is not possible to assign a category to the mistakes done by the classifier, since the false negative tweets are clearly hateful and the false positive ones are unambiguously nonhateful. These cases are prevalent in all the datasets, so they are independent from the language and also from the dataset size. The opaque mechanisms with which deep learning classifiers assign labels make it difficult to explain why these apparently trivial cases were misclassified, but we plan to exploit information conveyed by attention mechanisms to shed light into this issue [START_REF] Corazza | Inriafbk drawing attention to offensive language at germeval2019[END_REF].

Among the broad mistake categories found across the inspected datasets, there are some cases of implicit abuse. Such messages do not contain abusive words but rather convey their offensive nature through sarcasm, jokes, the usage of negative stereotypes or supposedly objective statements implying some form of offense.

We report few examples of false negatives for the hate speech class below:

4. It's not about any specific individuals, but about an ideology that will always produce terrorists.

5.

Molti ancora non vedono, ma quando attraversano un parco, se popolato da immigrati, si tengono stretta la borsa. (EN: Many do not see it, but when they cross a park populated with immigrants they hold their bag close).

6. Schau doch Pornos wenn du mehr Redeanteil von Frauen hören willst (EN: Watch porn if you want to hear more women talk).

We also observe that sentences with a complex syntactic structure, containing for example more than one negation, or questions, are frequent both among the false positives and the false negatives (see Sentence 7, which was wrongly classified as 'Not hate'). The same happens for tweets that contain anaphoric elements that hint at mentions probably present in previous messages, and for tweets which require some form of world knowledge to be understood. In some cases, a link to external media contributed to the hateful meaning of a tweet, as in Sentence 8. However, since we remove urls in the pre-processing step this information was not exploited for classification.

7.

No. You have proven your ignorance here to anyone who isn't as dumb as you. It's there for all to see but you don't know it..

8.

A quanto pare, il corano si può usare anche per questo. Ma pare non funzioni molto bene..... http: // t. co/ DcOSHfmfxK (EN: It seems that Quran can be used also for this. But apparently it does not work very well...http://t.co/DcOSHfmfxK).

Among false positives, the inspected examples confirm the remarks in [START_REF] Wiegand | Detection of Abusive Language: the Problem of Biased Datasets[END_REF] concerning the English dataset, and we observe a similar behaviour also for Italian tweets: since these datasets were collected starting from keywords concerning potential hate targets such as women, Roma and Muslims and then extended with not offensive tweets, classifiers tend to associate target mentions to hate speech, even if such messages are not offensive. This phenomenon is less evident on the German data, which indeed was created in a different way, starting from a list of users. Two examples of false positive are reported below. In [START_REF] Bassignana | Hurtlex: A multilingual lexicon of words to hurt[END_REF] the message is probably classified as hateful because of the mention of 'Jewish'. In [START_REF] Baziotis | DataStories at SemEval-2017 Task 4: Deep LSTM with Attention for Message-level and Topic-based Sentiment Analysis[END_REF] it may depend on the mention of 'migration'. 9. Fine by me. I had five Jewish friends in college. None ever went to a Synagogue.

10. l'immigrazione è un problema x tutti! Ma servono iniziative non comunicati (EN: Migration is a problem for everybody! But we need initiatives, not press releases).

Finally, we noted few mistakes in the gold standard annotation of the test sets, which were correctly classified by our system.

Conclusions

Targeting the hate speech detection task in social media messages, in this paper we have first identified a recurrent neural architecture that is rather stable and well-performing across different languages (i.e., English, German and Italian), and then we have evaluated the contribution of several components that are usually employed in the task, namely the type of embeddings, the use of additional features (text-based or emotion-based), the role of hashtag normalisation and that of emojis. Our comparative evaluation has been carried out on English, Italian and German available Twitter datasets for hate speech detection (annotated as either containing hate speech/offensive language or not). More precisely, in our detailed study we have compared 1,800 possible configurations for English, 1,080 for Italian and 1,224 for German. This allowed us to propose a set of findings, listed in Section 6, that could guide researchers in the design of hate speech detection systems, especially for languages different from English. To be exhaustive, we have also performed an additional set of experiments to investigate to what extent the size of the dataset affects the results.

Figure 2 :

 2 Figure 2: Results distribution with and without emoji transcription and using aligned emoji embeddings over the three languages.

Figure 3 :

 3 Figure 3: Results distribution with and without social network and emotion features over the three languages.

Table 1 :

 1 Statistics on the datasets both explicit and implicit, including sarcastic messages, rhetorical questions and false claims based on prejudices make them in our view comparable. The only difference is the set of messages labeled as 'Profanity' and included among the 'Offensive' ones in the German dataset, which covers slurs without a specific target. However, they account only for 1.4% messages in this training set.

	Dataset # hate speech/offensive (%)	# other (%)	# total
	English	5,006 (32%)	10,884 (68%)	16,000
	Italian	1,296 (32%)	2,704 (68%)	4,000
	German	1,688 (34%)	3,321 (66%)	5,009

Table 2 :

 2 Overview of the different embeddings used in our experiments

Table 3 :

 3 Best performing configuration on English data (Macro AVG). EMOJI = 'NO' means that no specific processing of emoji was applied

	P R F1	AVG AVG AVG	0.820 0.825 0.823	0.811 0.814 0.812	0.821 0.821 0.821	0.817 0.823 0.820	0.823 0.815 0.819	0.820 0.817 0.818	0.818 0.810 0.814	0.816 0.809 0.812	0.815 0.790 0.800	0.796 0.787 0.791	0.800 0.773 0.784	0.808 0.763 0.778
	F1	HATE	0.760	0.745	0.756	0.757	0.751	0.751	0.744	0.742	0.719	0.711	0.694	0.682
	F1 NO	HATE	0.885	0.879	0.886	0.883	0.887	0.885	0.884	0.883	0.881	0.871	0.873	0.875
	HASH.	SPLIT	YES	NO	YES	YES	YES	YES	YES	YES	YES	YES	YES
	EMBED. NETWORK TEXT FEATS SOCIAL EMOTIONS EMOJI	NO NO lstm Fasttext CCrawl emb NO	NO NO lstm Fasttext CCrawl emb NO	NO transcr. lstm Fasttext CCrawl emb NO	lstm NO NO Fasttext CCrawl emb YES	lstm EMOLEX NO Fasttext CCrawl emb YES	lstm EMOLEX NO Fasttext CCrawl emb NO	lstm HURTLEX NO Fasttext CCrawl emb NO	lstm HURTLEX NO Fasttext CCrawl emb YES	lstm NO NO Fasttext CCrawl emb+uni+bi NO	lstm NO NO NO Fasttext CCrawl emb+uni	lstm NO NO NO Fasttext CCrawl emb+bi	SVM baseline

Table 4 :

 4 Best performing configuration on Italian data (Macro AVG).

	P R F1	AVG AVG AVG	0.803 0.806 0.805	0.803 0.806 0.805	0.802 0.805 0.804	0.795 0.800 0.797	0.789 0.800 0.794	0.790 0.796 0.793	0.790 0.792 0.791	0.784 0.794 0.788	0.785 0.775 0.779	0.809 0.758 0.774	0.783 0.747 0.760	0.766 0.771 0.768	0.781 0.707 0.724
	F1	HATE	0.736	0.737	0.736	0.728	0.727	0.723	0.718	0.719	0.697	0.676	0.660	0.690	0.593
	F1 NO	HATE	0.867	0.872	0.871	0.867	0.861	0.863	0.864	0.858	0.862	0.872	0.860	0.847	0.855
	HASH.	SPLIT	YES	YES	YES	YES	NO	YES	YES	YES	YES	YES	YES	YES
	NETWORK EMBED. TEXT FEATS SOCIAL EMOTIONS EMOJI	NO transcription lstm Twitter emb+uni NO	NO transcription lstm Twitter emb+uni YES	NO NO lstm Twitter emb+uni NO	HURTLEX transcription lstm Twitter emb+uni NO	NO transcription lstm Twitter emb+uni NO	HURTLEX transcription lstm Twitter emb+uni YES	EMOLEX transcription lstm Twitter emb+uni YES	EMOLEX transcription lstm Twitter emb+uni NO	NO transcription lstm Twitter emb NO	NO embeddings lstm aligned emb+uni NO	NO transcription lstm Twitter emb+bi NO	NO transcription lstm Twitter emb+uni+bi NO	SVM baseline

Table 6 :

 6 Results of character based RNN using the best configurations for the three languages.

	Language	Emoji	AVG F1 Max F1 Standard deviation F1 Number of runs
	EN	NO	0.796	0.823	0.034	612
	EN	YES	0.797	0.821	0.009	576
	EN	Transcription	0.796	0.821	0.034	612
	IT	NO	0.764	0.804	0.021	468
	IT	YES	0.761	0.798	0.016	144
	IT	Transcription	0.763	0.805	0.021	468
	DE	NO	0.684	0.758	0.034	468
	DE	YES	0.678	0.745	0.028	288
	DE	Transcription	0.682	0.754	0.034	468

Table 7 :

 7 Mean and Standard deviation of macro averaged F1 scores without any specific processing of emojis ('NO'), using emojis obtained through alignment ('YES') and transcribing them ('TRANSCRIPTION')

Table 8 :

 8 Mean and Standard deviation of macro averaged F1 scores with and without social and emotion features

	Language Social & emotion features Mean F1 Max F1 Standard Deviation F1 Number of Runs
	EN	NO	0.794	0.823	0.011	300
	EN	YES	0.797	0.821	0.010	1500
	IT	NO	0.763	0.805	0.020	180
	IT	YES	0.763	0.805	0.021	900
	DE	NO	0.680	0.758	0.035	204
	DE	YES	0.682	0.754	0.033	1020

Table 9 :

 9 This led to the creation of a very challenging dataset, where lexical overlap between training and test data is limited (therefore unigram and bigram features do not work well) and where hate speech is not associated with specific topics or keywords. Performance evaluation on data sets of comparable size in English, Italian and German

	P R F1	AVG AVG AVG	0.763 0.769 0.765	0.794 0.811 0.801	0.726 0.690 0.700	0.763 0.780 0.770	0.789 0.796 0.793	0.712 0.680 0.689	0.767 0.783 0.773	0.780 0.785 0.783	0.736 0.703 0.713	0.780 0.792 0.785	0.767 0.766 0.767	0.732 0.710 0.718	0.776 0.791 0.782	0.755 0.767 0.760	0.741 0.702 0.713
	F1	HATE	0.683	0.739	0.578	0.696	0.723	0.563	0.701	0.708	0.598	0.713	0.684	0.611	0.711	0.683	0.596
	F1 NO	HATE	0.847	0.863	0.822	0.844	0.862	0.814	0.846	0.857	0.827	0.857	0.849	0.824	0.853	0.837	0.830
	LANG	EN	IT	DE	EN	IT	DE	EN	IT	DE	EN	IT	DE	EN	IT	DE
	HASH.	SPLIT		YES			YES			NO			YES			YES
	EMBED. TEXT FEATS SOCIAL EMOTIONS EMOJI NETWORK		Fasttext emb+uni NO NO transcription LSTM			Fasttext emb+uni NO NO NO LSTM			Fasttext emb NO NO NO GRU			Fasttext emb NO NO transcription LSTM			Fasttext emb NO NO NO LSTM

conduct_en.pdf

https://www.oeaw.ac.at/fileadmin/subsites/academiaecorpora/PDF/GermEval2018_ Proceedings.pdf

http://www.evalita.it/2018

http://www.di.unito.it/ ~tutreeb/sentipolc-evalita16/index.html

https://sites.google.com/view/ibereval-2018

http://www.di.unito.it/ ~tutreeb/haspeede-evalita18

https://www.oeaw.ac.at/ac/konvens2018/workshop/

https://fasttext.cc/docs/en/crawl-vectors.html

https://fasttext.cc/docs/en/english-vectors.html

https://fasttext.cc/docs/en/english-vectors.html

https://www.spinningbytes.com/resources/wordembeddings/

https://www.spinningbytes.com/resources/wordembeddings/

http://storage.googleapis.com/books/ngrams/books/datasetsv2.html

https://github.com/carpedm20/emoji

https://www.unicode.org/emoji/charts/full-emoji-list.html

http://creep-project.eu/

http://hatemeter.eu/

Acknowledgements

Part of this work was funded by the CREEP project 16 , a Digital Wellbeing Activity supported by EIT Digital in 2018 and 2019. This research was also supported by the HATEMETER project 17 within the EU Rights, Equality and Citizenship Programme 2014-2020, and by the ANR-19-P3IA-0002 -3IA Côte d'Azur -Nice -Interdisciplinary Institute for Artificial Intelligence.