Evan Dufraisse

Adrian Popescu

Julien Tourille

Armelle Brun

Jerome Deshayes

MAD-TSC: A Multilingual Aligned News Dataset for Target-dependent Sentiment Classification

Target-dependent sentiment classification (TSC) enables a fine-grained automatic analysis of sentiments expressed in texts. Sentiment expression varies depending on the domain, and it is necessary to create domain-specific datasets.

While socially important, TSC in the news domain remains relatively understudied. We introduce MAD-TSC, the first multilingual aligned dataset designed for TSC in news. MAD-TSC differs substantially from existing resources. First, it includes aligned examples in eight languages to facilitate a comparison of performance for individual languages, and a direct comparison of human and machine translation. Second, the dataset is sampled from a diversified parallel news corpus, and is diversified in terms of news sources and geographic spread of entities. Finally, MAD-TSC is more challenging than existing datasets because its samples are more complex. We exemplify the use of MAD-TSC with comprehensive monolingual and multilingual experiments. The latter shows that machine translations can successfully replace manual ones, and that performance for all included languages can match that of English by automatically translating test examples.

Introduction

Text analysis needs to address both objective aspects, such as topic extraction, and subjective aspects, such as sentiment and opinion classification. In spite of recent progress brought by the introduction of large language models [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Liu | Roberta: A robustly optimized bert pretraining approach[END_REF][START_REF] Yang | Xlnet: Generalized autoregressive pretraining for language understanding[END_REF], sentiment classification remains a challenging task. Expression of sentiments varies according to the data sources, languages and domain of the texts. These challenges are particularly important in targetdependent sentiment classification (TSC), which focuses on determining the sentiment expressed toward a given entity in a given context. The bulk of TSC-related research efforts are focused on major languages. This focus is an effect of the availability of generic and of task-specific resources in these languages [START_REF] Brauwers | A survey on aspect-based sentiment classification[END_REF][START_REF] Nazir | Issues and challenges of aspect-based sentiment analysis: a comprehensive survey[END_REF]. A majority of datasets are monolingual and, when they are multilingual [START_REF] Balahur | Comparative experiments using supervised learning and machine translation for multilingual sentiment analysis[END_REF][START_REF] Barriere | Improving sentiment analysis over non-English tweets using multilingual transformers and automatic translation for data-augmentation[END_REF][START_REF] Cortis | A dataset of multidimensional and multilingual social opinions for malta's annual government budget[END_REF][START_REF] Pontiki | SemEval-2016 task 5: Aspect based sentiment analysis[END_REF][START_REF] Severyn | Multilingual opinion mining on youtube[END_REF], the examples are not aligned across languages. Equally, a majority of existing datasets and methods are devised for texts such as tweets, reviews or comments [START_REF] Nakov | SemEval-2016 task 4: Sentiment analysis in Twitter[END_REF][START_REF] Pontiki | SemEval-2016 task 5: Aspect based sentiment analysis[END_REF][START_REF] Severyn | Multilingual opinion mining on youtube[END_REF] in which sentiment is most often expressed explicitly. Somewhat surprisingly, less attention is given to TSC in news, despite the usefulness of an automatic analysis of their content for the understanding of societally impactful processes such as disinformation or polarization [START_REF] Hamborg | News-MTSC: A dataset for (multi-)target-dependent sentiment classification in political news articles[END_REF].

Our main contribution is the introduction of MAD-TSC, the first large multilingual aligned dataset for TSC in news articles. It includes 5,110 annotated entity mentions from 4,714 unique sentences. Each sentence has professionally-translated and aligned versions in eight languages (English, Spanish, German, Italian, French, Portuguese, Dutch, and Romanian). Sentences originate from 286 news sources published in over 30 countries. These characteristics differentiate the proposed dataset from existing resources, and in particular from NewsMTSC [START_REF] Hamborg | News-MTSC: A dataset for (multi-)target-dependent sentiment classification in political news articles[END_REF], a monolingual dataset focused on American politics which is the closest to MAD-TSC. We first present the dataset creation process, and provide a qualitative analysis of its content. Then, we propose a thorough evaluation of state-of-the-art TSC methods on MAD-TSC in monolingual and multilingual settings, with particular focus on the usability of machine translation in TSC. The main findings are the following: • The proposed dataset is more challenging since it includes more complex examples compared to existing resources, as detailed in Subsection 3.5.

• Performance for individual languages varies due to the fact that the quality of available pretrained language models is itself variable, with the best scores being obtained for English. • Results with machine translation of training sets from English toward target languages and with manual translations are on par. • The performance level for other languages matches that of English by translating test examples to English in order to take advantage of the pretrained language models available in this language. The same is true when both the training and test sets are automatically translated to English.

The last two findings are particularly interesting since they show that if a domain-specific TSC dataset is available, it can be effectively used for multiple languages. Overall, the introduction of MAD-TSC will facilitate progress in multilingual TSC. The dataset and the full evaluation protocol are available online 1 to encourage further research, and to ensure reproducibility.

Related work

Target-dependent sentiment classification (also named aspect-based sentiment analysis [START_REF] Nazir | Issues and challenges of aspect-based sentiment analysis: a comprehensive survey[END_REF] or classification [START_REF] Brauwers | A survey on aspect-based sentiment classification[END_REF]) is a complex task due to the numerous factors which influence the way sentiments are expressed in texts [START_REF] Brauwers | A survey on aspect-based sentiment classification[END_REF], such as the language, the domain or the personal biases of the author/reader.

TSC is often evaluated on short texts such as tweets [START_REF] Nakov | SemEval-2016 task 4: Sentiment analysis in Twitter[END_REF], reviews [START_REF] Pontiki | SemEval-2016 task 5: Aspect based sentiment analysis[END_REF] or comments [START_REF] Severyn | Multilingual opinion mining on youtube[END_REF]. A common characteristic of these resources is that sentiment is often expressed in an explicit way. News are more challenging texts because sentiments are expressed implicitly or indirectly [START_REF] Hamborg | News-MTSC: A dataset for (multi-)target-dependent sentiment classification in political news articles[END_REF], often include multiple targets in a single sentence [START_REF] Brauwers | A survey on aspect-based sentiment classification[END_REF], and both negative and positive arguments about a target entity are combined due to the fact that journalists are supposed to be objective [START_REF] Balahur | Sentiment analysis in the news[END_REF][START_REF] Hamborg | Automated identification of media bias in news articles: an interdisciplinary literature review[END_REF][START_REF] Liu | Sentiment analysis and subjectivity[END_REF].

Multilinguality is important in order to be able to analyze texts in different languages. Multilingual datasets are proposed for tweets (Lam-1 https://github.com/EvanDufraisse/MAD_TSC pert and [START_REF] Lampert | Overcoming rare-language discrimination in multi-lingual sentiment analysis[END_REF][START_REF] Vilares | Supervised sentiment analysis in multilingual environments[END_REF]), reviews (Jiménez Zafra et al., 2015;[START_REF] Pontiki | SemEval-2016 task 5: Aspect based sentiment analysis[END_REF] and institutional texts [START_REF] Cortis | A dataset of multidimensional and multilingual social opinions for malta's annual government budget[END_REF]. While interesting, these datasets differ from MAD-TSC because they do not focus on news. Equally important, they are only aligned at a domain level, but not at an example level. This limits their utility in terms of comparative evaluation in monolingual and multilingual settings. Multilingual approaches were also explored for news representation. For instance, bilingual word embeddings were used to compensate data scarcity in under-resourced languages [START_REF] Md | Solving data sparsity for aspect based sentiment analysis using cross-linguality and multi-linguality[END_REF] or to transfer models between source and target languages in zero-shot settings [START_REF] Jebbara | Zero-shot cross-lingual opinion target extraction[END_REF].

Classical TSC methods rely on engineered features based on lexicons and syntactic analysis [START_REF] Biber | Styles of stance in english: Lexical and grammatical marking of evidentiality and affect[END_REF][START_REF] Baccianella | SentiWordNet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining[END_REF][START_REF] Jiang | Target-dependent twitter sentiment classification[END_REF][START_REF] Kiritchenko | Nrc-canada-2014: Detecting aspects and sentiment in customer reviews[END_REF][START_REF] Vo | Target-dependent twitter sentiment classification with rich automatic features[END_REF]. Strong progress in TSC was made possible by the introduction of deep language models, such as BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Zeng | Lcf: A local context focus mechanism for aspect-based sentiment classification[END_REF]. Improvements are obtained when pretraining includes a larger proportion of news (Hamborg and Donnay, 2021) -this is the case for English RoBERTa [START_REF] Liu | Roberta: A robustly optimized bert pretraining approach[END_REF] or XLNET [START_REF] Yang | Xlnet: Generalized autoregressive pretraining for language understanding[END_REF] -or when including an intermediate tuning on domain-related data [START_REF] Du | Adversarial and domain-aware BERT for cross-domain sentiment analysis[END_REF]. Naturally, further improvements are obtained by introducing TSC-specific architectures [START_REF] Brauwers | A survey on aspect-based sentiment classification[END_REF][START_REF] Nazir | Issues and challenges of aspect-based sentiment analysis: a comprehensive survey[END_REF][START_REF] Zhou | Deep learning for aspect-level sentiment classification: survey, vision, and challenges[END_REF]. We follow this trend and use pretrained language models in our experiments.

To our knowledge, there are only four datasets dedicated to TSC in news. The first one [START_REF] Balahur | Sentiment analysis in the news[END_REF] has 1,592 examples, and includes only quotes from political news. Quotes are interesting because they include a lot of sentiment expressions, but they are also easier since sentiment is often expressed explicitly [START_REF] Hamborg | News-MTSC: A dataset for (multi-)target-dependent sentiment classification in political news articles[END_REF]. The second one [START_REF] Steinberger | Large-scale news entity sentiment analysis[END_REF] has 1,274 examples. The size of these datasets makes them difficult to use with deep-learningbased TSC methods. The third dataset [START_REF] Hamborg | Automated identification of media bias in news articles: an interdisciplinary literature review[END_REF] includes 3,002 examples. However, as noted later by its authors [START_REF] Hamborg | News-MTSC: A dataset for (multi-)target-dependent sentiment classification in political news articles[END_REF], the dataset is imbalanced and its sentiment expressions are predominantly explicit. The dataset which is closest to ours is NewsMTSC [START_REF] Hamborg | News-MTSC: A dataset for (multi-)target-dependent sentiment classification in political news articles[END_REF]. Their common characteris-tics include: a focus on political news, an identical definition of the task, and a similar annotation process. Importantly, we follow [START_REF] Hamborg | News-MTSC: A dataset for (multi-)target-dependent sentiment classification in political news articles[END_REF] and instruct annotators to think from the author's perspective in a holistic way. They are instructed to consider the "what" of the sentence (events, facts) but also the "how" (choice of words, author's attitude). This choice contrasts with previous works [START_REF] Balahur | Sentiment analysis in the news[END_REF][START_REF] Steinberger | Large-scale news entity sentiment analysis[END_REF], which distinguish author-and reader-levels in TSC, and is important in order to minimize the influence of personal biases. The main differences between MAD-TSC and NewsMTSC relate to multilingualism, complexity of examples, political topics and geography of examples, while maintaining the same order of magnitude in the number of samples, with 5,110 for MAD-TSC , and 11,361 for NewsMTSC. These differences are detailed in Subsection 3.5, and they make MAD-TSC appropriate for a thorough evaluation of TSC in multilingual settings.

Dataset Construction

We build on previous works devoted to the creation of sentiment classification datasets [START_REF] Nakov | SemEval-2016 task 4: Sentiment analysis in Twitter[END_REF][START_REF] Pontiki | SemEval-2016 task 5: Aspect based sentiment analysis[END_REF], and particularly of TSC ones [START_REF] Balahur | Sentiment analysis in the news[END_REF][START_REF] Hamborg | News-MTSC: A dataset for (multi-)target-dependent sentiment classification in political news articles[END_REF][START_REF] Steinberger | Large-scale news entity sentiment analysis[END_REF]. We describe the main steps of the dataset creation process and analyze the resulting dataset.

Data Sources

Our objective is to create a dataset which: (1) is multilingual and aligned at a sentence level across languages to enable a comprehensive evaluation of TSC, (2) includes content from a large number of high-quality journalistic sources, which offer a diversified view of the included topics, (3) covers societally-impactful topics in different countries. Voxeurop2 is a multilingual news website which aims to offer interesting and high-quality news to European audiences. The project inherits from Presseurop, which was active from 2009 to 2013, and whose objective was to make Europe-related news from over 200 sources available. Voxeurop and Presseurop articles are available in up to ten languages. The content is translated by professional translators, thus ensuring high-quality texts in all available languages. The content is published using a Creative Commons BY-NC-ND, an open license which facilitates its redistribution and reuse for non-commercial purposes, which will be also used to distribute MAD-TSC. We have collected 7,370 news articles which have translations in all eight languages, amounting to a total of 122,263 sentences in English and comparable numbers in other languages. A wide majority of the examples are related to politics, with the others pertaining to business, culture and society. Most of the entities mentioned in articles refer to prominent public figures from different European countries at the time of publication (2009)(2010)(2011)(2012)(2013). Well-represented political sub-topics include: the economic crisis which started in 2008, European Union evolution process, election-related news, political crisis at a national level, and corruption scandals in different countries.

Sample Selection

Named entity detection was performed using the Flair model for English [START_REF] Akbik | Contextual string embeddings for sequence labeling[END_REF], which led to an initial pool of 30,303 sentences with at least one mention of a person. We combine entity linking with Blink [START_REF] Wu | Scalable zeroshot entity linking with dense entity retrieval[END_REF] and coreference resolution with neuralcoref3 to obtain reliable entity counts in articles. Entities which are mentioned a single time in an article are not kept for annotation because they are not considered in focus. This filtering led to 19,223 candidate sentences. The alignment of sentences for the eight languages is inspired by lingtrain4 . It is based on sentence embeddings from a multilingual-sentence BERT model [START_REF] Yang | Multilingual universal sentence encoder for semantic retrieval[END_REF], with English as source and the other languages as targets. Two matching criteria are used: (1) the need for reciprocal best matching (inter-match) between source and target sentences, and (2) a cosine similarity threshold of 0.5. Both criteria need to be met for all language pairs in order to select a sentence. Automatic alignment was manually checked for three languages (EN, FR, RO), with a sample of 1,000 examples. It was correct in 98.1% of cases. The remaining imperfections usually relate to additional context being provided by the translator in one of the languages. This does not affect the sentiment expressed about the target entity, and the obtained alignments of sentences are usable.

Following sentence alignment, it is also necessary to align entity mentions across languages, and we used a rule-based approach for this task. NFKD unicode normalization is first applied to examples in all languages. Then we computed a normalized Levenshtein distance between the English mention of the entity and the words from the target sentence. A similarity threshold of 80% between the English and the target mention in any of the other languages was used. To add flexibility to the matching process, we also considered nearly contiguous sequences as valid matches. We have checked this matching and it is correct in over 99% of cases on a subset of 1,000 mentions.

The sentiment classes are not evenly distributed in news, and we followed an initial selection procedure which is inspired by the one introduced in [START_REF] Hamborg | News-MTSC: A dataset for (multi-)target-dependent sentiment classification in political news articles[END_REF]. It involves an undersampling of potentially neutral mentions as predicted by a simple binary classifier. This led to a pool of 11,000 examples which were selected and proposed to annotators.

Sample Annotation

Annotations were crowdsourced using a custom web interface. Aware of the challenges of news annotations [START_REF] Balahur | Sentiment analysis in the news[END_REF] (i.e. low interannotator agreement and low suitability), [START_REF] Hamborg | News-MTSC: A dataset for (multi-)target-dependent sentiment classification in political news articles[END_REF] devised a process in which participants are asked to annotate following the news author's perspective. While some news articles express sentiments in a manner which is easy to recognize, others convey them in an intricate and/or implicit manner. Equally, the sentiment often depends on text parts distant from the target entity, which are not included in the text presented for annotations. The annotation guide made the annotators aware of the complexity of the task. They were presented with examples of sentences which include intricate and/or implicit sentiments expressions, as well as irony. Annotators also had the possibility to label examples as unknown whenever they could not determine the label of an example.

TSC annotations are usually collected using 3-, 5-or 7-points Likert scales [START_REF] Balahur | Sentiment analysis in the news[END_REF][START_REF] Hamborg | News-MTSC: A dataset for (multi-)target-dependent sentiment classification in political news articles[END_REF][START_REF] Nakov | SemEval-2016 task 4: Sentiment analysis in Twitter[END_REF][START_REF] Pontiki | SemEval-2016 task 5: Aspect based sentiment analysis[END_REF]. Following an initial experiment which involved 50 sentences, we opted for a 5-points scale which offers a good balance between annotation simplicity and expressiveness. Possible labels were: negative, weakly negative, neutral, weakly positive and positive. The annotation was supported by a Web interface which is illustrated in Appendix E. Examples were provided in English, French and Romanian to facilitate the annotation.

Annotations were provided by a total of 21 volunteer participants, whose demographics are presented in Appendix D. They were recruited via a call for participation which was circulated via group and personal e-mails. Participants provided explicit consent to use their annotations and demographic data at the beginning of the experiment. The choice to work with volunteers is motivated by the fact that crowdsourcing performance is similar for paid and volunteered participation [START_REF] Mao | Volunteering versus work for pay: Incentives and tradeoffs in crowdsourcing[END_REF]. Samples were presented randomly in order to avoid any ordering effect, and users were free to stop at any point. Each sample was labeled by three annotators in order to allow annotation consolidation. All users speak at least two of the three languages used in the annotation interface.

Label Aggregation

Since the annotation task is prone to disagreement [START_REF] Hamborg | News-MTSC: A dataset for (multi-)target-dependent sentiment classification in political news articles[END_REF][START_REF] Steinberger | Large-scale news entity sentiment analysis[END_REF], a consolidation of annotations is necessary. We first removed all samples for which there was at least one "unknown" label. Following [START_REF] Hamborg | News-MTSC: A dataset for (multi-)target-dependent sentiment classification in political news articles[END_REF], we reduced the five initial labels to three classes (negative, neutral and positive) by aggregating the two possible labels for the negative and positive sentiments. Finally, we kept only samples for which there was an unanimous voting or majority agreement with a third vote in a neighboring class (for instance two positive, one neutral). The inter-rater reliability, measured using Fleiss' kappa [START_REF] Fleiss | Measuring nominal scale agreement among many raters[END_REF], reaches K F = 0.58 and K F = 0.67, before and after consolidation, respectively. The two values indicate that the task is challenging, but the final reliability score corresponds to good agreement [START_REF] Hallgren | Computing inter-rater reliability for observational data: an overview and tutorial[END_REF]. This consolidation strategy leads to a coherent labeling of MAD-TSC, and is used in experiments.

MAD-TSC Analysis

MAD-TSC includes a total of 5,110 labeled target entity mentions for all eight languages, with 1,839, 2,011, 1,260 of them labeled as negative, neutral and positive, respectively. There can be more than one target entity per example, and there are 4,714 unique examples in MAD-TSC.

Figure 1 shows the total number of unique linkable entities, grouped by source language, from the original corpus of articles used to design MAD- TSC . These sources are generally correlated to journals from different countries, indicating that the proposed dataset comes from a variety of sources. The total number of unique linkable entities aggregated across all source languages is lower, with 1,007 entities. This is in contrast with NewsMTSC, which is sampled only from American newspapers.

We illustrate the content of the dataset by a number of quantitative and qualitative characteristics. We first present example-related statistics for MAD-TSC, and compare them to NewsMTSC (Hamborg and Donnay, 2021). These authors use sentence length as a proxy for the complexity of the dataset, and showed that texts in NewsMTSC are longer than those from previous datasets. We follow their approach and report the statistics regarding the number of characters for English examples included in MAD-TSC. The mean is 192.3 characters (72.1 stdev), while the corresponding value for NewsMTSC is 152.2 (109.1 stdev). The number of words per example is a related measure, but more informative from a semantic perspective. MAD-TSC examples include a mean of 31.1 words (11.6 stdev), while the corresponding values are 25.2 (15.5 stdev) for NewsMTSC. MAD-TSC can thus be considered as more complex than NewsMTSC.

A second analysis focuses on entities which can be linked to English Wikipedia articles using Blink [START_REF] Wu | Scalable zeroshot entity linking with dense entity retrieval[END_REF]. MAD-TSC contains 1,007 distinct linkable entities, with a mean of 6.3 mentions per entity (28.9 stdev). NewsMTSC includes 975 linkable entities, 5.5 mentions per entity (43.9 stdev). Donald Trump, Hillary Clinton, and Barack Obama, the top-3 entities from NewsMTSC cover 20.1%, 11.7% and 7.7% of mentions, respectively. In MAD-TSC, Angela Merkel, Silvio Berlusconi, and Nicolas Sarkozy cover 10.2%, 4.9%, and 3.5% of mentions, respectively. We conclude that MAD- TSC has a distribution of entity mentions which is less skewed.

Third, we examine the geographic distribution of linkable entities in both datasets. The obtained distributions are presented in Figure 2, and they confirm that MAD-TSC is much more diversified than NewsMTSC from a geographical point of view.

Finally, we present clouds of frequent words in the two datasets in Figure 3. This qualitative representation of the two datasets confirms that the main topics are different, with focus on European and on American topics, respectively.

Experiments

Compared TSC Methods

For English, we compare the performance of four existing TSC methods using English and multilingual pretrained language models. Whenever available, RoBERTa [START_REF] Liu | Roberta: A robustly optimized bert pretraining approach[END_REF] backbone is preferred due to its better performance for TSC in news [START_REF] Hamborg | News-MTSC: A dataset for (multi-)target-dependent sentiment classification in political news articles[END_REF]. Otherwise, a BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] [START_REF] Gao | Target-dependent sentiment classification with BERT[END_REF] only considers the last hidden states of the entities' tokens and merges their representation through a max-pooling layer. • PM [START_REF] Seoh | Open aspect target sentiment classification with natural language prompts[END_REF]) employs a prompt model for TSC. Our implementation of this method uses the simple prompt "<entity> is [mask]" with ["good", "ok", "bad"] proposed in the original paper as verbalizer. This prompt is translated in We implement the efficient fine-tuning process introduced by [START_REF] Mosbach | On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines[END_REF] to optimize our TSC models. This type of optimization was successfully used in a TSC context [START_REF] Seoh | Open aspect target sentiment classification with natural language prompts[END_REF].

We fix the learning rate to 2e-5 and train with earlystopping conditions up to 40 epochs using AdamW optimizer, and batches of size 32. Initial experiments confirmed that this approach is better than the more classical hyperparameter search used in BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF]. Details about the optimization process are provided in Appendix B. All results are reported by averaging scores of five runs launched with different seeds. Pytorch [START_REF] Paszke | Pytorch: An imperative style, high-performance deep learning library[END_REF] is used for all implementations.

Dataset Splits and Metrics

We run experiments with MAD-TSC in monolingual and multilingual settings, and also use News-MTSC for experiments in English. The training/validation/test subsets are sampled randomly and include 3,810/300/1,000 labeled mentions, respectively. Results for NewsMTSC are reported with the official splits from [START_REF] Hamborg | News-MTSC: A dataset for (multi-)target-dependent sentiment classification in political news articles[END_REF]. TSC evaluation can be performed with different metrics [START_REF] Hamborg | News-MTSC: A dataset for (multi-)target-dependent sentiment classification in political news articles[END_REF][START_REF] Nakov | SemEval-2016 task 4: Sentiment analysis in Twitter[END_REF][START_REF] Pontiki | SemEval-2016 task 5: Aspect based sentiment analysis[END_REF], we use macro F1 (F 1 m ) on all classes as primary metric. Performance with other metrics, such as F1 only on positive and negative classes (F 1 pn ), accuracy (acc), and average recall (rec) follow similar trends. A selection of such results, along with standard deviations are reported in Appendix C. 1: F 1 m results for individual train and test sets of the English subset of MAD-TSC (MAD) and News-MTSC (NM), and for their combination (MIX). TSC methods are applied on top of language models pretrained specifically for English (EN) or with a multilingual corpus (ML).

Experiments with MAD-TSC and NewsMTSC

We compare the TSC methods from Subsection 4.1 on both the English subset of MAD-TSC and on NewsMTSC. Models are trained and evaluated on each dataset and on their combination. Results with different train and test set combinations are reported in Table 1. Regardless of the specific combination, F 1 m scores obtained with SPC, TD and PM are similar. The fact that BASE has lower performance (approximately 6 and 3 points for News-MTSC and MAD-TSC, respectively) confirms that the need to provide the target entity in TSC. The performance of SPC and TD obtained when training and testing on NewsMTSC is 3 to 4 points better than the one originally reported in [START_REF] Hamborg | News-MTSC: A dataset for (multi-)target-dependent sentiment classification in political news articles[END_REF]. This is probably due to a better parametrization of these two methods here. SPC and TD scores are even on a par with those of the more complex GRU method [START_REF] Hamborg | News-MTSC: A dataset for (multi-)target-dependent sentiment classification in political news articles[END_REF], which needs an external knowledge source that is not available for languages other than English. In addition, results confirm that MAD- Table 2: F 1 m results for the eight languages included in MAD-TSC. SPC is applied on top of models pretrained specifically for each target language (T G) or with a multilingual corpus (M L) using SPC.

TSC is more challenging than NewsMTSC. The transfer of the models trained on one dataset toward the other test set gives suboptimal results. The combination of the two train sets has a marginal positive effect on each test set. Finally, the language model pretrained specifically for English is clearly better than its multilingual counterpart, probably due to the curse of multilinguality which affects multilingual models [START_REF] Conneau | Unsupervised cross-lingual representation learning at scale[END_REF].

Experiments with Individual Languages

Results for the eight MAD-TSC languages are presented in Table 2. They are reported using SPC, a commonly used TSC method [START_REF] Cao | Aspect is not you need: No-aspect differential sentiment framework for aspect-based sentiment analysis[END_REF][START_REF] Hamborg | News-MTSC: A dataset for (multi-)target-dependent sentiment classification in political news articles[END_REF][START_REF] Seoh | Open aspect target sentiment classification with natural language prompts[END_REF][START_REF] Song | Attentional encoder network for targeted sentiment classification[END_REF], and its performance is close to that of PM and TD in Table 1. The best F 1 m scores are obtained for English and French, and the lowest scores are reported for Dutch and Spanish. When using monolingual pretraining (T G), the difference between the best and worst scores is over 10 points. In contrast, the results obtained with multilingual pretraining (M L) are much more similar across all languages. Performance variability is explained by the quality of pretrained models, and in particular by the size of the datasets and that of the subsets relevant for politics. Interestingly, the multilingual pretraining is much better than the language-specific one for Dutch and Spanish, and is also better for Romanian. Inversely, monolingual models are clearly better for English and French, that are the two languages which have the best monolingual pretraining. The results from Table 2 indicate that strong monolingual pretraining is preferable in TSC, but it can be successfully replaced by multilingual pretraining when the dataset for a particular language is insufficient.

Experiments with Machine Translation

Machine translation (MT) has strongly progressed in recent years, notably due to the introduction of neural architectures [START_REF] Stahlberg | Neural machine translation: A review[END_REF]. A successful deployment of MT for sentiment classification would greatly facilitate the task in the multilingual setting because it would reduce, or even remove, the need for specific annotations in each language. Building on previous works which apply MT to TSC [START_REF] Balahur | Comparative experiments using supervised learning and machine translation for multilingual sentiment analysis[END_REF]Mohammad et al., 2016), we report results with English as pivot language. Test and/or train subsets of the other languages are translated to English. The translation is performed with two methods: (1) M2M100 [START_REF] Fan | Beyond english-centric multilingual machine translation[END_REF], a recent massively multilingual translation model, by using the largest available model (12B parameters); (2) the API of DeepL5 , a wellknown commercial machine translation service.

The F 1 m scores obtained with different MT configurations are reported in Table 3. The results are very interesting, particularly when translating the test set to English with DeepL (row with EN train and EN DL test). F 1 m scores are globally better than 72.3, the performance of SPC obtained with manual translations for English in Table 1. The maximum gain is 1.6 points for Spanish, and Dutch is the only language for which DeepL translations are slightly worse (-0.2 points). F 1 m is also interesting with M2M100, albeit lower than that of EN DL . These results surprised us initially, but we reach the same conclusions after running the experiments a second time in an independent manner. When translating the test set, all languages benefit from the strong pretraining available for English, and strong performance can be obtained for them if a good translation model is available from the target language toward English. This condition is met for all languages included in MAD-TSC . Qualitatively, the findings reported here could be explained by the fact that, while the polarity of sentiments is preserved by both machine and human translators, professional translators are more creative and sometimes add context in sentences for their international public. While useful for human readers, these changes can have a slight negative influence on sentiment classification.

Results are also interesting when the English training set is translated toward the target languages using DeepL and M2M100 (rows with T G DL / T G M 2M train and T G test). The associated F 1 m scores are on par with those obtained with the manual translations. However, the translation of training sets is less effective than that of test sets. This happens because TSC training is done in languages other than English, and is based on weaker pretrained language models. We also translated both the training and test sets from Dutch and Romanian to English, two of the languages which have low performance in Table 2, using DeepL. The F 1 m scores for the two languages are 72.3 and 73.6, respectively. We conclude that interesting performance can be obtained by automatically translating TSC datasets from other languages to English.

Analysis of Example Complexity

We complement the quantitative results by a qualitative analysis of factors which influence TSC performance. The analysis is done for English, and findings are similar for the other languages.

We first test the hypothesis that the complexity of examples is correlated to their length [START_REF] Hamborg | News-MTSC: A dataset for (multi-)target-dependent sentiment classification in political news articles[END_REF]. We split sentences in five subsets depending on the number of words per example and report F 1 m per subset: 75.0 for up to 20 words per sentence; 72.9 for 21 to 30 words; 70.9 for 31 to 40 words; 70.0 for 41 to 50 words; 70.0 for 50 words and more. These scores confirm that TSC difficulty increases with example length, but differences become smaller above 30 words.

The number of entities detected in each example is an interesting proxy for complexity since the expressed sentiments can vary for multiple entities. We compute F 1 m separately for examples which include 1, 2, 3 or more detected entities. The scores obtained for the three subsets are 73.6, 70.1, and 67.4, respectively. They confirm that the presence of more entities makes TSC more difficult.

Ablation of Training Set

The annotation of TSC datasets is a cumbersome task, and it is important to minimize this effort, while preserving the final performance. In Figure 4, we present the results obtained by sampling 1,000, 2,000, 3,000 and 4,000 training examples from NewsMTSC and MAD-TSC, and with the full datasets. F 1 m scores increase up to 3,000 samples, but the added benefit of more samples is reduced beyond this dataset size. The trend is similar for MAD-TSC , and the results reported in Figure 4 indicate that its size is sufficient to tackle the TSC task effectively.

Conclusion

We introduce MAD-TSC, a dataset for multilingual target-dependent sentiment classification. Compared to existing resources, the proposed dataset is aligned across languages, includes examples about geographically diversified entities. Examples are longer and more complex because sentiment is often expressed in an implicit way. Given its aligned character, MAD-TSC dataset enables a comparison of sentiment classification between languages. Performance varies significantly, and this variation is to a large extent explained by the quality of pretrained models available for each language.

Importantly, the MT experiments show that human translations can be replaced by automatic ones. The automatic translation of test sets from target languages to English is particularly interesting since it brings target-dependent sentiment classification in different languages to the same quality level as that of English. This allows TSC to be scaled for the languages included in this study without the need to develop language-specific training sets. The only condition is to have a labeled domain-related dataset in one language, English or other.

Future work will focus on limitations of the dataset: (1) the handling of example complexity, (2) the coverage of the dataset in terms of domains and entity types, (3) the number of included languages, and (4) the quality of pretrained language models. These limitations are discussed in more details in the dedicated section below.

Limitations

The analysis from Subsection 4.6 shows that the number of entities per example has an important influence on results. For now, the handling of examples with more than one entity includes the detection of the mention, but does not consider other criteria. It would be useful to adapt the TSC methods in order to determine whether the sentiment about all entities is expressed by the same part of the example or not. If sentiment is expressed in different parts of the example, a splitting of the example into parts which express the sentiment about each entity would prove useful.

Despite a more diversified coverage of the political domain compared to NewsMTSC, MAD-TSC remains focused on politics. It would be interesting to include other major news domains, such as environment, business, culture, technology, sports, etc. Equally important, all targets from MAD-TSC are person names. It would be useful to also include sentiment expressed about other types of named entities (organizations, locations, events, etc.), as well as other polarization-prone concepts in each domain. Such extensions of the dataset would provide a more complete view of TSC performance. Ultimately, they would make the analysis of sentiments expressed in news articles more comprehensive and reliable.

While MAD-TSC is the first multilingual aligned dataset designed for TSC in news, it would benefit from the inclusion of more languages, including non-European ones. This limitation is due to the unavailability of massively multilingual and aligned news datasets which could be used to include more languages. A potential solution to overcome these limitations would be to enrich the dataset with manual translations in other languages. However, this solution is costly and is left for future work.

Finally, the comparison of results between languages is biased because the effectiveness of available pretrained language models is variable. This is a limitation which is shared by most NLP works which focus on multilingual datasets and reuse pretrained models, themselves trained on whatever datasets available for each language.

Ethics Statement

The work presented here is part of a project which was reviewed and approved by our institution's ethical committee. This committee provided useful guidance regarding this specific work concerning the selection of news sources, and the annotation process. The recommendations were integrated in the dataset creation process.

One potential issue is that the dataset includes negative sentiments expressed about public figures. This is also applicable to any TSC dataset that focus on politics. Sentiment expressions were collected from publicly available news sources which have a right to freedom of expression in the European Union (EHCR Article 10). News articles were collected from diversified newspapers, which lean toward different parts of the political spectrum, and this reduces the risk of mischaracterizing any of the mentioned entities. Sentiment classification datasets are needed in order to understand how sentiment is expressed in the media, and thus contribute to the characterization of societal debates.

A Language models

All models used in this work are publicly available in the Hugging Face models repository. The main characteristics of the pretrained models used in our experiments are presented in Table 4. Considering the improvements brought by RoBERTa [START_REF] Liu | Roberta: A robustly optimized bert pretraining approach[END_REF] pretraining over subsequent fine-tuning tasks compared to BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF], models that replicate RoBERTa in another language are favored when possible. All models are cased versions. Those models are between 100M and 150M of parameters. Choice of training hyperparameters have been made resulting from the reading of [START_REF] Mosbach | On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines[END_REF]. However, we persist in the use of a validation set but add a loose early-stopping condition of 10 epochs without loss improvement. Results reported are for the model with the best validation loss found during training. Code will be made available upon publication for further implementation details.

The fine-tuning of a pretrained model for TSC takes approximately 3 hours on a Nvidia A100 card. We fine-tuned 52 models with 5 different seeds for each configuration. This leads to a budget of 780

Figure 1 :

 1 Figure 1: Distribution of linkable entities per original language of source article. "other" merges rare source languages. Linking is done with Blink (Wu et al., 2020).

Figure 2 :

 2 Figure 2: Distribution of linkable entities from MAD-TSC (top) and NewsMTSC (bottom) per country. The country associated to each entry is the most frequent one from the Wikipedia article of the entity.

Figure 3 :

 3 Figure 3: Frequent word clouds for MAD-TSC and NewsMTSC corpora (minimum frequency 20 occurrences).

Figure 4 :

 4 Figure 4: F 1 m scores with training sets of variable size for NewsMTSC (NM) and MAD-TSC (MAD). Results are reported with monolingual pretraining and SPC. The rightmost points represent scores for the full dataset.

Figure 5 :

 5 Figure 5: Top part of the instructions and registration page.

Figure 6 :

 6 Figure 6: Bottom part of the instructions and registration page.

Figure 7 :

 7 Figure 7: Page for an example annotation.

  backbone is selected. Details about pretrained models used in experiments are provided in Appendix A. A short description of the TSC methods is provided below: • SPC[START_REF] Song | Attentional encoder network for targeted sentiment classification[END_REF] is based on the classical

	Sentence Pair Classification task of Bert mod-
	els. The input is designed as "[CLS] <sentence>
	[SEP] <entity> [SEP]".
	• TD

  BASE EN 76.7 64.0 68.8 69.9 76.7 71.0 BASE M L 74.1 59.2 66.5 64.8 74.1 66.7

	Train	NM	MAD	MIX
	Test	NM MAD NM MAD NM MAD
	SPC EN SPC M L TD EN TD M L PM EN PM M L	83.2 67.1 76.7 72.3 83.4 72.7 81.0 61.5 70.5 67.8 80.3 67.9 83.6 69.7 75.2 73.2 83.0 74.1 81.6 63.3 71.9 68.5 80.4 68.4 83.6 67.3 77.3 72.8 82.6 72.2 81.7 60.9 69.3 66.6 82.0 67.0
	Table

Table 4 :

 4 List of pretrained models, with associated language, base architecture, URL and reference when available.

	The M2M model was downloaded from this repository 6 .
	B Training details
	Hyperparam	Fine-tuning
	Learning Rate	2e-5
	Batch Size	32
	Weight Decay	0.01
	Warmup	0.06
	Max Epochs	40
	Adam ϵ	1e-6
	β 1 β 2 Optimizer	0.9 0.98 AdamW
	Seeds	[42, 302, 668, 745, 343]

Table 5 :

 5 Values of hyperparameters used for training TSC models.

Table 6 :

 6 Scores and standard deviations of F 1 m , F 1 pn , an accuracy (acc) and macro averaged recall (rec) scores for different configurations train/test configurations of MAD-TSC (MAD) and NewsMTSC (NM). These results complement those presented in Table1of the main text.

	Model	Train	Test F 1 m std F 1m F 1 pn std F 1pn acc rec
	NM NM MAD 67.1 NM 83.2 NM NM 81.0 NM MAD 61.5 NM NM 83.7 NM MAD 69.7 NM NM 81.6 NM MAD 63.3 NM NM 83.6 NM MAD 67.3 NM NM 81.7 NM MAD 60.9 NM NM 76.8 NM MAD 64.0 NM NM 74.1 NM MAD 59.2 MAD NM 76.7 MAD MAD 72.3 MAD NM 70.5 MAD MAD 67.8 MAD NM 75.2 MAD MAD 73.2 MAD NM 71.9 MAD MAD 68.5 MAD NM 77.3 MAD MAD 72.8 MAD NM 69.3 MAD MAD 66.6 68.8 BASE EN MAD NM SPC EN SPC EN SPC M L SPC M L TD EN TD EN TD M L TD M L PM EN PM EN PM M L PM M L BASE EN BASE EN BASE M L BASE M L SPC EN SPC EN SPC M L SPC M L TD EN TD EN TD M L TD M L PM EN PM EN PM M L PM M L BASE EN MAD MAD 69.9 66.5 BASE M L MAD NM BASE M L MAD MAD 64.8	0.4 2.4 0.3 1.3 0.7 1.0 1.0 1.5 0.6 1.3 1.0 1.9 0.7 0.8 0.7 1.5 1.0 0.9 1.2 1.6 2.3 0.9 1.2 1.1 0.8 0.9 4.8 3.1 0.6 0.9 1.2 1.1	83.0 70.9 80.3 64.7 83.4 72.8 80.8 65.5 83.1 70.8 81.1 63.7 76.6 68.1 74.0 62.8 76.6 72.5 69.4 66.9 74.6 73.5 70.0 66.6 77.1 72.8 67.3 64.7 68.6 70.3 65.1 62.9	0.8 1.3 0.5 1.3 0.8 0.8 1.2 1.4 1.0 1.0 1.3 1.3 0.7 1.2 1.2 0.8 1.7 1.4 1.9 1.7 2.7 1.7 2.8 2.0 1.5 1.6 6.5 4.6 1.4 1.3 1.9 2.2	84.1 83.2 68.1 69.5 81.6 81.1 61.8 63.2 83.8 83.0 68.8 70.3 81.5 80.9 62.9 64.3 84.1 83.5 67.8 69.7 81.2 80.3 61.7 62.5 76.2 75.3 65.8 66.7 74.6 74.3 60.5 62.3 77.3 75.7 73.6 72.5 72.8 71.0 68.6 68.0 77.7 76.1 74.2 73.5 75.0 72.6 69.8 68.5 76.9 75.0 73.7 71.9 75.0 73.5 69.3 68.8 69.4 68.2 71.7 70.9 67.7 66.1 66.9 65.9

Table 7 :

 7 Pretrain F 1 m std F 1m F 1 pn std F 1pn acc recall EN Scores and standard deviations of F 1 m , F 1 pn , an accuracy (acc) and macro averaged recall (rec) scores for monolingual experiments.These results complement those presented in Table2of the main paper.

	72.3 EN M L 67.8 T G 63.9 ES T G 67.2 ES M L 66.1 DE T G DE M L 64.8 65.8 IT T G 65.1 IT M L 70.8 F R T G 67.2 F R M L 68.2 P T T G 66.2 P T M L 62.1 N L T G N L M L 66.4 66.9 RO T G RO M L 68.5	0.9 1.6 2.0 1.1 1.0 1.3 1.0 1.2 0.8 1.2 0.3 1.3 3.8 1.6 1.1 1.1	72.5 66.9 61.9 64.6 64.6 62.9 64.8 63.2 69.6 65.6 66.7 63.9 60.1 64.9 66.4 68.0	1.4 1.7 3.5 1.9 1.2 2.1 1.6 1.9 1.6 2.4 1.3 1.8 5.4 2.8 1.2 1.8	73.6 68.6 66.1 69.0 67.3 66.1 67.0 66.2 72.5 69.2 69.1 67.8 64.9 66.8 68.0 70.8	72.5 68.0 63.9 68.1 65.8 65.1 65.6 65.3 71.5 68.2 67.6 66.9 64.1 65.2 66.2 69.7

Table 8 :

 8 Scores and standard deviations of F 1 m , F 1 pn , an accuracy (acc) and macro averaged recall (rec) scores for machine translation experiments. These results complement those presented in Table3of the main paper.

https://voxeurop.eu

https://github.com/huggingface/neuralcoref

https://github.com/averkij/lingtrain-aligner

https://www.deepl.com/en/docs-api

https://huggingface.co/facebook/ m2m100-12B-last-ckpt

Acknowledgements

This work was supported by the European Commission under European Horizon 2020 Programme, grant number 951911 -AI4Media. This work has been funded by the BOOM ANR Project -ANR-20-CE23-0024. It was made possible by the use of the FactoryIA supercomputer, financially supported by the Ile-de-France Regional Council.

GPU-hours for training. The inference times for the various models used in this paper (Flair NER, Blink, translation models) amount to another 24 GPU-hours. The total budget spent for training and inference is 804 GPU-hours.

C Supplementary results

Tables 6, 7, 8 provide the standard deviation values for the experiments reported from Tables 6, 7, 8 of the main text. Notations from the main tables are preserved.

D Annotator Demographics

The main demographic characteristics of annotators are: (1) gender -5 female/16 male; (2) age groups -14 between 25 and 34, 5 between 35 and 44, 2 over 44, (3) countries of origin: France (16), Romania (4), Ivory Coast (1).

E Annotation interface

This section presents the top and bottom parts of the instructions and registration page (Figures 5 and6, respectively), and an example of annotation page (Figure 7). B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 2, Subsection 3.1 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Subsection 3.1, Ethics Statement B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3, Section 4 C Did you run computational experiments? Section 4 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix A, Appendix B

The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4, Appendix B C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4, Appendix C C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3, Section 4 D Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix E D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants' demographic (e.g., country of residence)? Section 3 D3. Did you discuss whether and how consent was obtained from people whose data you're using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Section 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?

Ethics Statement

D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?