Juan Manuel Coria
email: coria@limsi.fr

Sahar Ghannay
email: ghannay@limsi.fr

Sophie Rosset
email: rosset@limsi.fr

Hervé Bredin
email: bredin@limsi.fr

A Metric Learning Approach to Misogyny Categorization

The task of automatic misogyny identification has not received as much attention as other natural language tasks have, even though it is crucial for identifying hate speech in social Internet interactions. In this work, we address this sentence classification task from a representation learning perspective, using both a bidirectional LSTM and BERT optimized with the following metric learning loss functions: contrastive loss, triplet loss, center loss, congenerous cosine loss and additive angular margin loss. We set new state-of-the-art for the task with our fine-tuned BERT, whose sentence embeddings can be compared with a simple cosine distance, and we release all our code as open source for easy reproducibility. Moreover, we find that almost every loss function performs equally well in this setting, matching the regular cross entropy loss.

Introduction

Whether it is at the word or at the sentence level, learning robust representations allows neural networks to consolidate knowledge that can later be transferred to other tasks and domains. Many approaches have dealt with this problem in different ways, for instance with CBOW or skip-gram from word2vec [START_REF] Mikolov | Efficient Estimation of Word Representations in Vector Space[END_REF] for contextindependent word embeddings, or more recently with BERT's [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF] sentence embeddings and contextual word embeddings.

In order to learn sentence representations, a neural encoder enc needs to learn a mapping from an initial representation x i to a target vector space. In a metric learning approach, the distances between each pair of sentence embeddings (enc(x i ), enc(x j )) should be low if classes y i = y j (intra-class compactness) and high if y i = y j (interclass separability). This objective is achieved by modifying the model's loss function.

In the domain of face recognition, many loss functions [START_REF] Schroff | FaceNet: A Unified Embedding for Face Recognition and Clustering[END_REF][START_REF] Wen | A Discriminative Feature Learning Approach for Deep Face Recognition[END_REF][START_REF] Liu | Rethinking Feature Discrimination and Polymerization for Large-scale Recognition[END_REF][START_REF] Wang | CosFace: Large Margin Cosine Loss for Deep Face Recognition[END_REF][START_REF] Deng | ArcFace: Additive Angular Margin Loss for Deep Face Recognition[END_REF] have been proposed to learn better face representations. The effort is motivated by high intra-class variability due to lighting, position or background. Other studies have experimented with these methods in different domains with similar characteristics, like speaker verification [START_REF] Bredin | TristouNet: Triplet loss for speaker turn embedding[END_REF][START_REF] Joon | VoxCeleb2: Deep Speaker Recognition[END_REF][START_REF] Yadav | Learning Discriminative Features for Speaker Identification and Verification[END_REF], and even as an enhancement of BERT's sentence representations [START_REF] Reimers | Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks[END_REF] for semantic textual similarity. A recent study [START_REF] Srivastava | A Performance Comparison of Loss Functions for Deep Face Recognition[END_REF] has also focused on comparing these methods on face verification, showing that angular margin losses achieve superior performance.

On the other hand, during the AMI evaluation campaign (Fersini et al., 2018a), several participants have proposed neural network models, but none in a metric learning setting. The best system [START_REF] Ahluwalia | Detecting Hate Speech Against Women in English Tweets[END_REF] uses a bidirectional LSTM with word embeddings of size 100 for the binary misogyny identification task, and ensemble methods with feature engineering for category and target classification. They obtain a macro F1 score of 36.1 on the misogyny category identification part of sub-task B, which is the one we address as well. A different recurrent architecture [START_REF] Caselli | Tweetaneuse@ AMI EVALITA2018: Character-based Models for the Automatic Misogyny Identification Task[END_REF] uses a multi-layer character bidirectional LSTM with a classification layer to determine misogyny category. This model achieved a poor macro F1 score of 14.1.

In this paper, we focus on five metric learning losses for the task of misogyny category identification from tweets, using the AMI (Fersini et al., 2018a) dataset. Our hypothesis was that metric learning might reduce the natural intra-class variability within misogyny categories, making representations robust to writing styles, irony, insults, etc. The loss functions we experiment with are contrastive loss [START_REF] Hadsell | Dimensionality Reduction by Learning an Invariant Mapping[END_REF], triplet loss [START_REF] Schroff | FaceNet: A Unified Embedding for Face Recognition and Clustering[END_REF], center loss [START_REF] Wen | A Discriminative Feature Learning Approach for Deep Face Recognition[END_REF], congenerous cosine loss [START_REF] Liu | Rethinking Feature Discrimination and Polymerization for Large-scale Recognition[END_REF] and additive angular margin loss [START_REF] Deng | ArcFace: Additive Angular Margin Loss for Deep Face Recognition[END_REF], as well as cross entropy loss. We optimize these losses with two different architectures: a bidirectional LSTM [START_REF] Hochreiter | Long Short-Term Memory[END_REF] and BERT [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF], and we evaluate their performance using a simple K-nearest neighbors classifier to better measure representation quality.

Our main contributions consist of new stateof-the-art performance for the misogyny category identification task, as well as empirical evidence that these methods do not perform better than cross entropy loss on close-set sentence classification. Moreover, our code is released as open source for easy reproducibility.

Loss Functions

In this section, we present the loss functions chosen for our study, which can be separated into contrastbased and classification-based, according to how they are computed.

Contrast-based losses

The contrastive loss [START_REF] Hadsell | Dimensionality Reduction by Learning an Invariant Mapping[END_REF] uses pairs annotated as similar/dissimilar (also called positive/negative). It brings representations from similar examples closer together, while separating dissimilar ones explicitly:

L = P + i=1 (D i ) 2 + P - i=1 max(m -D i , 0) 2 (1)
where P + is the number of similar pairs, P -the number of dissimilar pairs, D i = 1 -cos θ i the distance between embeddings of the ith pair, and m a margin. The triplet loss [START_REF] Schroff | FaceNet: A Unified Embedding for Face Recognition and Clustering[END_REF] is calculated over triplets composed of a reference example known as the anchor, a positive and a negative, both the latter with respect to the anchor. Following the idea introduced by Gelly and Gauvain (2017), we define this loss using the sigmoid function:

L = T i=0 sigmoid(α (cos θ n i -cos θ p i ))) (2)
where T is the number of triplets, α a scaling hyperparameter, θ p i the angle separating the anchor and the positive embeddings, and θ n i the angle separating the anchor and the negative ones.

Classification-based losses

These loss functions derive from the cross entropy loss, either by modifying how the classification layer output is calculated or working as a penalization term. the cross entropy loss is defined as:

L CE = - 1 N N i=1 log softmax(σ i , y i ) (3)
where N is the number of training examples, σ i the output of the classification layer, and y i the class of the ith example.

The congenerous cosine (CoCo) loss [START_REF] Liu | Rethinking Feature Discrimination and Polymerization for Large-scale Recognition[END_REF] interprets the weights w k of the classification layer as class centroids, learning to maximize the cosine similarity between a representation and its centroid. The classification layer output σ i is redefined as:

∀k σ ik = α • cos θ iw k (4)
where θ iw k is the angle separating the ith representation and w k , and α a scaling hyper-parameter.

The additive angular margin (AAM) loss [START_REF] Deng | ArcFace: Additive Angular Margin Loss for Deep Face Recognition[END_REF] goes one step further adding a margin in angular space to penalize the distance between a representation and its centroid:

∀k σ ik = α • cos(θ iw k + δ ik m) (5)
where m is a margin, and δ ik = 1 if k = y i and 0 otherwise. Finally, the center loss [START_REF] Wen | A Discriminative Feature Learning Approach for Deep Face Recognition[END_REF] penalizes the cross entropy loss with the distance to jointly learned centroids c k external to the classification layer:

L = L CE + λ 2 N i=1 (1 -cos θ icy i ) 2 (6)
where λ is a hyper-parameter controlling the effect of penalization.

Task

The automatic misogyny identification (AMI) task was proposed in the context of the IberEval 2018 (Fersini et al., 2018b) and Evalita 2018 (Fersini et al., 2018a) We use the same dataset as in Fersini et al. (2018a) and we focus exclusively on misogyny category detection, using an additional class for non misogynous tweets. Our results are thus compared to the category identification part of sub-task B.

As the corpus does not provide a development set, one was constructed from the training set following the same class distribution. The final Train set is composed of 3200 tweets, and the Dev and Test sets of 800 and 1000 tweets respectively. Class distribution is described in detail in Table 1. Given the high class imbalance, the task is evaluated using the macro F1 score measure.

Experiments

Experimental protocol

As different losses rely on different hyperparameters, we perform a hyper-parameter search including learning rates, margins m, scalings α, and λ. The values we have experimented with are shown in Table 2. Each configuration is trained on Train for 60 epochs and validated using a Knearest neighbors classifier on Dev. As we deal with a rather small dataset, the best configuration for each loss and each architecture is then trained and validated from scratch 10 times to reduce the effect of randomness. Reported results are the mean macro F1 score and standard deviation on Test over these 10 runs.

In all experiments we use the cosine distance to compare embeddings, as congenerous cosine loss and additive angular margin loss can only be optimized in this way. Additionally, a linear classification layer is jointly trained with the sentence encoder when optimizing classification-based loss functions.

Parameter Values LR {10 -2 , 10 -3 , 10 -4 , 10 -5 , 10 -6 } • {10 -4 , 10 -5 , 10 -6 , 10 -7 } • m {0.02, 0.05, 0.25, 0.5, 0.75} α and λ {0.01, 0.1, 1, 10, 100, 1000}

Architecture

We experiment with two different encoder architectures. The first one is a one-layer bidirectional LSTM [START_REF] Hochreiter | Long Short-Term Memory[END_REF] with hidden size 768 (to match BERT) and word embeddings of size 300 obtained from a word2vec CBOW model [START_REF] Mikolov | Efficient Estimation of Word Representations in Vector Space[END_REF] trained on 2billion-word Wikipedia dumps. The second one is the standard monolingual uncased BERT [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF] from the huggingface library [START_REF] Wolf | HuggingFace's Transformers: State-of-the-art Natural Language Processing[END_REF] pretrained on Wikipedia.

To obtain a sentence embedding from an encoder, we perform a max pooling over the hidden states of the last layer, leaving us with sentence embeddings of size 768 on both models.

Implementation details

All sentences are pre-tokenized using the TweetTokenizer from the NLTK toolkit [START_REF] Bird | Natural Language Processing with Python[END_REF] in order to correctly deal with Twitterspecific tokens like hashtags, mentions, and even emojis. During this process we remove hashtags and URLs. When training BERT, we do a second pass of tokenization with BERT's pretrained tokenizer. We use a batch size of 32 sentences and RMSprop as optimizer, reducing the learning rate by half every 5 epochs of no improvement. The best configurations found during hyper-parameter search for each architecture and loss function are shown in Table 3.

To guarantee reproducibility, all sources will be released as open source upon paper publication.

Evaluation

We evaluate each model with the macro F1 score of a K-nearest neighbors classifier with K = 10 fit with all sentence embeddings from Train. However, given the high class imbalance, the a priori probability of a random embedding being closer to a non-misogynous embedding is higher than for Loss Hyper-parameters cross entropy a discredit one (see Table 1). To circumvent this issue, we penalize the vote for class k by the number of examples from k in Train. We believe this simple classifier to be a better measure for representation quality, as it relates to the separability and compactness properties that we expect from a metric learning model.

LR = 10 -3 • LR = 10 -5 • AAM LR = 10 -3 , m = 0.05, α = 100 • LR = 10 -5 , m = 0.05, α = 100 • center LR = 10 -4 , λ = 1000 • LR = 10 -5 , λ = 0.1 • congenerous LR = 10 -3 , α = 10 • cosine LR = 10 -5 , α = 100 • contrastive LR = 10 -4 , m = 0.25 • LR = 10 -6 , m = 0.25 • triplet LR = 10 -4 , α = 1000 • LR = 10 -6 , α = 1000 •

Results

The results are summarized in Figure 1. With a fixed architecture, it is clear that all loss functions perform equally, with the exception of LSTM with contrastive and triplet loss. As the LSTM encoder is rather shallow (4.4M parameters) in comparison to BERT (110M parameters), it is possible that contrast-based losses need bigger models to perform competitively.

The fact that almost all losses perform equally well shows that, contrary to what we thought, metric learning models perform no better than cross entropy, in contrast to other findings [START_REF] Srivastava | A Performance Comparison of Loss Functions for Deep Face Recognition[END_REF]) on face verification. One possible explanation is that the AMI dataset may not contain enough examples or classes for these models to exploit. However, another factor might be responsible for this behavior. One of the key differences of AMI with respect to face verification is the closeset nature of the problem. An open-set task is evaluated with unseen classes, while a close-set task is evaluated with unseen instances of the training classes. It is possible that open-set verification tasks are more suitable for metric learning than close-set tasks, meaning that the power of metric learning might in fact lie in generalizing to unseen classes rather than unseen class instances. The fact that verification tasks more closely resemble the training objective than exact class prediction could provide an explanation for this.

On the other hand, our fine-tuned BERT outperforms the Evalita winner baseline, setting new stateof-the-art for automatic misogyny category identification, with the added benefit of having comparable embeddings with a simple cosine distance.

As a final note, results in Table 3 suggest that congenerous cosine loss and center loss hyperparameters could be more sensitive to architecture changes than other losses. Perhaps not surprisingly, we also observe that additive angular margin loss works better with lower margins. This is consistent with the margin's role, serving as an upper bound for the distance between an embedding and its centroid, while the margin in contrastive loss serves as a lower bound for the distance between two negatives.

Conclusion

In this work we have addressed the problem of misogyny identification with a metric learning approach, comparing the performance of several loss functions. We hypothesized that reducing intraclass variability in this way would be beneficial. However, we have shown that none of the considered losses can outperform the regular cross entropy on the task. Our results suggest that metric learning approaches might not be suited to close-set sentence classification tasks.

Finally, our fine-tuned BERT sets new state-ofthe-art performance, with a macro F1 score of 40.5.

Figure 1 :

 1 Figure 1: F1 scores on Test for each architecture and loss function. Scores are calculated as the mean of 10 runs and standard deviation is shown as error bars. The baseline of the Evalita 2018 winner (Ahluwalia et al., 2018) is shown for reference.

Table 1 :

 1 Number of sentences per class for each partition of the AMI dataset. Note that classes are greatly imbalanced

	Class	Train Dev Test
	derailing	74	18	11
	discredit	811	203	141
	dominance	118	30	124
	sexual harassment 282	70	44
	stereotype	143	36	140
	non misogynous	1,772 443	540
	total	3,200 800 1,000

evaluation campaigns. It consists of an ensemble of tweets with three different types of annotations: misogyny (binary), misogyny category (derailing, discredit, dominance, sexual harassement and stereotype) and target (active or passive).

Table 2 :

 2 Values tested during initial hyper-parameter search, totaling 486 configurations (LR stands for learning rate). Values with • are LSTM only and values with • are BERT only.

Table 3 :

 3 Best hyper-parameter configurations found per loss function (LR stands for learning rate). Rows with • correspond to LSTM and rows with • to BERT.

Acknowledgements

This work has been partially funded by the LIHLITH project (ANR-17-CHR2-0001-03), and supported by ERA-Net CHIST-ERA, and the "Agence Nationale pour la Recherche" (ANR, France). It has also been made possible thanks to the Saclay-IA computing platform.


Pedro Ramaciotti Morales

Gabriel Muñoz Zolotoochin

Measuring the accuracy of social network ideological embeddings using language models

Keywords: Network embedding, multidimensional network scaling, textclassification, party systems, ideology, ideology scaling, polarization

The opinions of people on different issues are traditionally studied through polls. Recently, ideological embedding methods have proposed to position social network users in spaces where positions are informative of their opinions. These methods have been shown to be effective in the US and in European settings. However, validating their results is challenging, as required data must not rely on the social network structure used in the embedding (panel data, for example). In this article, we propose a validation method based on language models for classification of users in ideological spaces. We illustrate our methodology using political manifestos, political surveys on party positions, and text utterances produced by Twitter users in Chile and in France, two countries with different party structure. Using text classification related to issues relevant to the ideological spatialization, we show that positions can be shown to be highly accurate, thus allowing for the robust inference of opinions of users at large scales.

Introduction

Polls and surveys are traditional instruments for estimating opinions in populations. More recently, network ideological embedding -or scaling-methods have shown that digital traces in media platforms, such as friendship networks, can be used to infer opinions at massive scales. These methods embed users in spaces where dimensions stand for indicators of opinions, and where positions along dimensions indicate opinions ranging from most opposed to most favorable to different issues. Validating the opinion positions obtained with ideological embedding methods is a challenging task, as they require costly additional data such as vote registries or panel data. In this article we propose a method, based on the automated analysis of text written by users, to validate their ideological positions computed using the network structure via ideological embedding (and thus not dependent on language). We illustrate our method using Twitter data from two countries with different party structures and languages, Chile and France, showing that ideological embedding has demonstrable accuracy.

Ideological embedding methods have become an active research stream as they hold the promise to provide opinion models for studying many interesting phenomena including polarization and extremism [START_REF] Barberá | Birds of the same feather tweet together: Bayesian ideal point estimation using Twitter data[END_REF][START_REF] Barberá | Tweeting from left to right: Is online political communication more than an echo chamber?[END_REF][START_REF] Bond | Quantifying social media's political space: Estimating ideology from publicly revealed preferences on Facebook[END_REF], to produce contributions bridging opinion dynamics and social network analysis [START_REF] Morales | Auditing the Effect of Social Network Recommendations on Polarization in Geometrical Ideological Spaces[END_REF][START_REF] Morales | Your most telling friends: Propagating latent ideological features on Twitter using neighborhood coherence[END_REF], and finally to deliver valuable applications such as passive, inexpensive, and massive polling. These methods work by collecting social network data structures (i.e., the social graph), postulating latent ideological variables that may be structuring the network through some social mechanism such as homophily [START_REF] Mcpherson | Birds of a feather: Homophily in social networks[END_REF] (people with similar opinions, thus with similar variable values, tend to become friends), and use the network data to infer these ideological parameters. Through this procedure, users are embedded in the parameter space, called ideological space. When validating the results of these methods, different additional data have been leveraged, such as voting records and self-declared political affiliations [START_REF] Barberá | Birds of the same feather tweet together: Bayesian ideal point estimation using Twitter data[END_REF] (which may be difficult or impossible to obtain), and even panel data obtained through surveys [START_REF] Temporão | Ideological scaling of social media users: a dynamic lexicon approach[END_REF] (which may be expensive to obtain). Furthermore, newer network ideological embedding methods [START_REF] Morales | Unfolding the dimensionality structure of social networks in ideological embeddings[END_REF] propose the possibility of embedding users in multitudes of dimensions serving as indicators to a diversity of issues, which makes it difficult to rely on surveys or voting records for validation. While validations produced using text written by users themselves have been proposed (e.g., in the cited example from [START_REF] Barberá | Birds of the same feather tweet together: Bayesian ideal point estimation using Twitter data[END_REF]), these do not address quantitative measures of fit between identified issues related to dimensions of the ideological space, and the accuracy with which they capture opinions toward those issues.

In this article we propose the first validation method for the accuracy of positions computed using ideological network embedding, and that can be systematically applied to a wide range of issues. We begin by addressing the different previous works relating to our results. We then present the experimental Twitter data from Chile and France, and external opinion position data for parties that is generally used to compute the interpretation of the dimensions of the embedding space. We then present a family of ideological embedding methods and a computation of the interpretation of first dimensions of the embedding space in terms of issues of the political debate for which they act as indicators. Using the position of users and the text they have written in their online profiles to describe themselves, we propose our method for assessing the accuracy of their positions. Finally, we discuss our results and their consequences.

Related Work

Measuring opinions in populations often involves a priori distributional assumptions (e.g., normality) and establishing the parameters of polls. Alternatively, behavioral traces can be analyzed to infer opinions. The foremost example is the NOMINATE method [START_REF] Poole | A spatial model for legislative roll call analysis[END_REF] for the estimation of opinion positions (e.g., left-right scales [START_REF] Clinton | The statistical analysis of roll call data[END_REF]) of members of parliament (MPs) analyzing how they vote on pro-posed bills. Ideological network embedding extend the principle of these methods to social networks. Instead of inferring the ideological position of MPs that vote for bills, they infer the positions of users that follow politicians online [START_REF] Bond | Quantifying social media's political space: Estimating ideology from publicly revealed preferences on Facebook[END_REF][START_REF] Barberá | Birds of the same feather tweet together: Bayesian ideal point estimation using Twitter data[END_REF]. These methods have been used to infer the liberal or conservative attitudes of millions of US Twitter and Facebook users from the observed friendship -who follows whom-networks (replacing voting by following). To perform the inference of ideological parameters, some methods use the Correspondance Analysis (CA) method, shown to be equivalent to a log-spatialization for homophily models [START_REF] Lowe | Understanding wordscores[END_REF]. Importantly, these methods do not rely on textual data, making them language-independent. Very recently, these methods have been adapted to work on European settings, and inferring ideological dimensions other than Left-Right divides [START_REF] Barberá | Understanding the political representativeness of Twitter users[END_REF][START_REF] Morales | Unfolding the dimensionality structure of social networks in ideological embeddings[END_REF]. In terms of modeling, choice data (be it MPs choosing bills or users choosing friends online) is often leveraged as bipartite network structures. The reader is referred to [START_REF] Imai | Fast estimation of ideal points with massive data[END_REF] for survey on ideology estimation methods from bipartite network structures.

Empirical validation using external data (such as self-declared political affiliation or voting registries) has shown qualitatively that it is possible to recover ideological positions in the US and European countries. However, quantitative measurements of the accuracy of these methods have proved elusive, as it is difficult to obtain alternative data on the position of users to contrast the result of these methods. To obtain ideological positions without using social network graphs, researchers have relied on media outlet citation [START_REF] Bakshy | Exposure to ideologically diverse news and opinion on Facebook[END_REF][START_REF] Baumann | Modeling echo chambers and polarization dynamics in social networks[END_REF] and positions of parliamentarians as computed with the NOMINATE method based on voting on bills [START_REF] Ecker | Estimating policy positions using social network data: cross-validating position estimates of political parties and individual legislators in the Polish parliament[END_REF][START_REF] Cointet | What colours are the yellow vests? An ideological scaling of Facebook groups[END_REF]. Automated text analysis has been used to infer ideological positions, but in classification tasks. Researchers successfully developed methods to classify political text, e.g., by analysis whether different textual content can be classified coming from Democratic-or Republican-leaning sources in the US [START_REF] Jensen | Political polarization and the dynamics of political language: Evidence from 130 years of partisan speech [with comments and discussion[END_REF]. Other methods propose to extract positions of texts, and thus of users who produce them, discovering the objects towards which positions are taken [START_REF] Kwak | FrameAxis: characterizing microframe bias and intensity with word embedding[END_REF]. Yet, other families of methods aim at inferring the sentiment of an uttered text [START_REF] Liu | A survey of opinion mining and sentiment analysis[END_REF][START_REF] Ravi | A survey on opinion mining and sentiment analysis: tasks, approaches and applications[END_REF], but without identifying the issues to which text is referring nor latent ideological positions of the person uttering it. None of these methods has yet been combined with ideological embedding procedures to assess their accuracy in a way that tackles the challenge of required flexibility in dealing with multiple issues towards which users take positions.

Social Network Data

To test our method we use Twitter data. We select two countries with different attitudinal reference data for parties, Chile and France, and we manually identify the Twitter accounts of MPs. We build on the identification criteria and collections of the "Twitter Parliamentarian Database" [22, Section 3] to identify 187 Chilean and 515 French parliamentarians on Twitter. We then collect the followers of these accounts using Twitter's API 3 . Following [START_REF] Barberá | Birds of the same feather tweet together: Bayesian ideal point estimation using Twitter data[END_REF], we filter out followers that have posted less than 100 tweets or have less than 25 followers to improve the ratio of true users to bots, and retain users that follow at least 3 MPs, and we call them politicized users. This collection was conducted on October 2020 and resulted in 418.979 politicized users in Chile and 288.496 in France (see Table 1), for which we also collected their Twitter bios ( a 160 character profile description written by users). The bipartite network of the MPs and their followers will be used for the ideological embedding procedure, and the text bios to validate the ideological positions.

For each country, we consider attitudinal references for parties, which allow to position them in multi-dimensional opinion spaces where positions are indicators of attitudes towards separated, well-identified issues of public debate, and will allow for the interpretation of the content of dimensions in the ideological embedding space. For Chile we use the 2020 Manifesto Project (MARPOR) data [START_REF] Krause | The Manifesto Data Collection: South America. Version 2020b[END_REF], which is derived from human annotations of political manifesto texts produced by political parties; annotators associate phrases of manifestos as addressing any of a number of predefined issues, and the number of phrases per issue is used to compute positions. For France, we use the 2019 Chapel Hill Expert Survey (CHES) data [START_REF] Bakker | Chapel Hill Expert Survey[END_REF], which is derived from answers provided by a survey answered by 421 political scientists specializing in European politics. The MARPOR data allows us to position Chilean political parties in 147 attitudinal scales, while the CHES data allows us to position French political parties in 48 scales. Scales mark with a numerical value a favorable or opposed position towards issues such as, e.g., attitudes towards the political left or right, European integration, immigration, or welfare state. The MARPOR dataset includes ideological positions for parties in more than 50 countries, while the CHES dataset includes positions for 34 European countries. We chose to use CHES for France because it is generally considered as more reliable [START_REF] Mikhaylov | Coder reliability and misclassification in comparative manifesto project codings[END_REF], and to show that our approach works when the embedding procedure is used with either of the two referential sources: surveys for France and annotated manifestos for Chile.

Social Network Ideological Embedding

Using the bipartite networks collected for Chile and France, embed individuals in ideological spaces, which we will validate using text bios in the next section.

For each country we represent this bipartite sub-graph of the Twitter network as an adjacency matrix A ∈ {0, 1} F ×P , where P is the number of politicians or MPs, and F is the number their politicized followers. We then produce a reduced-dimensionality representation of the MPs and their followers using the coordinates of accounts in the principal components of the CA of matrix A. We denote by δ i the i-th dimension of this reduced-dimensionality representation.

Figure 1 shows the positions of politicized followers on the first two dimensions (δ 1 & δ 2 ) along with the position of the MPs and their parties (computed as the mean position of MPs from each party), and a point cloud scatterplot in the first three dimensions (δ 1 , δ 2 & δ 3 ). We manually inverted the direction of δ 2 in France for convenience of visualization. This has no effect in the rest of our analysis. To associate an interpretation to dimensions of this ideological space (an issue for which dimensions act as indicators of opinions) we use external attitudinal reference from CHES and MARPOR datasets. For the purposes of validating these spatial positions, we will focus on the single, broadly-investigated issue of Left-Right cleavages [START_REF] Barberá | Tweeting from left to right: Is online political communication more than an echo chamber?[END_REF]. To identify which dimension is related to Left-Right cleavages in each country we compare the position of parties according to the dimensions of ideological space, and the position of parties as provided by chosen external sources (MARPOR in Chile and CHES in France). Figure 2 show the correlation of positions using ideological space and external sources for the issue of Left-Right cleavages, present in both CHES and MARPOR data. Readers familiarized with Chilean and French politics, will have identified what correlations confirm: first Chilean dimension δ 1 and second French dimension δ 2 are latent variables related to users' positions on some Left-Right scale. For further details on the dimensionality structure of the French political space (e.g., showing how δ 1 relates to attitudes towards the European Union), the reader is referred to [START_REF] Morales | Unfolding the dimensionality structure of social networks in ideological embeddings[END_REF].

Validating User Positioning Via Text Classification and Language Models

Thus far, analyses have focused on validating ideological embeddings using the position of MPs, as illustrated by In each country, we classify users as being Left-and Right-leaning based exclusively on the text of their Twitter bios using automated text analysis. First, we filter users by the language in which they write their bio (Spanish in Chile and French in France) using CLD3 model for language identification [START_REF]Compact Language Detector v3 (CLD3)[END_REF]. This allowed us to identify 230.834 users in Chile (out of 418.979) and 148.145 users in France (out of 288.496) that write their bios in the language of each country. For each bio, we infer the sentiment of the text using the BERT multilingual base model for uncased words [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF], which classify sentiments in 5 levels, from most negative to most positive. We classify bios as having positive sentiment if the sentiment predicted using the BERT language model is equivalent to 4 or 5 (on a scale from 1 to 5), and as having negative sentiment if the BERT-predicted sentiment is equivalent to 1 or 2. Additionally, we label users as belonging to two groups: 1) Left if they use the keywords "izquierda" or "izquierdista" in Spanish, and "gauche" or "gauchiste" in French ("left" or "leftist") with their variations in gender and number, and the bio has an identified positive sentiment, and 2) Right if they use the keywords "derecha" or "derechista" in Spanish, and "droite" in French ("right") with their variations, and the bio has an identified positive sentiment. This resulted in 1.886 users labeled Left and 1.550 users labeled Right in Chile (for a total of 3.436 users using "left" or "right" with positive sentiment in their profile bios), and in 891 users labeled Left and 739 users labeled Right in Chile (for a total of 1.630 users using "left" or "right" with positive sentiment in their profile bios). We consider these labeled users as the subset of users for which left or right stances can be determined using automated text analysis. We aim to show that our ideological dimensions serve as a good classifier for them too. Figure 3 shows the empirical densities of these groups of users per country along the pertinent ideological space dimension selected in Section 4. In order to show that the selected ideological dimension containing Left-Right cleavages (δ 1 in Chile and δ 2 in France), we fit a logistic model for the classification of users based the positions of the labeled users identified in the Left and Right groups. To fit a logistic model for users in the Right based on ideological positions from Section 4, we consider Right users as successes of the model, and users on the Left as failures of the model. Because groups are complementary (individuals in our model construction are either in the Left or in the Right), we use the same logistic model to classify them (instead of using a separate model with inverted direction of the ideological dimension and in which the Left would be the success group and the Right the failure group). Figure 4 shows, for each country, the positions of users in both groups (represented by colored dots), the cumulative distribution function, and the fitted logistic model for classification. For each country, and for each group, we compute the goodness of fit of our logistic models as the precision and the recall if they were used to classify our labeled users. After fitting a logistic model, we classify users with it by establishing a 0.5 threshold: users below the 0.5 threshold are classify as Left, while users over the threshold are classified as Right. We compute the performance as the comparison between their groups as classified using text analysis, and as resulting from the logistic model. The values from Figure 4

Conclusions and Discussion

We presented ideological embedding methods for Chile and France, two countries with different socio-political systems using the Correspondence Analysis method. Using external attitudinal data we provided meaning to dimensions of this space identifying for both countries dimensions that capture Left-Right cleavages. We proposed a method based on language models to classify users (independently form ideological embedding procedures) as being Left or Right sympathizers, we showed that dimensions of the ideological space can provide accurate classification of users, and we provided measurements of accuracy of the positions. This amounts to the first validation of the positions achieved by ideological embedding in way that is applicable to a wide range of issues that may be captured by other dimensions by changing associated keywords (e.g., "migrant", "immigration", etc., to test the accuracy of dimensions capturing attitudes towards immigration). This method allows for the validation of the position of a large quantity of users (as opposed to validating the positions or MPs), paving the way for the use of ideological embedding methods as a massive, passive, frequent, and inexpensive polling tool. Besides polling, validated ideological positions of social network users in multidimensional ideological spaces in which the substance and meaning of dimensions is well-identified, opens an interesting field of research into many online phenomena at the intersection of Social Network Analysis, Opinion Dynamics, sociology, and political sciences. The positions of users can be used to measure polarization in different online arenas and the effect of algorithmic recommendation on them [START_REF] Morales | Auditing the Effect of Social Network Recommendations on Polarization in Geometrical Ideological Spaces[END_REF]. The positions of MPs and party sympathizers opens an interesting path for conducting studies of party competitions and ideological party diversity [START_REF] Grossman | Economic internationalization and the decline of the leftright dimension[END_REF][START_REF] Morales | Measuring diversity in heterogeneous information networks[END_REF].

Fig. 1 .

 1 Fig. 1. Ideological embedding of Twitter networks collected for Chile and France. Parliamentarians, party positions computed as mean of parliamentarians per party, and density of their followers (left), and three-dimensional cloud points (right).

Fig. 2 .

 2 Fig. 2. Party positions in ideological embedding (δ1 in Chile and δ2 in France) and in attitudinal sources for Left-Right cleavages (MARPOR for Chile and CHES for France), with reported Pearson r, p-value, slope m, and intersect b.

Figure 2 .

 2 In this section we propose a method to validate the position of their followers by assessing the accuracy of positioning in the Left-Right scale. To validate the ideological positions on the Left-Right scale we propose a model based on text classification showing the accuracy achievable by using latent dimensions (δ 1 in Chile and δ 2 in France) as indicators for Left and Right stances of users.

Fig. 3 .

 3 Fig. 3. Empirical density of distribution of the groups of users labeled Left or Right using automated text analysis along the ideological dimensions previously identified as containing Left-Right cleavages.

Fig. 4 .

 4 Fig. 4. Users in groups • Left and • Right, their cumulative distribution function, and the fitted logistic model for classification. Precision and recall for logistic classification (0.5 threshold) is reported in the embedded table.

  are read as follows. The Left-Right scale axis can classify users in Left and Right groups such that, in Chile 88% of users classified Left or Right are rightly classified, and 89% of true Left and 86% of true Right users are identified as such. In France, 79% of users classified Left and 83% classified Right are rightly classified, and 86% of true Left and 74% of true Right users are identified as such.

Table 1 .

 1 Twitter datasets used in ideological embedding and validation.

	MPs Twitter Acc. (out of) Followers Politized Fol. Attitudinal References
	Chile 187 (198)	3.507.374 418.979	MARPOR
	France 515 (925)	4.894.333 288.496	CHES

In compliance with GDPR, the data for this study was declared on July the 15th, 2021, to Sciences Po's data processing registry.

Acknowledgments. This work has been funded by the French National Agency for Research under grant ANR-19-CE38-0006 "Geometry of Public Issues" (GOPI) and by CIVICA's "European Polarisation Observatory" (EPO).