Nejat Arinik

Rosa Figueiredo

Vincent Labatut

Characterizing measures for the assessment of cluster analysis and community detection

Keywords: Cluster analysis, Community detection, External evaluation measures

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

The problem of comparing two partitions of the same set occurs in a number of situations, the most widespread being probably the assessment of cluster analysis and community detection results. In these contexts, one has computed the clusters of a dataset, or the community structure of a network. This result takes the form of a partition of the set of data points or set of nodes, respectively. One then wants to compare this estimation with some ground-truth also taking the form of a partition. Alternatively, one has computed several such estimations, for instance using several algorithms, and wants to compare them to each other. This comparison is traditionally performed through some measures able to quantify the similarity between two such partitions, and often called external measures (measures, hereafter), as they allow comparing the output of the partitioning method to some form of ground truth. There are many ways to formalize what one means by "similarity", resulting in the proposition of a very large number of such measures over the years, as well as surveys comparing them [START_REF] Nguyen | Information theoretic measures for clusterings comparison: Variants, properties, normalization and correction for chance[END_REF].

In the literature, authors proposing new external measures follow a relatively standard workflow. First, they list some mathematical properties which they deem desirable in such measures, e.g. not being sensitive to the number of clusters k [START_REF] Liu | Evaluation of community detection methods[END_REF]. They then show that existing measures do not possess these properties. This task is frequently performed through an empirical approach, which consists in applying some predefined transformations to certain partitions, both designed in a way that is related to the property of interest, and to study how the measure reacts to these perturbations by using it to compare those partitions. Finally, the authors solve this issue by proposing a new measure having these properties, or modifying an existing one to this end.

Each application case is likely to bring its own constraints and requirements, so there is no such thing as a "best" measure that would fit all situations. One trait considered as positive in one case could very well be perceived as a drawback in another. However, due to the profusion of available measures, selecting the most appropriate one for a given situation is a challenge for the end user. As mentioned before, some survey articles try to compare them, but they focus only a small number of measures [START_REF] Nguyen | Information theoretic measures for clusterings comparison: Variants, properties, normalization and correction for chance[END_REF] and/or properties [START_REF] Albatineh | On similarity indices and correction for chance agreement[END_REF]. More importantly, the evaluations they perform are specific to these measures and properties [START_REF] Nguyen | Information theoretic measures for clusterings comparison: Variants, properties, normalization and correction for chance[END_REF], preventing the end user to include additional measures or properties in the comparison. In practice, the problem of selecting an appropriate measure to compare partitions is generally overlooked, and researchers tend to follow tradition and use the measures frequently appearing in the literature of their field.

In this work, we propose a new framework to solve this issue. It is based on the empirical approach presented above, and consequently relies on a set of predefined partitions and parametric partition transformations. We study the effect of each parameter on the measure through multiple linear regression, in order to produce results that the end user can interpret. Our framework is not tied to any specific measure, property, or transformation, so it can be applied to any situation. In the full work, we will illustrate its relevance by applying it to a selection of popular measures, and show how it can be put in practice through two concrete use cases. Here, we summarize the framework and give some preliminary results.

Method

The framework we propose is designed to analyze the behavior of a set of measures, and it is constituted of two parts.

The first part aims at generating the data required to study and compare the considered measures. We proceed in three steps, and unlike the common approach adopted in the literature, we do so in a fully parametric way in order to get a greater control. For the same reason, our approach is deterministic. The first step is to create a so-called original partition, controlled by three parameters: number of elements n, number of clusters k and cluster size heterogeneity h. The second step consists in applying a transformation to the original partition, in order to produce a so-called transformed partition. This step is controlled by two parameters: type t and intensity q of the transformation. The third step is to compute the selected external measures in order to assess how similar each pair of original and transformed partitions are. We normalize the resulting values so that they all express dissimilarity between the partitions, in order to make them comparable. We repeat the process with an adequate number of different parameter values in order to cover the parameter space.

The second part of our framework consists in analyzing all the dissimilarity values obtained at the first part. To do so, we design a multiple linear regression model able to take into account the interactions between the parameters. We perform a relative importance analysis [START_REF] Saltelli | Global sensitivity analysis[END_REF] in order to assess how much the framework parameters affect the measures.

Preliminary Results

To show the interest of our framework, we discuss here a few preliminary results, directly obtained from the first part of our framework (and preceding the work conducted in the second part). (a) Scores obtained with all measures, as functions of q, for the Singleton Clusters transformation. First, we consider how measure scores are affected by parameter q (transformation intensity) when applying the so-called Singleton Cluster transformation, while other parameters are fixed to arbitrary values. This transformation consists in turning each element concerned by the transformation into a new cluster. Figure 1a shows that all the measure scores increase with q, albeit in different ways. Overall, RI [START_REF] Rand | Objective criteria for the evaluation of clustering methods[END_REF] has the smallest slope coefficient, followed by NMI sum [START_REF] Strehl | A knowledge reuse framework for combining multiple partitions[END_REF], and they are therefore the least sensitive to this transformation. We observe that JI [START_REF] Jaccard | Étude comparative de la distribution florale dans une portion des alpes et des jura[END_REF], ARI [START_REF] Hubert | Comparing partitions[END_REF], FM [START_REF] Fowlkes | A method for comparing two hierarchical clusterings[END_REF] and the F -measure [START_REF] Artiles | The semeval-2007 weps evaluation: Establishing a benchmark for the web people search task[END_REF] get similar scores for extreme q values, but are relatively different when q gets closer to 0.5.

In Figure 1b, we focus on the RI and show how it is affected by changes in q (transformation intensity) and k (number of clusters in the transformed partition), for the so-called 1 New Cluster transformation. This transformation consists in adding to the original partition a single new cluster containing all the elements affected by the transformation. As a function of k, the RI score is always monotonic, however the nature and slope of the trend depends on q: increase trend for q ≥ 0.7 vs. decrease for q < 0.7. This means there is a significant interaction between q and k. This type of joint effect between parameters will be captured by the interaction terms present in our regression model.

The goal of the second part of our framework, whose results are not presented here, is to summarize the information conveyed by figures such as Figure 1, in order to ease the characterization and comparison of the considered measures.

Fig. 1 :

 1 Fig. 1: Preview of two types of results obtained with our framework. Figures available at 10.6084/m9.figshare.13109813 under CC-BY license.


Noé Cécillon

Richard Dufour

Vincent Labatut

Georges Linarès

Tuning Graph2vec with Node Labels for Abuse Detection in Online Conversations

Introduction

In recent years, online social media have allowed people to meet and discuss world-wide. These popular platforms are confronted with increasing abusive content. In order to automate the detection of abusive content in such social media, researchers have proposed various methods based on Natural Language Processing (NLP) [START_REF] Nobata | Abusive language detection in online user content[END_REF][START_REF] Badjatiya | Deep learning for hate speech detection in tweets[END_REF], and have leveraged behavioral information about users and the structure of conversations [START_REF] Chatzakou | Mean birds: Detecting aggression and bullying on twitter[END_REF][START_REF] Mishra | Author profiling for abuse detection[END_REF].

In our previous work [START_REF] Papegnies | Conversational networks for automatic online moderation[END_REF][START_REF] Cécillon | Abusive language detection in online conversations by combining content-and graph-based features[END_REF], we proposed to combine NLP and conversational graph-based features to detect abusive messages in chat logs extracted from an online game. These conversational graphs model interactions between users (i.e. who is arguing with whom?), while completely ignoring the language content of the messages. We characterized the structure of these graphs by computing a large set of manually selected topological measures, and used them as features to train a classifier into detecting abusive messages.

Graph embedding methods allow representing graphs as low-dimensional vectors while preserving at least a part of their topological properties. In addition to the plain structure, certain methods are able to capture additional information such as node labels or the weight and direction of edges. These representations are automatically learned, so they have the advantage of not requiring to perform any feature selection or feature engineering. One can distinguish four main categories of graph embedding methods, depending on the nature of the considered objects: node [START_REF] Grover | Node2vec: Scalable feature learning for networks[END_REF][START_REF] Perozzi | Don't walk, skip! online learning of multi-scale network embeddings[END_REF], edge [START_REF] Bordes | Learning structured embeddings of knowledge bases[END_REF], subgraph [START_REF] Yanardag | Deep graph kernels[END_REF] and whole-graph [START_REF] De Lara | A simple baseline algorithm for graph classification[END_REF][START_REF] Narayanan | graph2vec: Learning distributed representations of graphs[END_REF] embeddings. Each category better fits the needs of different applications and problems.

In this paper, we focus on the information that is used in addition to the plain structure by some embedding approaches. Especially, we study the impact of the node labels that are used by Graph2vec [START_REF] Narayanan | graph2vec: Learning distributed representations of graphs[END_REF], a whole-graph embedding method. We study the effectiveness of such additional information in the context of online abuse detection.

Data and Method

We focus on a task consisting in detecting abusive messages in chat logs. This can be formulated as a classification problem consisting in deciding if a message is abusive or not. Our dataset is the same as the one described in [START_REF] Papegnies | Conversational networks for automatic online moderation[END_REF]. It is composed of 1, 320 messages, equally distributed between the Abuse and Nonabuse classes.

In order to turn a chat log into a graph, we rely on an extraction method that we previously introduced in [START_REF] Papegnies | Conversational networks for automatic online moderation[END_REF]. For each message, it produces a conversational graph whose nodes represent users and links model their interactions. It is built by leveraging the targeted message itself, but also the neighboring messages constituting the conversation to which it belongs. Classifying a message amounts to classifying the corresponding conversational graph. In our previous work [START_REF] Papegnies | Conversational networks for automatic online moderation[END_REF], we experimented with a large set of 459 topological measures in order to get the most exhaustive representation of the graphs that we could, and fetched them to the classifier. We had to leverage feature selection to identify the most discriminant ones.

Here, instead, we use graph embedding approaches to automatically learn a representation of conversational graphs. Specifically, we use Graph2vec [START_REF] Narayanan | graph2vec: Learning distributed representations of graphs[END_REF], a method that is able to represent a whole graph as a low-dimensional vector while preserving some of its topological properties. The algorithm takes the set of graphs to represent, and outputs their representations by applying a two-step process. It first identifies the subgraphs surrounding each node and constituting the graph. More precisely, it looks for so-called rooted subgraphs, i.e. node neighborhoods of a certain order. Second, these subgraphs are considered as the vocabulary and fetched to a doc2vec SkipGram [START_REF] Le | Distributed representations of sentences and documents[END_REF] model. This embedding method captures structural equivalence, i.e. graphs whose nodes tend to possess this form of similarity will be close in the representation space.

In addition, this method requires the user to provide a label associated to each node. It does not have to be unique, for instance by default the method uses the node degree. We propose a few alternative labeling strategies, and assess how much they fit our specific situation. The first three can be considered as baselines. First, Degree is the default approach, that uses the degree of each node as its label. Second, Random multiple assigns a random label to every node by considering each graph separately. The same label can thus be assigned to distinct nodes over the whole corpus. Third, Random unique assigns a random label to every node, each label being unique in the whole corpus (and not only in the considered graph). The last three strategies are designed specifically for our situation. Fourth, as nodes correspond to users in our graphs, and users have unique IDs in our dataset, Author ID consists in using the ID of each node as its label. Fifth, Distance to target uses the distance to the targeted node (author of the targeted message) as the label. Sixth and finally, Targeted is a binary label depicting whether the node is the targeted node (here, the message to classify) or not.

Results

We use graph embedding approaches to generate vector representations of the conversational graphs, then fetch these representations to an SVM to perform the classification. We set-up our experiments using a 10-fold cross-validation, using 70% of the data for training and the remaining 30% for testing.

Table 1. F -measures obtained by Graph2vec with different labeling strategies. The last two rows correspond to our previous method from [START_REF] Papegnies | Conversational networks for automatic online moderation[END_REF], which relies on topological measures and its subset of Top Features (TF). The total runtime is expressed as h:min:s.

Node Table 1 presents the micro F -measure values obtained by Graph2vec on our dataset with the different labeling strategies. It shows that node labels have an important impact on the classification performances. Unsurprisingly, the two random strategies yield the lowest performance, as their labels do not bring any information. In the case of Random multiple strategy, they can even introduce incorrect relations between nodes representing different users but sharing the same label. Degree is better than both other baselines, which suggests that nodes with the same degree might have a similar role in the graph. Maybe one does not need to distinguish between individual nodes, but can instead adopt a more generic approach and deal with classes of nodes. Strategies Author ID and Targeted are both above Degree, and perform approximately at the same level. This indicates that introducing a problem-related but non-structural information seems to improve the representation. Furthermore, Targeted performs slightly better than Author ID which corroborates our assumption regarding the possibility to handle nodes in a more generic way. The best performance is reached with the Distance to target strategy, which is centered around the targeted node, a key element of our problem. It allows combining both a generic description of the nodes and some problem-related aspects in the same representation.

The method we presented previously [START_REF] Papegnies | Conversational networks for automatic online moderation[END_REF][START_REF] Cécillon | Abusive language detection in online conversations by combining content-and graph-based features[END_REF], which relies on topological measures, outperforms all the strategies proposed in this article, with a 88.08 Fmeasure. However, our best strategy, Distance to target, achieves a promising 84.03 F -measure while being much more time efficient. Topological measures TF yields a better performance than Distance to target while having a runtime close to that of this strategy. However, it requires an important effort to determine the top features. Our experiments show that using appropriate labels with Graph2vec allows improving the quality of the generated embeddings. Based on this observation, we can suppose that other graph embedding methods that only focus on the graph structure to construct representations could benefit from such information. Hence, adapting such methods in order to incorporate additional information in the embedding process could reduce the gap between embedding approaches and our Topological Measures method.


Nejat Arinik

Rosa Figueiredo

Vincent Labatut

Multiple Optimal Solutions but Single Search: A Study of the Correlation Clustering Problem

Keywords: Signed Graph, Graph Partitioning, Correlation Clustering, Structural Balance

Introduction

A signed graph, whose links are labeled as positive (+) and negative (-), is considered structurally balanced [START_REF] Davis | Clustering and structural balance in graphs[END_REF] if it can be partitioned into a number of clusters, such that positive (negative) links are located inside (in-between) the clusters. Due to the imbalanced nature of real-world networks, various measures have been defined to quantify the amount of imbalance. Such measures are expressed relatively to a graph partition, so processing the graph balance amounts to identifying the partition corresponding to the lowest imbalance measure. A well-known measure among them corresponds to the definition of the Correlation Clustering problem (CC), and it consists in counting the numbers of misplaced links [START_REF] Davis | Clustering and structural balance in graphs[END_REF]. 1: 3 (out of 22) different optimal CC solutions obtained for the same network: a) P a = {{n 1 ,n 5 ,n 6 ,n 7 }, {n 2 ,n 3 ,n 4 }}; b) P b = {{n 5 ,n 6 ,n 7 }, {n 1 ,n 2 ,n 3 ,n 4 }}; and c) P c = {{n 1 ,n 2 ,n 5 }, {n 6 ,n 7 }, {n 3 ,n 4 }}. Red and green lines represent negative and positive links, respectively. The graph is complete, but for clarity, some negative links between clusters are intentionally omitted.

In the literature, a large number of applied works solve the CC problem to get a better understanding of some studied real-world system (e.g vote analysis [1], etc.). Most of them rely on heuristic approaches, especially when dealing with large networks, due to the complexity of the problem. But a non-negligible number of studies are also concerned with optimality. In any case, the standard approach is to find a single partition, even if other optimal or high scoring solutions possibly exist. Figure 1 illustrates such situation: solving the CC problem for this network of 7 nodes yields 22 distinct optimal partitions. We show only a few of them to highlight how similar these can be. For instance, P a (left) and P b (middle) are very similar partition-wise, as they are both bisections differing only in the cluster assignment of n1. By comparison, P c (right) is less similar: it contains an extra cluster obtained by separating an element from each cluster of the previous solutions, in addition to n1.

This focus on a single solution raises several questions. First, many optimal partitions may coexist. If so, are they all equally relevant to the application problem at hand? If not, it may be necessary to design a more appropriate version of CC, in order to distinguish them, possibly based on some external criteria related to the application context. Second, how different are these partitions? Application-wise, very similar partitions (e.g. single node differences as between P a and P b ) could be assimilated to the same solution, whereas significantly different partitions might correspond to dramatically different interpretations. Finally, if groups of similar partitions exist, how to identify a representative partition to summarize them?

To the best of our knowledge, no one has ever tried to answer these questions in the literature. Our goal here is to do by characterizing the optimal solution space associated with a given network for the CC problem. Put differently, we want to study the CC problem itself, and not some of its existing resolution methods. As this preliminary work, we focus on complete unweighted graphs, and consider only optimal solutions.

Method

We propose a 4-stepped method. The first step is to obtain all optimal partitions. For this purpose, we use an integer programming-based method able to solve the CC problem exactly. The second step consists in computing the similarity between the partitions through the measure Variation of Information (VI) 1 , in order to perform the third step, which is a cluster analysis. This leads to a set of clusters (or only one), each one gathering similar partitions. The fourth step is to process what we call the representative partition for each cluster, which is supposed to be the most similar over all the partitions belonging to the cluster.

Preliminary Experimental Results

We investigate into the space of optimal solutions based on synthetic signed networks, which are complete and unweighted. Network generation relies on three parameters: n (graph size), k (number of cluster) and q (proportion of misplaced links). Once perfectly balanced networks are created with n and k, the sign of links are inverted based on q. This process is repeated 10 times for replication. Figure 2 shows the number of optimal solutions by varying n and q, while fixing k = 4. Interestingly, a large number of optimal solutions (1093) are obtained for n = 28, q = 0.3. However, there is no direct link between the number of optimal solutions and n, since increasing n can also increase the number of cluster assignment combinations.

FIG. 2: Evolution of the number of optimal partitions found in synthetic signed networks generated with parameters n ∈ {12, 16, 20, 24, 28, 32, 36}, q ∈ {0.1, 0.2, 0.3} and k=4.


Noé Cecillon

Vincent Labatut
email: vincent.labatut@univ-avignon.fr

Richard Dufour

Georges Linarès

Georges Linar

Abusive Language Detection in Online Conversations by Combining Content-and Graph-based Features

Keywords: Automatic abuse detection, Content analysis, Conversational graph, Online conversations, Social networks

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

Internet widely impacted the way we communicate. Online communities, in particular, have grown to become important places for interpersonal communications. They get more and more attention from companies to advertise their products or from governments interested in monitoring public discourse. Online communities come in various shapes and forms, but they are all exposed to abusive behavior. The definition of what exactly is considered as abuse depends on the community, but generally includes personal attacks, as well as discrimination based on race, religion or sexual orientation.

Abusive behavior is a risk, as it is likely to make important community members leave, therefore endangering the community, and even trigger legal issues in some countries. Moderation consists in detecting users who act abusively, and in taking action against them. Currently this moderation work is mainly a manual process, and since it implies high human and financial costs, companies have a keen interest in its automation. One way of doing so is to consider this task as a classification problem consisting in automatically determining if a user message is abusive or not.

A number of works have tackled this problem, or related ones, in the literature. Most of them focus only on the content of the targeted message to detect abuse or similar properties. For instance, [START_REF] Spertus | Smokey: Automatic recognition of hostile messages[END_REF] applies this principle to detect hostility, [START_REF] Dinakar | Modeling the detection of textual cyberbullying[END_REF] for cyberbullying, and [START_REF] Chen | Detecting offensive language in social media to protect adolescent online safety[END_REF] for offensive language. These approaches rely on a mix of standard NLP features and manually crafted application-specific resources (e.g. linguistic rules). We also proposed a content-based method (Papegnies et al., 2017a) using a wide array of language features (Bag-of-Words, tf -idf scores, sentiment scores). Other approaches are more machine learning intensive, but require larger amounts of data. Recently, [START_REF] Wulczyn | Ex Machina: Personal attacks seen at scale[END_REF] created three datasets containing individual messages collected from Wikipedia discussion pages, annotated for toxicity, personal attacks and aggression, respectively. They have been leveraged in recent works to train Recursive Neural Network operating on word embeddings and character n-gram features [START_REF] Pavlopoulos | Deep learning for user comment moderation[END_REF][START_REF] Mishra | Neural character-based composition models for abuse detection[END_REF]. However, the quality of these direct content-based approaches is very often related to the training data used to learn abuse detection models. In the case of online social networks, the great variety of users, including very different language registers, spelling mistakes, as well as intentional users obfuscation, makes it almost impossible to have models robust enough to be applied in all cases. [START_REF] Hosseini | Deceiving google's perspective api built for detecting toxic comments[END_REF] have then shown that it is very easy to bypass automatic toxic comment detection systems by making the abusive content difficult to detect (intentional spelling mistakes, uncommon negatives...).

Because the reactions of other users to an abuse case are completely beyond the abuser's control, some authors consider the content of messages occurring around the targeted message, instead of focusing only on the targeted message itself. For instance, [START_REF] Yin | Detection of harassment on Web 2.0[END_REF] use features derived from the sentences neighboring a given message to detect harassment on the Web. [START_REF] Balci | Automatic analysis and identification of verbal aggression and abusive behaviors for online social games[END_REF] take advantage of user features such as the gender, the number of in-game friends or the number of daily logins to detect abuse in the community of an online game. In our previous work [START_REF] Papegnies | Conversational networks for automatic online moderation[END_REF], we proposed a radically different method that completely ignores the textual content of the messages, and relies only on a graph-based modeling of the conversation. This is the only graph-based approach ignoring the linguistic content proposed in the context of abusive messages detection. Our conversational network extraction process is inspired from other works leveraging such graphs for other purposes: chat logs [START_REF] Mutton | Inferring and visualizing social networks on Internet Relay Chat[END_REF] or online forums [START_REF] Forestier | Extracting social networks to understand interaction[END_REF] interaction modeling, user group detection [START_REF] Camtepe | A tool for Internet chatroom surveillance[END_REF]. Additional references on abusive message detection and conversational network modeling can be found in [START_REF] Papegnies | Conversational networks for automatic online moderation[END_REF].

In this paper, based on the assumption that the interactions between users and the content of the exchanged messages convey different information, we propose a new method to perform abuse detection while leveraging both sources. For this purpose, we take advantage of the content- [START_REF] Papegnies | Impact of content features for automatic online abuse detection[END_REF] and graph-based [START_REF] Papegnies | Conversational networks for automatic online moderation[END_REF]) methods that we previously developed. We propose three different ways to combine them, and compare their performance on a corpus of chat logs originating from the community of a French multiplayer online game. We then perform a feature study, finding the most informative ones and discussing their role. Our contribution is twofold: the exploration of fusion methods, and more importantly the identification of discriminative features for this problem.

The rest of this article is organized as follows. In Section 2, we describe the methods and strategies used in this work. In Section 3 we present our dataset, the experimental setup we use for this classification task, and the performances we obtained. Finally, we summarize our contributions in Section 4 and present some perspectives for this work.

METHODS

In this section, we summarize the content-based method from [START_REF] Papegnies | Impact of content features for automatic online abuse detection[END_REF]) (Section 2.1) and the graph-based method from [START_REF] Papegnies | Conversational networks for automatic online moderation[END_REF] (Section 2.2). We then present the fusion method proposed in this paper, aiming at taking advantage of both sources of information (Section 2.3). Figure 1 shows the whole process, and is discussed through this section. [START_REF] Papegnies | Impact of content features for automatic online abuse detection[END_REF] (content-based method) and [START_REF] Papegnies | Conversational networks for automatic online moderation[END_REF] (graph-based method), whereas the contribution presented in this article appears on the right side (fusion strategies). Figure available at 10.6084/m9.figshare.7442273 under CC-BY license.

Content-Based Method

This method corresponds to the bottom-left part of Figure 1 (in green). It consists in extracting certain features from the content of each considered message, and to train a Support Vector Machine (SVM) classifier to distinguish abusive (Abuse class) and non-abusive (Non-abuse class) messages [START_REF] Papegnies | Impact of content features for automatic online abuse detection[END_REF]. These features are quite standard in Natural Language Processing (NLP), so we only describe them briefly here.

We use a number of morphological features. We use the message length, average word length, and maximal word length, all expressed in number of characters. We count the number of unique characters in the message. We distinguish between six classes of characters (letters, digits, punctuation, spaces, and others) and compute two features for each one: number of occurrences, and proportion of characters in the message. We proceed similarly with capital letters. Abusive messages often contain a lot of copy/paste. To deal with such redundancy, we apply the Lempel-Ziv-Welch (LZW) compression algorithm [START_REF] Batista | Texture classification using the lempel-ziv-welch algorithm[END_REF] to the message and take the ratio of its raw to compress lengths, expressed in characters.

Abusive messages also often contain extra-long words, which can be identified by collapsing the message: extra occurrences of letters repeated more than two times consecutively are removed. For instance, "looooooool" would be collapsed to "lool". We compute the difference between the raw and collapsed message lengths.

We also use language features. We count the number of words, unique words and bad words in the message. For the latter, we use a predefined list of insults and symbols considered as abusive, and we also count them in the collapsed message. We compute two overall tf -idf scores corresponding to the sums of the standard tf -idf scores of each individual word in the message. One is processed relatively to the Abuse class, and the other to the Non-abuse class. We proceed similarly with the collapsed message. Finally, we lower-case the text and strip punctuation, in order to represent the message as a basic Bag-of-Words (BoW). We then train a Naive Bayes classifier to detect abuse using this sparse binary vector (as represented in the very bottom part of Figure 1). The output of this simple classifier is then used as an input feature for the SVM classifier.

Graph-Based Method

This method corresponds to the top-left part of Figure 1 (in red). It completely ignores the content of the messages, and only focuses on the dynamics of the conversation, based on the interactions between its participants [START_REF] Papegnies | Conversational networks for automatic online moderation[END_REF]. It is three-stepped: 1) extracting a conversational graph based on the considered message as well as the messages preceding and/or following it; 2) computing the topological measures of this graph to characterize its structure; and 3) using these values as features to train an SVM to distinguish between abusive and non-abusive messages. The vertices of the graph model the participants of the conversation, whereas its weighted edges represent how intensely they communicate. The graph extraction is based on a number of concepts illustrated in Figure 2, in which each rectangle represents a message. The extraction process is restricted to a so-called context period, i.e. a sub-sequence of messages including the message of interest, itself called targeted message and represented in red in Figure 2. Each participant posting at least one message during this period is modeled by a vertex in the produced conversational graph. A mobile window is slid over the whole period, one message at a time. At each step, the network is updated either by creating new links, or by updating the weights of existing ones. This sliding window has a fixed length expressed in number of messages, which is derived from ergonomic constraints relative to the online conversation platform studied in Section 3. It allows focusing on a smaller part of the context period. At a given time, the last message of the window (in blue in Figure 2) is called current message and its author current author. The weight update method assumes that the current message is aimed at the authors of the other messages present in the window, and therefore connects the current author to them (or strengthens their weights if the edge already exists). It also takes chronology into account by favoring the most recent authors in the window. Three different variants of the conversational network are extracted for one given targeted message: the Before network is based on the messages posted before the targeted message, the After network on those posted after, and the Full network on the whole context period. Figure 3 shows an example of such networks obtained for a message of the corpus described in Section 3.1. Once the conversational networks have been extracted, they must be described through numeric values in order to feed the SVM classifier. This is done through a selection of standard topological measures allowing to describe a graph in a number of distinct ways, focusing on different scales and scopes. The scale denotes the nature of the characterized entity. In this work, the individual vertex and the whole graph are considered. When considering a single vertex, the measure focuses on the targeted author (i.e. the author of the targeted message). The scope can be either micro-, meso-or macroscopic: it corresponds to the amount of information considered by the measure. For instance, the graph density is microscopic, the modularity is mesoscopic, and the diameter is macroscopic. All these measures are computed for each graph, and allow describing the conversation surrounding the message of interest. The SVM is then trained using these values as features. In this work, we use exactly the same measures as in [START_REF] Papegnies | Conversational networks for automatic online moderation[END_REF].

Fusion

We now propose a new method seeking to take advantage of both previously described ones. It is based on the assumption that the content-and graph-based features convey different information. Therefore, they could be complementary, and their combination could improve the classification performance. We experiment with three different fusion strategies, which are represented in the right-hand part of Figure 1.

The first strategy follows the principle of Early Fusion. It consists in constituting a global feature set containing all content-and graph-based features from Sections 2.1 and 2.2, then training a SVM directly using these features. The rationale here is that the classifier has access to the whole raw data, and must determine which part is relevant to the problem at hand.

The second strategy is Late Fusion, and we proceed in two steps. First, we apply separately both methods described in Sections 2.1 and 2.2, in order to obtain two scores corresponding to the output probability of each message to be abusive given by the content-and graph-based methods, respectively. Second, we fetch these two scores to a third SVM, trained to determine if a message is abusive or not. This approach relies on the assumption that these scores contain all the information the final classifier needs, and not the noise present in the raw features.

Finally, the third fusion strategy can be considered as Hybrid Fusion, as it seeks to combine both previous proposed ones. We create a feature set containing the content-and graph-based features, like with Early Fusion, but also both scores used in Late Fusion. This whole set is used to train a new SVM. The idea is to check whether the scores do not convey certain useful information present in the raw features, in which case combining scores and features should lead to better results.

EXPERIMENTS

In this section, we first describe our dataset and the experimental protocol followed in our experiments (Section 3.1). We then present and discuss our results, in terms of classification performance (Sections 3.2) and feature selection (Section 3.3).

Experimental protocol

The dataset is the same as in our previous publications [START_REF] Papegnies | Impact of content features for automatic online abuse detection[END_REF][START_REF] Papegnies | Conversational networks for automatic online moderation[END_REF]. It is a proprietary database containing 4,029,343 messages in French, exchanged on the in-game chat of SpaceOrigin1 , a Massively Multiplayer Online Role-Playing Game (MMORPG). Among them, 779 have been flagged as being abusive by at least one user in the game, and confirmed as such by a human moderator. They constitute what we call the Abuse class. Some inconsistencies in the database prevent us from retrieving the context of certain messages, which we remove from the set. After this cleaning, the Abuse class contains 655 messages. In order to keep a balanced dataset, we further extract the same number of messages at random from the ones that have not been flagged as abusive. This constitutes our Non-abuse class. Each message, whatever its class, is associated to its surrounding context (i.e. messages posted in the same thread).

The graph extraction method used to produce the graph-based features requires to set certain parameters. We use the values matching the best performance, obtained during the greedy search of the parameter space performed in [START_REF] Papegnies | Conversational networks for automatic online moderation[END_REF]. In particular, regarding the two most important parameters (see Section 2.2), we fix the context period size to 1,350 messages and the sliding window length to 10 messages. Implementation-wise, we use the iGraph library [START_REF] Csardi | The igraph software package for complex network research[END_REF] to extract the conversational networks and process the corresponding features. We use the Sklearn toolkit [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF] to get the text-based features. We use the SVM classifier implemented in Sklearn under the name SVC (C-Support Vector Classification). Because of the relatively small dataset, we set-up our experiments using a 10-fold cross-validation. Each fold is balanced between the Abuse and Non-abuse classes, 70% of the dataset being used for training and 30% for testing.

Classification Performance

Table 1 presents the Precision, Recall and F -measure scores obtained on the Abuse class, for both baselines (Content-based [START_REF] Papegnies | Impact of content features for automatic online abuse detection[END_REF] and Graph-based [START_REF] Papegnies | Conversational networks for automatic online moderation[END_REF]) and all three proposed fusion strategies (Early Fusion, Late Fusion and Hybrid Fusion). It also shows the number of features used to perform the classification, the time required to compute the features and perform the cross validation (Total Runtime) and to compute one message in average (Average Runtime). Note that Late Fusion has only 2 direct inputs (content-and graph-based SVMs), but these in turn have their own inputs, which explains the values displayed in the table.

Our first observation is that we get higher F -measure values compared to both baselines when performing the fusion, independently from the fusion strategy. This confirms what we expected, i.e. that the information encoded in the interactions between the users differs from the information conveyed by the content of the messages they exchange. Moreover, this shows that both sources are at least partly complementary, since the performance increases when merging them. On a side note, the correlation between the score of the graph-and content-based classifiers is 0.56, which is consistent with these observations. Next, when comparing the fusion strategies, it appears that Late Fusion performs better than the others, with an F -measure of 93.26. This is a little bit surprising: we were expecting to get superior results from the Early Fusion, which has direct access to a much larger number of raw features (488). By comparison, the Late Fusion only gets 2 features, which are themselves the outputs of two other classifiers. This means that the Content-Based and Graph-Based classifiers do a good work in summarizing their inputs, without loosing much of the information necessary to efficiently perform the classification task. Moreover, we assume that the Early Fusion classifier struggles to estimate an appropriate model when dealing with such a large number of features, whereas the Late Fusion one benefits from the pre-processing performed by its two predecessors, which act as if reducing the dimensionality of the data. This seems to be confirmed by the results of the Hybrid Fusion, which produces better results than the Early Fusion, but is still below the Late Fusion. This point could be explored by switching to classification algorithm less sensitive to the number of features. Alternatively, when considering the three SVMs used for the Late Fusion, one could see a simpler form of a very basic Multilayer Perceptron, in which each neuron has been trained separately (without system-wide backpropagation). This could indicate that using a regular Multilayer Perceptron directly on the raw features could lead to improved results, especially if enough training data is available.

Regarding runtime, the graph-based approach takes more than 8 hours to run for the whole corpus, mainly because of the feature computation step. This is due to the number of features, and to the compute-intensive nature of some of them. The content-based approach is much faster, with a total runtime of less than 1 minute, for the exact opposite reasons. Fusion methods require to compute both content-and graph-based features, so they have the longest runtime.

Table 1. Comparison of the performances obtained with the methods (Content-based, Graph-based, Fusion) and their subsets of Top Features (TF). The total runtime is expressed as h:min:s. See text for details.

Method

Feature Study

We now want to identify the most discriminative features for all three fusion strategies. We apply an iterative method based on the Sklearn toolkit, which allows us to fit a linear kernel SVM to the dataset and provide a ranking of the input features reflecting their importance in the classification process. Using this ranking, we identify the least discriminant feature, remove it from the dataset, and train a new model with the remaining features. The impact of this deletion is measured by the performance difference, in terms of F -measure. We reiterate this process until only one feature remains. We call Top Features (TF) the minimal subset of features allowing to reach 97% of the original performance (when considering the complete feature set).

We apply this process to both baselines and all three fusion strategies. We then perform a classification using only their respective TF. The results are presented in Table 1. Note that the Late Fusion TF performance is obtained using the scores produced by the SVMs trained on Content-based TF and Graphbased TF. These are also used as features when computing the TF for Hybrid Fusion TF (together with the raw content-and graph-based features). In terms of classification performance, by construction, the methods are ranked exactly like when considering all available features.

The Top Features obtained for each method are listed in Table 2. The last 4 columns precise which variants of the graph-based features are concerned. Indeed, as explained in Section 2.2, most of these topological measures can handle/ignore edge weights and/or edge directions, can be vertex-or graph-focused, and can be computed for each of the three types of networks (Before, After and Full).

There are three Content-Based TF. The first is the Naive Bayes prediction, which is not surprising as it comes from a fully fledged classifier processing BoWs. The second is the tf -idf score computed over the Abuse class, which shows that considering term frequencies indeed improve the classification performance. The third is the Capital Ratio (proportion of capital letters in the comment), which is likely to be caused by abusive message tending to be shouted, and therefore written in capitals. The Graph-Based TF are discussed in depth in our previous article [START_REF] Papegnies | Conversational networks for automatic online moderation[END_REF]. To summarize, the most important features help detecting changes in the direct neighborhood of the targeted author (Coreness, Strength), in the average node centrality at the level of the whole graph in terms of distance (Closeness), and in the general reciprocity of exchanges between users (Reciprocity).

We obtain 4 features for Early Fusion TF. One is the Naive Bayes feature (content-based), and the other three are topological measures (graph-based features). Two of the latter correspond to the Corenessof the targeted author, computed for the Before and After graphs. The third topological measure is his/her Eccentricity. This reflects important changes in the interactions around the targeted author. It is likely caused by angry users piling up on the abusive user after he has posted some inflammatory remark. For Hybrid Fusion TF, we also get 4 features, but those include in first place both SVM outputs from the content-and graph-based classifiers. Those are completed by 2 graph-based features, including Strength (also found in the Graph-based and Late Fusion TF) and Coreness (also found in the Graph-based, Early Fusion and Late Fusion TF).

Besides a better understanding of the dataset and classification process, one interesting use of the TF is that they can allow decreasing the computational cost of the classification. In our case, this is true for all methods: we can retain 97% of the performance while using only a handful of features instead of hundreds. For instance, with the Late Fusion TF, we need only 3% of the total Late Fusion runtime.

CONCLUSION AND PERSPECTIVES

In this article, we tackle the problem of automatic abuse detection in online communities. We take advantage of the methods that we previously developed to leverage message content (Papegnies et al., 2017a) and interactions between users [START_REF] Papegnies | Conversational networks for automatic online moderation[END_REF], and create a new method using both types of information simultaneously. We show that the features extracted from our content-and graph-based approaches are complementary, and that combining them allows to sensibly improve the results up to 93.26 (F -measure). One limitation of our method is the computational time required to extract certain features. However, we show that using only a small subset of relevant features allows to dramatically reduce the processing time (down to 3%) while keeping more than 97% of the original performance.

Another limitation of our work is the small size of our dataset. We must find some other corpora to test our methods at a much higher scale. However, all the available datasets are composed of isolated messages, when we need threads to make the most of our approach. A solution could be to start from datasets such as the Wikipedia-based corpus proposed by [START_REF] Wulczyn | Ex Machina: Personal attacks seen at scale[END_REF], and complete them by reconstructing the original conversations containing the annotated messages. This could also be the opportunity to test our methods on an other language than French. Our content-based method may be impacted by this change, but this should not be the case for the graph-based method, as it is independent from the content (and therefore the language). Besides language, a different online community is likely to behave differently from the one we studied before. In particular, its members could react differently differently to abuse. The Wikipedia dataset would therefore allow assessing how such cultural differences affect our classifiers, and identifying which observations made for Space Origin still apply to Wikipedia.

Figure 3 .

 3 Figure 3. Example of the three types of conversational networks extracted for a given context period: Before (left), After (center), and Full (right). The author of the targeted message is represented in red. Figure available at 10.6084/m9.figshare.7442273 under CC-BY license.

  Figure 1. Representation of our processing pipeline. Existing methods refers to our previous work described in

		Graph-based Method			Graph			EARLY	Score Fusion Early
	Conversation	CONVERSATIONAL	Conversational Graph	COMPUTING	Based Features	GRAPH		FUSION SVM
		NETWORK		TOPOLOGICAL		BASED
		EXTRACTION		MEASURES		SVM	Graph Based Score	LATE	Score Fusion Late
	User3: quoi ?!							FUSION
	User2: naaaan !		FEATURE EXTRACTION			CONTENT BASED SVM	Content Based Score	SVM HYBRID	Score Fusion Hybrid
		BoW EXTRACTION		NAIVE BAYES	Content Based Features			FUSION SVM
		Content-based Method	Bag-of-Words
			EXISTING METHODS				CONTRIBUTION

Table 2 .

 2 Top features obtained for our 5 methods. The letters in the Graph column stand for Before (B), After (A) and Full (F). Those in the Weights and Directions columns stand for: Unweighted or Undirected (U), Weighted (W), Directed (D), Incoming (I) and Outgoing (O). Those in the Scale column mean Graph-scale (G) or Vertex-scale (N).

	Method Top Features	Graph Weights Directions Scale
		Naive Bayes	-	-	-	-
	Content-Based	tf -idf Abuse Score	-	-	-	-
		Character Capital Ratio	-	-	-	-
		Coreness Score	F	-	I	G
		PageRank Centrality	A	U	D	N
		Strength Centrality	F	W	O	N
		Vertex Count	F	-	-	G
	Graph-Based	Closeness Centrality Closeness Centrality	B B	W W	O O	G N
		Authority Score	B	W	D	G
		Hub Score	B	U	D	N
		Reciprocity	A	-	D	G
		Closeness Centrality	A	W	U	N
		Coreness Score	A	-	O	G
	Early Fusion	Coreness Score Eccentricity	B B	--	I I	G G
		Naive Bayes	-	-	-	-
	Late Fusion Content-Based TF ∪ Graph-Based TF -	-	-	-
		Graph-based output	-	-	-	-
	Hybrid Fusion	Content-based output Strength Centrality	-A	-W	-O	-N
		Coreness Score	B	-	I	G

This is a provisional file, not the final typeset article

Frontiers

https://play.spaceorigin.fr/ This is a provisional file, not the final typeset article


Adrien Gresse

Mickael Rouvier

Richard Dufour

Vincent Labatut

Jean-Francois Bonastre

Acoustic Pairing of Original and Dubbed Voices in the Context of Video Game Localization

Keywords: voice casting, voice similarity, speaker recognition, i-vector, video game

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Voice casting aims at selecting the most appropriate voice to dub a character [START_REF] Obin | On automatic voice casting for expressive speech: Speaker recognition vs. speech classification[END_REF]. In a multilingual context, this voice selection process does not simply rely on the character but also on the voice of the original actor. Voice casting is mainly performed by a human operator and raises two main problems. First, the process is very sensitive to the operator's subjectivity. Second, the operator cannot take a decision based on all available voices, since their number is usually too large. Above all, because of his expertise and his past, he is unconsciously encouraged to consider a small subset of voices with which he is used to work.

We focus our work on the domain of voice dubbing from a source language, the dubbing being the targeted language (for example, dubbing in French from an English voice), in the context of video games. Indeed, video games are now not simply a playful experience but also a medium offering a close immersion, akin to movies [START_REF] Bateman | Game Writing: Narrative Skills for Videogames[END_REF]. One of the critical aspects for these scripted games is the voice attribution process for the different characters. Since it directly impacts the gaming experience by influencing the player's immersion, video game producers want their characters to have a voice that mostly respects its specificities, like role, character's traits, attitudes, etc. This importance translates to game localization, the process of dubbing the audio tracks from a source to a target language. The dubbed voice is not just a simple transposition of the original voice to the target one, but more based on the speaker's ability to trans-pose his voice in order to act like the original one. In addition, the human experts have to take into account more sophisticated aspects, like cultural or social values related to the considered language. They also have to take care of the mental representations of voices, stereotypes for short. As an illustration, a soldier could be stereotypically described as having a slightly hoarse and screamed voice.

Our final goal is to approximate the human operator expertise through the definition of appropriate statistical models. In such an automatic recommendation system for assisted voice casting, different sources of variability should be considered. The first concerns the speaker, which can be easily confused with the second: the character. One of our objectives is to build a system where these two dimensions can be observed independently. In other words, when considering a single speaker, we want to be able to identify all the different characters he can play. Note that a one-to-one association between actors and characters would just lead to a voice similarity system -close to a speaker recognition system-working on multiple languages. This brings us to the third source of variability, the language, which we want to decrease together with the linguistic content of voice segments. We propose a preliminary work to automatically pair original and dubbed voices using character voice segments. We extend an i-vector based automatic speaker recognition approach to the characterization of higher-level informations representing acted voices similarity. Given the fact that speakers use different languages, our proposed contribution for video game voice localization aims at reducing the language variability in order to better extract acoustic information relying on characters. To do so, we propose to perform a transformation in order to project the dubbed segments to the original ones.

The rest of this paper is organized as follows. Firstly, we present related work about voice casting and perceptual voice similarity in Section 2. We then describe the proposed acoustic pairing approach in Section 3. The experimental protocol as well as the experiments and discussions about our system performance are presented in Sections 4 and 5 respectively. Finally, Section 6 concludes and gives some perspectives.

Related work

Due to its subjective nature, automatic voice perception is a very difficult problem. The first step of this process is to handle how humans perceive voices, and the literature abounds of such work, especially in the sociology and psychology domains [START_REF] Zuckerman | What sounds beautiful is good: The vocal attractiveness stereotype[END_REF][START_REF] Scherer | Vocal communication of emotion: A review of research paradigms[END_REF][START_REF] Krauss | Inferring speakers' physical attributes from their voices[END_REF]. They show different factors that impact judgment and perception, not directly for voice perceptual similarity but for perception of speaker, state, traits, or even emotions. These factors are voice indicators (e.g. pitch, intensity, . . . ) and speech cues (e.g. non-fluency, speech rate, . . . ). In addition, these work present particular acoustic features that could clearly help distinguishing personality traits. [START_REF] Nolan | The role of voice quality 'settings' in perceived voice similarity[END_REF] showed that percep-tual voice similarity is strongly correlated with particular voice quality settings. However, this work only considers speakers with the same accents. Others work have also been conducted on the automatic attribution of acoustic features to personality traits [START_REF] Mohammadi | Automatic Attribution of Personality Traits Based on Prosodic Features[END_REF]. The research on voice perception has focused much more on the automation of personality trait attribution than on voice similarity. Work on personality traits are strongly related to our own work, because video game characters also have personality traits that must be supported by their voice. In addition, these aspects have to be preserved in the dubbed version.

There are few papers relating to voice casting, and the problem has recently started gaining interest. In [START_REF] Adachi | Perceptual similarity measurement of speech by combination of acoustic features[END_REF], the authors increase the users satisfaction of computer-generated characters through their instant casting movie system, called Future Cast, by using perceptual similar voices selection. They estimate the voice similarity by performing a weighted sum of acoustic distances between cepstral features computed with DTW. More recent work [START_REF] Obin | On automatic voice casting for expressive speech: Speaker recognition vs. speech classification[END_REF], [START_REF] Obin | Similarity search of acted voices for automatic voice casting[END_REF] dive deeply into the automatic voice casting problem, by exploring different models for perceptual voice similarity search. The authors compare GMM-based acoustic models and multi-label classification of paralinguistic features, with an objective experiment in order to measure the perceptual similarity of acted voices. In addition, they carry out a subjective experimentation by presenting to different human evaluators the 3 most similar voice samples according to both GMM-based acoustic models and multi-label recognition systems. Their results show an increased performance with respect to paralinguistic features recognition instead of classical speaker recognition systems. However, they finally conclude by considering the use of an acoustic model that would be more specific to voice casting than the one used for speaker recognition.

To our knowledge, there is no other scientific work that propose to look for acoustic representations of acted voices and even less between different languages. We propose an original approach in the form of a preliminary study that seeks to extend speaker recognition systems to the purpose of voice casting.

Proposed approach

In this paper, we focus on video game localization, because of the strongly typed voices present in this kind of multimedia products. Video games editors largely make use of stereotyped voices for their characters. In this approach, we explore the relation between original and dubbed voices by performing character verification given two different voice excerpts.

General system presentation

The desired system is coarsely represented in Figure 1, where two audio files represent the main inputs. The first one is referred to as the original audio segment, and the second one as the dubbed audio segment. As we want to compare voice similarity, our desired system outputs scores corresponding to the proximity of the two voice segments. The voice similarity system is voluntarily presented as a black box, as its composition will be discussed here. In this preliminary approach, we chose to measure the voice similarity using off-the-shelf ivector/PLDA components. We build three variants of our system, as illustrated in Figure 2. The different approaches are denoted System A (details in Section 3.3), and Systems B and C (details in Section 3.4). All of them consider two voice segments referred to by Src and Tgt, which correspond to the source (i.e. original) and target (i.e. dubbed) voice segments, respec- tively. In addition, all our systems consider different language configurations. An i-vector is extracted from an audio segment thanks to a total variability matrix (the T matrix). We use different T -matrix, adapted to the two considered languages (English and French). We thus have three language configurations for ivector extraction: EN-EN, EN-FR and FR-FR. The first part of the configuration name refers to the language used to represent the source voice segments, and the second part to the dubbed voice segments.

An i-vector/PLDA based approach

The voice segments are represented by i-vectors and the similarity between two i-vectors is estimated by a PLDA approach (a probabilistic version of Linear Discriminant Analysis [START_REF] Prince | Probabilistic linear discriminant analysis for inferences about identity[END_REF]), thus the obtained score is a likelihood ratio. An i-vector is a low-dimensional feature that characterizes speakers [START_REF] Dehak | Front end factor analysis for speaker verification[END_REF]. It is extracted from a variable length sequence of acoustic parameters representing a speech segment. In the i-vectors space, no distinction is made between the variability sources.

Baseline i-vector based similarity

Our approach considers a baseline i-vector system using a language-dependent T -matrix to extract the i-vector. So, each T -matrix has been trained on a corpus corresponding to a single language. Since our problem involves two languages, we are distinguishing two cases. In the first case, paired voice segments are represented by i-vectors extracted using a T -matrix trained for English (i.e. T matrix learned on an English corpus).

In the second case, both original and dubbed voice segments are represented by i-vectors extracted for the French language (i.e.

T matrix learned on a French corpus). This approach is referred to as System A in Figure 2.

Dubbed adapted i-vector comparison system

Since this work is dedicated to source/target-language paired audio segments, we have to deal with two languages in each voice similarity measure. We need to extract the i-vector for each audio file using the corresponding language-dependent Tmatrix. So original voice segments are represented by i-vectors with a total variability matrix learned on English data, and dubbed segments are represented by i-vectors learned on French data. Thus, an adaptation is needed in order to allow a comparison between these two i-vectors using PLDA. The heart of our contribution is dedicated to this language adaptation. We consider projecting i-vectors from a language to an other in order to perform comparison and scoring between them. The approach is inspired from [START_REF] Mikolov | Exploiting Similarities among Languages for Machine Translation[END_REF], where authors carry out word translation, and has been adapted to our problem. The projection is learned on a set of segment pairs that are represented by {xm, ym}, where x is a voice segment ivector in the source language, and y is its counterpart in the target language. Here, both segment x and y convey the same message m. We want to find a transformation function such that xm ≈ L(ym). We use the least squares method [START_REF] Legendre | Nouvelles méthodes pour la détermination des orbites des comètes[END_REF] to solve this linear equation, by computing the transformation matrix W that minimizes xm -ymW 2 . This can be seen as a linear mapping between two different spaces: source and target languages i-vector spaces. Here, the i-vectors are projected into the target language, but the inverse is also possible, depending on how we see the task. We perform the projection in System B and System C and it refers to W in Figure 2. In the same way as for System A, we use i-vectors extracted from different languages and we use the three configurations that have been described previously: EN-EN, EN-FR and FR-FR.

Experimental Protocol

Corpus Description

Our corpus has been built by extracting dialogs from the role playing video game Mass Effect 3. It was originally released in English, but it has also been translated into several others languages such as French. We extracted the audio files composing both the English and French versions of the dialogs. Each language dialog subsets contains 10,000 audio segments that are all hight-quality studio recorded audio files. A segment generally refers to a sentence. Our corpus includes 38 different characters dubbed by at least 30 different actors. We consider a set of characters all belonging to the video game, denoted P. Since two different actors act as the voice for the character k, we represent the character as a pair P k = (X k , Y k ) where X k is the set of its original voice segments and Y k its dubbed ones. We also dispose of all the different pairs of possible combinations of utterances as a set E = {(x l,m,i,k , y l,m,i,k )}, which constitutes our corpus, where x is the English segment i-vector and y the French one. Here, variable m refers to the linguistic content (i.e. the message), i to the speaker, and k to the character. Finally, variable l denotes the language used to extract the i-vector. We finally split the corpus into 3 subsets with 70%, 10% and 20% for train, dev, and test respectively.

The corpus approximately contains a total of 7.5 hours of speech, for each language. The different segments are 3.5 seconds long, in average. The amount of speech by character is related to the importance of the character in the game, and it is thus not uniformly distributed: its mean is 12 minutes and its standard deviation is 20 minutes.

Voice similarity estimation

Our experiments operate on 20 MFCC parameters (including log-energy) augmented with 20 first (∆) and 20 second (∆∆) derivatives, providing 60 dimensional feature vectors. We apply a cepstral mean normalization using a sliding window size of 3 seconds. We remove the low-energy frame, which corresponds mainly to silence. The low-energy algorithm is based on thresholding the log-energy and taking the consensus of threshold decisions within a window of 11 frames centered on the current frame. Finally, we train a Gender-dependent 2048 full component UBM and total variability matrix of low rank 400.

We set up two speaker recognition systems: English and French. For the English system, we train the UBM, total variability matrix and PLDA on NIST SRE 2004, 2005and 2006. For the French one, we train them on ESTER-1, ESTER-2, EPAC, ETAPE and REPERE.

Evaluation Metric

Our metric evaluates the system capacity to detect paired segments belonging to a same character. We chose to measure this capacity by computing the overall accuracy of the system given all the pairs of segments being scored. The accuracy is computed as a ratio between the number of correctly classified pairs and the total number of classified pairs. We chose to measure the correctness of a test by considering the k-best scores of the system among all possible pairs {(xi,n, yj,m)}. We put 2 constraints for the test validation. The first is that i = j, ∀n, m, which means both paired segments belong to the same character. The second is that the considered pairs must have been ranked among the k-best scores of the system for the set of pairs {(xi,n, Y )}, where xi,n is the segment n of character i, and Y refers to every segment from every character of the target language.

Results & Discussion

The results of voice similarity measures are presented in Table 1. The evaluation has been performed using a 3-fold crossvalidation on the overall corpus. To better analyze the systems performance, accuracies are computed with respect to the 3-best scores (i.e. pairing is considered as correct if the correct dubbed voice has been retrieved in the 3-best hypothesis). First of all, the baseline i-vector system (System A) is mainly evaluated given 2 different language configurations: EN-EN and FR-FR. Indeed, the EN-FR configuration, which obtained 53.9% for information, has no real interest in this approach since the comparison of i-vectors extracted from different languages is not relevant (no transformation is performed). As we can see, the first results show a reasonable accuracy for voice similarity detection, with an accuracy of 60.9% for EN-EN and 60.7% for FR-FR.

We now compare the results of System A with those of Systems B and C. Our first observation is that the performance is higher than the baseline system A for both Systems B and C.

The best results are obtained on the language configuration EN-EN, with accuracies of 71.7% and 70.7% for System B and C, respectively; whereas the language configuration EN-FR leads to a 69.6% accuracy for System B and 68.9% for System C. For the configuration FR-FR, the accuracy is slightly lower, with 69.4% for System B and 69.6% for System C. To get a global idea of the measured performance, Figure 3 presents the accuracies obtained by each system when varying the k-best number using the best configuration (EN-EN). In comparison to Table 1, similar tendencies can be observed no matter the k value chosen. We observe that in Systems B and C, there is no increased representational capacity when using English i-vectors to represent original voice segments and French i-vectors for dubbed ones. However, according to our results, the i-vector projection significantly increases the system performance. It seems that the projection operates like a speaker mapping system. The W matrix has learned to map speakers in the source language to ones in the target language. A possible explanation for the better results obtained with the EN-EN configuration is that 1) the comparison is performed in the same language i-vector space, and 2) it might benefit from the speaker mapping learned by the projection matrix. Since this projection is just an approxima- tion of xm = L(ym), it could explain why the system performances with the FR-FR configuration are not as good as with the EN-EN configuration. Finally, Figure 4 shows the overall confusion matrix obtained for the characters using System B and the EN-EN configuration for k = 1 (global accuracy of 43%). We observe good performances on the diagonal in the bottomright corner, that corresponds to the most represented characters. More important, this figure displays some errors which can be explained by the different numbers between actors and characters in the corpus. Indeed, we know some actor can lend his voice to several characters, and the matrix shows that some of the errors are associated to the different characters acted by a same voice actor. For example, the same actor plays one character at two different ages (new and old), and is considered as two different characters by the system. This could also be the reason why good results are observed in Table 1 (correct character being retrieved in the 3-best hypothesis). Finally, it reveals that this proposed preliminary work on voice similarity does not allow to completely answer the global problematic, namely make the different dimensions independent, but is a first step in the automatic voice casting problem.

Conclusion & Perspectives

In this paper, we present an automatic pairing system of original and dubbed voice segments for the detection of acted voice similarity. The proposed approach explores i-vectorsbased systems for automatic voices comparison between different languages and use projection matrix to reduce the language variability. Experiment have showed that a mapping between speakers can be learned from a source language to speakers from a target language, since the projection approach has significantly increased the accuracy of the system. These preliminary results for acted voice similarity is a first step toward a dedicated framework for automatic voice recommendation.

These initial encouraging results, however, highlighted some weaknesses in the proposed approach. First, the system could be improved by using similar acoustic data for training, the corpus used by now being far from the targeted data. In further work, we will explore acoustic features representation for character verification. In order to do so, we will investigate the use of other speech feature representations. As an illustration, Figure 5 shows different variability dimensions of our problematic. According to our experimental results, the approach presented here suffers from an ambiguity between the speaker and the character axes. Further researches will perform a contrastive experiment to observe how the system reacts. Finally, we will consider the use of classes of characters in order to reduce the variability of this dimension.

Figure 1 :Figure 2 :

 12 Figure 1: Similarity measure between two audio files is performed by our system.

Figure 3 :

 3 Figure 3: Accuracy on EN-EN configuration using different values of k.

Figure 4 :

 4 Figure 4: Characters confusion matrix using System B with EN-EN configuration (k=1).

Figure 5 :

 5 Figure 5: Voice casting variability dimensions. Here p represents a character in the variability space.

Table 1 :

 1 Results of the different approaches (3-best accuracy).

	System EN-EN EN-FR FR-FR
	A	60.9	53.9	60.7
	B	71.7	69.6	69.4
	C	70.7	68.9	69.6

Acknowledgements

This work is supported by the Foundation of the University of Avignon.