Emmanuele Chersoni
email: emmanuelechersoni@gmail.com

Adrià Torrens Urrutia

Philippe Blache
email: philippe.blache@univ-amu.fr

Alessandro Lenci
email: alessandro.lenci@unipi.it

Modeling Violations of Selectional Restrictions with Distributional Semantics

Distributional Semantic Models have been successfully used for modeling selectional preferences in a variety of scenarios, since distributional similarity naturally provides an estimate of the degree to which an argument satisfies the requirement of a given predicate. However, we argue that the performance of such models on rare verb-argument combinations has received relatively little attention: it is not clear whether they are able to distinguish the combinations that are simply atypical, or implausible, from the semantically anomalous ones, and in particular, they have never been tested on the task of modeling their differences in processing complexity. In this paper, we compare two different models of thematic fit by testing their ability of identifying violations of selectional restrictions in two datasets from the experimental studies.

Introduction

In recent years, Distributional Semantic Models (henceforth DSMs) have been at the core of one of the most active research areas in NLP, and have been applied to a wide variety of tasks. Among these, distributional modeling of selectional preferences [START_REF] Erk | A Flexible, Corpus-Driven Model of Regular and Inverse Selectional Preferences[END_REF][START_REF] Baroni | Distributional Memory: A General Framework for Corpus-based Semantics[END_REF] has been quite popular in computational psycholinguistics, since the similarity estimated by DSMs works very well for predicting the thematic fit between an argument and a verb. That is to say, the more the argument vector is similar to some kind of vector representation of the ideal filler of the verb slot (it can be either an abstract prototype, or a cluster of exemplars), the more the argument will satisfy the semantic requirements of the slot. The notion of thematic fit, as it has been proposed by the recent psycholinguistic research1 , is related to, but not totally equivalent to the classical notion of selectional preferences, since the former refers to a gradient compatibility between verb and role, whereas the latter conceives such compatibility as as boolean constraint evaluated on discrete semantic features (Lebani and [START_REF] Gianluca | A Distributional Model of Verb-Specific Semantic Roles Inferences[END_REF].

The distributional models of thematic fit have been evaluated by comparing the plausibility scores produced by the models with human-elicited judgements [START_REF] Erk | A Flexible, Corpus-Driven Model of Regular and Inverse Selectional Preferences[END_REF][START_REF] Baroni | Distributional Memory: A General Framework for Corpus-based Semantics[END_REF][START_REF] Greenberg | Improving Unsupervised Vector-Space Thematic Fit Evaluation via Role-Filler Prototype Clustering[END_REF][START_REF] Santus | Measuring Thematic Fit with Distributional Feature Overlap[END_REF], showing significant correlations. Moreover, they have been used to predict the composition and the update of argument expectations [START_REF] Lenci | Composing and Updating Verb Argument Expectations: A Distributional Semantic Model[END_REF][START_REF] Chersoni | Towards a Distributional Model of Semantic Complexity[END_REF], and for modeling reading times of experimental studies on complement coercion [START_REF] Zarcone | Fitting, not Clashing! A Distributional Semantic Model of Logical Metonymy[END_REF]. However, an issue regarding their evaluation has not been addressed yet, i.e. their ability of capturing different levels of implausibility. 2Our processing system is sensitive to minimal variations in predictability between highly unpredictable word combinations, and such sensitivity has been shown to have an influence on reading times [START_REF] Nathaniel | The Effect of Word Predictability on Reading Time Is Logarithmic[END_REF]. Moreover, word combinations that are simply rare and/or unlikely and word combinations that are semantically deviant have been shown to have different consequences on processing complexity [START_REF] Paczynski | Multiple Influences of Semantic Memory on Sentence Processing: Distinct Effects of Semantic Relatedness on Violations of Real-World Event/State Knowledge and Animacy Selection Restrictions[END_REF][START_REF] Warren | Comprehending the Impossible: What Role Do Selectional Restriction Violations Play? In Language[END_REF].

From this point of view, thematic fit models represent an interesting alternative to the traditional probabilistic ones: they use distributional information about typical arguments to create an abstract representation of the "ideal" filler of the argument slot, and thus they are more capable of generalizing to the unseen. In other words, it does not matter if a specific verb-argument combination is attested in the training corpus of our system or not: its plausibility will still be computed on the basis of the similarity of the argument with the words that typically satisfy the requirements of the verb. It is important to stress that the inability to work with rare expressions has been for a long time a general point of criticism of statistical approaches to language, precisely because they could not explain why a given linguistic expression is not attested in the data [START_REF] Eva | Linear) Maps of the Impossible: Capturing Semantic Anomalies in Distributional Space[END_REF].

In the present contribution, we take the first step toward the evaluation of thematic fit models on semantic anomaly detection. We set up a simple classification task on two datasets that have been recently introduced in the literature, and we test two different models on their ability to discriminate between a typical anomalous condition, i.e. the violation of a selectional restriction, and other highly unpredictable conditions.

2 Related Work

Distributional Semantic Models

All the DSMs rely on some version of the Distributional Hypothesis [START_REF] Lenci | Distributional Semantics in Linguistic and Cognitive Research[END_REF], which can be stated as follows: The semantic similarity between two linguistic expressions A and B is a function of the similarity of the linguistic contexts in which A and B occur.

The idea of analyzing meaning by measuring the similarity of distributional patterns turned out to be one of the most successful in the computational semantics research of the last two decades. Thanks to the improvements of automatic tools for language analysis and to the online availability of huge corpora of text, it has become easier and easier to automatically derive semantic representations of linguistic expressions in the form of vectors recording their contexts of occurrence. The closer two vectors in a distributional space, the more similar the meanings of the corresponding words.

Depending on the task, different definitions can be given to the notion of context: the contexts for a target word can be simply other words co-occurring within a sentence, within a word window with a fixed size or, as in our case, words that are syntactically related. In their most classical form, the so-called Structured DSMs use syntactic relation: word pairs as contexts to represent linguistic expressions. For example, subject:baby, adverb: loudly are possible contexts for the distributional representation of the verb to cry.

Since most DSMs of selectional preferences are structured and based on dependencies, also the models presented in this work will share the same features.

Thematic Fit and Distributional Semantics

Given a specific verb role-argument combination, the thematic fit task generally consists in predicting a value that expresses how well the argument fits the requirements of the role, e.g. how good is burglar as a patient for arrest. Since [START_REF] Erk | A Flexible, Corpus-Driven Model of Regular and Inverse Selectional Preferences[END_REF], thematic fit models have been typically evaluated in terms of correlation of the model-derived scores with human-elicited judgements that have been collected for the purpose of psycholinguistic experiments [START_REF] Mcrae | Modeling the Influence of Thematic Fit (and Other Constraints) in Online Sentence Comprehension[END_REF][START_REF] Ferretti | Integrating Verbs, Situation Schemas, and Thematic Role Concepts[END_REF][START_REF] Padó | The Integration of Syntax and Semantic Plausibility in a Wide-Coverage Model of Human Sentence Processing[END_REF][START_REF] Hare | Reading Time Evidence for Enriched Composition[END_REF]. Erk and colleagues computed the fit of the candidate nouns by assessing their similarity with previously attested fillers of the respective roles. Going back to the previous example, if burglar is distributionally similar to the nouns of the entities that are typically arrested, then it should get a high score. [START_REF] Baroni | Distributional Memory: A General Framework for Corpus-based Semantics[END_REF] similarly evaluated their Distributional Memory (DM) framework on the same task, adopting an approach that has became very popular in the literature: for each verb role, they built a single prototype vector by averaging the dependency-based vectors of its most typical fillers. The higher the similarity of a noun with a role prototype, the higher its plausibility as a filler for that role. Their model inspired several other studies: some of them tried to refine their DSM by using semantic roles-based vectors instead of dependency-based ones (Sayeed and Demberg, 2014;[START_REF] Greenberg | Improving Unsupervised Vector-Space Thematic Fit Evaluation via Role-Filler Prototype Clustering[END_REF] or by using multiple prototypes, obtained through hierarchical clustering of the role fillers, in order to deal with verb polysemy [START_REF] Greenberg | Improving Unsupervised Vector-Space Thematic Fit Evaluation via Role-Filler Prototype Clustering[END_REF].

An extension of the original model, introduced by [START_REF] Lenci | Composing and Updating Verb Argument Expectations: A Distributional Semantic Model[END_REF], has also been used to compute the dynamic update on the expectations for an argument filler, depending on how other roles have been filled in the previous part of the sentence (i.e., engine and spelling are both good patients for to check, but if the agent slot is filled by mechanic, then the former becomes a more predictable patient than the latter), and tested his system in a binary classification task on the subject-verb-object triples of the Bicknell dataset [START_REF] Bicknell | Effects of Event Knowledge in Processing Verbal Arguments[END_REF]. More recently, [START_REF] Chersoni | Towards a Distributional Model of Semantic Complexity[END_REF] integrated a similar mechanism of thematic fit computation in a more general model of semantic complexity, and obtained results comparable to [START_REF] Lenci | Composing and Updating Verb Argument Expectations: A Distributional Semantic Model[END_REF] on the same dataset.

Finally, [START_REF] Zarcone | Fitting, not Clashing! A Distributional Semantic Model of Logical Metonymy[END_REF] made use of the notion of thematic fit in their study on complement coercion. Typically, we have a complement coercion when an event-selecting verb takes an entity-denoting NP as its direct object (i.e. the author began the book), so that a hidden verb has to inferred in order to satisfy the selectional restrictions of the verb (the author began writing the book). These authors computed the thematic fit for different verb-object combinations, corresponding to the experimental items used in the psycholinguistic experiments of [START_REF] Mcelree | Reading Time Evidence for Enriched Composition[END_REF][START_REF] Matthew | Coercion in Sentence Processing: Evidence from Eye-Movements and Self-Paced Reading[END_REF], and showed that the scores mirrored very closely the differences across conditions that were found in the above-mentioned studies. The coercion condition is particularly interesting for the present work, since it consists of an apparent violation of selectional restrictions. Therefore, the discrimination between actual violations and cases of complement coercion will be one of the tests for our models.

Experimental Evidence on Selectional Restrictions

Selectional restrictions can be defined as the set of semantic features that a verb requires of its arguments [START_REF] Warren | Comprehending the Impossible: What Role Do Selectional Restriction Violations Play? In Language[END_REF]. Modular theories argued that they were represented in the lexicon, which was seen as a specialized module [START_REF] Katz | The Structure of a Semantic Theory[END_REF][START_REF] Fodor | The Modularity of Mind[END_REF]: it was generally assumed that the human comprehension system initially uses the knowledge available in such modules, and only later uses general world knowledge.

Since now there is evidence speaking against the modularity of the lexicon [START_REF] Mante | When Peanuts Fall in Love: N400 Evidence for the Power of Discourse[END_REF] and in favor of the access to world knowledge in the early stages of the comprehension process [START_REF] Mcrae | Modeling the Influence of Thematic Fit (and Other Constraints) in Online Sentence Comprehension[END_REF][START_REF] Mcrae | People Use their Knowledge of Common Events to Understand Language, and Do So as Quickly as Possible[END_REF], it was questioned whether selectional restrictions have an independent reality, instead of being just part of a general world knowledge about events and participants [START_REF] Hagoort | Integration of Word Meaning and World Knowledge in Language Comprehension[END_REF][START_REF] Kuperberg | Neural Mechanisms of Language Comprehension: Challenges to syntax[END_REF].

However, an EEG experiment by Pacyznski and [START_REF] Paczynski | Multiple Influences of Semantic Memory on Sentence Processing: Distinct Effects of Semantic Relatedness on Violations of Real-World Event/State Knowledge and Animacy Selection Restrictions[END_REF] showed that the processing difficulty of a sentence is affected differently by violation of selectional restrictions, with respect to simple event knowledge violation. The authors recorded ERPs on post-verbal Agent arguments as participants read passive English sentences, and they noticed that the N400 evoked by incoming animate Agent arguments violating event knowledge (e.g. The bass was strummed by the drummer) was strongly attenuated when they were semantically related to the context (e.g. the drummer is related to a concert-type scenario). In contrast, semantic relatedness did not modulate the N400 evoked by inanimate Agent arguments that violated the preceding verbs animacy selection restrictions (e.g. The bass was strummed by the drum). Such a result led the researchers to the conclusion that the two types of violations are actually distinct at the brain processing level.

Moreover, [START_REF] Warren | Comprehending the Impossible: What Role Do Selectional Restriction Violations Play? In Language[END_REF] recently brought new evidence that the violation of a selectional restriction determines higher processing complexity than simple event implausibility. In an eye-tracking experiment, the authors compared the reading times between sentences in three different experimental conditions: a plausible condition (i.e. The hamster explored a backpack), an implausible condition with no violation of selectional restrictions (The hamster lifted a backpack) and an impossible condition with violation (The hamster entertained a backpack). Although the difference in human possibility ratings was not statistically significant between the last two conditions, eye-movements evidenced longer disruption in the violation condition compared to the other two. They concluded suggesting that selectional restrictions could actually be coarse-grained semantic features, derived by means of abstractions over exemplar-type representations of events in memory. Violations of coarse-grained semantic features are likely to be detected earlier by the readers and cause more difficulty also in the later stages of processing, as they lead to such a degree of semantic anomaly that it becomes hard to build a coherent discourse model for the sentence [START_REF] Warren | Investigating Effects of Selectional Restriction Violations and Plausibility Violation Severity on Eye-Movements in Reading[END_REF].

Most importantly, from a computational perspective, word combinations corresponding to the violations either of world knowledge (the implausible condition in Warren's data) or of selectional restrictions are not likely to be found in corpora of natural language data, and thus they cannot be distinguished on the basis of probabilistic methods. In our work we aim at testing the ability of thematic fit models to spot the difference and to assign different degrees of anomaly to the two conditions. The idea, intuitively, is that the degree of semantic anomaly goes hand in hand with an increase in processing complexity.

Experiments

For our experiments, we used two evaluation datasets: the sentences from the studies of [START_REF] Pylkkänen | An MEG Study of Silent Meaning[END_REF] and [START_REF] Warren | Comprehending the Impossible: What Role Do Selectional Restriction Violations Play? In Language[END_REF]. The first study presented a magnetoencephalography experiment, with the goal of investigating the brain response to anomaly and to complement coercion, i.e. the case of a type clash between an event-selecting verb and an entity-denoting direct object. The experimental subjects were exposed to sentences in three different conditions: i) sentences with a typical verb-object combination (The journalist wrote the article after his coffee break); ii) sentences with a complement coercion (The journalist began the article after his coffee break); iii) sentences with a selectional restriction violation (The journalist astonished the article after his coffee break). This dataset is interesting for us because it will allow a direct comparison between violations of selectional restrictions and a similar phenomenon, the only difference being that a coercion involves the inference of a hidden verb (in the case of the example above, writing) that is not present in the linguistic input, leading to a sort of 'repair' of the violation. Discriminating between the two conditions is likely to be a difficult task.

The Warren dataset is the same of the study mentioned in Section 2.2. We are going to compare the items in the three conditions (plausible, implausible with no violation and impossible violation: see the examples in Section 2.2) of the experiment of Warren and colleagues, and we are particularly interested in the ability of the models to set the violation condition apart from the others. As declared by the authors themselves, they have built the sentences in a way than even the events described in the plausible condition are rare, or very unlikely. The test on this dataset will be particularly indicative of the performance of thematic fit models when they have to deal with different types of rare verb-argument combinations.

In both the datasets, we expect our thematic fit models to assign the lowest score to the violation condition, thus being able to distinguish between combinations that are simply unlikely and others that are really anomalous. Datasets The Pylkkänen dataset is composed by 33 triplets of sentences, while the Warren dataset is composed by 30 triplets. We converted the experimental sentences in subject-verb-object triples. Here is one example from the Pylkkänen dataset (1) and one from the Warren dataset (2):

(1) a. journalist-write-article (typical) b. journalist-begin-article (coercion) c. journalist-astonish-article (violation)

(2) a. hamster-explore-backpack (plausible) b. hamster-lift-backpack (implausible) c. hamster-entertain-backpack (violation)

Before building our dependency-based DSM, we had to exclude three triplets from the Warren dataset since one or more words in the triplets had frequency below 100 in the training corpus. On the other hand, we have full coverage for the Pylkkänen dataset. DSM We built a dependency-based DSM by using the data in the BNC corpus [START_REF] Neil | 100 Million Words of English: The British National Corpus (BNC)[END_REF] and in the Wacky corpus [START_REF] Baroni | The WaCky Wide Web: A collection of Very Large Linguistically Processed Web-Crawled Corpora[END_REF]. Both the corpora were POS-tagged with the Tree Tagger ( 1994) and parsed with the Maltparser [START_REF] Nivre | Maltparser: A Data-Driven Parser-Generator for Dependency Parsing[END_REF]. 3 We extracted all the dependencies for the 20K most frequent words in the corpora, including the words of our datasets. Every co-occurrence between a target word and another context word in a given syntactic relation was weighted by means of Positive Local Mutual Information [START_REF] Evert | The Statistics of Word Cooccurrences: Word Pairs and Collocations[END_REF]. 4 Given a target t, a relation r and a context word c occurring in the relation r with the target (e.g. t = bark, r = sbj, c = dog), we computed both their cooccurrence O trc , and the expected co-occurrence E trc under the assumption of statistical independence.

The Positive Local Mutual Information (henceforth PLMI) is then computed as follows:

LM I(t, r, c) = log Otrc Etrc * Otrc (1)

P LM I(t, r, c) = max(LM I(t, r, c), 0) (2)
Finally, each target word is represented by a vector of PLMI-weighted syntactic co-occurrences. Each contextual dimension corresponds to the co-occurrence of the target with a word in a given syntactic relation. For example, the vector of the verb write-v has dimensions such as journalist-n:subj,articlen:obj etc.5 Method As in [START_REF] Baroni | Distributional Memory: A General Framework for Corpus-based Semantics[END_REF], the thematic fit of a word for a given verb role is computed as the distributional similarity of that word with a prototype representation of the typical role filler. Such representation is obtained by averaging the vectors of the most typical fillers, i.e. words that are strongly associated with that verb-specific role. More concretely, the authors used syntactic functions to approximate thematic roles, and considered the most typical subjects of a verb as the fillers for the agent role, and the most typical objects as the fillers for the patient role. Typicality was measured by means of PLMI values: given a target verb t and a syntactic relation r, the typical fillers for the corresponding role were the 20 words with the highest PLMI association score with (t, r). Some examples of the extracted fillers are provided in Table 1. 6 Once built the prototype, the thematic fit of each candidate filler is assessed as the cosine similarity between the filler vector and the prototype itself. For example, the prototype for the patient of entertain-v will be built out of the typical objects of the verb, such as public, player etc. Words that are distributionally similar to such fillers (i.e. fan) are likely to have a high thematic fit for the role. Models In our experiments, we compared two different models of thematic fit. B&L2010 is a 'classical' model of thematic fit, and it consists of a direct reimplementation of [START_REF] Baroni | Distributional Memory: A General Framework for Corpus-based Semantics[END_REF]: since we are scoring sentences which differ for the degree of typicality of the verb-object combination, the scores assigned by this model will be the thematic fit scores θ of the object of each sentence given the verb and the patient role. In Equation 3, t is the target verb and c is a word occurring as an object (obj) of t:

θ = - → c |obj, - → t (3)
For example, the score of the sentence of the example 1a will be the thematic fit of the object article-n as a patient of write-v.

The second model is inspired by the proposal of [START_REF] Chersoni | Towards a Distributional Model of Semantic Complexity[END_REF] who, instead of seeing the thematic fit as a simple measure of congruence between a predicate and an argument, considered it as a more general measure of the semantic coherence of an event. The global degree of semantic coherence is given by the product of the partial θ scores of all the event participants.

Similarly to Baroni and Lenci's model, each θ score is defined as the cosine similarity between an argument vector and the prototype vector for the slot, built as the centroid of its typical fillers. Once computed the partial θ scores, they are combined to find the global score θ e .

θ e = - → t ,r, - → c ∈e θ( - → c |r, - → t ) (4)
where t is a target word in the event e7 , r is a syntactic relation and c is a context word occurring in the relation r with t (it is read as: the thematic fit score of c given the word t and the relation r).

For example, for the verb-argument triple of the example 1a, the three partial components of the final score would be: i) the thematic fit of the subject journalist-n as an agent of write-v; ii) the thematic fit of the object article-n as a patient of write-v; iii) the thematic fit of the object article-n as a co-argument of the subject journalist-n. 8The intuition of the authors was that the semantic coherence of an event does not depend simply on predicate-argument congruence scores, taken in isolation, but on a general degree of mutual typicality between all the participants. We will refer to this variant of the thematic fit model as CBL2016. Task We evaluate the accuracy of the models in a classification task: for each triplet in the datasets, we compute the thematic fit scores for the subject-verb-object triples in the three conditions. We score a hit for a model each time it assigns the lowest score to the triple in the violation condition. The performance of both thematic fit models is compared to the one of a random baseline (since we have three different conditions, the accuracy is estimated to be 33.33%). We also use statistical tests to check in what measure the scores between the violation and the other conditions differ.

Results

The results of our experiments on the classification task are shown in Table 2 andTable 3. On the Warren dataset, the CBL2016 model performs extremely well, managing to assign the lowest thematic fit score to the violation condition in more than 80% of the triples of the dataset and reporting a highly significant advantage over the random baseline (p < 0.001)9 . Although inferior in accuracy to the other model, B&L2010 manages as well to significantly outperform the baseline (p < 0.05). The Kruskal-Wallis test revealed a strong main effect of the condition on the scores assigned by both models (B&L2010: χ 2 = 20.502, p < 0.01; CBL2016: χ 2 = 14.117, p < 0.01). Post-hoc comparisons with the Wilcoxon rank sum test showed that, for both models, the scores differ significantly between the plausible and the violation condition and between the not plausible and the violation condition (in both cases, p < 0.01). These results are extremely relevant: although all the events of the Warren dataset have very low probabilities (for an explicit design choice of the authors), both the thematic models proved to be able to discriminate between events violating selectional restrictions and events that are simply unlikely (see also Figure 1, left side). They do not differ significantly for their ability to discriminate between the violation and the other conditions, as the violation consists of a mismatch of semantic features between the patient role of the verb and its filler (typically an animacy violation), and this information is available to both B&L2010 and CBL2016 in the form of an extremely low thematic fit for the patient. With respect to B&L2010, CBL2016 has also information on the thematic fit of the other event fillers. In theory, this should be an an advantage for distinguishing between the plausible and the not plausible condition: as it can be seen in Example 2, it is difficult to account for the difference in plausibility between a. and b. by only looking at the verb-patient combination. In practice, none of the models has assigned significantly different scores to the conditions a. and b., in line with the results of [START_REF] Warren | Comprehending the Impossible: What Role Do Selectional Restriction Violations Play? In Language[END_REF], who also reported the absence of significant differences in reading times between plausible and not plausible sentences. This suggests that, for very rare events, different degrees of plausibility do not determine big changes in processing complexity, at least when selectional restrictions are not violated.

Model

not plausible violation 0.0000 0.0005 0.0010 0.0015 0.0020 0.0025 0.0030 coercion violation 0.0000 0.0005 0.0010 0.0015 0.0020 0.0025 0.0030 As for the Pylkkänen dataset, both models were again able to outperform the random baseline on the classification task with a significant margin (p < 0.05) and, also on this dataset, the Kruskal-Wallis test showed a strong effect of the condition (B&L2010: χ 2 = 40.114, p < 0.001; CBL2016: χ 2 = 13.804, p < 0.01). The Wilcoxon test revealed that they are both efficient in discriminating between the typical and the other two conditions (B&L2010: p < 0.001 for both the typical-coercion and the typicalviolation comparison; CBL2016: p < 0.01 for the same comparisons), but it revealed also an important difference: while B&L2010 assigns significantly higher scores to coerced sentences with respect to their counterparts containing violations (p < 0.01), CBL2016 fails to detect such a distinction (p > 0.1; see also Figure 1, right side). This result may seem surprising, since the less informed B&L2010 turns out to be the most efficient in detecting the fine-grained distinction between coercions and violations, simply on the basis of the typicality of the verb-patient argument combination.

A possible explanation is that the thematic fit was conceived in CBL2016 as a general index of se-mantic coherence. If we limit ourselves to compute the fit between the event and the participants that are present in the linguistic input, it is not surprising that coercions and violations have similarly low coherence levels. After all, coercions can be described as violations of selectional restrictions that are repaired by inferring a hidden verb from the context (e.g. writing in The journalist began the article): since the model has no way to infer the hidden verb, it assigns a similarly low coherence score to the two experimental conditions.

Conclusion

In this paper, we have evaluated two thematic fit models in a classification task for the identification of violations of selectional restrictions. Our models had to deal with extremely rare word combinations (in the case of the Warren dataset) or to distinguish between violations and a similar phenomenon, i.e. complement coercion (in the case of the Pylkkänen dataset). On the Warren data, the performance of both models was very solid, clearly showing that they are able to discriminate between unlikely and anomalous inputs. Typically, such rare verb-argument combinations are not attested at all in corpora. We think this is a proof that the role characterization in thematic fit models allows generalizations on potential fillers that go well beyond the observable evidence. On the Pylkkänen dataset, the classical model by [START_REF] Baroni | Distributional Memory: A General Framework for Corpus-based Semantics[END_REF] manages to distinguish between coercion and violation, whereas the more recent model by [START_REF] Chersoni | Towards a Distributional Model of Semantic Complexity[END_REF] does not. Still, the predictions of the latter could find some justification in the rationale behind its notion of thematic fit, and in the particular nature of the coercion phenomenon, describable as an apparent violation that is repaired by inferring a covert event.

More in general, the notion of thematic fit turns out to be very useful for modeling processing complexity, measured as in the experimental studies (mostly) in terms of processing times. Since thematic fit quantifies how a given argument fits a given semantic role, or a given event scenario, the low values correspond to situations in which it is extremely difficult to build a coherent semantic representation for the sentence. Given these promising results, future research should aim at building larger datasets to evaluate distributional models on anomaly detection tasks.

Another issue that deserves further investigation is the effect of the general discourse context on event plausibility, since contextual information in the current datasets is often limited to the other argument fillers. 10 As shown by studies like [START_REF] Warren | Effects of Context on Eye Movements when Reading about Possible and Impossible Events[END_REF], a context such as a fantasy world scenario can modulate the plausibility of an event and consequently the processing times, and the same could be true also for some specific real world scenarios (i.e. a psychiatric hospital, a circus etc.). Future efforts in modeling semantic anomalies have to take into account the acquisitions of the rich experimental literature on the topic, and to try to integrate as many as possible types of contextual manipulation in building new gold standards.

Figure 1 :

 1 Figure 1: CBL 2016 score comparison between the NOT PLAUSIBLE and the VIOLATION condition on the Warren dataset (left) and between the COERCION and the VIOLATION condition on the Pylkkänen dataset (right).

Table 1 :

 1 Schmid, Verb roles and examples of fillers extracted by means of a corresponding syntactic relation.

	Verb and Role	Fillers
	Agent of to play	actor, gamer, violinist
	Agent of to arrest cop, policeman, superhero
	Patient of to eat	pizza, sandwich, ice-cream
	Patient of to shoot	enemy, soldier, prey

Table 2 :

 2 Accuracy scores for the Warren dataset.

	Model	Hits	Accuracy
	Random	11/33	33.33%
	B&L2010 21/33	63.63%
	CBL2016 19/33	57.57%

Table 3 :

 3 Accuracy scores for the Pylkkänen dataset.

See[START_REF] Mcrae | People Use their Knowledge of Common Events to Understand Language, and Do So as Quickly as Possible[END_REF] for an overview.

A partial exception is the study on semantic deviance by[START_REF] Eva | Linear) Maps of the Impossible: Capturing Semantic Anomalies in Distributional Space[END_REF]. However, they focus on the acceptability of adjectival phrases, rather than on selectional preferences.

We used the scripts of the DISSECT framework to build the distributional space[START_REF] Dinu | Dissect-Distributional Semantics Composition Toolkit[END_REF].

As context words, we took into account only the 20K words of our target list, in order to limit the size of the distributional space.

Obviously, including all the syntactic relations would have hugely increased the dimensionality of the vector space. Therefore, we took into account only the following relations: subject, direct and indirect object, prepositional complement. For each relation, we also considered its inverse: for example, the target apple-v has a dimension eat-v:obj-1, meaning that apple occurs as a direct object of eat-v.

 6 In the literature, 20 is a common choice for the number of fillers[START_REF] Baroni | Distributional Memory: A General Framework for Corpus-based Semantics[END_REF][START_REF] Greenberg | Improving Unsupervised Vector-Space Thematic Fit Evaluation via Role-Filler Prototype Clustering[END_REF]. Thus, we decided to keep this value for our experiments.

Keep in mind that, in the above-mentioned work, sentences are seen as linguistic descriptions of events and situations.

The latter component was introduced because nouns, according to recent psycholinguistic studies[START_REF] Hare | Reading Time Evidence for Enriched Composition[END_REF][START_REF] Bicknell | Effects of Event Knowledge in Processing Verbal Arguments[END_REF], activate expectations about arguments typically co-occurring in the same events. In order to model the relationship between agents and patients of the same events, we introduced in our DSM the generic relation verb to link subjects and objects that tend to occur together, independently of the

predicate. 9 p-values computed with the χ 2 statistical test.

Acknowledgements

This work has been carried out thanks to the support of the A*MIDEX grant (nANR-11-IDEX-0001-02) funded by the French Government "Investissements d'Avenir" program.


Enrico Santus
email: esantus@mit.edu

Hongmin Wang
email: hongminwang@cs.ucsb.edu

Emmanuele Chersoni
email: emmanuelechersoni@gmail.comyuezhang@sutd.edu.sg

Yue Zhang

A Rank-Based Similarity Metric for Word Embeddings

Word Embeddings (WE) have recently imposed themselves as a standard for representing word meaning in NLP. Semantic similarity between word pairs has become the most common evaluation benchmark for these representations, with vector cosine being typically used as the only similarity metric. In this paper, we report experiments with a rank-based metric for WE, which performs comparably to vector cosine in similarity estimation and outperforms it in the recently-introduced and challenging task of outlier detection, thus suggesting that rank-based measures can improve clustering quality. 1

Introduction

"All happy families resemble one another, but each unhappy family is unhappy in its own way." Anna Karenina, Leo Tolstoy Distributional Semantic Models (DSMs) have received an increasing attention in the NLP community, as they constitute an efficient data-driven method for creating word representations and measuring their semantic similarity by computing their distance in the vector space [START_REF] Peter | From Frequency to Meaning: Vector Space Models of Semantics[END_REF].

The most popular similarity metric in DSMs is the vector cosine. Compared to Euclidean distances, vector cosine scores are normalized on each dimension and hence are robust to the scaling effect. On the other hand, one limitation of this metric is that it regards each dimension equally, without taking into account the fact that some dimensions might be more relevant for characteriz-1 Enrico Santus and Hongmin Wang equally contributed to this work, which was started while they were both affiliated to the Singapore University of Technology and Design.

ing the semantic content of a word. Such a limitation led to the introduction of alternative metrics based on feature ranking, which have been reported to outperform vector cosine in several similarity tasks (Santus et al., 2016a,b).

Recently, the focus of the research on word representations has been shifting onto the so-called word embeddings (WE), which are dense vectors obtained by means of neural network training that achieved significant improvements in several similarity-related tasks (Mikolov et al., 2013a;[START_REF] Baroni | Don't Count, Predict! A Systematic Comparison of Context-counting vs. Context-predicting Semantic Vectors[END_REF]. Although the representation type of the embeddings was helpful for reducing the sparsity of traditional count vectors, their nature does not sensibly differ [START_REF] Levy | Improving Distributional Similarity with Lessons Learned from Word Embeddings[END_REF]. Most research works involving WE still adopt vector cosine for similarity estimation, yet little experimentation has been done on alternative metrics for comparing dense representations (exceptions include [START_REF] Camacho-Collados | NASARI: a Novel Approach to a Semantically-Aware Representation of Items[END_REF]).

Some attempts to directly transfer rank-based measures from traditional DSMs to WE have faced difficulties (see, for example, [START_REF] Jebbara | Extracting Common Sense Knowledge via Triple Ranking Using Supervised and Unsupervised Distributional Models[END_REF]). In this paper, we suggest a possible solution to this problem by adapting AP Syn, a rank-based similarity metric originally proposed for sparse vectors (Santus et al., 2016b,a), to low-dimensional word embeddings. This goal is achieved by removing the parameter N (the extent of the feature overlap to be taken into account) and adding a smoothing parameter that is proven to be constant under multiple settings, therefore making the measure unsupervised. Our experiments show performance improvements both in similarity estimation and in the more challenging outlier detection task (Camacho-Collados and Navigli, 2016), which consists in cluster and outlier identification. 2 2 Similarity, Relatedness and Dissimilarity: Current Issues in the Evaluation of DSMs A classical benchmark for DSMs is represented by the estimation of word similarity: evaluation datasets are built by asking human subjects to rate the degree of semantic similarity of word pairs, and the performance is assessed in terms of the correlation between the average scores assigned to the pairs by the subjects and the cosines of the corresponding vectors (similary estimation task). Similarity as modeled by DSMs has been under debate, as its definition is underspecified. It in fact includes an ambiguity with the more generic notion of semantic relatedness, which is present also in many popular datasets (i.e. the concepts of coffee and cup are certainly related, but there is very little similarity about them), as opposed to 'genuine' semantic similarity (i.e. the relation holding between concepts such as coffee and tea) [START_REF] Agirre | A study on Similarity and Relatedness Using Distributional and Wordnet-based Approaches[END_REF][START_REF] Hill | Simlex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation[END_REF][START_REF] Gladkova | Intrinsic Evaluations of Word Embeddings: What Can We Do Better[END_REF]. Therefore, when testing a DSM, it is important to pay attention to what type of semantic relation is actually modeled by the evaluation dataset. Moreover, researchers pointed out that similarity estimation alone does not constitute a strong benchmark, as the inter-annotator agreement is relatively low in all datasets and the performance of several automated systems is already above the upper bound [START_REF] Batchkarov | A Critique of Word Similarity as a Method for Evaluating Distributional Semantic Models[END_REF]. As a consequence, workshops such as RepEval have been organized with the explicit purpose of finding alternative evaluation tasks for DSMs.

A recent proposal is the challenging outlier detection task [START_REF] Camacho-Collados | Find the Word that Does not Belong: A Framework for an Intrinsic Evaluation of Word Vector Representations[END_REF][START_REF] Blair | Automated Generation of Multilingual Clusters for the Evaluation of Distributed Representations[END_REF], which consists in the recognition of cluster membership, as well as of a relative degree of semantic dissimilarity. The task is described as follows: given a group of words, identify the outlier, namely the word that does not belong to the group (i.e. the one that is less similar to the others). On top of its potential applications (e.g. ontology learning), detecting outliers in clusters is a goal that poses a more strict quality requirement on the distributional representations compared to tests based simply on pairwise comparisons, as it is required that similar words group into semantically meaningful clusters. Clearly, the task involves the identification of discriminative semantic dimensions, which could set the cluster members apart from non-members. Outliers are not necessarily unrelated to the other words: rather, they have a lower degree of similarity with respect to some prominent property of the cluster (e.g. the case of Los Angeles Lakers as an outlier in a cluster of football teams). In our view, a similarity metric has to exploit such discriminative dimensions to form cohesive clusters.

A Rank-Based Metric for Embeddings

We use cosine as a baseline and we test an adaptation of a rank-based measure to the dense features of the word embeddings.

Vector cosine computes the correlation between all the vector dimensions, independently of their relevance for a given word pair or for a semantic cluster, and this could be a limitation for discerning different degrees of dissimilarity. The alternative rank-based measure is based on the hypothesis that similarity consists of sharing many relevant features, whereas dissimilarity can be described as either the non-sharing of relevant features or the sharing of non-relevant features [START_REF] Santus | Taking Antonymy Mask Off in Vector Space[END_REF](Santus et al., , 2016b)).

This hypothesis could turn out to be very helpful for a task like the outlier detection, where prominent features might be the key to improve clustering quality: semantic dimensions that are shared by many of the cluster elements should be weighted more, as they are likely to be useful for setting the outliers apart. In fact, a cohesive cluster should be mostly characterized by the same 'salient' dimensions, and thus, basing word comparisons on such dimensions should lead to more reliable estimates of cluster membership.

In our contribution, we propose to adapt AP Syn, a metric originally proposed by Santus et al. (2016a,b), to dense word embeddings representations.3 AP Syn was shown to perform well on both synonymy detection and similarity estimation tasks, and it was recently adapted to achieve state-of-the-art results in thematic fit estimation [START_REF] Santus | Measuring Thematic Fit with Distributional Feature Overlap[END_REF]. The original AP Syn formula is shown in equation 1.

AP Syn(wx, wy) = i=N i=0 1 AV G(rs x (fi), rs sy (fi)) (1)
For every feature f i in the intersection between the top N features of two vectors w x and w y , we add the inverse of the average rank of such feature, r sx (f x ) and r sy (f y ), in the two decreasingly value-sorted vectors s x and s y (in traditional vectors, often the parameter N ≥ 1000, but in WE N = |f |). AP Syn scores low if the features of the two vectors are inversely ranked and high if they are similarly ranked. AP Syn maps the average feature ranks to a non-linear function, emphasizing the contribution of top-ranked features. Its direct application to dense embeddings would shrink too much the contribution of lower ranks (see Figure 1), with the score mostly affected by the top ∼ 25 features. While this is reasonable for the traditional vectors derived from co-occurrence counts, where thousands of smaller contributions can still affect the final score, dense vectors need a smoother curve. While preserving the idea of the non-linear weight allocation across the average feature ranks during the summation, we modify the original AP Syn formula by taking the exponential of the feature ranks to a power of a constant value ranging between 0 and 1 (excluded), as shown in equation 2, such that now the number of ranks contributing to the final score is widen to all features (see the smoother curve of AP SynP ower in Figure 1). We name this variant AP SynP ower or, shortly, AP SynP .

AP SynP (wx, wy)

= i=|f | i=0 1 AV G(rs x (fi) p , rs y (fi) p ) (2)
The power p added to AP SynP formula is a trainable parameter. We trained it on the similarity subset of WordSim dataset, obtaining the optimal value of p = 0.1, which has been successfully used in all evaluations, under all settings (i.e. embedding types and training corpora). Such regularity allows us to consider p = 0.1 as a constant, therefore dropping p. Since in WE we can drop also the parameter N by defining N = |f |, AP SynP can be not parametrized at all.

Evaluation Settings

Embeddings

For our experiments, we used two popular word embeddings architectures: the Skip-Gram with negative sampling (Mikolov et al., 2013a,b) and the GloVe vectors [START_REF] Pennington | Glove: Global Vectors for Word Representation[END_REF] (standard hyperparameter settings: 300 dimensions, context size of 10 and negative sampling). 4For comparison with Camacho-Collados and Navigli (2016) on outlier detection, we used the same training corpora: the UMBC [START_REF] Han | UMBC EBIQUITY-CORE: Semantic Textual Similarity Systems[END_REF] and the English Wikipedia.5

Datasets

As for the similarity estimation task, we evaluate the Spearman correlation between systemgenerated scores and human judgments. We used three popular benchmark datasets: WordSim-353 [START_REF] Finkelstein | Placing Search in Context: The Concept Revisited[END_REF], MEN [START_REF] Bruni | Multimodal Distributional Semantics[END_REF] and SimLex-999 [START_REF] Hill | Simlex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation[END_REF]. It is important to point out that SimLex-999 is the only one specifically built for targeting genuine semantic similarity, while the others tend to mix similarity and relatedness scores.

As for outlier detection, we evaluate our DSMs on the 8-8-8 dataset [START_REF] Camacho-Collados | Find the Word that Does not Belong: A Framework for an Intrinsic Evaluation of Word Vector Representations[END_REF]. The dataset consists of eight clusters, each one with a different topic and consisting in turn of eight lexical items belonging to the cluster and eight outliers (with four degrees of relatedness to the cluster members: C1, C2, C3, C4). In total, the dataset includes 64 sets of 8 words + 1 outlier for the evaluation. For each word w of a cluster W of n words, the authors defined a compactness score c(w) corresponding to the average of all pairwise similarities of the words in W \ {w}. On the basis of the compactness score, they proposed two evaluation metrics: Outlier Position (OP) and Outlier Detection (OD). Given a set W of n + 1 words, OP is the rank of the outlier w n + 1 according to the compactness score. Ideally, the rank of the outlier should be n, mean- ing that it has the lowest average similarity with the other cluster elements. The second metric, Outlier Detection (OD), is indeed defined as 1 iff OP (w n + 1) = n, 0 otherwise. Finally, the performance on a dataset composed of |D| sets of words was estimated in terms of Outlier Position Percentage (OP P , Eq. 3) and Accuracy (Eq. 4):

OP P = W ∈D OP (W ) |W |-1 D × 100 (3)
Accuracy = W ∈D OD(W ) D × 100 (4)

Pairwise and Prototype Approaches to Outlier Detection

While for the similarity task scores are always calculated pairwise, for spotting the outlier two different methods were tested: the pairwise comparisons and the cluster prototype.

In the first case, we reimplemented the method of Camacho-Collados and Navigli (2016): (i) compute the average similarity score of each word with the other words in the cluster; (ii) pick as the outlier the word with the lowest average score. An alternative consists in creating a cluster prototype: (i) for a cluster of N words, we create N prototype vectors by excluding each time one of the words and averaging the vectors of the other ones; (ii) we pick as the outlier the word with the lowest similarity score with the prototype built out of the vectors of the other words in the cluster.

Results and Discussion

Table 1 summarizes the correlations for the similarity task. AP SynP outperforms both vector cosine and AP Syn in all the datasets described in 4.2 when GloVe embeddings are used. The advantage is statistically significant over the cosine on the MEN dataset (p < 0.05) and over AP Syn on all datasets (p < 0.01).6 With Skip-Gram embeddings, AP SynP performs comparably to vector cosine for relatedness, dominant in WordSim and MEN, while retaining a significant advantage over AP Syn on the same datasets (p < 0.05). It also performs slightly better than cosine in SimLex-999, and this complies with previous findings of Santus et al. (2016a), who showed that AP Syn performs better on genuine similarity datasets. Apparently, the top-ranked vector dimensions (those contributing more to APSyn scores) are more often shared by similar words, than by simply related ones.

Table 2 shows the results for the outlier detection task. The line CC-Cos contains the scores by Camacho-Collados and Navigli (2016) as a baseline. The models are divided into pairwise comparison and cluster prototype (see Section 4.3).

As it can be easily noticed by looking at the bold line, AP SynP outperforms the baselines in all settings for both Skip-Gram and GloVe, obtaining higher accuracies and OPPs. Not only AP SynP is better at identifying the outlier, but when it is not able to do so, its error is minimum (e.g. the outlier is eventually the second ranked candidate). The best accuracy (73.4 vs. SOA of 70.3) and the best OPP (94.9 vs. SOA of 93.8) are both obtained by AP SynP with the prototype approach, using the Skip-Gram trained on Wikipedia. We also tested the significance of the accuracy improvements with the χ 2 test but, also given the small size of the 8-8-8 dataset, the result was negative.

Finally, we observe that the two approaches described in 4.3 do not lead to sensitively different results. The major factors of difference can be found instead in the embeddings (with Skip-Gram outperforming Glove) and in the training corpus (the smaller Wikipedia, 1.7B words, outperforms the bigger UMBC, 3B words).

Error Analysis

In Table 3, we report the 5 outliers that were most difficult to detect by AP SynP . Most of them are related to the German Car Manufacturers topic, which was ambiguous and populated by rare terms. All outliers in the Months and in the South American countries clusters (except for the two South-American cities Rio de Janeiro and Bogotá) are successfully identified under all experimental settings. Finally, the reader can notice that most errors belong to C1 and C2, which are the most challenging classes in the dataset, as these outliers are either very related or very similar to other cluster members.

Conclusions

We have introduced AP SynP , an adaptation of the rank-based similarity measure AP Syn (Santus et al., 2016a,b) for word embeddings. This adaptation introduces a power parameter p, which is shown to be constant in multiple tasks (i.e. p = 0.1). The stability of this parameter, together with the possibility of dropping the parameter N of AP Syn when using WE by setting N = |f |, makes the measure unsupervised. We have tested it on the tasks of similarity estimation and outlier detection, obtaining similar or better performances than vector cosine and the original AP Syn. AP SynP performs more consistently on SimLex-999, showing a preference for genuine similarity, as already noticed by Santus et al. (2016a). We also introduced a new approach to the outlier detection task, based on a cluster prototype. The prototype method is competitive and computationally less expensive than pairwise comparisons.

We leave to future work a systematic comparison of AP SynP and other rank-based measures. Pilot tests have shown that other rank-based metrics (e.g. Spearman's Rho) also outperform vector cosine in multiple settings and tasks.

Figure 1 :

 1 Figure 1: Comparison of weight per feature rank in AP Syn and AP SynP (p = 0.1) across feature ranks ranging from 1 to 300.

Table 1 :

 1 Similarity Estimation, Spearman Correlation by Setting. Embeddings trained on Wikipedia.

			Skip-Gram			GloVe
	WordSim-353 MEN Simlex-999 WordSim-353 MEN Simlex-999
	Cosine	0.736	0.758	0.364	0.511	0.640	0.311
	AP Syn	0.599	0.643	0.343	0.356	0.393	0.174
	AP SynP	0.710	0.737	0.369	0.607	0.670	0.335
			Skip-Gram		GloVe
			UMBC	Wiki	UMBC	Wiki
			OPP Acc. OPP Acc. OPP Acc. OPP Acc.
	CC -Cos 92.6 64.1 93.8 70.3 81.6 40.6 91.8 56.3
					Pairwise
	AP Syn		93.0 67.2 94.0 68.8 78.7 40.6 89.3 53.1
	AP SynP	94.0 68.8 94.5 73.4 81.8 42.2 92.8 61.0
				Prototype
	P T -Cos 93.4 65.6 93.8 68.8 80.3 40.6 90.6 54.7
	AP Syn		92.6 70.3 91.0 62.5 81.6 40.6 88.7 54.7
	AP SynP	94.0 70.3 94.9 73.4 82.2 43.8 92.0 60.9

Table 2 :

 2 Outlier Detection, Performance by Setting. CC-Cos refers to Camacho-Collados and Navigli (2016)'s pairwise method, while PT-Cos refers to the prototype-based one. In bold, best scores per method; in bold and underlined, best scores per corpus-embedding combination.

Table 3 :

 3 Outlier Detection: Top 10 common errors across settings and their difficulty class (i.e. C1, C2, C3 and C4). (GCM: German Car Manufac-

turers; AJC: Apostles of Jesus Christ; SSP: Solar System Planets; BC: Big Cats; ITC: IT Companies).

Code and vectors used for the experiments are available at https://github.com/esantus/Outlier Detection.

The number of dimensions in word embeddings is in the scale of hundreds, and thus the dimensionality is way lower than in the original DSMs used by Santus and colleagues.

We also performed experiments with CBOW embeddings(Mikolov et al., 2013b), but results were irregular and inconsistent. We leave therefore their analysis to future work.

5 Dump of Nov. 2014, approx. 1.7 billion words.

p-values computed with Fisher's r-to-z transformation.

Acknowledgments

The authors thank the reviewers for the constructive reviews.

Enrico Santus' research is supported by the Singapore University of Technology and Design (SUTD) and by the Massachusetts Institute of Technology (MIT). Emmanuele Chersoni's research is supported by an A*MIDEX grant (nANR-11-IDEX-0001-02), funded by the French Government "Investissements d'Avenir" program.


Christophe Cerisara
email: cerisara@loria.fr

Somayeh Jafaritazehjani

Adedayo Oluokun

Hoa T Le
email: hoa.le@loria.fr

Multi-task dialog act and sentiment recognition on Mastodon

come

Introduction

Social media are a gold mine for researchers in many domains and especially in natural language processing, because of the endless stream of linguistic content produced every day. However, a major issue faced by every researcher working with Twitter data concerns the accessibility of datasets previously extracted. Indeed, license restrictions limit the possibility to store tweets in a database for a long period of time, and the proportion of tweets that are continuously deleted from the Twitter company servers makes any Twitter corpus quickly obsolete and impossible to retrieve after a few months. This is a very serious issue for the ethics of experimental research, of which reproducibility is a foundational feature. Despite these limitations, many research and development works use Twitter data for a wide variety of applications.

We propose in this work to exploit another social medium with language data, the Mastodon network, which closely resembles Twitter, except for two important differences: Mastodon is a decentralized social network, and it adopts permissive licenses that are compatible with reproducible research. In particular, everyone may create his own Mastodon server with his cohort of registered users, and have his server automatically federated within the worldwide social network. To enable this, the Mastodon software is released with the open-source AGPL license, and the user-generated content in the Mastodon network typically follows a Creative Commons license 1 , which allows redistribution of posts.

Mastodon is very similar to Twitter with regard to content and usage, except that the user posts are limited by default to 500 characters, which gives the user the possibility to write longer sentences than on Twitter. From a sociological point of view, Mastodon users are mostly composed of people who are either attracted by the technical challenge of setting up their own server, or who have been disappointed by Twitter, sometimes because of advertisements, changes of policies and privacy threats, or who feel a strong sense of belonging to a minority group, such as LGBT, because Mastodon naturally enables and favors the emergence of local communities. Another difference, which is relevant for researchers in Linguistics and Natural Language Processing, is that Mastodon is more international than Twitter, in the sense that the majority of Mastodon servers are located in Japan and in the West of Europe (France, Spain and Germany in particular), and although these servers are federated together, each of them brings together a community, which is often linguistic. It is thus frequent to observe on some server a majority of posts that are not written in English.

In terms of the amount of language data produced every day, Mastodon is still less developed than Twitter, but it is growing and is already large enough (with one and a half million of users as of February 2018) to be useful for most research works. We demonstrate this by scrapping and annotating a corpus of dialogues in English from Mastodon and training and validating a deep learning model on this data. Both the corpus and software described in this work are distributed freely at https://github.com/ cerisara/DialogSentimentMastodon. We focus in this work on two aspects of Mastodon posts: dialog acts and sentiment recognition, and on the correlations between both of these tasks.

One of the most studied natural language processing task on Twitter is sentiment recognition, which is for example a recurrent task every year at the SEMEVAL evaluation campaign [START_REF] Nakov | Evaluation measures for the semeval-2016 task 4: Sentiment analysis in twitter[END_REF]Bethard et al., 2017). We thus have also annotated our Mastodon corpus with sentiments, in order to maximize the potential impact and usage of this new corpus. Conversely, there are fewer works that study dialogues on Twitter. Dialogues on social media have quite a different form than on other media, such as vocal, sms and meetings interactions, because the participants in the dialogue are often unknown in advance, they may come and leave at any time and live in different time zones. The structure of social media dialogues thus typically forms a tree, or a directed acyclic graph if we take into account @-mentions. We focus next on dialog acts, also known as speech acts, which characterize the function of a phrase in the course of a dialog. Typical dialog acts are questions, answers, disagreements... The scientific hypothesis that is studied in this work is that sentiments are correlated to dialog acts on social media, and that this correlation may be exploited to enable transfer learning between both tasks. Intuitively, consider a dialog where user A writes something positive (sentiment) about a given smart phone brand. Then, user B disagrees (dialog act) and points out a negative aspect of this brand. An obvious correlation between dialog acts and sentiments can be found on this trivial example. Surprisingly, the majority of works about sentiment analysis ignores such relations between dialog acts and sentiments. We propose next to demonstrate experimentally that this correlation is strong enough to enable transfer learning between both tasks, and we further analyze both quantitatively and qualitatively some of the observed correlation patterns.

Corpus annotation

We hereafter call post the Mastodon equivalent of a tweet, i.e., a single message that is limited to 500 chars maximum. From the user perspective, Mastodon is very similar to Twitter, and we can thus find the same type of content that is found on Twitter. We have crawled about 800,000 posts from the octodon.social Mastodon instance, which is one amongst thousands of existing instances, and have filtered out non English posts automatically with the langdetect python library. We have then followed the reply-to links to structure all posts into dialog trees: dialogues in social media are structured as trees, because anyone can get involved in a dialog from any post that composes the dialog so far. About half of the dialogues have two posts, 1/4 three, and so on. The longest dialog is composed of 44 posts. The tree-dialogues are then split into a training and test set.

Before annotation, all dialogues are linearized, following the work of [START_REF] Zarisheva | Dialog act annotation for twitter conversations[END_REF], which means that each branch of the tree forms a unique dialog. Two students with a Master degree in linguistics and fluent in English independently assigned two tags to each post in a dialog:

• A sentiment tag with 3 possible values: positive (26% of the corpus), negative (31%) and neutral (43%); The baseline performances, when always classifying posts as neutral, are F1=24.4%.

• A dialog act with 27 possible values (the bold labels on the right of The dialog act tags have been derived from the seminal work on the Switchboard corpus (Core and Allen, 1997) and have been adapted to take into account specificities of social media, taking inspiration from the work of [START_REF] Zarisheva | Dialog act annotation for twitter conversations[END_REF]. The recently proposed ISO standard [START_REF] Bunt | Dialogue act annotation with the iso 24617-2 standard[END_REF] has also been taken into account in the design phase of the annotation guide. This process has lead to the definition of the tags listed in Table 2. After a first round of annotations and in order to avoid rare labels, these 27 tags have been further merged into 15 tags, whose distribution is shown in Table 2.

The inter-annotator agreement is 88.6% for dialog acts and 90.2% for sentiments. The Cohen's kappa coefficient is 85.1% for dialog acts and 90.2% for sentiments.

The final training corpus is composed of 239 dialogues for a total of 1075 posts, and the test corpus of 266 dialogues for a total of 1142 posts. The vocabulary size is 5330 words.

3 Multi-task model

Model description

The proposed model is shown in Figure 1. It is a two-level hierarchical recurrent network similar to the one proposed in [START_REF] Sordoni | A hierarchical recurrent encoder-decoder for generative context-aware query suggestion[END_REF]:

• The first level (post level) takes as input the sequence of word embeddings in a post. This first level is composed of a bi-LSTM, which outputs a single vector per post.

• The second level (dialog level) takes as input the sequence of vectors produced at the first level, i.e., one vector per post for every post within a dialog. The second level is composed of a standard RNN, because the length of dialogues rarely exceeds 10 posts and so the LSTM cells do not offer a clear advantage over standard recurrent cells. The output of this RNN at every post is passed to two MLPs, one for dialog acts and another for sentiment labels.

Training procedure

Two cross-entropy losses are used to train the model, one for dialog act recognition and another for sentiment classification. In our corpus, every post is manually annotated with both a sentiment and dialog act labels. So standard multi-task training simply consists in backpropagating the gradient of both losses with equal weights. However, we also realize some transfer learning experiments with fewer annotations in one of the tasks. In such cases, we artificially remove the gold sentiment label for training and only backpropagate the gradient of the dialog act loss, and vice versa when transferring from the sentiment to the dialog act task. A development corpus is extracted using 10-fold cross-validation. Three values of the learning rate (0.1, 0.01 and 0.001) have been tried on this development corpus and the best one has been kept. The number of epochs, up to a maximum of 500 epochs, is also tuned on this development corpus for each experiment. Furthermore, because some badly initialized weights may fail to converge, every experiment Table 2: List of dialog acts with the 27 labels used by annotators (letters in bold) and their distribution in the corpus (percentage within parentheses). Some labels have been merged a priori (before the annotation process), in order to simplify the annotation process. Very rare labels have been merged after the annotation process. The remaining 15 labels used to compute the F1 metric are the ones with a distribution percentage. The baseline recognition performance obtained when always answering I is F1=35%. on the development corpus is run twice with different random initializations, and the best one on the development set is kept. Running only twice does not guarantee that at least one training is correct, but it reduces the chances of failure, without increasing too much the computation requirements. All other hyper-parameters have been set a priori to reasonable values: 100-dim LSTM hidden state size, 100-dim word embeddings, 0.4 dropout, ReLU activations.

Related work

Pluwak (2016) demonstrates that some expressions of sentiments might not be detected with traditional methods of opinion mining, and that exploiting dialog acts may partly solve this challenge. Nevertheless, very few works have investigated this joint modeling of dialog acts and opinion mining: Clavel and Callejas ( 2016) study the impact and usage of both dialog act recognition and sentiment analysis in human-agent conversational platforms and affective conversational interfaces, while Novielli and Strapparava (2013) perform a dialog act clustering of a lexicon and study the emotional load of dialog acts. The authors further try to improve dialog act recognition by exploiting affective lexicon, but without convincing results. Conversely, [START_REF] Herzig | Classifying emotions in customer support dialogues in social media[END_REF] include dialog features (e.g., time elapsed between turns...) into an emotion classifier, but do not model the dialog and emotion jointly. [START_REF] Boyer | An affect-enriched dialogue act classification model for task-oriented dialogue[END_REF] exploit affects that are visually expressed on faces to better classify dialog acts.

While many works have investigated sentiment analysis on Twitter (Bethard et al., 2017), a few works only handle dialog act recognition on Twitter, such as in [START_REF] Vosoughi | Tweet acts: A speech act classifier for twitter[END_REF], (Ritter et al., 2010), (Forsythand and[START_REF] Eric | Lexical and discourse analysis of online chat dialog[END_REF] and [START_REF] Zarisheva | Dialog act annotation for twitter conversations[END_REF]. Each of these works have annotated their own corpus, because, to the best of our knowledge, no large corpus with dialog act annotations on Twitter exist. [START_REF] Zarisheva | Dialog act annotation for twitter conversations[END_REF] describe a detailed and precise procedure for annotating and evaluating dialog acts on German tweets, which we have tried to reproduce as closely as possible for English. Hence, we have manually segmented our own corpus into functional segments -following the ISO definition of a functional segment [START_REF] Bunt | Dialogue act annotation with the iso 24617-2 standard[END_REF] -and assumed that this gold segmentation is known during training and testing of our models. We have also linearized the dialog trees on Mastodon by splitting and reforming every branch of a dialog tree from the root of the tree as a complete independent dialog. However, because several such dialogues may share some posts at the beginning of dialogues, we have paid attention that our train, dev and test splits never contain the same sub-dialog, and are thus completely independent. We finally took inspiration from the conclusions of [START_REF] Zarisheva | Dialog act annotation for twitter conversations[END_REF], where it is shown that dialog act recognition is quite reliable for small taxonomies to design our own taxonomy adapted from the new ISO-standard [START_REF] Bunt | Dialogue act annotation with the iso 24617-2 standard[END_REF].

A joint model of sentiments and dialog acts is proposed in [START_REF] Kim | Integrated neural network model for identifying speech acts, predicators, and sentiments of dialogue utterances[END_REF]. It exploits one convolutional network per task, which output vectors are concatenated and passed to one classifier per task. It makes strong simplifying assumptions to merge the tasks, in particular:

• The dialog act at time t does not depend on sentiments, but does depend on the dialog act at time t -1;

• The sentiment at time t only depends on the sentiment and dialog act at time t (no Markov hypothesis); Despite these strong hypothesis, the authors report better results when considering jointly dialog acts and sentiments on a Korean corpus. As far as we know, their corpus and code is not distributed, which makes comparative evaluations with this model difficult. Compared to this work, our proposed network jointly models both tasks at deeper layers, just after the word embeddings layer. Furthermore, our hypothesis are weaker, because both labels at time t depend on both tasks at time t and t -1, and recurrence is applied at both word and dialog levels. Finally, we distribute our corpus and software under an open-source and free license to make future comparison easier.

Experimental validation

Model evaluation is performed with metrics used in related works:

• For sentiment recognition, following SemEval 2016 [START_REF] Nakov | Evaluation measures for the semeval-2016 task 4: Sentiment analysis in twitter[END_REF], this is the macro-average of the positive and negative F1 scores;

• For dialog act recognition, following [START_REF] Zarisheva | Dialog act annotation for twitter conversations[END_REF], this is the average of the dialog-act specific F1 scores weighted by the prevalence of each dialog act.

The model has been written in pytorch. The source code, along with all the data used in this work, is released as open-source and is available at https://github.com/cerisara/ DialogSentimentMastodon.

Multi-task experiments

The development corpus is composed of 28 dialogues (average size across all 10 folds). We experiment next with a training corpus of varying size: 1, 10, 50, 100, 150, 200 and the full set of 239 dialogues. The x-axis in Figures 2 and3 In Figure 2, both the mono-task and multi-task models have similar performance. So considering an additional label for the second task does not help, as compared to when only the label of the target task is given. This might be due to the fact that the model already captures all available information with a single task, and the auxiliary label does not bring enough new information to help recognition of the target label.

Transfer between tasks

We study next unbalanced cases, when the number of annotations differ between tasks. Figure 3 compares the evolution of the sentiment analysis F1 score when:

• Both tasks are trained on the same number of annotated dialogues (both-rich); this is the same experiment as realized in Section 5.1.

• The sentiment recognition task is limited to a maximum of 38 (10 dialogues for training plus the development corpus) annotated dialogues, while the training size of the dialog act task is not limited (sentiment-poor);

• Both tasks are limited to only 38 annotated dialogues (both-poor). The curves show that information is largely transferred from the richer dialog act recognition task to the target sentiment classification task. Hence, although the sentiment-poor and both-poor systems have only access to 38 dialogues annotated with sentiment labels, the accuracy of the sentiment-poor model keeps on increasing when additional dialog act labels are considered. On the right, when the full dialog act training corpus is used, its accuracy is quite close to the one obtained with all sentiment labels, and is better than the both-poor model by a large margin, thanks to multi-task transfer learning.

Conversely, we also validate transfer learning from a rich sentiment recognition task to a poor dialog act recognition task. The resulting curves are shown in Figure 4.

Similar gains in performance are obtained through transfer learning for the dialog act recognition task than for the sentiment analysis task. We can thus conclude that transfer learning is efficient between both tasks in both directions.

Analysis

Beyond quantitative experiments that demonstrate transfer learning between dialog act and sentiment recognition, we analyze next some properties of our Mastodon corpus.

Dynamics of sentiments and dialog acts

In the course of a dialog, the sentiment globally changes at a slower rate than the dialog act. While dialog acts change nearly at every segment, often, a single sentiment is expressed per dialog, sometimes two or three, but rarely more. This is understandable, but this also implies that the correlation between both tasks is not very strong, otherwise, they would have a much similar dynamic. On the other hand, they are not completely independent, as shown in the transfer learning experiments. So we try and exhibit next some patterns where correlation between tasks is explicit in the corpus.

Both tasks are sparsely correlated

In order to validate our initial hypothesis that, in the course of a dialog, sentiments may be correlated to agreements and disagreements, we have computed the transition log-probabilities between the previous sentiment s t-1 and the current sentiment s t as a function of the current dialog act d t . These log-probabilities are shown respectively for s t-1 = neutral, positive and negative in Figures 5 and6. We observe that:

• With agreements (A), a positive sentiment mainly stays positive and a negative sentiment mainly stays negative.

• With disagreements (D), a positive sentiment becomes either neutral or negative, a negative sentiment becomes either positive or stays negative, a neutral sentiment either stays neutral or becomes negative.

Figure 6: log p(s t |d t , s t-1 = positive) (left) and log p(s t |d t , s t-1 = negative) (right)

These observations globally follow our intuition, even though more complex patterns than expected seem to occur. Another observation concerns the evolution of sentiments from the start to the end of a dialog. Previous studies have shown that, with regard to dialogues, the sentiment on social media is related to:

• The topic of discussion: for instance, discussions about politics tend to have more negative sentiments (Thelwall et al., 2012a);

• The length of the dialog: negative sentiments tend to lead to longer dialogues (Thelwall et al., 2012b), which we suggest may be also related to the correlation between sentiments and disagreements that we have observed in this work; indeed, an increased proportion of disagreements may lead to longer dialogues.

Based on our corpus, another pattern seems to emerge, which is in favor of negative sentiments at the beginning of dialogues and positive sentiments at the end. This is intuitively plausible, as at least a small proportion of the sources of discords are likely to be resolved during dialogues, but this interpretation is still to be confirmed on a more diverse set of corpora.

Conclusion

We have studied a multi-task model for joint sentiment and dialog act recognition on social media. We have shown that transfer learning is quite efficient when the number of annotated labels for one task is smaller than for the other task. We have further analyzed the correlation between both tasks and shown that although there is enough mutual information to enable transfer learning, both tasks are not strongly correlated globally. They are actually characterized by different dynamics, but we have nevertheless found a few specific patterns that exhibit a strong correlation. All these studies are realized on a new corpus extracted from the Mastodon social network. Conversely to other corpora based on Twitter, this corpus enables reproducible experiments and both the annotated corpus and source code of our model are available with an open-source license on github. Obviously, the privacy issues concerned with extracting corpora from social media are not only related to licenses, and other aspects must be considered, including dissemination, usage and ethical considerations with respect to citizen privacy. Even though the proposed approach does not totally solve all such issues, it initiates an original alternative to the current dominant experimental methodology that hopefully leads to a better experimental reproducibility for natural language researches.

  ............................................................................... S (3.0%) -Directive & commissive functions -Accept offer request suggest .......................................... P (merged in A) -Decline offer request suggest ........................................ L (merged in D) -Feedback functions -auto-positive acknowledgement ..................................................... F (0.2%) -allo-positive acknowledgement .................................. (merged a priori in F) -auto-negative acknowledgement ........................................... B (merged in F) -allo-negative acknowledgement .................................. (merged a priori in B) -Social obligation functions -Initial greetings .................................................................... H (2.0%) -Return greetings .................................................... (merged a priori in H) -Initial goodbye .............................................................. G (merged in H) -Return goodbye ...................................................... (merged a priori in G) -Apology ......................................................................... X (merged in M) -Accept apology ............................................................... C (merged in A) -Thanking ................................................................................. T (2.0%) -Accept thanking .............................................................. K (merged in A) -Exclamation ................................................................................... J (1.5%) -Explicit performative ( hope you get better) .............................................. V (1.6%) -Sympathy ( I'm sorry to hear that) ............................................................. M (0.6%) -Miscellaneous / other .................................................................. * (absent) -UNK (too ambiguous to decide) ...................................................... U (removed) -Malformed input ......................................................................... Z (removed)

Figure 1 :

 1 Figure 1: Multi-task hierarchical recurrent model. The number of posts per dialog and the number of words per post may vary and are handled dynamically without any padding.

  are labeled with the total number of annotated dialogues used at training time (train + dev set). The number of epochs for training is tuned on the development set by optimizing either the sentiment loss (target sentiment) or the dialog act loss (target dialog act).

Figure 2 :

 2 Figure 2: Sentiment and dialog act F1s as a function of the number of dialogues used for training.

Figure 3 :

 3 Figure 3: Sentiment F1 as a function of the number of dialogues used for training. both-rich: both tasks have the same training size; sentiment-poor: only 38 dialogues maximum are annotated with sentiment labels; both-poor: both tasks are limited to 38 training dialogues.

Figure 4 :

 4 Figure 4: Dialog act F1 as a function of the number of dialogues used for training. both-rich: both tasks have the same training size; dialog act-poor: only 38 dialogues maximum are annotated with dialog act labels; both-poor: both tasks are limited to 38 training dialogues.

Figure 5 :

 5 Figure 5: log p(s t |d t , s t-1 = neutral). The dialog acts d t are listed at the bottom of each histogram column. Because log-probabilities are plotted, boxes with the smallest height are the most likely. The start of a dialog, which has no previous sentiment, is a special case that is added on the left.

Table 2

 2

	).

Table 1 :

 1 Example of a typical dialog from the Mastodon corpus.

  No question ............................................................... Q (8.3%) -Check question .............................................. (merged a priori in Q) -Set question -Wh * / Open question .......................................................... O (7.4%) -Choice question -Open with choices .......................................... (merged a priori in O) -Information providing functions -Inform -Statement ......................................................................... I (49.3%) -Agreement .......................................................................... A (7.9%) -Disagreement ...................................................................... D (1.9%) -Correction .................................................... (merged a priori in D) -Answer -Open + choice answer ........................................................ W (9.9%) -Confirm answer ....................................................... Y (merged in A) -Disconfirm answer ................................................... N (merged in D) -Commissive functions -Offer .................................................................................. E (1.4%) -Promise ........................................................... (merged a priori in E) -Directive functions -Request ................................................................................ R (3.3%) -Instruct ......................................................... (merged a priori in R) -Suggest .

	-Communicative functions
	-General purpose functions
	-Information seeking functions
	-Question
	-Propositional question
	-Yes/

Acknowledgments

The authors thank the "Programme d'Investissements d'Avenir" of the French government, the French National Research Agency (ANR) and the Lorraine Université d'Excellence (LUE) initiative for funding. Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr).