Vincent Segonne
email: vincent.segonne@linguist.univ-paris-diderot.fr

Marie Candito
email: marie.candito@linguist.univ-paris-diderot.fr

Benoît Crabbé
email: benoit.crabbe@linguist.univ-paris-diderot.fr

Using Wiktionary as a Resource for WSD : The Case of French Verbs

. In this paper, we investigate which strategy to adopt to achieve WSD for languages lacking data that was annotated specifically for the task, focusing on the particular case of verb disambiguation in French. We first study the usability of Eurosense (Bovi et al. 2017), a multilingual corpus extracted from Europarl (Kohen, 2005) and automatically annotated with BabelNet (Navigli and Ponzetto, 2010) senses. Such a resource opened up the way to supervised and semi-supervised WSD for resourceless languages like French. While this perspective looked promising, our evaluation showed the annotated senses' quality was not sufficient for supervised WSD on French verbs. Instead, we propose to use Wiktionary, a collaboratively edited, multilingual online dictionary, as a new resource for WSD. Wiktionary provides both sense inventory and manually sense tagged examples which can be used to train supervised and semi-supervised WSD systems. Yet, because senses' distribution differ in lexicographic examples as found in Wiktionary with respect to natural text, we then focus on studying the impact on WSD of the training data size and senses' distribution. Using state-of-the art semi-supervised systems, we report experiments of wiktionary-based WSD for French verbs, evaluated on FrenchSemEval (FSE), a new dataset of French verbs manually annotated with wiktionary senses.

Introduction

Word Sense Disambiguation (WSD) is a NLP task aiming at identifying the sense of a word occurrence from its context, given a predefined sense inventory. Although the task emerged almost 70 years ago with the first work on Automatic Machine Translation [START_REF] Weaver | Translation. Machine translation of languages[END_REF], it remains unresolved. The recent breakthrough in neural net models allowed a better representation of the context and thus improved the quality of supervised disambiguation systems [START_REF] Melamud | context2vec: Learning generic context embedding with bidirectional lstm[END_REF][START_REF] Yuan | Semi-supervised word sense disambiguation with neural models[END_REF][START_REF] Peters | Deep contextualized word representations[END_REF]. Nevertheless, although WSD has the advantage of providing interpretable senses (as opposed to the unsupervised task of word sense induction), it also has the drawback of heavily relying on the availability and quality of sense-annotated data, even in the semi-supervised setting. Now, such data is available in English, essentially with SemCor [START_REF] Miller | A semantic concordance[END_REF], a corpus manually sense-annotated with Wordnet [START_REF] Miller | Wordnet: a lexical database for english[END_REF] senses. But for most languages, sense disambiguated data are very rare or simply don't exist. This is mainly due to the fact that manual semantic annotation is very costly in time and resources [START_REF] Navigli | Word sense disambiguation: A survey[END_REF]. Nevertheless, [START_REF] Bovi | Eurosense: Automatic harvesting of multilingual sense annotations from parallel text[END_REF] recently presented Eurosense, a multilingual automatically sense-disambiguated corpus extracted from Europarl (Koehn, 2005) and annotated with BabelNet [START_REF] Navigli | Babelnet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network[END_REF] senses.

In this article, we focus on supervised WSD for French verbs and investigate a way to perform the task when no manually sense-annotated training data specifically designed for the task are available.We focus on verbs because they are known to be central to understanding tasks, but also known to lead to lower WSD performance [START_REF] Raganato | Word sense disambiguation: A unified evaluation framework and empirical comparison[END_REF]. In section 2 we report a study on the suitability of using Eurosense as training data for our task. Because the results of our evaluation were inconclusive, we decided to explore Wiktionary, a free collaboratively edited multilingual online dictionary which provides a sense inventory and manually sense tagged examples, as resource for WSD. We give a general description of Wiktionary in section 3. In section 4, we present FrenchSemEval, a new manually sense annotated dataset for French verbs, to serve as evaluation data for WSD experiments using Wiktionary as sense inventory and training examples. Because senses' distribution differ in the lexicographic examples found in Wiktionary with respect to natural text, we provide in section 5 a descriptive statistical comparison of the wiktionary example corpus and SemCor. The WSD first experiments are reported in section 6 and we provide an analysis of the results in section 7. We finally conclude and give insights of future work in section 8.

Eurosense

In this section we present our investigation on the suitability of using Eurosense as training data for supervised WSD on French verbs. We first present Eurosense and then describe our manual evaluation of the resource regarding French verbs.

Eurosense is a multilingual Europarl-based corpus that was automatically sense-annotated using the BabelNet [START_REF] Navigli | Babelnet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network[END_REF] multilingual sense inventory. This sense inventory was originally built by merging the English Wordnet and the English Wikipedia. Subsequent releases integrate senses (mapped or added) from other resources, as listed in BabelNet statistics page1 . Two versions of Eurosense are available: the "high coverage" corpus and the "high precision" corpus. Both result from jointly disambiguating the parallel Europarl corpus using the Babelfy [START_REF] Moro | Entity linking meets word sense disambiguation: a unified approach[END_REF] WSD system. A further refinement step was then performed to obtain the high precision version. The refinement aims at enforcing the intra-sentence coherence of annotated senses, in terms of similarity of their corresponding Nasari distributional vectors [START_REF] Camacho-Collados | Nasari: Integrating explicit knowledge and corpus statistics for a multilingual representation of concepts and entities[END_REF]. The resource was evaluated both intrinsically and extrinsincally.

The intrinsic evaluation was carried out through a manual evaluation of the annotation on four languages (English, French, German and Spanish). Fifty sentences present in all four languages were randomly sampled and manually evaluated both before and after the refinement step. The results showed a good inter-annotator agreement the judges agreed 85% of the time, and the average Kappa score [START_REF] Cohen | Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit[END_REF] was 67.7 %) and an improvement of the precision of the annotation after refinement at the expense of a lower coverage. For English, the high-precision Eurosense annotations cover 75% of the content words and have a 81.5% precision2 . As For French results are lower though: coverage is 71.8% and precision is 63.5%.

Closer to our WSD objective, [START_REF] Bovi | Eurosense: Automatic harvesting of multilingual sense annotations from parallel text[END_REF] report an extrinsic evaluation of Eurosense that uses it as additional training data for all-words WSD, evaluated on two standard datasets for English, the SemEval 2013 task 12 [START_REF] Navigli | Semeval-2013 task 12: Multilingual word sense disambiguation[END_REF] and the SemEval 2015 task 13 [START_REF] Moro | Semeval-2015 task 13: Multilingual all-words sense disambiguation and entity linking[END_REF]. The authors compared results of the It Make Sense (IMS) system (Zhong and Ng, 2010) when trained on SemCor alone versus SemCor augmented with examples sampled from the high precision Eurosense corpus (up to 500 additional training examples per sense). They report a slight improvement in the latter case, the Fscore rising from 65.3 to 66.4 on SemEval-2013, and from 69.3 to 69.5 on SemEval-2015.

These results give a contrasted picture of the usability of Eurosense as training data for WSD for French: the extrinsic evaluation concerns English, and a setting that uses Eurosense as additional data (picking examples from Eurosense annotated with senses present in SemCor only). For French, the situation is necessarily worse, given that (i) the intrinsic evaluation of Eurosense is lower for French, (ii) we focus on verbs, whose disambiguation is known to be more difficult [START_REF] Raganato | Word sense disambiguation: A unified evaluation framework and empirical comparison[END_REF], and (iii), most importantly, Eurosense would be used as sole training data and not as additional data. This poses extra constraints on the quality of the annotations, hence in order to further investigate the usability of Eurosense for our purpose, we decided to first perform another evaluation of the Eurosense annotations, focused on French verbs.

Evaluation of Eurosense on French verbs We randomly selected 50 sentences from the French version of Eurosense's high coverage corpus, and for all the non-auxiliary verbs (160 occurrences), we manually judged whether the Eurosense automatic sense tag was correct : we split the 160 occurrences in three sets, each set being independently annotated by two judges and adjudicated by a third one. The judges were asked to answer "correct" if the Eurosense tag seemed correct, even if some other tag in the BabelNet inventory3 was possible or even more precise. The agreement between the two judges was 0.72, and the kappa was 0.67, a level described as good in the literature. Note this is a binary task only, which is different from asking the judges to annotate the correct tag as [START_REF] Bovi | Eurosense: Automatic harvesting of multilingual sense annotations from parallel text[END_REF] did for all parts-of-speech. Yet our agreement score is even lower, shedding light on an overall greater difficulty of judging the annotated sense of verbs. Indeed, we were then able to measure that the proportion of Eurosense automatic annotations that we judged correct after our adjudication is 44% only. Moreover, during this annotation task we could notice that because BabelNet aggregates several data sources (including Wordnet, Verbnet, FrameNet, Wiktionary among others), the BabelNet sense inventory exhibits a high number of senses per verb. To better quantify this observation, we sampled 150 sentences, and measured that the average number of BabelNet senses per verb type occurring in these sentences is 15,5. More importantly, we could notice that the frontiers of the various senses sometimes appeared difficult to judge, making it difficult to grasp the exact perimeter of a sense.These mixed results led us to investigate other sources of sense-annotated data for French.

Wiktionary as data for WSD

Wiktionary is a collaboratively edited, open-source multilingual online dictionary, hosted by the Wikimedia Foundation. It provides an interesting open-source resource and several studies already showed its usefulness for various NLP tasks (e.g. lemmatization [START_REF] Liebeck | Iwnlp: Inverse wiktionary for natural language processing[END_REF]), especially in the lexical semantic field, for extracting or improving thesauri [START_REF] Navarro | Wiktionary and nlp: Improving synonymy networks[END_REF][START_REF] Henrich | Semi-automatic extension of germanet with sense definitions from wiktionary[END_REF][START_REF] Miller | Wordnet-wikipedia-wiktionary: Construction of a three-way alignment[END_REF]. In this section we briefly present Wiktionary's most interesting features along with our motivations to investigate the use of this resource for WSD on French verbs.

Wiktionary's main advantages is that it is entirely open-source, multilingual and has a good coverage for a substantial number of languages (according to wiktionary statistics4 , 22 languages have more than 50, 000 wiktionary entries each). Each entry consists of a definition and one or several examples, either attested or created, each example being a potential sense-annotated example for the lemma at hand. Definitions and examples point to other wiktionary pages, which can be useful, although not as useful as if links to wiktionary senses (not pages) would be provided. The structured nature of wiktionary makes it possible to extract wordnets rather easily (as was done for English, German and French by [START_REF] Sérasset | Dbnary: Wiktionary as a lmf based multilingual rdf network[END_REF], in the RDF format). On the qualitative level, our interest for wiktionary rose after studying random verbal entries for French: we could observe that in general the granularity level is "natural" and that the sense distinctions are easy to grasp. On the quantitative level, we report in table 1 several statistics for the French wiktionary5 , in which it can be seen that the resource is large (we will see in the next section that the coverage in corpus is good indeed). These advantages come at the cost of Wiktionary's main potential drawback, namely its crowdsourced nature. Firstly, this means that it is constantly evolving, since any user can edit pages at any time (unless pages that users with more editing rights might have protected). Indeed, new pages are created every day while already existing pages are deleted, modified, merged (note though that every change occurring in the resource is kept in track). Secondly, this means that the resource is not curated by skilled lexicographers only, and the "guidelines" are themselves collaboratively built.

POS Nb of entries

Despite this potential disadvantage, several features of Wiktionary seemed particularly suitable for the task of WSD and this, combined with the fact that sense-annotated data for French verbs are quasiinexistent6 , makes it a serious candidate for a new resource of WSD. To investigate this opportunity for our objective of French verb WSD, we present FrenchSemEval, a new dataset manually annotated for WSD of French verbs which we used to carry out several evaluations, we describe the new resource in the next section.

FrenchSemEval : An evaluation corpus for French verb disambiguation

Since the first Senseval evaluation serie in 1998 [START_REF] Kilgarrif | Senseval: An exercise in evaluating word sense disambiguation programs[END_REF], a various number of evaluation frameworks were proposed to evaluate different WSD tasks, but only a few include French test datasets [START_REF] Lefever | Semeval-2010 task 3: Cross-lingual word sense disambiguation[END_REF][START_REF] Navigli | Semeval-2013 task 12: Multilingual word sense disambiguation[END_REF] and unfortunately these only focus on nouns7 . In this section we present FrenchSemEval8 a new French dataset in which verb occurrences were manually annotated with Wiktionary senses. Our objective was to evaluate whether Wiktionary's sense inventory is operational for humans to sense-annotate a corpus, and if so, to use it as evaluation data for WSD experiments. We describe the annotation process along with several statistics about the resulting dataset and the quality of the annotations.

Data selection

To build FrenchSemEval, we chose to focus on moderately frequent and moderately ambiguous verbs. Rare verbs are often monosemous, and very frequent verbs tend to be very polysemous and extremely difficult to disambiguate (we thus left these for future work). FrenchSemEval was built using the following steps: we first selected a vocabulary of verbs based on their frequency in corpus. We selected verbs appearing between 50 and 1000 times in the French Wikipedia (dumped on 2016-12-12 hereafter fr-Wikipedia). Secondly, from this pre-selected list of verbs we extracted those having a number of senses comprised between two and ten in Wiktionary's sense inventory. For these verbs, we chose to extract 50 occurrences primarily from corpora comprising other annotations (the French TreeBank (FTB) (Abeillé and [START_REF] Abeillé | Enriching a french treebank[END_REF]) and the Sequoia (Candito and Seddah, 2012) treebank9 ), supplementing the corpus when necessary by occurrences sampled from fr-Wikipedia, in order to reach 50 occurrences per verb.

Annotation process

The annotation has been performed by three students10 for nearly a month. We used WebAnno [START_REF] Yimam | Automatic annotation suggestions and custom annotation layers in webanno[END_REF][START_REF] De Castilho | A web-based tool for the integrated annotation of semantic and syntactic structures[END_REF] an open-source adaptable annotation tool. Sentences had already been pre-processed into CoNLL format [START_REF] Nivre | The conll 2007 shared task on dependency parsing[END_REF] with the Mind The Gap (MTG) parser [START_REF] Coavoux | Incremental discontinuous phrase structure parsing with the gap transition[END_REF] and were plugged in WebAnno. We were thus able to provide files (one file per verb) containing sentences in which occurrences of the specific verb were marked for annotation. The annotators were asked to annotate only the marked occurrences. We integrated in WebAnno the sense inventory from Wiktionary, including definitions and examples of senses, and added two extra tags: "OTHER POS" and "MISSING SENSE". The former was to use when an occurrence was wrongly tagged as verb, and the latter was to use when the sense of an occurrence didn't exist in the sense inventory. As Wiktionary is constantly evolving through time, we used the 04-20-2018 dump available via Dbnary [START_REF] Sérasset | Dbnary: Wiktionary as a lmf based multilingual rdf network[END_REF]. The annotation was performed in double annotation and adjudication.

Resulting resource

Table 2 reports various statistics about the resulting dataset. It contains 3199 occurrences for 66 different verbs, which means nearly 50 annotated instances per verb (about 100 OTHER POS occurrences were discarded). The annotators agreed more than 70% of the time and obtained a Kappa score of 0.68 which is good according to the literature. We believe that these metrics indicate an annotation quality which may not be extremely high but still sufficient to validate the coherence of the Wiktionary sense inventory, definitions and examples, despite its non-expert crowd-sourced nature.

Descriptive statistical study of the datasets

The best-suited data for training a supervised WSD system is a corpus with sense tags for all content words. Training on such a corpus benefits from basic frequency information found in the corpus. This is particularly striking for WSD, as the "most frequent sense" baseline is known to be very high. FSE). AMBIG trainSI corresponds to using for the number of senses the sense inventory in the corresponding training corpus, whereas AMBIG fullSI corresponds to using the full sense inventory.

Comparison of the sense distribution in training examples

We study here the distribution of the annotated senses in training data. When looking at the number of training examples per sense, we obtain an average of 9.6 and a mean absolute deviation of 11.9 for SemCor, whereas the average is only 2.0 for FR-Wiktionary, and the mean absolute deviation is 0.9. It is clear that using wiktionary examples will lack the genre-dependent but nonetheless very informative information of sense frequency in corpus.

Evaluation of the task difficulty: comparison of ambiguity rates

We now turn to comparing the difficulty of the WSD task, when tested on English SenseEval datasets versus on FrenchSemEval. Note that performance of WSD systems cannot be used for that purpose, given that it is not comparable across languages and datasets. For a corpus consisting in a sequence of tokens t 1 . . . t N , we rather compute the average ambiguity rate that a WSD system has to face, in two settings:

• token AMBIG fullSI: the ambiguity rate per token, using the full sense inventory:

1 N N i=1
n senses(t i )

• token AMBIG trainSI: the ambiguity rate per token, using the sense inventory found in the training corpus

1 N N i=1 attested n senses(t i )
For further information, although not directly measuring corpus WSD difficulty, we also provide the ambiguity rate per verb type, both using the full inventory or that attested in the training set (shown in the "type" columns in Table 3).

We report these metrics in Table 3. When studying the difference between the "fullSI" versus "trainSI" modes, namely when using the full sense inventory versus that found in the training set, we have a different trend for the English corpora (containing natural text) and the French ones: for Sem-Cor and the English evaluation sets, there is a drop of ambiguity in trainSI mode. This illustrates the usual difficulty to cover rare senses in a corpus of natural text. Note though that for the French corpora, based on the wiktionary inventory, there is almost no difference between the two modes of computation, illustrating that almost all senses have examples in wiktionary.

When comparing, for each language, the figures for the training corpora (SemCor and Wiktionary examples) and for the evaluation datasets, it can be noted that the average ambiguity per token is similar for training and evaluation datasets, but the average ambiguity per type is much smaller for the training corpora (3.24 for SemCor, and 1.74 for Wiktionary). This is because the lexicon covered in the training corpora is much larger, and contains many more monosemic verbs. . As far as training corpora are concerned, it can be seen that the overall average ambiguity is higher for SemCor than for Wiktionary (e.g. in fullSI mode, 10.94 per token ambiguity for SemCor, versus 5.68 for Wiktionary). It shows that the sense inventory for Wiktionary is slightly less ambiguous than Wordnet's (both for the senses found in SemCor, and overall).

Experiments on supervised WSD

To investigate the suitability of using Wiktionary for supervised WSD on French verbs, we evaluated state-of-the-art supervised WSD systems on FrenchSemEval, using the examples of Wiktionary's senses as training data. As for the representation of the instances we used two different models that we describe below. We then applied a supervised disambiguation method to evaluate the performance of the models. We first describe the models used to obtain vector representations of the instances and the disambiguation algorithm we used for evaluation. Then we propose several experiments based on FSE and finally we evaluate the models using Wiktionary as input for disambiguation.

Models for context representations AWE

We implemented a simple model that we use as baseline. We first train a word2vec [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] model on fr-Wikipedia11 to obtain non contextual word vectors. We then represent the context of an occurrence by averaging the vectors of the words found in its context window, which we defined as the 5 words on the left and 5 words on the right of the target word. This is a common model often referred in the literature as averaged-word-embeddings (AWE).

C2V Context2vec [START_REF] Melamud | context2vec: Learning generic context embedding with bidirectional lstm[END_REF] is a recurrent neural model that learns a function mapping the context around a target word to a vectorial representation. The context2vec model represents the context using a bi-directional recurrent neural network [START_REF] Hochreiter | Long short-term memory[END_REF]) that allow us to take the context of the sentence into account, thus contrasting with AWE. All codes and implementations are available publicly so we only adapted it and trained the model on the whole French Wikipedia. We then applied the learnt model to obtain vectorial representations of our target verb occurrences.

Supervised disambiguation algorithm

We replicated the supervised WSD method used in [START_REF] Yuan | Semi-supervised word sense disambiguation with neural models[END_REF]: a sense representation is computed from annotated data by averaging the context vector representation of its instances, in our case the Wiktionary examples or instances from FSE. Then each test instance is sense tagged with the sense whose representation is the closest, based on cosine similarity.

Protocol

Wiktionary experiment We did a first experiment simply using the examples of the senses in the Wiktionary sense inventory as training data and then we performed disambiguation on FSE.

In domain experiments In order to better identify the potential error sources, we also performed experiments with "in-domain" training instances, namely instances directly taken from FSE. To evaluate the impact of the number of training examples per sense, a property that is quite different for a lexicographic training set as opposed to a corpus-based training set, we performed experiments on different sets, using N max a varying maximum number of examples per sense. More precisely, for each verb we selected respectively 1, 2, 5 and 10 maximum training examples per sense from the dataset and evaluated the disambiguation on the remaining examples.

Results and Analysis

The results of our experiments on Wiktionary are presented in table 4. Although both automatic systems perform better than the Most Frequent Sense baseline (MFS), using only the literary Wiktionary examples to build a classifier for newspaper and Wikipedia test instances remains a rather adversarial setup 12 To study the effect of the amount of training instances, since Wiktionary is limited in terms of number of examples per sense, we switched to using FSE both for training and testing. We used a variable number of maximum training examples per sense from N max = 1 to N max = 10 and we used the remaining examples as test set 13 . The results of these experiments are summarized in Table 5 and illustrated in Figure 1.

Let us observe first the impact the amount of training data. The mean number of examples in Wiktionary for the verbs occurring in FSE is N avg = 3.1 and the results show that all classifiers dramatically improve when the available training examples per verb grows up to N max = 10. This means that if we were able to expand with absolute certainty the small amount of examples in Wiktionary, we could get a much higher disambiguation performance.

Second, using the same setup we can compare the behaviour of the classifiers when predicting out of domain (Table 4) with in domain predictions ( Third, we can observe that the 3 classifiers (MFS, AWE,C2V) do behave consistently in the different configurations. To understand why MFS is a rather weak predictor in our setup, we have to recall that Figure 1: Illustration of results reported in table 5. contrary to Semcor and Senseval, FSE is built following a lexicographic perspective: the sentences are sampled in a non natural way (e.g. monosemic words are excluded). We observe (table 5) that in these conditions the MFS is a weaker (although still strong) baseline and that standard sequential neural models are able to outperform it, especially when there are few training examples. Among the neural models C2V performs consistently better than AWE but we that these models or some extensions of these models to be designed in the future might well still show significant improvements.

Lastly, we can observe that among all models, C2V performs best on every test setup. It supports [START_REF] Melamud | context2vec: Learning generic context embedding with bidirectional lstm[END_REF]'s results, especially regarding the fact that Context2vec succeeds better in capturing context information than the common averaging of word vector representations.

Conclusions and future work

Word Sense Disambiguation is a task rarely seen for languages other than English. One obvious reason to explain that is the of costly sense annotated resources for those languages. In this paper we provide some elements seeking to set up a methodology to perform word sense disambiguation for other languages than English, such as French, without requiring the cost of annotating sense disambiguated corpora.

For this purpose we considered using Eurosense and Wiktionary as training data for Verb Sense Disambiguation. As our first experiments with Eurosense turned out to be inconclusive, we then turned our attention to Wiktionary. We studied how to use it as a resource for Word Sense Disambiguation and we develop FrenchSemEval, a new French WSD evaluation dataset, thanks to which we were able to extract preliminary evaluation results. Our current results showed that the Wiktionary sense inventory has an appropriate granularity for a good quality sense annotation, and that training on Wiktionary examples only leads to encouraging WSD results for verbs. But we could also quantify the gain in performance that could be obtained by adding a moderate number of seed instances. Hence automating the selection and annotation of additional instances might pay off to improve verb sense disambiguation. Zhong, Z. and H. T. Ng (2010). It makes sense: A wide-coverage word sense disambiguation system for free text. In Proceedings of the ACL 2010 System Demonstrations, pp. 78-83. Association for Computational Linguistics.

Table 1 :

 1 Nb of senses Mean nb of senses per entry Nb of examples Statistics from French Wiktionary of the 04-20-2018 dump available via the tool of Sérasset(2012)

	Noun	81099	112428	1.39	1511517
	Verb	27271	41207	1.51	55206
	Adj	25865	33732	1.30	46212
	Adv	5904	6012	1.29	5904

Table 2 :

 2 Statistics for the FrenchSemEval corpus (FSE).

	Number of sentences	3121
	Number of annotated verb tokens	3199
	Number of annotated verb types	66
	Mean number of annotations per verb type 48.47
	Mean number of senses per verb type	3.83

Table 3 :

 3 Ambiguity rates for verbs, in the English usual training set (SemCor) and usual evaluation sets, and in the French training set (Wiktionary) and evaluation set (

	In

Table 4 :

 4 . We thus two potential ways to leverage the hardness of this initial setup: domain adaptation and the amount of training examples. WSD accuracies when training on Wiktionary examples, and testing on FSE.

	Models score
	MFS	0.30
	AWE	0.40
	C2V	0.43

Table 5 )

 5 . Recall that Wiktionary examples are often long literary sentences whereas the test instances are sampled from newspaper or Wikipedia. Again as Wiktionary has N avg ≈ 3 training examples per sense we can see that the domain adaptation effect is worth roughly 20 points in accuracy.

	Models	N max = 1 N max = 2 N max = 3 N max = 5 N max = 10
	MFS	0.32	0.38	0.45	0.52	0.70
	AWE	0.44	0.53	0.58	0.64	0.70
	C2V	0.5	0.57	0.62	0.68	0.74
	Mean number of training ex. per sense	1	1.81	2.61	3.86	6.30
	Mean size training data per verb	3.83	6.95	9.81	14.8	24.15
	Mean size test data per verb	44.63	41.51	38.65	33.66	24.31

Table 5 :

 5 Training on FSE examples, with varying maximum number of examples per sense (N max ). Top: WSD accuracies. Bottom: training / test sets statistics.

https://babelnet.org/stats

Note the precision is computed as an average between the two annotators, and not with respect to an adjudicated version.

The sense inventory had been previously extracted via the HTTP API see https://babelnet.org/guide

https://en.wiktionary.org/wiki/Wiktionary:Statistics

In all the following, all the statistics and work on the French wiktionary corresponds to the 04-20-2018 dump available via Dbnary[START_REF] Sérasset | Dbnary: Wiktionary as a lmf based multilingual rdf network[END_REF].

Verbs are annotated with frames in the French FrameNet data[START_REF] Djemaa | Corpus Annotation within the French FrameNet: a Domain-by-domain Methodology (regular paper)[END_REF], but in such data, only some notional domains were considered, and verb occurrences not pertaining to such domains were not disambiguated.

Except for SensEval1 but only the English dataset was given to public domain.

The dataset is available here http://www.llf.cnrs.fr/dataset/fse/

The FTB contains 18500 sentences, from articles from Le Monde newspaper, and Sequoia contains 3099 sentences from Europarl, the European Medicine Agency, a regional newspaper (L'Est Républicain) and fr-Wikipedia.

None of them had previous experience in annotation.

We used the fr-Wikipedia dump of 10-20-2017

As a comparison, results for supervised WSD for English verbs are around 0.55 in the benchmark of[START_REF] Raganato | Word sense disambiguation: A unified evaluation framework and empirical comparison[END_REF].

We say that we use Nmax as a maximum number of training examples because some senses may have only K < Nmax annotated instances in the whole data set.


Silvio Ricardo Cordeiro

Marie Candito
email: marie.candito@linguist.univ-paris-diderot.fr

Syntax-based identification of light-verb constructions

 for the known cases, and couple it with another simple baseline to handle the unknown cases. We additionally present two other classifiers based on a richer set of features, with results surpassing these best results by 7 percentage points.

Introduction

Light-verb constructions (LVCs), such as the expression pay visit, are a linguistic phenomenon coupling a verb and a stative or eventive noun, in which the verb itself is only needed for morphosyntactic purposes, its syntactic dependents being semantically related to the noun. For instance in the sentence John paid me a visit, the subject and object of paid play the roles of the visitor and the visited. The verb's semantics is either bleached or redundant with that of the noun (as in commit crime) [START_REF] Savary | PARSEME multilingual corpus of verbal multiword expressions[END_REF]. This mismatch between syntax and semantics has to be taken care of for semantically-oriented tasks to recover the full predicate-argument structure of the noun, since at least one of its semantic arguments of the noun is generally attached to the verb in plain syntactic treebanks. 1 Moreover, the fact that the verb choice is conventionalized and semantically bleached makes LVC identification an important requirement in semantic tasks such as machine translation [START_REF] Cap | How to account for idiomatic German support verb constructions in statistical machine translation[END_REF].

Because of their syntactico-semantic characteristics, LVCs are generally considered difficult to circumscribe and annotate consistently [START_REF] Bonial | Comprehensive and consistent propbank light verb annotation[END_REF]). Yet recently, the PARSEME 2018 shared-task has brought forth a collection of corpora containing verbal multiword expression (VMWE) annotations across 19 languages [START_REF] Ramisch | Edition 1.1 of the PARSEME shared task on automatic identification of verbal multiword expressions[END_REF], including LVCs. The reported inter-annotator agreement is variable across languages, but the macro-averaged chancecorrected kappa is overall 0.69, which is generally considered to denote a good agreement. In the annotated corpora, the category of LVCs2 accounted for a third of all expressions [START_REF] Savary | PARSEME multilingual corpus of verbal multiword expressions[END_REF]. The annotation was performed in a separate layer, largely independent from the underlying syntactic framework, and relied on semantic properties of the verb (bleached or redundant in the given context) and both semantic and syntactic properties of the noun: it should be stative or eventive and take at least one semantic argument, and it should be possible to have all the syntactic arguments of the verb realized within a NP headed by the noun (for instance out of John paid me a visit, one can create the NP John's visit to me).

A total of 13 systems participated in the PARSEME shared-task, predicting VMWEs occurrence in the test corpora. Results for each system varied across different systems and target languages, in which expressions that had been seen in the test corpus were predicted with variable accuracy. However, expressions that had never been seen in the test corpus were hardly ever predicted by most systems (the best F-score on unseen-in-train expressions in the closed track is below 20%).

In this paper, we investigate the task of LVC identification in running text. The main contributions of this paper are: (1) we propose handling the task of LVC identification differently depending on whether it was seen in the training corpus;

(2) we present a simple baseline that surpasses all systems for seen LVCs; (3) we propose and evaluate different techniques for the prediction of unseen LVCs, which we then compare to the state of the art.

The remainder of this paper is structured as follows: Section 2 presents the related work; Section 3 describes the methodology that will be employed; Section 4 describes the results; and finally, Section 5 presents our conclusions.

Related Work

LVC identification may follow one of two strategies: (a) LVC candidates are initially proposed based on lexicosyntactic patterns, and are then classified as LVC or non-LVC based on other criteria; (b) a variant of the BIO scheme [START_REF] Lance | Text chunking using transformation-based learning[END_REF] is employed so as to directly classify each token as belonging or not to an LVC. The former method allows the use of features that encompass the LVC as a whole, while the latter can be more easily implemented in the framework of some machine learning algorithms.

Most works in the literature concerning LVC identification focus on annotations in a particular language, often with a language-specific understanding of LVCs. [START_REF] Vincze | Dependency parsing for identifying Hungarian light verb constructions[END_REF] adapt a dependency parser so as to identify Hungarian LVC candidates as a byproduct of parsing, which they then evaluate on the Szeged Dependency Treebank with LVC annotations. [START_REF] Nagy | Full-coverage identification of English light verb constructions[END_REF] extract English LVC candidates involving a verb and a dependent noun with a specific dependency label. A J48 and an SVM classifier are then considered, using lexical and morphosyntactic features from the corpus, as well as semantic features from WordNet. The latter was found to contribute to better results when compared to earlier works that relied purely on morphosyntactic and statistical features [START_REF] Tu | Learning English light verb constructions: contextual or statistical[END_REF]. [START_REF] Chen | English light verb construction identification using lexical knowledge[END_REF] detect English LVCs in the BNC and OntoNotes corpora, using the PropBank layer to select LVC candidates composed of an eventive noun linked one of 6 known light verbs. The candidates are then filtered based on semantic features, including WordNet synsets and hypernym relations.

More recently, the PARSEME shared-task saw 13 system submissions that tried to predict LVCs along with other VMWEs for annotated corpora in 19 languages [START_REF] Ramisch | Edition 1.1 of the PARSEME shared task on automatic identification of verbal multiword expressions[END_REF]. Overall, the best F 1 scores across all languages in the open track were obtained by the SHOMA system, which employed a pipeline of CNNs, a Bi-LSTM, and and optional CRF layer [START_REF] Taslimipoor | SHOMA at PARSEME Shared Task on automatic identification of VMWEs: Neural multiword expression tagging with high generalisation[END_REF]. MWE prediction followed a variant of the BIO scheme that allowed multiple tags per token, with input features including a set of pretrained embeddings (leading the system to compete in the open track category), POS tags, and a set of word-shape features.

In the closed track, the TRAVERSAL system obtained the best overall results for MWEs in general as well as for LVCs. It uses a syntax-based approach, in which each node in the syntax tree was classified as part of an MWE or not [START_REF] Waszczuk | TRAVERSAL at PARSEME Shared Task 2018: Identification of verbal multiword expressions using a discriminative treestructured model[END_REF]. The classifier resembles a second-order CRF, but rather than considering the previous 2 tokens at each point, it considers the parent and leftsibling. Features included the lemma, POS tag and dependency relation.

Rather than predicting each token as being part of an LVC or not, the varIDE system use a Naive Bayes classifier to tag LVC candidates [START_REF] Pasquer | VarIDE at PARSEME shared task 2018: Are variants really as alike as two peas in a pod?[END_REF]. These were extracted based on all possible token combinations whole multi-set of lemmas corresponded to an LVC that had been seen in the training corpus (no attempts were made at predicting unseen LVCs). Classifier features included POS tags and morphological information.

Graph convolutional neural networks have also been used in the identification of VMWE candidates for subsequent classification [START_REF] Omid Rohanian | Bridging the gap: Attending to discontinuity in identification of multiword expressions[END_REF]. In this work, the network is combined with an attention mechanism so as to improve the accuracy of long-range predictions, and a Bi-LSTM layer is used to classify these predictions and produce the final output. The system uses contextualized embeddings (ELMo) and outperforms the state of the art for the four languages for which results are reported.

Methods and materials

Our LVC identification technique consists of two main stages: (1) extraction of LVC candidates based on syntactic patterns; and (2) classification of candidates based on a set of lexical, morphological, syntactic and semantic features, concerning both the candidate as a whole and its components.

Extraction of candidates

The first step of LVC identification in a target corpus is to identify candidates. While LVCs are commonly thought of as a combination of a verb and noun acting as its direct object, other configurations can be attested in the PARSEME corpora. This may be due to morphosyntactic variations (e.g. passive voice), the presence of more complex noun phrases (instead of a single noun), nonstandard analyses (e.g. verbs that are tagged as adjectives) or other language-specific idiosyncrasies.

A robust candidate extraction method should handle this variation, and we do so by using the morphosyntactic patterns of the LVCs in the training corpora, using the provided UD parses.

So we start by extracting language-specific morphosyntactic patterns from the training LVCs. More specifically, for each LVC annotated in a training corpus, we retain a representation involving the POS tag and the syntactic relation between components (henceforth referred to as a "pattern"). If the LVC does not form a connected tree (e.g. to give a series of lectures), the pattern will additionally include the minimum number of nodes that makes the tree connected (if two nodes are only connected by the root node, we discard the occurrence instead). In the example above, the extracted pattern would be:

VERB 1 obj --→(NOUN 2 )
nmod --→NOUN 3 (the components of the LVC being those not within brackets).

The number of extracted patterns ranges from 14 (Slovene) to 185 (Farsi), with an average of 90 patterns per language. We then sort the patterns based on how many occurrences of LVCs led to each pattern. As expected, the patterns follow a Zipfian distribution. For example, for the French training data, the most common pattern is VERB 1 obj --→NOUN 2 with 977 occurrences; the second is NOUN 1 acl --→VERB 2 with 150 occurrences (as in for instance a picture taken yesterday); the third is VERB 1 nsubj:pass ------→NOUN 2 with 58 occurrences (as in this picture was taken yesterday); and so on3 .

The most common patterns are then used to identify LVC candidates in the train, development and test data, using the Grew tool [START_REF] Bonfante | Application of Graph Rewriting to Natural Language Processing[END_REF]. Obviously, using unlexicalized patterns results in getting a vast majority of candidates that are not LVCs, and this is even more true for rare patterns. We experimented with two pattern selecting stategies: topN, in which we take the N most common patterns (we considered values of N ∈ {1, 5, 10, 20, 50} ); and atleastNoccurs, in which we take all patterns that originated from at least N occurrences in the training corpus (we considered N ∈ {2, 5, 10, 50}). Moreover, for each pattern p containing label ---→NOUN i , we add a pattern p replacing this subpattern by label ---→(NOUN j ) conj --→NOUN i4 . Using the selected patterns, we identify LVC candidates in the development and test corpora, but also in the training corpora, so as to obtain positive and negative LVC candidates to train a binary classifier. For each identified candidate, we produce a set of features which may be related either to the whole LVC, or to its components.5

Features

The PARSEME 1.1 data contains test/development and training data for 19 languages. The training data contained an average of 1171 LVCs per language (σ=948, ranging from 78 for English to 2952 for Turkish). Most corpora contain morphosyntactic information (in most cases obtained by an external parser, and in most cases representing data using the POS and dependency tagsets recommended by UD).

For a given candidate c, we first extract the verb component v and predicative noun component n. This is in general trivial, but in order to cover all cases, v is taken to be the leftmost token that has POS tag VERB, or the leftmost AUX, or the leftmost ADJ, or the leftmost token in c, while n is the leftmost NOUN, leftmost PROPN, or leftmost token that is not v. In all the features, we use the lemmas of v and n. We then extract the following features:

• F 1 : One-hot representing the pattern used to predict the candidate (see Section 3.1). dates in train that have the same pattern and lexical items (lemma-wise comparison) as c (-1 if unseen in train).

• F 3 : POS tag of v and n.

• F 4 : Dependency relation between v and n (NONE if not directly connected).

• F 5 : One-hot for the number of components of c (with the rationale that LVCs of length higher than 2 may display more non-standard behavior due to the additionally lexicalized words).

• F 6 : One-hot for the number of gaps (extraneous words that do not belong to the LVC), between the leftmost and rightmost components of c, in the underlying sentence.

• F C : Binary contextual features from the underlying UD parses. Features are defined for every observed key, value pair in the morphological CoNLLU column (e.g. Tense, Past ), as well as every observed column, value pair for the UD columns FORM, LEMMA, XPOS, UPOS and DEPREL (e.g. FORM, took , LEMMA, picture , POS, NOUN ). These features are binary in value, and indicate whether the key, value pair is present for c. A feature is considered present if it appears in at least one of the direct dependents of n or v. We consider only the top t features with the highest mutual information and whose underlying pairs appear in at least LVCs.

While it is clear that LVC identification would greatly benefit from fine-grained semantic clues such as noun predicativeness, such information is not readily available for most languages under study. We consider instead on a set of unsupervised features that can be constructed for all languages based on distributional semantic models. In particular, we consider the fasttext [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF] set of pretrained word embeddings (which is also used by the SHOMA system) as a basis for semantic features.

• F E : Word embeddings for the lemma of the verb and noun (300 dimensions each).

• F 1 k : k-nearest neighbors of the underlying noun n. Considered neighbors are all nouns that are paired up with the underlying verb v in at least one LVC candidate in the training set, whether true LVC or not. We select the top k neighbors whose embedding has highest cosine against n's embedding. Each neighbor is either seen-in-LVC (it is part of at least one true LVC) or an unseen-in-LVC (it is part of false positives only). The final value of the feature is the sum of the scores of the k neighbors, where a seen-in-LVC neighbor has score +1 and an unseen-in-LVC neighbor has score -1.

• F c

k : Same as F 1 k , but each neighbor's score that is being summed up is additionally weighted by the underlying cosine.

LVC classifiers

We present below two LVC candidate binary classifiers based on the features above: SVM and FFN. We compare them against two simple baselines: Majority, which only predicts LVCs seen in train, and kNN, which we use either for all LVCs or for those unseen in train, in combination with Majority for the seen LVCs. Note we consider a predicted or gold LVC to be seen in train when the training corpus contains at least one gold LVC with same lemmas, in whatever order and with whatever syntactic pattern. Table 2: F 1 scores on Majority and kNN baselines (F c k with k = 2) , along with the best configuration for the SVM and FFN classifiers -on the development sets.

meaning that within the k nominal neighbors of the noun n of the candidate, the total cosine of seen-in-LVC surpasses that of the unseen-in-LVC neighbors.

• SVM: Support vector machine with RBF kernel. Positive and negative examples are balanced through compensating class weights. We use a 3-fold grid-search to select for the best combination of classifier hyperparameters for each language; we consider the values C ∈ {1, 10, 20, 50, 100} and γ ∈ {0.5, 0.1, 0.05, 0.01}.

• FFN: Feed-forward network with a 100neuron hidden layer, using tanh as an activation function and 50% dropout. The network uses an SGD optimizer 6 and negative log-likelihood loss. Positive training examples are duplicated as much as needed so as to be balanced against negative examples. The final list of examples is shuffled, and fed into the classifier in batches of size B ∈ {1, 2, 4, 8, 16}. Training is performed for a number e of epochs, such that epoch e + 1 would have had higher loss on the validation set (10% of train). One-hot features are implemented as a layer of trainable embeddings instead (300 dimensions for lemmas; 5 dimensions for dependency relations, for F 5 and F 6 ).

Evaluation

We explore hyperparameters on the 16 languages that contained a development set, and evaluate the final systems on the test set for all 19 languages (using both training and development set for training). Evaluation of LVC predictions for each language uses the MWE-based F 1 score from of the PARSEME shared task [START_REF] Ramisch | Edition 1.1 of the PARSEME shared task on automatic identification of verbal multiword expressions[END_REF]. We modified its evaluation script so as to output scores 6 Basic tuning of the learning rate led us to use 0.01.

for seen and unseen LVCs: it first labels a LVC (whether gold or predicted) as "seen" if there exists at least one gold LVC occurrence with the same set of lemmas in the training set, and unseen otherwise. The two labels are then evaluated separately.

We also present a micro-average score (µAvg), in which the F 1 scores of all languages are averaged with a weight that is proportional to the number of LVCs in that languages test (or development) set. 7 On test sets, we compare our results with SHOMA and TRAVERSAL, the two highestscoring systems in the shared-task. 8

Results

Table 1 presents the fraction of LVCs in the development set that can also be seen in the training set. In the lower end, German dev LVCs were only seen in train 26% of the time, mostly due to the small training set in this language. In the higher end, 90% of Romanian LVCs had a counterpart in the training set, suggesting that a simple baseline focusing on seen LVCs should already yield good results for this language.

The last two rows in Table 1 present the coverage (i.e. recall) in the initial step of LVC candidate extraction, for the strategy atleastNoccurs with N = 2. This strategy was found to yield the best results in both SVM and FFN settings during early experiments. It can be seen that, despite variation across languages, mainly due to training corpus size differences, the micro-averaged coverage is 95% for dev LVCs seen in train, and slightly Table 3: F 1 scores (split for seen LVCs, unseen LVCs and overall) for the Majority and kNN baselines, the best configuration of our SVM and FFN classifiers, and the highest-scoring systems in the shared-task (SHOMA and TRAV(ERSAL)) -evaluated on the test sets. lower (90%) for unseen ones.

We tuned the hyperparameters on the development sets. For every system, the same configuration is used for all languages. The best kNN configuration is F c k with k=2; the best SVM and FFN configurations are both F 1..6 , F C (t=30, =30), F E .

Table 2 presents the scores obtained by these best configurations on the development sets. Across seen LVCs, both the Majority and kNN baselines have considerably high scores (F 1 =81 and 78 respectively, but the highest results are obtained by FFN and SVM (F 1 =85 and 84). For the unseen LVCs, results are quite lower, and there is a bigger gap between the kNN baseline (F 1 =0.22) and the best system on unseen, namely the SVM (F 1 =0.34). Table 3 presents system results for the same configurations when evaluated against the test sets. On seen LVCs here again, the Majority baseline is slightly higher than the kNN baseline. However, both baselines beat the best systems from the shared-task (that we recomputed for LVCs only). Results for SVM (F 1 =81) are comparable to the Majority baseline (F 1 =81) while FFN obtains the highest score (F 1 =83).

When we consider LVCs that were not seen in training data, results are much lower. The kNN baseline obtains an F 1 =0.22, while SHOMA obtains F 1 =0.31, as does our SVM, while results for FFN are slightly weaker. When predictions for both seen and unseen LVCs are taken together, FFN and SHOMA have comparable scores (F 1 =56), while the baselines (either Major-ity+kNN or kNN alone) is slightly higher. The best system overall is the SVM (F 1 =63).

Conclusion

In this paper, we considered the task of identifying LVCs in running text. We propose to use data-driven language-specific syntactic patterns for the extraction of LVC candidates out of syntactic parses, followed by a binary classification of the candidates into LVC or not.

We proposed a strong baseline combining different methods for LVC candidates depending on whether they were seen in the training set or not ("seen" meaning a LVC with same lemmas is annotated at least once in the training set). The baseline for seen cases tags a candidate as LVC if the training occurrences with same lemmas are more often tagged as LVC than not. The baseline for unseen cases uses the similarity of the predicative noun with the nouns of the training candidates, in a distributional semantic model. We also proposed supervised classifiers (a SVM and a feed-forward neural network) trained using internal and contextual morphosyntactic and semantic features, and working independently of the seen/unseen status.

Overall the SVM system is our best one, surpassing the best shared task system on LVCs (SHOMA, [START_REF] Taslimipoor | SHOMA at PARSEME Shared Task on automatic identification of VMWEs: Neural multiword expression tagging with high generalisation[END_REF]) by 7 percentage points. When evaluating performance separately on seen and unseen LVCs, the feed-forward network performs a little better on seen LVCs, but less well on unseen ones. It also appears that our results for seen LVCs surpass the best shared-task results even in the case of the baseline, in spite of a much simpler technique of supervised learning. For unseen LVCs, results are globally quite lower. The best performance is F 1 =31%, achieved both by the SHOMA system and our SVM. Our kNN-inspired baseline achieves F 1 =22% only, a performance that would rank second for unseen LVCs in the shared task.

Given the quality of predictions for seen LVCs, future works should focus on improving prediction for the unseen expressions. Such task could be achieved through an evaluation of different types of neural network. Other semanticallymotivated language-independent features should also be considered, so as to estimate the candidate noun's abstractness and predicativeness, as well as the level of semantic bleaching in the use of the verb. Finally, future works should investigate using a model for contextualized word embeddings such as BERT [START_REF] Devlin | BERT: pre-training of deep bidirectional transformers for language understanding[END_REF]), despite the difficulty of covering the 19 languages of the PARSEME datasets.

Table 1 :

 1 • F 2 : Fraction of true LVCs among all candi-Fraction of LVC annotations that were seen in train, and LVC candidate coverage (highest recall achievable, if all candidates are predicted as LVC) -evaluated on the development sets.

		BG DE EL ES EU FA FR HE HR HU IT PL PT RO SL TR µAvg
	%seen	60 26 50 48 86 61 68 45 29 75 71 66 74 90 57 44	62
	Coverage (seen)	98 100 100 95 94 91 99 82 93 86 90 97 95 96 100 98	95
	Coverage (unseen) 73 84 91 98 98 90 91 78 96 86 72 90 93 33 92 95	90

For instance,[START_REF] Nivre | Light verb constructions in universal dependencies[END_REF] report that for the majority of the 18 UD languages at that time, in a structure like X takes a photo of Y in English, X is attached to the verb, but the Y argument is attached to the noun. In some annotation schemes, the Y would be attached to the verb too. Note though that some treebanks do annotate the LVC status (e.g. in Hungarian). Additional semantic annotation of LVC can be found e.g. in propbank(Bonial and Palmer,

2016).

Unless otherwise stated, this paper refers to the 1.1 edition of the PARSEME shared task, and to the category LVC.full (as opposed to LVC.cause).

Note that the majority of LVCs has two components only, but some do contain additional components, such as prepositions when they are required to connect the verb and the noun.

This alternative pattern would cover the expression make adjustment in make an effort and an adjustment, for which two occurrences of LVCs would be annotated according the PARSEME guidelines.

For the training data, we take the union of gold LVCs and LVC candidates identified through syntactic patterns, since the patterns do not cover all gold LVCs.

We chose to use micro-average, since the test sets across languages don't have the same number of sentences, for reasons that are independent of the linguistic properties of each.

We used the predicted test sets of all participating systems (made available by the shared task organizers at https://gitlab.com/parseme/sharedtask-data/ tree/master/1.1/system-results), filtering them to consider LVCs only. The best systems in open and closed tracks (SHOMA and TRAVERSAL) are the same when considering all verbal MWEs or LVCs only.

Acknowledgements

We would like to thank Bruno Guillaume for advice in using Grew. This work was partially funded by the French National Research Agency (PARSEME-FR ANR-14-CERA-0001).


Hazem Al Saied
email: halsaied@atilf.fr

Marie Candito
email: marie.candito@gmail.com

Mathieu Constant
email: mathieu.constant@univ-lorraine.fr

Comparing linear and neural models for competitive MWE identification

In this paper, we compare the use of linear versus neural classifiers in a greedy transition system for MWE identification. Both our linear and neural models achieve a new state-of-the-art on the PARSEME 1.1 shared task data sets, comprising 20 languages. Surprisingly, our best model is a simple feed-forward network with one hidden layer, although more sophisticated (recurrent) architectures were tested. The feedback from this study is that tuning a SVM is rather straightforward, whereas tuning our neural system revealed more challenging. Given the number of languages and the variety of linguistic phenomena to handle for the MWE identification task, we have designed an accurate tuning procedure, and we show that hyperparameters are better selected by using a majority-vote within random search configurations rather than a simple best configuration selection. Although the performance is rather good (better than both the best shared task system and the average of the best perlanguage results), further work is needed to improve the generalization power, especially on unseen MWEs.

Introduction

Multi-word expressions (MWE) are composed of several words (more precisely of elements that are words in some contexts) that exhibit irregularities at the morphological, syntactic and/or semantic level. For instance, "prendre la porte" is a French verbal expression with semantic and morphological idiosyncrasy because (1) its idiomatic meaning ("to leave the room") differs from its literal meaning ("to take the door") and (2) the idiomatic reading would be lost if "la porte" were used in the plural. Identifying MWE is known to be challenging [START_REF] Constant | Multiword expression processing: A survey[END_REF], due to the highly lexical nature of the MWE status, the various degrees of the MWE irregularities and the various linguistic levels in which these show. In this paper we focus on the task of identifying verbal MWEs, which have been the focus of two recent shared tasks, accompanied by data sets for 20 languages: PARSEME shared task ST.0 [START_REF] Savary | The parseme shared task on automatic identification of verbal multiword expressions[END_REF] and ST.1 [START_REF] Ramisch | Edition 1.1 of the parseme shared task on automatic identification of verbal multiword expressions[END_REF]. Verbal MWEs are rather rare (one every 4 sentences overall in ST1.1 data sets) but being predicates, they are crucial to downstream semantic tasks. They are unfortunately even more difficult to identify than other categories of MWEs: they are more likely to be discontinuous sequences and to exhibit morphological and structural variation, if only the verb generally shows full inflectional variation, allows adverbial modification and in some cases syntactic reordering such as relativization.

Our starting point to address the MWE identification task is to reuse the system of Al [START_REF] Hazem | A transition-based verbal multiword expression analyzer[END_REF], an enhanced version of the winning system of ST.0, a transition system using a linear (SVM) model. Our objective has been to incorporate neural methods, which are overwhelming in current NLP systems. Neural networks have brought substantial performance improvements on a large variety of NLP tasks including transitionbased parsing (e.g. [START_REF] Kiperwasser | Simple and accurate dependency parsing using bidirectional lstm feature representations[END_REF] or [START_REF] Andor | Globally normalized transition-based neural networks[END_REF]), in particular thanks to the use of distributed representations of atomic labels, their ability to capture contextual information. Moreover, neural methods supposedly learn combinations from simple feature templates, as an alternative to hand-crafted task-specific feature engineering.

Yet, using neural methods for our task is challenging, the sizes of the available corpus are relatively modest (no ST.1 language has more than 5000 instances of training MWEs), albeit neural models generally have more parameters to learn than linear models. Indeed, the best systems at the shared tasks ST.0 and ST.1 [START_REF] Hazem | The ATILF-LLF system for parseme shared task: a transition-based verbal multiword expression tagger[END_REF][START_REF] Waszczuk | Traversal at parseme shared task 2018: Identification of verbal multiword expressions using a discriminative tree-structured model[END_REF] (in closed track) are not neural and surpassed some neural approaches.

In this paper, we carefully describe and compare the development and tuning of linear versus neural classifiers, to use in the transition system for MWE identification proposed in Al [START_REF] Hazem | A transition-based verbal multiword expression analyzer[END_REF], which itself built on the joint syntactic / MWE analyzer of [START_REF] Constant | A transition-based system for joint lexical and syntactic analysis[END_REF]. We set ourselves the constraints (i) of building systems that are robust across languages, hence using the same hyperparameter configuration for all languages and (ii) of using lemma and POS information but not syntactic parses provided in the PARSEME data sets, so that the resulting systems require limited preprocessing. We report a systematic work on designing and tuning linear and neural transition classifiers, including the use of resampling, vocabulary generalization and several strategies for the selection of the best hyperparameter configuration. We address both the open and closed tracks of the PARSEME ST.1, i.e with and without external resources (which in our case amount to pre-trained word embeddings).

The contributions of our work are:

• a new state-of-the art for the MWE identification task on the PARSEME ST1.1 data sets.

Our neural model obtains about a four-point error reduction on an artificial score mixing the best results for each language, and 4.5 points compared to the best participating system (even though we do not use syntactic parses);

• a report on which hyperparameters proved crucial to obtain good performance for the neural models, knowing that a basic feedforward network without class balancing showed high instability and achieves very poorly (average F-score between 15% and 30%);

• an alternative strategy for tuning the hyperparameters, based on trends in random search [START_REF] Bergstra | Random search for hyper-parameter optimization[END_REF];

• a fine-grained analysis of the results for various partitions of the MWE, shedding light on the necessity to address unknown MWE (not seen in train);

• a negative result concerning the basic semisupervised strategy of using pre-trained word embeddings.

We discuss the related work in Section 2, data sets in Section 3 and the transition system in Section 4. Linear and neural models are described in Sections 5 and 6, and the tuning methodology in Section 7. We present experiments and discuss results in Sections 8 and 9, and conclude in Section 10.

Related work

Supervised MWE identification has made significant progress in the last years thanks to the availability of new annotated resources [START_REF] Schneider | Semeval-2016 task 10: Detecting minimal semantic units and their meanings (dimsum)[END_REF][START_REF] Savary | The parseme shared task on automatic identification of verbal multiword expressions[END_REF][START_REF] Ramisch | Edition 1.1 of the parseme shared task on automatic identification of verbal multiword expressions[END_REF]. Sequence tagging methods have been largely used for MWE identification. In particular, first studies experimented IOB or IOB-like annotated corpora to train conditional random fields (CRF) models [START_REF] Blunsom | Multilingual deep lexical acquisition for hpsgs via supertagging[END_REF][START_REF] Constant | Mwuaware part-of-speech tagging with a crf model and lexical resources[END_REF][START_REF] Vincze | Multiword expressions and named entities in the Wiki50 corpus[END_REF] or other linear models [START_REF] Schneider | Discriminative lexical semantic segmentation with gaps: running the MWE gamut[END_REF].

Recently, [START_REF] Gharbieh | Deep learning models for multiword expression identification[END_REF] experimented on the DiMSUM data set various IOB-based MWE taggers relying on different deep learning models, namely multilayer perceptron, recurrent neural networks and convolutional networks. They showed that convolutional networks achieve better results. On the other hand, [START_REF] Taslimipoor | Shoma at parseme shared task on automatic identification of vmwes: Neural multiword expression tagging with high generalisation[END_REF] used pre-trained non-modifiable word embeddings, POS tags and other technical features to feed two convolutional layers with window sizes 2 and 3 in order to detect n-grams. The concatenation of the two layers is then passed to a Bi-LSTM layer. [START_REF] Legrand | Phrase representations for multiword expressions[END_REF] used a phrase representation concatenating word embeddings in a fixed-size window, combined with a linear layer in order to detect contiguous MWEs. They reach state-of-the-art results on the French Treebank [START_REF] Abeillé | Building a treebank for French[END_REF][START_REF] Seddah | Overview of the SPMRL 2013 shared task: A cross-framework evaluation of parsing morphologically rich languages[END_REF]. [START_REF] Omid Rohanian | Bridging the gap: Attending to discontinuity in identification of multiword expressions[END_REF] 2018) extends a sequential CRF to tree structures, provided that MWEs form connected syntactic components and that dependency parse trees are given as input. Dependency trees are used to generate a hypergraph of possible traversals and a binary classifier labels nodes as MWEs or not using local context information. A multi-class logistic regression is then used to determine the globally optimal traversal. This method has showed very competitive scores on the data sets of the PARSEME ST1.1, by ranking first overall on the closed track. By contrast, some authors have used Transition systems, introducing a greedy structured method that decomposes the MWE prediction problem into a sequence of local transition predictions. [START_REF] Constant | A transition-based system for joint lexical and syntactic analysis[END_REF] proposed a two-stack transition system to jointly perform MWE identification and syntactic parsing. Al Saied et al. (2017) experimented a partial implementation of this system for identifying and categorizing verbal MWEs. This system eliminates the syntactic aspects of Constant and Nivre (2016)'s system and learn a SVM model using linguistic and technical features to classify transitions. Relying on Al [START_REF] Hazem | The ATILF-LLF system for parseme shared task: a transition-based verbal multiword expression tagger[END_REF], [START_REF] Stodden | Trapacc and trapaccs at parseme shared task 2018: Neural transition tagging of verbal multiword expressions[END_REF] replaced the linear model with a convolutional module that transforms the sparse feature vectors into continuous ones and connect them to a dense layer.

Name

Cond. Action SHIFT

β = ∅ (σ, i|β, γ) ⇒ (σ|i, β, γ) REDUCE σ = ∅ (σ|i, β, γ) ⇒ (σ, β, γ) MERGE |σ| > 1 (σ|i, j, β, γ) ⇒ (σ|(i, j), β, γ) MARK σ = ∅ (σ|i, β, γ) ⇒ (σ|i, β, γ ∪ (i))
Figure 1: Set of transitions, each with its precondition.

Data sets

For our investigation, we focus on the data sets of the PARSEME Shared Task on verbal MWE identification edition 1.1 [START_REF] Ramisch | Edition 1.1 of the parseme shared task on automatic identification of verbal multiword expressions[END_REF], thereafter ST.1.

System description

Transition system A transition system incrementally builds the expected output structure by sequentially applying a transition to a configuration that encodes the state of the system, outputting a new configuration. It has been used in particular to build a syntactic tree for a given input sentence [START_REF] Nivre | Incrementality in deterministic dependency parsing[END_REF], and to build both the syntactic tree and the MWE list [START_REF] Constant | A transition-based system for joint lexical and syntactic analysis[END_REF]. We use such a system here to build the list of MWEs only.We reuse the transition system of Al Saied et al. ( 2018), simplified in that we do not predict the MWE types.

In this system, a configuration is a triplet c = (σ, β, γ), where β is a buffer of (remaining) tokens, σ is a stack of "elements", which are either single tokens or binary trees of tokens, and γ is the list of elements that have been identified as MWEs so far3 . To build the list of MWEs for a given input sentence w 1 , w 2 , ...w n , the system starts by the initial configuration

Trans Configuration = (σ, β, γ) Fi(s) ⇒ [ ], [Take, .., account], [ ] SHIFT ⇒ [Take], [the, .., account], [ ] SHIFT ⇒ [Take, the], [fact, .., account], [ ] REDUCE ⇒ [Take], [fact, .., account], [ ] SHIFT ⇒ [Take, fact], [that, .., account], [ ] ... SHIFT ⇒ [Take, give], [up, into, account], [ ] SHIFT ⇒ [Take, give, up], [into, account], [ ] MERGE ⇒ [Take, (give, up)], [into, account], [ ] MARK ⇒ [Take, (give, up)], [into, account] , [(give, up)] REDUCE ⇒ [Take], [into, account], [(give, up)] SHIFT ⇒ [Take, into], [account], [(give, up)] MERGE ⇒ [(Take, into)], [account], [(give, up)] SHIFT ⇒ [(Take, into), account], [ ], [(give, up)] MERGE ⇒ [((Take, into), account)], [ ], [(give, up)] MARK ⇒ [((Take, into), account)], [ ], (give, up), ((Take, into), account)] REDUCE ⇒ [ ], [ ], [(give, up), ((Take, into), account)]
(σ = [ ], β = [w 1 , ..., w n ], γ = [ ])
, and applies a sequence of transitions until a terminal configuration is reached, namely here when both the buffer and stack are empty. The transition set, and their precondition is described in Figure 1. Note the MERGE transition creates complex stack elements, by merging the top 2 elements of the stack4 .

The identification of a MWE made of m components t 1 , ..., t m necessitates m -1 MERGEs, and one final MARK. The REDUCE transition allows to manage discontinuities in MWEs. Note that MARK identifies S 0 as MWE, but does not remove it from the stack, hence enabling to identify some cases of embedded MWEs (we refer to Al Saied et al. ( 2018) for the precise expressive power). At prediction time, we use a greedy algorithm in which the highest-scoring applicable transition according to a classifier is applied to the current configuration. Figure 2 shows the application of the oracle transition sequence for a sentence with two MWEs.6

Learning algorithm and oracle To learn this

Linear model

In order to compare linear and neural models for MWE identification, we reused the best performing linear model of Al [START_REF] Hazem | A transition-based verbal multiword expression analyzer[END_REF], namely a SVM, in a one versus rest scheme with linear kernel and squared hinge loss. We used the feature templates of Al Saied et al. ( 2018) minus the syntactic features, since we focus on MWE identification independently of syntactic parsing. Table 2 displays the list of feature templates. We detail the "S 0 in MWT dictionary" and "MWE component dictionary" templates, the other features names being rather transparent: "S 0 in MWT dictionary" feature fires when S 0 lemma is a MWT at least once in train, and binary features fire when either S 0 , S 1 , B 0 , B 1 or B 2 belong to at least one train multi-token MWE.

We ran some preliminary experiments which led us to set some hyperparameters once and for all (first four lines of Table 2). In particular, we ended up not using resampling to balance the class distribution, because it proved quite detrimental for the linear model, contrary to the neural models. We then performed tuning for all the other features (cf. section 7).

MLP model

Though we investigated various neural architectures 7 , the "baseline" multi-layer perceptron (hereafter MLP) proved to be the best in the end. It is a plain feed-forward network, with an embedding layer concatenating the embeddings for the POS of S 0 , S 1 , B 0 and B 1 and either their word form or lemma (hyperparameter), fully connected to a dense layer with ReLU activation, in turn connected to the output layer with softmax activation.

Table 3 provides the exhaustive list of MLP hyperparameters, along with their possible values and their optimal values for the most performing 7 We tried in particular (1) a MLP with several hidden layers; (2) a MLP fed with a bidirectional recurrent layer to represent the sequence of elements S0S1B0; (3) We also built a model inspired by [START_REF] Kiperwasser | Simple and accurate dependency parsing using bidirectional lstm feature representations[END_REF] in which the recurrent (LSTM) embeddings of certain focus elements (S0, S1, B0 and B1) are dynamically concatenated and fed into a MLP, with back-propagation for a whole sentence instead of for each transition. The recurrent representations of the focus elements are supposed to encode the relevant contextual information of these elements in the sentence. These models suffered from either a non-competitive performance or a very unstable loss (36.7 for the bidirectional MLP and 8.4 for kiperwasser on test data sets of ST1.1).

configurations. Lines 1 to 9 correspond to embedding and initialization hyperparameters: Lines (1, 2) concern which elements to include as additional input (Use B 2 , Use B -1 )8 , (3) which form for input tokens (Lemmatization), (4, 5) which size for token and POS tag embeddings (Token and POS dimensions), (6) whether the embeddings are initialized randomly or pre-trained (pre-trained), (7) whether the embeddings are Trainable or not, and (8) how to generate embedding vectors for stack elements: as the average of tree token embeddings or as their sum (Averaging).

Vocabulary For the neural model, when S i or B i are missing, a special dummy word is used instead. Moreover, we investigated an aggressive reduction of the known vocabulary. We compared 2 strategies to define it: in exhaustive vocabulary mode, hapaxes are replaced at training time by a UNK symbol, with probability 0.5. In compact vocabulary mode, any token (or complex element) whose lemma is never a component of a MWE in the training set is replaced by UNK. Note that in both modes, the used vocabulary contains the concatenated symbols in case of complex S i elements.

Resampling Given that tokens are mostly not part of a VMWE, the transitions for their identification are very rare, leading to a very skewed class distribution. 9 Resampling techniques aiming at balancing class distribution are known to be efficient in such a case [START_REF] Nitesh | Data mining for imbalanced datasets: An overview[END_REF]. Moreover, preliminary experiments without resampling showed unstable loss and rather low performance. We thus used in subsequent experiments a hybrid resampling method composed of (1) under sampling, that removes training sentences non containing any MWE, and (2) random oversampling, that forces a uniform distribution of the classes by randomly duplicating minority class instances (all but SHIFT) [START_REF] Nitesh | Data mining for imbalanced datasets: An overview[END_REF]. Preliminary experiments showed that without these strategies, the systems suffered from very unstable loss and low performance, which led us to systematically use these two strategies in the subsequent experiments. Tuning explored two supplementary resampling techniques: "focused" oversampling which aims at mimicking a minimum number of occurrences for all MWEs. When set, training instances with MERGE and MARK transitions are duplicated for each training MWE below a frequency threshold. "Over loss" hyperparameter penalizes the model when it fails to predict MERGE and MARK, by multiplying the loss by a coefficient (see Table 3).

Tuning methodology

The tuning phase served us to choose a hyperparameter configuration for the linear model and the neural model, in closed and open track. In our case, we experimented open track for the neural model only, by using pre-trained embeddings instead of random initialization. We thus consider three cases: closed track linear, closed track MLP and open track MLP.

For each of these three cases, in order to enforce the robustness across languages of the selected hyperparameters, we aimed at selecting the same hyperparameter configuration for all the languages.

Yet, to reduce the tuning time, we have chosen to work on three pilot languages, from three different language families. But because the various training sets have various sizes, we tried to neutralize this variation by using training sets of average size. This led us to choose three languages (Bulgarian, Portuguese and Turkish) among ST.1 languages having training sets bigger than average and to tune the hyperparameters using training sets truncated to that average size (270k tokens) and evaluating on dev sets.

Multilingual metric: the official metric for the PARSEME shared task is the macro average of the F-scores over all languages (hereafter F AV G ). Yet we noted that although macro-averaging precision and recall is appropriate because the number of dev and test MWEs is almost the same for all languages, averaging the F-scores of all languages sometimes substantially differs (e.g. by 2 points) from taking the F-score of the macro-averaged precision and the macro-averaged recall (hereafter F G ). We thus use F AV G for comparability with the shared task results, but also report the F G score, and use the latter during tuning.

Random search: To tune the hyperparameters on the three pilot languages, we used random search, which proved to be more efficient than grid search when using the same computational budget, because it allows to search larger ranges of values [START_REF] Bergstra | Random search for hyper-parameter optimization[END_REF]. We thus run about 1000 trials for SVM, closed track MLP and open track MLP. For the SVM, random search used a uniform distribution for the hyperparameters, which are all boolean. For the MLP, the random hyperparameter values are generated from either a set of discrete values using a uniform distribution or from a range of continuous values using logarithmic distribution. For the MLP, each resulting random hyperparameter configuration was run on each pilot language twice, using always the same two seeds 0 and 1 10 . We then averaged the precision and recall on the dev sets, for the three languages and the two seeds (i.e. use the global F-score F G ).

Selecting hyperparameter configurations:

Random hyperparameter search for the three pilot languages led us to use two strategies to select the hyperparameter sets. The first one is simply to select the best performing hyperparameter sets (shown in column BoR in Table 2 for the linear model, and in the BoR c and BoR o columns in Table 3). Yet, we noted that some hyperparameters varied a lot among the top performing systems. We thus investigated to build a "trend-based" configuration, by selecting each hyperparameter value according to the observed trend among the top k best configurations (with k=500/250 for MLP/SVM) 11 . This results in two sets for the linear model (best-of-random and trend-based, in closed mode) and four configurations for the MLP: best-of-random or trend-based, in closed or open mode.

We then trained these six configurations on the full-size training sets for all ST.1.1 languages, using two seeds (0 and 1), and retaining the best seed for each language. For the MLP case, the global F-scores on dev sets are provided in the last row of Table 3. Interestingly, the trend-based configuration largely beats the best-of-random configurations, both in closed and open tracks 12 . This shows that choosing hyperparameter values independently of each other is compensated by choosing more robust values, by using the top k best performing systems instead of one.

Note that for the linear case, the trend-based configuration does not surpass the best performing random search configuration (the last line of 10 Preliminary experiments showed a relative stability when changing seeds, hence we used only two seeds in the end. Changing seeds was useless for the linear model which is more stable. 11 We chose the values using an approximate majority vote, using a graphical visualization of the hyperparameter values in the top k best performing systems.

12 Moreover, the best-of-random open configuration showed instability when switching from the three pilot languages to all languages, leading to a null score for Hindi (hence the rather low global F-score of 57.8).

Language

Closed FG is the global F-score (see Section 7). In the FAV G and FG lines, the best ST.1 per-language scores are used, whereas the last line concerns the FG score of the best ST.1 systems [START_REF] Waszczuk | Traversal at parseme shared task 2018: Identification of verbal multiword expressions using a discriminative tree-structured model[END_REF][START_REF] Taslimipoor | Shoma at parseme shared task on automatic identification of vmwes: Neural multiword expression tagging with high generalisation[END_REF].

Table 2). This asymmetry could mean that the number of random trials is sufficient for the linear case, but not for the neural models, and that a trend-based strategy is advantageous within a limited computational budget.

Experiments and results

Table 4 provides identification scores on test sets, for our tuned SVM and MLP models for each ST.1 language, along with the best score of ST.1 for each language, in open and closed tracks. It also displays overall scores using both the official ST.1 metrics (F AV G ) and the more precise F G score introduced in section 7. This F G score for the ST.1 results is computed in two modes: in line F G , the ST.1 columns correspond to artificially averaging the best result of each language (in closed / open tracks), whereas "F G best sys" is the score of the best system of ST.1. The differences between SVM and MLP c results are significant 13 for all languages except EU, HU, LT and PL.

For both F AV G and F G metrics, results show that MLP models significantly outperforms all other systems both in the closed and open tracks. In the closed track, MLP surpasses SVM by 1.8 points, the best ST.1 systems per language by 4.8 points, and the best ST.1 system [START_REF] Waszczuk | Traversal at parseme shared task 2018: Identification of verbal multiword expressions using a discriminative tree-structured model[END_REF] by 8.6 points. In the open track, MLP beats the best ST.1 system [START_REF] Taslimipoor | Shoma at parseme shared task on automatic identification of vmwes: Neural multiword expression tagging with high generalisation[END_REF] by 4.2 points, and the best ST.1 systems per language by 3.6 points14 .

In the closed track, MLP ranks first for 11 languages, while the SVM model and the best ST.1 systems per language reach the first position respectively for three and five languages. In open track, MLP achieves the highest scores for 13 languages while ST.1 systems beat it for six languages. These results tend to validate the robustness of our approach across languages. Regarding language families, MLP reports remarkable gains for Slavic languages and languages from the other family, but achieve lower performance on Indo-Iranian languages when compared with best ST.1 results. For Romance languages, our models surpass the ST.1 best results (except for RO), and the SVM model is globally better than the MLP.

Comparing the results of the open and the closed track, we can observe that the use of pretrained word embeddings has no significant impact on the MLP results. This might mean that static embeddings are not well-suited for representing tokens both when used literally and within MWE. This tendency would deserve more investigation using other word embedding types, in particular contextualized ones [START_REF] Devlin | BERT: pre-training of deep bidirectional transformers for language understanding[END_REF].

Discussion

Performance analysis In order to better understand the strengths and weaknesses of the various systems, we provide in Table 5 an in-depth performance analysis of our models, on dev sets, broken-down by various classifications of MWEs, namely (1) whether a dev MWE was seen in train (and if so, more than 5 times or not) or unseen;

(2) whether the MWE is continuous or has gaps; and (3) according to the MWE length. The table provides the proportions of each subclass within the gold dev set and within the predictions of each model (% columns), in addition to the average precision and recall over all languages, and the global F G score, for each model. Overall, neural models (in closed and open tracks) tends to get better recall than the SVM model (56 and 57, versus 49) but lower precision (70 versus 86), which is coherent with the use of embeddings.

Generalization power Without surprise, the global F-score on seen MWEs is high for all our systems (> 80), and it is still above 75 for MWEs with frequency ≤ 5. Yet this masks that the neural models have comparable precision and recall on seen MWEs, whereas the SVM has better precision than recall. Now when turning to the unseen category, we can observe that all systems get very low performance.

In comparison with MLP models, the most important advantage of SVM is its (little) ability to generalize (F G = 12 on unseen MWEs), whereas the MLPs have none at all. Note that frequency ≤ 5 is sufficient for the MLP models to surpass the linear model. For comparison, the average F-scores on test sets of the PARSEME ST.1 for unseen MWEs range from 0 to almost 20. This very low generalization of our MLP models is understandable since tuning led us to favor the compact vocabulary mode, which agressively reduces the known vocabulary to seen MWE components. Yet our best result on unseen MWEs with a MLP with exhaustive vocabulary mode only achieves F G = 4 on unseen MWEs.

It appears that for all models, more than 90% of the unindentified MWEs (the silence) are either unseen or with frequency ≤ 5, which clearly shows that the frequency of a MWE in train set is the crucial trait for identification. Further analysis is needed to study the performance according to the literal versus MWE ambiguity rate.

Continuous/discontinuous

MWEs MLP models show better performances for discontinuous MWEs than SVM, whereas they reach comparable scores for continuous MWEs. In particular, they display a 5-point gain in F-score, due to a 6-point gain in recall on discontinuous MWEs.

MWE length

The three systems display comparable scores regarding MWE length. Results validate the intuition that the shorter the MWE, the easier it is to identify.

Conclusion

We described and compared the development of linear versus neural classifiers to use in a transition system for MWE identification [START_REF] Hazem | A transition-based verbal multiword expression analyzer[END_REF]. Surprisingly, our best neural architecture is a simple feed-forward network with one hidden layer, although more sophisticated architectures were tested. We achieve a new state-of-the art on the PARSEME 1.1 shared task data sets, comprising 20 languages. Our neural and linear models surpass both the best shared task system [START_REF] Waszczuk | Traversal at parseme shared task 2018: Identification of verbal multiword expressions using a discriminative tree-structured model[END_REF] and the artificial average of the best per-language results. Given the number of languages and the variety of linguistic phenomena to handle, we designed a precise tuning methodology. Our feedback is that the development of the linear (SVM) system was pretty straightforward, with low variance between the configurations. For the neural models on the contrary, preliminary runs led to low and unstable performance. Class balancing proved crucial, and our proposal to select hyperparameter values using majority vote on the top k best performing systems in random search also proved beneficial.

Although our systems are competitive, their generalization power reveals disappointing: performance on unseen MWEs is very low for the linear model (F-score=12) and almost zero for the neural models (whereas the shared task results range from 0 to 20 for unseen MWEs). Basic semi-supervised experiments, consisting in using pre-trained word embeddings, did not bring any improvement. Static embeddings might not be suitable representations of MWE components, as their behavior differs when used literally or within a MWE. This definitely calls for future work that can incorporate information on semantic irregularity.

Figure 2 :

 2 Figure 2: Application of the oracle transition sequence for the sentence Take the fact that I didn't give up into account, containing two verbal MWEs: Take into account and give up.

  Table 1 provides statistics on this data set, which includes 20 languages 1 covering a wide range of families and corpus sizes. All languages come with train and test sets, and all but EN, HI and LT have a development set. They contain tokenized sentences in which MWEs are annotated.

Each token comes with its word and lemma forms and its part of speech (POS) tag. ST.1 also has extra linguistic annotations such as morphological features and syntactic dependency trees, but we do not use them for the purpose of the paper. One MWE instance is either a set of several potentially non-continuous tokens, or a single token compounding multiple words (namely a multiword token, hereafter MWT). 2 Data sets also contain rare MWEs embedded in another one, and overlapping MWEs.

Table 2 :

 2 Linear model feature hyperparameters. First column: prelim if the hyperparameter was fixed once and for all given preliminary tests vs. Rdm Sch for tuning via random search (see Section 7). Best of random BoR column: whether the template is activated (+) or not (-) in the best performing hyperparameter set of the random search. Trend-based TB: same but for the trend-based hyperparameter set (cf. section 7). The last line provides the corresponding global Fscores on dev sets of the three pilot languages (BG, PT and TR).

transition classifier, we use the static deterministic oracle of Al

[START_REF] Hazem | A transition-based verbal multiword expression analyzer[END_REF]

. For any input sentence and list of gold MWEs, the oracle defines a unique sequence of transitions, providing example pairs (config / next transition to apply). Transitions have a priority order (MARK > MERGE > REDUCE > SHIFT), and the oracle greedily applies the highest-priority transition that is compatible with the gold analysis. MERGE is gold-compatible whenever S 0 and S 1 are part of the same gold MWE.

5

For REDUCE to be gold-compatible, S 0 must not be strictly included in a gold MWE. Moreover, either S 0 is not a gold MWE, or it is already marked as MWE.

Table 3 :

 3 MLP hyperparameters and their possible values ("range or set" column). Best-of-random closed (BoRc) and Bestof-random open (BoRo) columns: hyperparameter values in best configurations according to random search on the three pilot languages, in closed and open tracks. Last column: Trend-based (TB) configuration (see text in section 7). Last line: global F-scores for these configurations, calculated using the average precision and recall for all ST.1 languages. The models are fit on truncated training sets of the three pilot languages (BG, PT and TR) (cf. section 7).

	Type	Hyperparameter	Range or set	BoRc	BoRo	TB
	Embedding and initialisation	Use B2 Use B-1 Lemmatization Token dimension POS dimension Pre-trained Trainable Averaging Vocabulary	{True, False} {True, False} {True, False} [100, 600] [15, 150] {True, False} {True, False} {True, False} {Compact, Exhaustive}	True True True 157 147 False True False True	True False True 300 132 True True True False	True True True 300 35 True | False True True True
	Dense	Unit number Dropout	[25, 600] {.1, .2,.. .6}	85 0.3	56 0.1	75 0.4
	Sampling	Focused / Frequency threshold Over loss / Loss coefficient	{True, False} / {5, 10,.. 30} False / -{True, False} / [1, 40] False / 1 False / 1 False / 1 False / -False / -
	Train	Learning rate Batch size	[.01, .2] {16, 32, 48, 64, 128}	0.017 128	0.095 16	0.03 48
		FG on all languages (on dev sets if available or 20% of train)	61.2	57.8	63.5 | 64.3

Table 4 :

 4 MWE-based F-scores for ST.1 languages on test sets using our tuned SVM and MLP models, fit on train and dev sets when available. ST.1 stands for the most performing scores of the shared task for each language in closed and open tracks. All ST.1 systems fit training and development sets except the system that produced the best score of BG on closed track. Languages are grouped according to their linguistic families (Slavic, Germanic, Romance, Indo-Iranian and other).) FAV G is the official metric (average F-scores).

			track		Open track
		SVM MLPc ST.1	MLPo ST.1
	BG	63.3	66.8	62.5	67.7	65.6
	HR	55.4	59.3	55.3	59.0	47.8
	LT	38.0	45.7	32.2	45.3	22.9
	PL	69.4	71.8	67.0	72.2	63.6
	SL	53.5	62.7	64.3	61.2	52.3
	DE	49.5	51.5	45.3	49.9	45.5
	EN	28.4	31.4	32.9	31.9	33.3
	ES	39.2	40.0	34.0	39.7	38.4
	FR	61.1	59.0	56.2	58.8	60.9
	IT	55.7	55.0	49.2	56.5	45.4
	PT	68.9	67.8	62.1	70.4	68.2
	RO	80.9	83.5	85.3	82.0	87.2
	HI	66.8	64.9	73.0	64.9	72.7
	FA	75.4	70.6	77.8	70.6	78.4
	EL	57.8	62.2	49.8	61.4	58.0
	EU	80.7	82.1	75.8	80.2	77.0
	HE	43.3	45.2	23.3	47.3	38.9
	HU	91.7	92.4	90.3	92.6	85.8
	TR	47.5	52.1	45.2	47.9	58.7
	FAV G	59.3	61.3	56.9	61.0	57.9
	FG	60.8	62.6	57.8	62.3	58.7
	FG best sys			54.0		58.1

We used all languages but Arabic due to licence issues.

MWTs are extremely marginal for all ST.1 languages except German (30%) and Hungarian(75%)

In all the following, we use σ|i to denote a stack with top element i and remainder σ, and i|β for a buffer with first token i followed by the elements in β. Si and Bi denote the ith element of the stack and buffer respectively, starting at 0.

Hence Si elements are either single tokens or binary trees of tokens. In the latter case, their linguistic attributes (lemma, POS, word form) are obtained by simple concatenation over their components.

Note that this order will lead to left-branching binary trees for elements in the stack.

The system is implemented in Python 2.7, using Keras and Scikit-learn libraries. The code is available at https://github.com/hazemalsaied/MWE. Identification/releases/tag/v.1 under MIT licence.

B-1 is the last reduced element (its right-most token if it is a complex stack element).

For all ST.1 languages, the transitions in training sets are approximately distributed as follows: 49% for SHIFT, 47% for REDUCE, 3% for MERGE and 1% for MARK.

It is worth noting that the model of[START_REF] Omid Rohanian | Bridging the gap: Attending to discontinuity in identification of multiword expressions[END_REF], published while writing this paper, outperforms our scores for the languages they use for evaluating their model (EN:41.9, DE:59.3, FR:71.0, FA:80.0) on the ST.1 test sets. However, this model exploits syntactic information (See Section 2).

Acknowledgement

This work was partially funded by the French National Research Agency (PARSEME-FR ANR-14-CERA-0001).

), we report for each subclass: the proportion of the subclass in the system prediction, the global F-score (FG), Precision (P) and Recall (R).