Mathieu Dehouck
email: mathieu.dehouck@inria.fr

Pascal Denis
email: pascal.denis@inria.fr

Joakim Nivre

Mitchell Abrams

Željko Agić

Lars Ahrenberg

Lene Antonsen

Maria Jesus Aranzabe

Gashaw Arutie

Masayuki Asahara

Luma Ateyah

Mohammed Attia

Aitziber Atutxa

Liesbeth Augustinus

Elena Badmaeva

Miguel Balles- Teros

Esha Banerjee

Sebastian Bank

Verginica Barbu Mititelu

John Bauer

Sandra Bellato

Kepa Bengoetxea

Riyaz Ahmad Bhat

Erica Biagetti

Eckhard Bick

Rogier Blokland

Victoria Bobicev

Carl Börstell

Cristina Bosco

Gosse Bouma

Sam Bowman

Adriane Boyd

Aljoscha Burchardt

Marie Candito

Bernard Caron

Gülşen Gauthier Caron

Cebiroğlu Eryiğit

Giuseppe G A Celano

Savas Cetin

Fabricio Chalub

Jinho Choi

Yongseok Cho

Jayeol Chun

Silvie Cinková

Aurélie Collomb

Çağrı Çöltekin

Miriam Connor

Marine Courtin

Elizabeth Davidson

Marie-Catherine De Marn- Effe

Valeria De Paiva

Arantza Diaz De Ilarraza

Carly Dickerson

Peter Dirix

Kaja Dobrovoljc

Timothy Dozat

Kira Droganova

Puneet Dwivedi

Marhaba Eli

Ali Elkahky

Binyam Ephrem

Tomaž Erjavec

Aline Etienne

Richárd Farkas

Hector Fernandez Alcalde

Jennifer Foster

Cláudia Freitas

Katarína Gajdošová

Daniel Galbraith

Marcos Garcia

Moa Gärdenfors

Kim Gerdes

Filip Gin- Ter

Iakes Goenaga

Koldo Gojenola

Memduh Gökırmak

Yoav Goldberg

Xavier Gómez

Berta Gonzáles Saavedra

Matias Grioni

Normunds Grūzītis

Bruno Guillaume

Céline Guillot-Barbance

Nizar Habash

Jan Hajič

Linh Hà My

Na-Rae Han

Kim Harris

Dag Haug

Barbora Hladká

Jaroslava Hlaváčová

Florinel Hociung

Petter Hohle

Jena Hwang

Radu Ion

Elena Irimia

Tomáš Jelínek

Anders Johannsen

Fredrik Jørgensen

Hüner Kaşıkara

Sylvain Kahane

Hiroshi Kanayama

Jenna Kanerva

Tolga Kayade- Len

Václava Kettnerová

Jesse Kirchner

Natalia Kotsyba

Simon Krek

Sookyoung Kwak

Veronika Laippala

Lorenzo Lambertino

Tatiana Lando

Septina Dian Larasati

Alexei Lavrentiev

Phương John Lee

Lê Hôǹg

Alessandro Lenci

Saran Lertpradit

Cheuk Herman Leung

Ying Li

Josie Li

Keying Li

Kyungtae Lim

Nikola Ljubešić

Olga Loginova

Olga Lyashevskaya

Teresa Lynn

Vivien Macketanz

Aibek Makazhanov

Michael Mandl

Christopher Manning

Ruli Manurung

Cătălina Mărănduc

David Mareček

Héctor Katrin Marheinecke

Martínez Alonso

André Martins

Jan Mašek

Yuji Matsumoto

Ryan Mcdonald

Gustavo Mendonça

Niko Miekka

Anna Missilä

Cătălin Mititelu

Yusuke Miyao

Simonetta Montemagni

Amir More

Laura Moreno Romero

Shinsuke Mori

Bjartur Mortensen

Bohdan Moskalevskyi

Kadri Muischnek

Yugo Murawaki

Kaili Müürisep

Pinkey Nainwani

Juan Ignacio

Navarro Horñiacek

Anna Nedoluzhko

Gunta Nešpore-Bērzkalne

Nguyêñ Lương

Huyêǹ Thị

Thị Nguyêñ

Vitaly Minh

Rattima Nikolaev

Hanna Nitisaroj

Stina Nurmi

Adédayọ̀Olúòkun Mai Ojala

Petya Omura

Robert Osenova

Lilja Östling

Niko Øvrelid

Elena Partanen

Marco Pascual

Agnieszka Passarotti

Siyao Patejuk

Cenel-Augusto Peng

Guy Perez

Slav Perrier

Jussi Petrov

Emily Piitulainen

Barbara Pitler

Thierry Plank

Martin Poibeau

Lauma Popel

Sophie Pretkalniņa

Prokopis Prévost

Adam Prokopidis

Tiina Przepiórkowski

Sampo Puolakainen

Andriela Pyysalo

Alexandre Rääbis

Loganathan Rademaker

Taraka Ramasamy

Carlos Rama

Vinit Ramisch

Livy Ravishankar

Siva Real

Georg Reddy

Michael Rehm

Larissa Rießler

Laura Rinaldi

Luisa Rituma

Mykhailo Rocha

Rudolf Romanenko

Da- Vide Rosa

Valentin Rovati

Olga Roșca

Shoval Rudina

Shadi Sadde

Tanja Saleh

Stephanie Samardžić

Manuela Samson

Baiba Sanguinetti

Yanin Saulīte

Nathan Sawanakunanon

Sebas- Tian Schneider

Djamé Schuster

Wolfgang Seddah

Mojgan Seeker

Mo Seraji

Atsuko Shen

Muh Shimada

Dmitry Shohibussirri

Natalia Sichinava

Maria Silveira

Radu Simi

Katalin Simionescu

Mária Simkó

Kiril Šimková

Aaron Simov

Isabela Smith

Antonio Soares-Bastos

Milan Stella

Jana Straka

Alane Strnadová

Umut Suhr

Zsolt Sulubacak

Dima Szántó

Yuta Taji

Takaaki Takahashi

Isabelle Tanaka

Trond Tellier

Anna Trosterud

Reut Trukhina

Francis Tsarfaty

Sumire Tyers

Zdeňka Uematsu

Larraitz Urešová

Hans Uria

Sowmya Uszkoreit

Daniel Vajjala

Gertjan Van Niekerk

Viktor Van Noord

Veronika Varga

Lars Vincze

Jonathan Wallin

Washington North

Mats Wirén Seyi Williams

Tsegay Woldemariam

Tak-Sum Wong

Chunxiao Yan

Marat M Yavrumyan

Zhuoran Yu

Zdeněk Žabokrtský

Amir Zeldes

Daniel Zeman

Manying Zhang

Hanzhi 2018 2 Zhu

Phylogenetic Multi-Lingual Dependency Parsing

Languages evolve and diverge over time. Their evolutionary history is often depicted in the shape of a phylogenetic tree. Assuming parsing models are representations of their languages grammars, their evolution should follow a structure similar to that of the phylogenetic tree. In this paper, drawing inspiration from multi-task learning, we make use of the phylogenetic tree to guide the learning of multi-lingual dependency parsers leveraging languages structural similarities. Experiments on data from the Universal Dependency project show that phylogenetic training is beneficial to low resourced languages and to well furnished languages families.

Introduction

Languages change and evolve over time. A community that spoke once a single language can be split geographically or politically, and if the separation is long enough their language will diverge in direction different enough so that at some point they might not be intelligible to each other. The most striking differences between related languages are often of lexical and phonological order but grammars also change over time.

Those divergent histories are often depicted in the shape of a tree in which related languages whose common history stopped earlier branch off higher than languages that have shared a longer common trajectory [START_REF] Jäger | Support for linguistic macrofamilies from weighted sequence alignment[END_REF]. We hypothesize that building on this shared history is beneficial when learning dependency parsing models. We thus propose to use the phylogenetic structure to guide the training of multi-lingual graph-based neural dependency parsers that will tie parameters between languages according to their common history.

As our phylogenetic learning induces parsing models for every inner node in the phylogenetic tree, it can also perform zero-shot dependency parsing of unseen languages. Indeed, one can use the model of the lowest ancestor (in the tree) of a new language as an approximation of that language grammar.

We assess the potential of phylogenetic training with experiments on data from the Universal Dependencies project version 2.2. Our results show that parsers indeed benefit from this multi-lingual training regime as models trained with the phylogenetic tree outperform independently learned models. The results on zero-shot parsing show that a number of factors such as the genre of the data and the writing system have a significant impact on the quality of the analysis of an unseen language, with morphological analysis being of great help.

The remaining of this paper is organized as follows. Section 2 presents both the neural parsing model as well as the phylogenetic training procedure. Section 3 presents some experiments over data from UD 2.2. Section 4 presents some related works on multi-task learning and multilingual parsing. Finally, Section 5 closes the paper and gives some future perspectives.

Model

We propose a multi-task learning framework that shares information between tasks using a tree structure. The tree structure allows us to both share model parameters and training samples between related tasks. We instantiate it with a graph-based neural parser and use the language phylogenetic tree to guide the learning process, but it can in principle be used with any tree that encodes tasks relateness and any learning algorithm that supports fine-tuning.

In this section we first describe the intuition be- hind phylogenetic training, then the neural parser and then the phylogenetic training itself.

Phylogenetic Hypothesis

Languages evolve from earlier stages and sometimes a language will change differently in different places leading to different languages with a common ancestor. This evolution process is often depicted in the shape of a tree in which leaves are actual languages and inner nodes can be either attested ancestral languages or their idealized reconstruction. Figure 1 gives an example of such a tree for a subset of the Slavic family of Indo-European languages (Simons and Fennig, 2018).

Just as languages evolve and diverge, so do their grammars. Assuming a parsing model is a parameterized representation of a grammar, then we can expect those models to evolve in a similar way. We thus take a multi-task approach to the problem of multi-lingual dependency parsing. What was once a single problem (e.g. parsing sentences in Proto-West-Slavic) becomes a set of distinct but related problems (parsing sentences in Czech, Polish, Slovak and Sorbian) as Proto-West-Slavic was evolving into its modern descendants.

We assume that the grammar of the last common ancestor is a good approximation of those languages grammars. Thus it should be easier to learn a language's grammar starting from its ancestor grammar than from scratch. There are however some issues with this assumption. First, a language grammar can be very different from its ancestor one from two millennia earlier. Consider the difference between modern French and early Classical Latin for example, in two millennia Latin has wit-nessed the loss of its case system and a complete refoundation of its verbal system. And the "last common" ancestors can have very different age depending on the languages we consider. We expect the common ancestor of Tagalog and Indonesian to be much much older than the common ancestor of Portuguese and Galician. Second, a lot of languages have only started to be recorded very recently thus lacking historical data all together. And when historical records are available, much work still needs to be done to render those data usable by parsers. For example the Universal Dependencies Project [START_REF] Stymne | Parser training with heterogeneous treebanks[END_REF] only has annotated corpora for Latin, old Greek, old Church Slavonic and Sanskrit. And even for those classical languages, it is not clear to which extent their modern counterparts really descend from them. Thus we need to find another way to access the ancestor language grammar than using historical data.

We propose to use all the data from descendent languages to represent an ancestor language. In principle, one could give more weight to older languages or to languages that are known to be more conservative, but this knowledge is not available for all languages families. Thus we resort to using all the available data from descendent languages without distinction.

Another problem is that the tree view is too simple to represent the complete range of phenomena involved in language evolution, such as language contacts. Furthermore, languages do not evolve completely randomly, but follow some linguistic universals and have to keep a balance between speakability, learnability and understandability. Thus, languages can share grammatical features without necessarily being genetically related, either by contact or by mere chance. However, the tree model is still a good starting point in practice and language families align well with grammatical similarity as recent works on typological analysis of UD treebanks have shown [START_REF] Chen | Classifying languages by dependency structure. typologies of delexicalized universal dependency treebanks[END_REF][START_REF] Schluter | Empirically sampling universal dependencies[END_REF]. We thus make the simplifying assumption that a language grammar evolves only from an older stage and can be approximated by that previous stage.

Neural Model

Our scoring model is an edge factored graph-based neural model in the vein of recent works by Dozat et al. [START_REF] Dozat | Stanford's graph-based neural dependency parser at the conll 2017 shared task[END_REF]. There are two major differences here compared to the parser of Dozat Let x = (w 1 w 2 ...w l ) be a sentence of length l. Each word w i is represented as the concatenation of 3 subvectors, one for its part-of-speech, one for its morphological attributes and one for its form:

• • • ⊕ • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • w o
w i = pos i ⊕ morph i ⊕ char i .
The part-of-speech vector (pos i ) is from a look up table. The morphological vector (morph i ) is the sum of the representation m m of each morphological attribute m of the word given by the treebanks:

morph i = m∈morph i m m .
We add a special dummy attribute representing the absence of morphological attributes. The form vector (char i ) is computed by a character BiLSTM [START_REF] Hochreiter | Long short-term memory[END_REF]. Characters are fed one by one to the recurrent neural network in each direction. The actual form vector is then the concatenation of the outputs of the forward character LSTM and of the backward character LSTM as depicted in Figure 2. Once, each word has been given a representation in isolation, those representations are passed to two other BiLSTMs. Each word is then represented as the concatenation of its contextualised vector from the forward and backward layers:

• • • • • • • • • • • ⊕ • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • <ROOT>
c i = f orward(w 1 , ..., w i )⊕backward(w i , ..., w l ).
We actually train two different BiLSTMs, one representing words as dependents (c) and one words as governors (ĉ). An edge score is then computed as follows. Its governor word vector ĉi and its dependent word vector c j are concatenated and fed to a two layer perceptron (whose weights are L 1 and L 2 ) with a rectifier (noted [...] + ) after the first layer in order to compute the score s ijl of the edge for every possible relation label l:

s ij = max l s ijl = max l (L 2 • [L 1 • (ĉ i ⊕ c j )] + ) l .
All the neural model parameters θ (part-ofspeech, character and morphological embeddings, character, dependant and governor BiLSTMs and the two layer perceptron weights) are trained end to end via back propagation one sentence at a time. Given a sentence x, we note j the index of the governor of w i and l the relation label of w i , the loss function is:

loss(x) = w i j =j j =i max(0, s ij -s ij + 1) 2 + l =l max(0, s ijl -s ijl + 1) 2
For each word, there are two terms. The first term enforces that for all potential governors that are neither the word itself nor its actual governor, their highest score (irrespective of the relation label) should be smaller than the score of the actual governor and actual label by a margin of 1. The second term is similar and enforces that for the actual governor, any label that is not the true label should have a score smaller than the score of the actual label again by a margin of 1.

Phylogenetic Training

Let L = {l 1 , l 2 , ..., l n l } be a set of n l languages and let P = {p 1 , p 2 , ..., p np } be a set of n p protolanguages (hypothesized ancestors of languages in L). Let T be a tree over L * = L ∪ P such that languages of L are leaves and proto-languages of P are inner nodes. This means that we assume no two languages in L share a direct parenthood relation, but they at best descend both from a hypothesized parent. We could in principle have data appearing only in inner nodes. Tree T has a single root, a proto-language from P that all grammars descend from. This ancestor of all languages shall model linguistic universals3 and ensure we deal with a well formed tree. We use the notation p > l for the fact that language/node l descends from language/node p.

For each language l ∈ L, we assume access to a set of n annotated examples D l . For each protolanguage p ∈ P, we create an annotated set D p = p>l D l as the union of its descendent sets. For each language l ∈ L * , we want to learn a parsing model θ l .

Model Evolution

The main idea behind phylogenetic training is to initialize a new model with the model of its parent, thus effectively sharing information between languages and letting models diverged and specialize over time. The training pocedure is summarized in Algorithm 1.

At the beginning, we initialize a new blank/random model that will be the basic parsing model for all the world languages. Then, we sample sentences (we will discuss sampling issues in next section) randomly from all the available languages, parse them, compute the loss and update the model accordingly. Since the training sentences are sampled from all the available languages, the model Data: a train set D l and a dev set D l per language, a tree T , two sampling sizes k, k and a maximum number of reboot r Result:

a model θ per node in T begin Instantiate empty queue Q Q.push(T .root) while Q is not empty do l = Q.pop() if l = T .root then initialize θ 0 T .root randomly else θ 0 l = θ l.parent reboot = 0, i = 1, a 0 = 0 while reboot < r do θ i l = train(θ i-1 l , D l , k) a i = test(θ i l , D l , k ) if a i ≤ a i-1 then reboot += 1 else reboot = 0, i += 1 θ l = θ i l for c in l.children do Q.push(c)
Algorithm 1: Phylogenetic training procedure.

will learn to be as good as possible for all the languages at the same time.

When the model θ p has reached an optimum (that we defined hereafter), we pass a copy of it to each of its children. Thus, for each child c of p, we initialize θ 0 c = θ p to its parent (p) final state. Each model θ c is then refined on its own data set D c which is a subset of D p , until it reaches its own optimum state and is passed down to its own children. This process is repeated until the model reaches a leaf language, where the model θ c is eventually refined over its mono-lingual data set D c .

By passing down optimal models from older/larger languages sets to newer/smaller ones, models get the chance to learn relevant information from many different languages while specializing as time goes by.

The question now is to find when to pass down a model to its children. In other words, at which stage has a model learned the most it could from its data and should start to diverge to improve again?

Following the principle of cross-validation, we propose to let held-out data decide when is the right time to pass the model down. Let D p be a set of held-out sentences from the same languages as D p . Then, after every epoch i of k training examples, we freeze the model θ i p , and test it on k sentences from D p . This gives a score a i (UAS/LAS) to the current model. If the score is higher than the score of the previous model θ i-1 p then training goes on, otherwise we discard it and retrain θ i-1 p for another k sentences. If after having discarded r epochs in a raw we have not yet found a better one, then we assume we have reached an optimal model θ i-1 p and pass it on to its children (unless it is a leaf, in which training is over for that language).

Sentence Sampling

There are a few things we should consider when drawing examples from a proto-language distribution. Beside the question of whether some languages are more conservative than others with respect to their ancestor, which we have decided to simplify saying that all languages are as representative of their ancestors, there is the problem of data unbalance and tree unbalance.

Sampling sentences uniformly across languages is not a viable option for the size of datasets varies a lot across languages and that they do not correlate with how close a language is to its ancestor. For example, there are 260 Belarusian training sentences against 48814 Russian ones. The basic question is thus whether one should draw examples from languages or branches. Basic linguistic intuition tells us that drawing should be performed on branches. Modern languages distribution has no reason to reflect their proximity to their ancestor language. Amongst Indo-European languages, there are one or two Armenian languages as well as one or two Albanian languages (depending on the criteria for being a language), while there are tens of Slavic languages and Romance languages. However, there is no reason to believe that Slavic or Romance languages are better witnesses of proto-Indo-European than Armenian or Albanian.

Drawing examples from languages would bias the intermediate models toward families that have more languages (or more treebanks). It might be a good bias depending on the way one compute the overall accuracy of the system. If one uses the macro-average of the individual language parsers, then biasing models toward families with many members should improve the accuracy overall.

In this work, at a given inner node, we decided to sample uniformly at random over branches spanning from this node, then uniformly at random over languages and then uniformly at random over sentences. It boils down to flattening the subtree below an inner node to have a maximum depth of 2. For example in Figure 1, at the root (Proto-Slavic) we pick a branch at random (e.g. Proto-South-Slavic), then a language at random (e.g. Croatian) then a sentence at random. Given that we have picked the Proto-South-Slavic branch, all South-Slavic languages are then as likely to be chosen. This biases slightly the model toward bigger subfamilies. In our example, Croatian and Serbian have the same chances to be sampled than Slovenian, therefore their family, Proto-Serbocroatian is twice as likely to be chosen as Slovenian is, while being at the same depth in the tree. We could otherwise sample over branches, then over sub-branches again and again until we reach a leaf and only then pick a sentence. In this case, Proto-Serbocroatian and Slovenian would have the same probability to be chosen. This would give much more weight to languages high in the tree than languages low in the tree. While this would give more balance to the actual model, it could be detrimental to the averaged results since the data distribution is itself unbalanced. It would of course be possible to try any variation between those two, picking sub-branches according to a probability that would depend on the number of languages in that family for example, therefore mitigating the unbalance problem.

Zero-Shot Parsing

An interesting property of the phylogenetic training procedure is that it provides a model for each inner node of the tree and thus each intermediary grammar. If one were to bring a new language with its position in the tree, then we can use the pretrained model of its direct ancestor as an initialization instead of learning a new model from scratch. Similarly, one can use this ancestor model directly to parse the new language, effectively performing zero-shot dependency parsing. We investigate this possibility in the experiment section.

Experiments

To assess the potential of phylogenetic training both in terms of multi-task learning and zero-shot parsing capabilities, we experimented with data from the Universal Dependencies project version 2.2 [START_REF] Stymne | Parser training with heterogeneous treebanks[END_REF]. When several corpora are available for a language, we chose one to keep a good balance between morphological annotation and number of sentences. For example, the Portuguese GSD treebank has slightly more sentences than the Bosque treebank but it is not well morphologically annotated. The zero-shot parsing models have been directly tested on languages that lack of training set. The treebanks names are given in the tree 4 and the result table 1.

Setting

As some languages have no training data and unique writing systems making the character model inefficient for them, we resorted to use gold parts-of-speech and morphological attributes rather than predicted ones. For example, Thai has no training data, no language relative and a unique script, which altogether make it really hard to parse (from a phylogenetic perspective).

The phylogenetic tree used for the experiment is adapted from the Ethnologue (2018). For space reasons, it is reported in the appendix in Figures 4 and5. We tried to have a tree as consensual as possible, but there are still a few disputable choices, mostly about granularity and consistency. Sanskrit could have its own branch in the Indic family just as Latin in the Romance family, but because Sanskrit has no training data, that would not actually change the results. Likewise, as Czechoslovak and Dutch-Afrikaans have their own branches, Scandinavian languages could also distributed between east and west Scandinavian. As an English based Creole, Naija could as well be put in the Germanic family, but we kept it as a separate (Creole) family.

Regarding model training proper, we used k = 500 training sentences per iteration, k = 500 held-out sentences from the developpement set to compute running LAS and a maximum number of reboot r = 5. Following [START_REF] Dozat | Stanford's graph-based neural dependency parser at the conll 2017 shared task[END_REF], we use Eisner algorithm [START_REF] Eisner | Three new probabilistic models for dependency parsing: An exploration[END_REF] at test time to ensure outputs are well formed trees. The neural model is implemented in Dynet [START_REF] Neubig | Dynet: The dynamic neural network toolkit[END_REF] and we use Adadelta with default parameters as our trainer. We averaged the results over 5 random initializations. Independent models are trained in the same manner but with mono-lingual data only. We report both labeled and unlabeled edge prediction accuracy (UAS/LAS). In the appendix we also report results averaged per family.

Multi-Task Learning

Table 1 reports parsing results for languages that have a training set. Note that a few languages do not have a separate developpement set, then we used the training set for both training and validation. The training set size of those languages is reported in square brackets. This has low to no impact on other languages results but it can be problematic for the language itself as it can over-fit its training data especially when they are very few as is the case of Buryat for example. To be fair, we report two different averages. Avg is the average over languages that have a separate developpement set, and Avg No Dev is the average over languages that do not have a separate developpement set. For each language, the best UAS/LAS are reported in bold.

On average, phylogenetic training improves parsing accuracy, both labeled and unlabeled. This is especially true for languages that have very small training sets (50 sentences or less) and lack of developpement set. Those languages show an averaged 7 points improvement and up to 15 points (hsb, kmr). Since independent mono-lingual models follow the exact same training procedure but without phylogenetic initialization and that every sentence will be seen several times both at training and validation, the sampling method cannot explain such a difference. This shows that the ancestor's model is a good initialization and acts as a form of regularization, slowing down over-fitting.

Phylogenetic training is also beneficial as one gains information from related languages. Indo-European languages gain from sharing information. This is especially true for Balto-Slavic (sk +5.82, lt +5.07 UAS) and Indo-Iranian languages (mr +2.05 UAS). It is less consistent for Romance and Germanic languages. This might be due to the tree not representing well typology for those families. Typically, English tends to group syntactically with Scandinavian languages more than with West-Germanic. Turkic and Uralic languages show the same benefits overall (ug +2.67, fi +3.39 UAS).

Dravidian and Afro-Asiatic languages are not as consistent. While Telugu seems to gain from Tamil data, the reverse is not true. Result variation for Arabic, Hebrew and Coptic are marginal. This is likely due to the fact that we only have three quite different languages from that family and that they all have their own script.

Similarly, phylogenetic training is not consistently useful for languages that do not have rela- ja, ko, vi, zh).

Overall results are a bit below the state of the art, but the model is very simple and relies on gold morphology, so it is not really comparable.

Zero-Shot Parsing

Table 2 reports parsing results for languages that do not have a training set. Because of phylogenetic training and the tree structure that guides it, it can happen that a language ancestor's model is in fact trained on data only accounting for a narrow range of later stages. For example, while Faroese uses the North-Germanic model refined on both Norwegians, Swedish and Danish data, Tagalog uses the Austronesian model only refined with Indonesian data thus making it more an Indonesian model than an actual Austronesian model. Those cases are marked by an asterisk in the table. Komi (kpv) model is refined on Finno-Samic data, Breton (br) model on Irish data, Cantonese (yue) model on Mandarin data.

Looking at Table 2, we make the following observations. As expected scores are on average lower than for languages with training data, however the UAS/LAS gap is substantially bigger from 6.781 to 17.08 points. It is hard to compare to other works on zero-shot parsing since they use different data and scores span a big range, but our results are comparable to those of [START_REF] Aufrant | Zero-resource Dependency Parsing: Boosting Delexicalized Cross-lingual Transfer with Linguistic Knowledge[END_REF] and [START_REF] Naseem | Selective sharing for multilingual dependency parsing[END_REF], while our zero-shot models are given for free by the phylogenetic training method.

On a language per language basis, we see that there are a few important factors, the most striking being genre. Tagalog (tl) and more surprisingly Warlpiri (wbp) have relatively high parsing accuracy despite being either completely isolated or having only one relative (Indonesian). This is likely because their data are well annotated stereotypical sentences extracted from grammars, thus making them easy to parse.

Then we see that Naija (pcm) and Yoruba (yo) are about 25 points higher than Thai (th) despite them three having low morphology (in the treebanks). As they have different genres (spoken, bible, news and wiki), without a deeper look at the trees themselves, our best guess is that this is due to Thai having a different script. Naija and Yoruba both use the Latin alphabet, and as such they can rely to some extent on the character model to share information with other languages, to at least organise the character space. This analysis would also carry for Cantonese (yue). It is a morphologically simple language, and despite having a relative (Mandarin), its score is rather low. The genre alone (spoken) would not explain everything as Naija has also a spoken treebank and a higher score. The writing system might be at blame once again. Indeed, Chinese characters are very different from alphabetic characters and are much harder to use in character models because of sparsity. Comparing Mandarin and Cantonese test sets with Mandarin train set, the amount of out-of-vocabulary words is 32.47% of types (11.90% of tokens) for Mandarin and 54.88% of types (56.50% of tokens) for Cantonese. The results for out-of-vocabulary characters are even more striking with 3.73% of types (0.49% of tokens) for Mandarin and 12.97% of types (34.29% of tokens) for Cantonese. This shows that not only there are a lot of OOV in Cantonese test set, but that those words/characters are common ones as 12.97% of character types missing make up for more than a third of all character tokens missing, where on the contrary Mandarin OOV are seldom and account for less tokens percentage than types. This is one more argument supporting the importance of the character vector.

Other important factors are typology and morphology. Amharic (am) despite its unique script has a higher score than Cantonese that actually shares its scripts (to some extent as we have seen) with Mandarin. The key point for Amharic score, is that all its relatives (Hebrew, Arabic and Cop-tic) have their own scripts and are morphologically rich, thus the model learns to use morphological information. The analysis is similar for Komi which on top of sharing morphology with its relatives also share the writing system which provides it an extra gain. However, this might word in the opposite direction as well, as we can see with Faroese, Breton and Sanskrit. Faroese (fo) is morphologically rich and that should help, however its North-Germanic relatives are morphologically much simpler. Thus the model does not learn to rely on morphological attributes nor on word endings for the character model as much. The same is true for Sanskrit (sa), which is morphologically richer than its modern Indic relatives, with an extra layer of specific writing systems. Eventually, Breton model (br) is refined over Irish data only and while Irish is a typological outlier amongst Indo-European languages because of its Verb-Subject-Object word order, Breton has the standard Subject-Verb-Object, thus using Irish data might actually be detrimental.

These arguments show the respective importance of the writing system, the genre of the data, the morphological analysis and the typology in phylogenetic zero-shot dependency parsing. Those factors can either work together positively (Komi) or negatively (Cantonese) or cancel each other out (Amharic, Faroese).

Related Work

The goal of multi-task learning is to learn related tasks (either sharing their input and/or output space of participating of the same pipeline) jointly in order to improve their models over independently learned one [START_REF] Caruana | Multitask learning[END_REF]. In [START_REF] Søgaard | Deep multitask learning with low level tasks supervised at lower layers[END_REF], task hierarchy is directly encoded in the neural model allowing tasks with different output space to share parts of their parameters (POS tagging comes at a lower level than CCG parsing and only back-propagates to lower layers). Likewise, in [START_REF] Johnson | Google's multilingual neural machine translation system: Enabling zero-shot translation[END_REF], the encoder/decoder architecture allows to learn encoders that target several output languages and decoders than handle data from various input languages. However, in multitask learning literature, task relationships are often fixed. In [START_REF] Cavallanti | Linear algorithms for online multitask classification[END_REF] tasks with the same output spaces share parameter updates through a fixed similarity graph. In this work, changing level in the tree can be seen as splitting the similarity graph into disjoint sub graphs. It is a way to have tasks relationships evolving during training and to encode information about task evolution that lacks in other multi-task methods.

In multi-lingual parsing, [START_REF] Ammar | Many languages, one parser[END_REF] propose to train a single model to parse many languages using both typological information, crosslingual word representations and language specific information. While their model gives good results, they only apply it to 7 Germanic and Romance languages. It would be worth doing the experiment with 50+ languages and see how the results would change. However, because of language specific information their model would probably become very big. In this work, language specific information is not added on the top of the model, but is just language generic information that refines over time. [START_REF] Che | The hit-scir system for end-to-end parsing of universal dependencies[END_REF]2018) and [START_REF] Stymne | Parser training with heterogeneous treebanks[END_REF] propose to train parsers on several concatenated treebanks either from the same language or from related languages and to fine-tune the parsers on individual treebanks afterward to fit specific languages/domains. The main difference with our method, is that instead of one step of fine-tuning, we perform as many fine-tuning as there are ancestors in the tree, each time targeting more and more specific data. This in turn requires that we handle data imbalance therefore using sampling rather than plain concatenation. [START_REF] Aufrant | Zero-resource Dependency Parsing: Boosting Delexicalized Cross-lingual Transfer with Linguistic Knowledge[END_REF] propose to tackle zeroshot parsing by rewriting source treebanks to better fit target language typology. Assuming that typology is homogeneous in a language family, the phylogeny should drive models to be typologically aware. However, as we have seen for Breton and Irish, that assumption might not always hold.

Eventually, the closest work from our in spirit is the one of [START_REF] Berg | Phylogenetic grammar induction[END_REF]. They use a phylogenetic tree to guide the training of unsupervised dependency parsing models of several languages, using ancestor models to tie descendent ones. The main difference here beside supervision, is that we do not use ancestor models as biases but rather as initialization of descendent models.

Conclusion

We have presented a multi-task learning framework that allows one to train models for several tasks that have diverged over time. Leveraging their common evolutionary history through a phylogenetic tree, models share parameters and train-ing samples until they need to diverge. As a by product of this phylogenetic training, we are provided with intermediary models that can be used to zero-shot a new related task, given its position in the evolutionary history.

We have applied this framework to dependency parsing using a graph-based neural parser and the phylogenetic tree of the languages from UD 2.2 to guide the training process. Our results show that phylogenetic training is beneficial for well populated families such as Indo-European and Uralic. It also helps generalization and prevents over-fitting when very few data are available. For zero-shot parsing, genre, writing system and morphology are crucial factors for the quality of parse predictions.

Some works have been done on automatically learning task relationship in multi-task setting. It would be interesting to see how the algorithm could figure out when and how to cluster languages automatically as phylogenetic trees do not directly depict grammar evolution.

Our model does not know that Latin came before Old French and before modern French, or that despite being Germanic, English underwent a heavy Romance influence. It would be worth investigating softening the tree constraints and instigating more evolutionary information in the structure.

Another important point is that we use gold partof-speech and morphological information which is unlikely to be available in real scenarios. However, our new training procedure can be applied to any task, so a future work would be to use it to perform phylogenetic POS tagging.

Other directions for the future are designing better sampling methods as well as better ways to measure training convergence at each level.

Figure 1 :

 1 Figure 1: A possible phylogenetic tree for languages in the Slavic family.

Table 1 :

 1 Parsing results for languages with a training set for phylogenetic models and independent models. The training set size of languages without a developpement set are reported in brackets.

		Phylogenetic	Independent	Lang	Model	UAS LAS
	UAS LAS UAS LAS 74.81 70.32 75.07 71.08 85.51 79.28 86.03 80.15 81.89 75.36 81.59 75.57 48.72 30.68 37.88 18.09 76.81 69.51 78.61 72.76 85.15 80.94 85.44 81.66 78.50 72.50 79.16 74.13 80.37 73.54 79.48 72.37 79.25 74.34 79.27 74.66 77.83 71.54 79.91 74.33 84.62 78.78 83.82 78.09 80.36 74.79 80.14 74.52 65.59 61.15 66.02 61.74 88.88 74.24 87.01 72.05 75.05 66.94 76.19 67.93 80.13 76.40 78.34 74.25 79.51 72.88 80.15 74.31 78.05 72.20 74.66 68.22 75.32 68.13 73.91 66.96 66.33 48.20 63.66 46.07 59.64 50.66 59.00 50.54 70.82 55.42 62.81 44.59 73.38 68.35 74.23 69.81 91.22 87.31 91.40 87.37 81.08 74.97 80.83 74.69 65.03 51.76 59.27 46.67 75.91 67.54 76.20 67.72 73.82 67.88 71.68 66.05 86.44 83.30 86.88 83.96 84.32 77.02 84.82 78.19 78.65 68.97 76.60 64.04 69.08 59.64 54.77 45.07 89.32 82.89 88.75 82.60 78.76 72.95 79.93 74.07 79.83 70.46 79.93 70.88 84.93 79.37 84.90 79.83 66.25 58.88 65.07 57.80 87.03 81.67 87.10 82.27 gl [600] treegal 83.80 78.06 83.60 77.63 ar nuyad cop he bxr [19] eu af da de gsd en ewt got nb nl alpino nn nynorsk sv talbanken be bg cs pdt cu hr hsb [23] lt lv pl lfg ru syntagrus sk sl ssj sr uk ca es ancora fr gsd fro it isdt la proiel pt bosque ro rrt fa hi hdtb kmr [20] mr ur el grc proiel ga [566] hy [50] id gsd ja gsd ko kaist kk [31] tr imst ug et fi ftb hu sme [2257] ta te vi zh 82.32 74.24 78.91 69.95 84.35 77.59 84.21 77.94 85.11 79.52 85.61 80.18 84.67 78.81 85.69 80.11 78.16 73.50 74.96 70.91 85.85 79.86 86.17 80.47 87.15 83.43 88.39 85.21 84.91 79.17 79.09 73.20 77.91 72.72 77.33 72.85 92.88 88.53 91.07 86.49 78.39 70.14 76.69 68.89 61.42 50.88 56.35 46.14 74.24 66.01 58.59 50.37 81.70 74.73 81.05 73.95 82.98 77.19 83.31 78.32 79.78 71.71 77.45 69.88 86.01 79.16 86.40 79.79 80.18 74.11 78.09 72.76 80.46 74.62 81.17 75.47 82.39 76.44 82.58 77.32 77.19 68.55 76.52 68.40	am br fo sa kpv lattice Finno-Permiac * 65.16 52.11 Semitic 57.27 26.25 Celtic * 61.36 43.89 North-Germanic 52.40 46.52 Indic 56.18 40.46 pcm World 60.43 43.80 th World 29.14 17.61 tl Austronesian * 70.89 50.38 wbp World 87.67 65.66 yo World 56.16 37.51 yue Sino-Tibetan * 41.68 25.02 Avg 58.04 40.83
	Avg Avg No Dev	80.05 73.35 79.47 73.02 70.97 60.69 63.93 53.05

Table 2 :

 2 Accuracy of languages without a training set.

	tives. While Buryat (bxr) that has a very small
	training set benefits from universal linguistic in-
	formation and gain almost 11 points UAS, Basque
	(eu) that has a very different grammatical struc-
	ture than other languages and enough training
	data (5396 sentences) looses 3.25 LAS. Gains and
	losses are marginal for the other five languages (id,

Universal part-of-speech for a set of 17 tags. Does not encode morphology.

Language specific part-of-speech. Might include morphological information, but is not available for all languages.

It does not imply anything about our belief or not in the monoglotto genesis hypothesis.

Acknowledgement

This work was supported by ANR Grant GRASP No. ANR-16-CE33-0011-01 and Grant from CPER Nord-Pas de Calais/FEDER DATA Advanced data science and technologies 2015-2020. We also thank the reviewers for their valuable feedback.

A Appendix

Figures 4 and5 represent the phylogenetic tree used for guiding the training process. As we only use data from the UD project 2.2, we collapse unique child so that Vietnamese is not an Austro-Asiatic language, it is just Vietnamese. We also only use well attested families, thus Buryat, a Mongolic language, is alone and not linked to Turkic languages. Maybe, the most disputable choice is to put Naija in its own Creole family instead of the Germanic family.


Bo Li
email: bo.li@inria.fr

Mathieu Dehouck
email: mathieu.dehouck@inria.fr

Pascal Denis
email: pascal.denis@inria.fr

Modal sense classification with task-specific context embeddings

Sense disambiguation of modal constructions is a crucial part of natural language understanding. Framed as a supervised learning task, this problem heavily depends on an adequate feature representation of the modal verb context. Inspired by recent work on general word sense disambiguation, we propose a simple approach of modal sense classification in which standard shallow features are enhanced with task-specific context embedding features. Comprehensive experiments show that these enriched contextual representations fed into a simple SVM model lead to significant classification gains over shallow feature sets.

Introduction

Modal sense analysis plays an important role in NLP tasks such as sentiment analysis [START_REF] Liu | Sentiment analysis of sentences with modalities[END_REF], hedge detection [START_REF] Morante | Learning the scope of hedge cues in biomedical texts[END_REF] and factuality recognition [START_REF] Sauri | Are you sure that this happened? assessing the factuality degree of events in text[END_REF]. As an important part of modality analysis, sense classification of modal verbs has attracted notable interest of researchers in recent years. Modal sense classification (MSC) can be treated as a special type of Word Sense Disambiguation (WSD) [START_REF] Navigli | Word sense disambiguation: A survey[END_REF], but differs from general WSD tasks in that MSC deals with a small set of modal verbs with a limited set of senses. For instance, the English modal verb can has a limited sense set containing dynamic, deontic, and epistemic senses.

MSC is typically modeled in a supervised learning framework, which in turn raises the question of how to build the most effective feature representation of the modal context. Existing studies have proposed various context representations. A first set of approaches rely on manually engineered features derived from sophisticated linguistic pre-processing. For instance, [START_REF] Ruppenhofer | Yes we can!? annotating english modal verbs[END_REF] uses syntax-based features in addition to shallow features such as POS and lemma. Pushing this line of research even further, [START_REF] Zhou | Semantically Enriched Models for Modal Sense Classification[END_REF][START_REF] Marasovic | Modal sense classification at large: Paraphrase-driven sense projection, semantically enriched classification models and cross-genre evaluations[END_REF] propose to use semantic-rich features that are related to lexical, proposition-level and discourse-level semantic factors. Taking a drastically different approach, [START_REF] Marasovic | Multilingual modal sense classification using a convolutional neural network[END_REF] employs Convolution Neural Networks (CNNs) to automatically extract modal sense features, and show that their approach is competitive with hand-crafted feature-based classifiers.

In this paper, we explore an intermediate line of research, attempting to leverage both shallow feature sets and pre-trained embeddings for the context words in a very simple learning architecture. That is, we propose to enrich a very simple feature set (basically, POS tags, uni-, bi-and tri-grams), known to be very effective for WSD, with embeddings for the context words. Inspired by the recent work [START_REF] Iacobacci | Embeddings for word sense disambiguation: An evaluation study[END_REF] for general WSD, we propose several weighting schemes, including a new task-specific one, for constructing context embeddings from pre-trained word embeddings. Experimental results prove that this approach outperforms state-of-the-art methods over two standard MSC datasets.

Related work

Modal sense classification. Similar to WSD, MSC has been framed as a supervised learning problem. [START_REF] Ruppenhofer | Yes we can!? annotating english modal verbs[END_REF] presents an annotation schema for modal verbs in the MPQA corpus and represent modal senses with word-based and syntactic features. On top of feature sets, they perform machine learning with a maximum entropy classifier. Since the corpus used in [START_REF] Ruppenhofer | Yes we can!? annotating english modal verbs[END_REF] is relatively small and unbalanced, [START_REF] Zhou | Semantically Enriched Models for Modal Sense Classification[END_REF][START_REF] Marasovic | Modal sense classification at large: Paraphrase-driven sense projection, semantically enriched classification models and cross-genre evaluations[END_REF] develop a paraphrase-driven, cross-lingual projection approach to create large and balanced corpora automatically. In addition, they develop semantically grounded features such as lexical features of the modified verb and subjectrelated features for sense representation. [START_REF] Marasovic | Multilingual modal sense classification using a convolutional neural network[END_REF] casts modal sense classification as a novel semantic sentence classification task using CNNs to model propositions.

Word sense disambiguation. Word sense disambiguation is a long-standing challenge in natural language processing [START_REF] Navigli | Word sense disambiguation: A survey[END_REF]. We focus here on most recent advances in WSD which come from the use of neural networks. One category of neural WSD works use neural networks to learn word embeddings [START_REF] Taghipour | Semi-supervised word sense disambiguation using word embeddings in general and specific domains[END_REF][START_REF] Iacobacci | Embeddings for word sense disambiguation: An evaluation study[END_REF] or context representation [START_REF] Yuan | Semisupervised word sense disambiguation with neural models[END_REF][START_REF] Kågebäck | Word sense disambiguation using a bidirectional lstm[END_REF] that can be further used separately or as additional features to shallow sense features. The other category of approaches [START_REF] Raganato | Neural sequence learning models for word sense disambiguation[END_REF] model WSD as an end-to-end sequence learning problem. These two categories of approaches show similar performance as reported in [START_REF] Raganato | Neural sequence learning models for word sense disambiguation[END_REF]. Our work is inspired by [START_REF] Iacobacci | Embeddings for word sense disambiguation: An evaluation study[END_REF] which has shown state-of-the-art performance in WSD and which is convenient for adaptation to MSC.

Approach

Following [START_REF] Iacobacci | Embeddings for word sense disambiguation: An evaluation study[END_REF] , we will use word embeddings trained on unlabeled data to enrich shallow features used in general WSD tasks. As far as we can tell, such a combination has never been investigated for MSC.

Shallow features (SF)

We refer to standard features used in WSD tasks and make use of three shallow feature sets as general features for MSC. For all feature sets, we compute with the same context window surrounding a target modal verb with size 2n (n left, n right). The first feature set is POS tags of context words. The second feature set is context words excluding stop words. The third feature set is the local collocations of a target word which are ordered sequences of words appearing in the context window. We use several uni-gram, bi-gram and tri-gram patterns as the collocation patterns in the context window. We use these simple yet efficient features to do away with heavy feature engineering.

Word embedding features

Embeddings of context words can be combined in a way such that a comprehensive context embedding vector can be obtained to represent the modal context. The context vector can then be combined with those shallow features. Inspired by [START_REF] Iacobacci | Embeddings for word sense disambiguation: An evaluation study[END_REF], we obtain context representation by taking a weighted sum of embedding vectors of all context words. Given a target modal verb w 0 and a context window size 2n, let us denote the context word at the relative position i(i ∈ [-n, 0) ∪ (0, n]) as w i . The context embedding of w 0 (denoted as w c ∈ R d ) can be written as the weighted sum of embedding vector w i ∈ R d of each context word w i , which is:

w c = n i=-n&i =0 f (w i )w i (1)
where f (w i ) is a weighting function measuring the importance of each context word w i w.r.t. the modal verb w 0 . For simplicity, we assume that f (w i ) only depends on i (i.e., the relative position of w i w.r.t. w 0 ). Average weighting (WE av ). The most intuitive weighting schema simply treats all the context words equally, which corresponds to:

f (w i ) = 1 2n (2)
Linear weighting (WE li ). In a linear manner, the importance of a word is assumed to be inversely proportional to its distance from the target word, which can be formulated as:

f (w i ) = 1 - |i -1| n (3)
Exponential weighting (WE ex ). Compared to linear manner, the exponential weighting schema put more emphasis on the nearest context words. The importance of a word decreases exponentially w.r.t. its distance from the target word, which is:

f (w i ) = α i-1 n-1 (4)
where α is a hyper-parameter. The above weighting schema have been investigated in [START_REF] Iacobacci | Embeddings for word sense disambiguation: An evaluation study[END_REF]. Modal-specific weighting. In order to adapt the weighting function f (w i ) to MSC, we want to assign more weights to context words which have closer connections to modal senses. Inspired by findings in [START_REF] Zhou | Semantically Enriched Models for Modal Sense Classification[END_REF], we note that the embedded verb in the scope of the modal and the subject noun phrase are two components acting as strong hints to determine the modal verb sense. Let us consider as an example the sentence the springbok can jump very high. When the embedded verb jump appears, we prefer a dynamic reading of can. Based on the intuition above, we assign more weights to context word which is the embedded verb or which is the head in the subject noun phrase. Formally, we define a modal-specific weighting function g such that:

g(w i ) =    β 1 f (w i ), w i is embedded verb β 2 f (w i ), w i is the head in the subj noun phrase f (w i ), else
where β 1 and β 2 are hyper-parameters indicating that words that are more relevant to modal verb senses take more responsibility, and f is one of the weighting schema defined above. We can obtain a context representation w c that is optimized for MSC task by replacing f in equation 1 with g defined here.

Experiments

In this section, we perform comprehensive experiments in order to evaluate the usefulness of word embedding features in MSC.

Datasets. We make use here of two representative corpora for MSC. One is the small and unbalanced corpus annotated manually based on the MPQA corpus in [START_REF] Ruppenhofer | Yes we can!? annotating english modal verbs[END_REF]. The other one is the larger and balanced corpus EPOS which is constructed automatically via paraphrase-driven modal sense projection in [START_REF] Zhou | Semantically Enriched Models for Modal Sense Classification[END_REF]. We use two training/testing settings: (1) both training and testing sets picked from MPQA. ( 2) training set from EPOS and testing set from MPQA. Characteristics of training/testing sets are not given here due to space reasons.

Experimental settings. We have used Stanford parser to pre-process the original corpora. For classification, we employ SVM implemented in LibSVM [START_REF] Chang | Libsvm: A library for support vector machines[END_REF]. The context window size 2n is picked from {2, 6, 10, 20}. The parameter α is set to {0.1, 0.2, 0.5, 0.8}, and the parameters β 1 and β 2 are selected from {1, 2, 5, 10}. The word embeddings are trained with word2vec [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] on Wikipedia. The hyper-parameters are optimized via 5-fold cross validation. We employ accuracy as the performance measure and McNemar's test (p < 0.05) to test significance.

Competing Systems. we use max frequent sense (MFS) baseline for unbalanced classifier and random sense (RS) baseline for balanced classifier. The other baseline systems are RR [START_REF] Ruppenhofer | Yes we can!? annotating english modal verbs[END_REF] and ZH [START_REF] Zhou | Semantically Enriched Models for Modal Sense Classification[END_REF]. In addition, we compare different combinations of features presented in section 3. SF is the shallow features, and +WE * * denotes the concatenation of word embedding features WE * * to SF. +WE * * stands for the modal specific weighting version.

Results on unbalanced corpus. We firstly show the results on the unbalanced MPQA corpus in table 1. The performance of SF is comparable to RR and slightly inferior to ZH, given that SF contains similar features as RR. Furthermore, when we consider word embeddings as extra features, we obtain results that are at least as good as SF. The best performance comes from +WE ex where, compared to SF, we have gotten significant improvement on can and insignificant improvement on could, and slightly decrease that is not significant on should. However, with any of +WE av , +WE li and +WE ex , we cannot achieve any significant improvement. We further note that MFS is a strong baseline on the unbalanced corpus. Neither RR nor ZH is able to beat the MFS baseline with ZH being slightly better than RR, which is coincident with conclusions in previous work [START_REF] Zhou | Semantically Enriched Models for Modal Sense Classification[END_REF]. Results on balanced corpus. Since the MPQA training set is small in size and unbalanced in sense distribution, machine learning algorithms tend to over fit the training set. To circumvent this problem, we perfom additional experiments and train on the balanced EPOS corpus. The results are reported in table 2. Not surprisingly, the RS baseline is inferior to any other system in the table. When word embedding features are combined to SF, we note improvement on most of the six modal verbs, and the best results are obtained with +WE ex . The improvements of +WE ex over SF are significant on can. Both results on balanced and unlabeled corpora reveal that task-specific embedding features on top of WE ex are superior to their counterparts without task-specific information, which shows the best performance overall.

Conclusion

In this paper, we make use of word embeddings learned from unlabeled data as additional features to more adequately represent the context of modal verbs. Our main experimental result is that the best weighting scheme for combining pre-trained word embeddings into context embeddings is a corrected version of exponentially decaying weighting which attributes higher weights to the verb modified by the modal and its subject.

Table 1 :

 1 Accuracy with various features on MPQA. + or -indicates that the improvements or degradations with respect to SF are statistically significant. The highest value in each row is marked in bold.

		MFS	RR	ZH	SF	+WE av +WE li +WE ex +WE ex
	can	69.9	66.6 66.1	63.5	66.2	66.7	67.1	70.1 +
	could	65.0	62.5 67.9	67.5	67.8	68.9	70.4	70.5
	may	93.6 93.6 93.6 93.6	93.6	93.6	93.6	93.6
	must	94.3 94.3 94.3 94.3	94.3	94.3	94.3	94.3
	shall	84.6 83.3 83.3	83.3	83.3	83.3	83.3	83.3
	should 90.8	90.8 92.9 91.1	90.9	90.8	90.8	90.8

Table 2 :

 2 Accuracy with various features on EPOS+MPQA. + or -indicates that the improvements or degradations with respect to SF are statistically significant. The highest value in each row is marked in bold.

		RS	RR	ZH	SF	+WE av +WE li +WE ex +WE ex
	can	33.3 57.8 60.4	65.6	64.4	64.7	64.7	67.3 +
	could 33.3 49.2 56.3	56.7	57.9	57.9	60.0	59.6
	may	50.0 92.1 92.1	93.6	94.2	93.6	94.2	95.0
	must	50.0 71.7 85.6 76.8	78.7	79.3	79.9	80.6
	shall	50.0 53.9 53.9 61.5	46.2	46.2	46.2	53.8
	should 50.0 76.3 88.3 90.8	90.8	90.8	90.8	90.8

* This work was supported by the ANR-16-CE93-0009 REM project.