Aina Garí
email: aina.gari@limsi.fr

Marianna Apidianaki
email: marianna@limsi.fr

Alexandre Allauzen
email: allauzen@limsi.fr

A comparative study of word embeddings and other features for lexical complexity detection in French

Keywords: Complexité lexicale, lisibilité, ordonnancement de synonymes, plongements lexicaux Lexical complexity, readability, synonym ranking, word embeddings

Etude comparative de plongements lexicaux et autres traits pour la détection de la complexité lexicale en français Détecter la complexité lexicale est une étape importante pour la simplification automatique de textes, servant lors de l'identification des éléments lexicaux à substituer. Dans ce travail, nous explorons l'utilité des plongements lexicaux pour mesurer la complexité de mots en français, en les combinant avec d'autres traits reconnus comme étant utiles pour cette tâche. Nos résultats sur une tâche d'ordonnancement de synonymes selon leur complexité montrent que les plongements seuls donnent de meilleurs résultats que nombreux autres traits, bien que leur performance reste inférieure à celle de systèmes basés sur la fréquence pour cette langue.

Introduction

Complex word identification (CWI) is an important step in text simplification systems which aim at modifying the complexity level of texts to make them more accessible to readers with reading difficulties. There are many factors that can make a text complex or difficult, at the syntactic level (e.g. texts using passive voice), but also at the pragmatic and lexical levels (e.g. texts with a high number of domain-specific or rare words). Lexical complexity is the property of words that may pose comprehension difficulties to some readers, and which would not be part of the vocabulary of young children or low-proficiency foreign language learners. Complex words have a high impact on the readability of a text. [START_REF] Arya | The effects of syntactic and lexical complexity on the comprehension of elementary science texts[END_REF] showed that lexical complexity affected comprehension of elementary science texts by children, and manual lexical simplification has been shown to improve understanding and reading speed in dyslexics [START_REF] Rello | Frequent words improve readability and short words improve understandability for people with dyslexia[END_REF]. CWI can serve to identify the lexical elements in a text that need to be substituted by simpler synonyms or paraphrases for the meaning of the text to become more accessible.

We carry out a comparative study of different types of systems for measuring the complexity of French words. We experiment with features that have been identified as good indicators of complexity in the literature and word embeddings which, although successful in numerous semantics-related tasks,

have not yet been applied to lexical complexity prediction. We assume that complex words appear in complex contexts, and simple words occur in texts that are easier to understand. We consider lexical complexity to be a property of words that is reflected in their context of use. Word embeddings encode rich contextual information, so we expect them to be particularly useful for complexity detection.

We train our systems on two French lexical resources that encode the distribution of words across different levels of difficulty : Manulex, where categories correspond to school levels [START_REF] Lété | MANULEX : A grade-level lexical database from french elementary school readers[END_REF], and the FLElex database where words are categorized into second language proficiency levels [START_REF] François | FLELex : a graded lexical resource for french foreign learners[END_REF]. We evaluate our systems on lists of synonyms that have been manually ordered by complexity.

Related Work

Due to the importance of lexical complexity detection for simplification, there have been numerous approaches to automatically predicting complexity. The task is often regarded as a classification problem into two or more levels of difficulty. [START_REF] Shardlow | A Comparison of Techniques to Automatically Identify Complex Words[END_REF] performs binary classification of words into simple or complex by means of a Support Vector Machine (SVM) classifier based on features such as frequency, number of syllables and senses. [START_REF] François | A model to predict lexical complexity and to grade words (un modèle pour prédire la complexité lexicale et graduer les mots)[END_REF] classify French words into three or six complexity levels, which indicate the distribution of words in the three school levels and six second language proficiency levels found in Manulex and FLElex, respectively. Gala et al.'s (2014) systems achieve accuracies close to 63% and 43% when trained and evaluated on the two resources. Interestingly, both [START_REF] Shardlow | A Comparison of Techniques to Automatically Identify Complex Words[END_REF]Gala et al.'s (2014) systems obtain results that are close to those of baseline models based on frequency only, and show that complex words occur more rarely in texts.

Another approach proposed by [START_REF] François | Bleu, contusion, ecchymose : tri automatique de synonymes en fonction de leur difficulté de lecture et compréhension[END_REF] consists in ordering synonyms by complexity, creating thus a ranking instead of a classification. Their system is trained on pairs of French synonyms based on a complexity score derived from Manulex, and uses 21 features. We use in this study three of their best performing features : number of characters, number of phonemes and frequency.

Other works have focused on creating readability lexicons. [START_REF] Kidwell | Statistical estimation of word acquisition with application to readability prediction[END_REF] develop a model that infers the age of aquisition of a word from its presence in graded texts and use it to assess text readability. [START_REF] Brooke | Building readability lexicons with unannotated corpora[END_REF] build a readability lexicon based, among others, on word co-occurrence information, which is the closest feature to the word embeddings we use in our experiments.

In the SemEval 2012 shared task on lexical simplification [START_REF] Jauhar | Uow-shef : Simplex-lexical simplicity ranking based on contextual and psycholinguistic features[END_REF], where systems had to choose a simpler synonym for words in context, the frequency baseline was very powerful and was only beaten by one system using psycholinguistic features including concreteness, imageability, familiarity and age of acquisition [START_REF] Jauhar | Uow-shef : Simplex-lexical simplicity ranking based on contextual and psycholinguistic features[END_REF] and word length is the best approach for choosing a substitute for a complex word.

Data and Resources

We train and evaluate our systems on two large lexical resources for French, Manulex and FLElex.

Manulex [START_REF] Lété | MANULEX : A grade-level lexical database from french elementary school readers[END_REF]) is a lexical database with word frequency information calculated from 54 French textbooks used in three school levels, with a total of 1.9 million word tokens. The levels correspond to ages of 6 (CP), 7 (CE1) and 8 to 11 (CE2-CM2) years old. We use its non-lemmatized version, which contains 48,886 word forms, and keep only unigrams with open parts of speech (nouns, adjectives, verbs and adverbs), which reduces it to 39,839 words.

FLElex [START_REF] François | FLELex : a graded lexical resource for french foreign learners[END_REF] is a graded lexicon based on textbooks and simplified readers for learners of French as a foreign language which contains a total of 777,000 tokens. This resource is organized in 6 levels corresponding to the proficiency levels of the CEFR scale (Conseil de l'Europe, 2001), ranging from A1 to C2. The resource contains 14,236 lemmas from which we retain 13,250 after filtering for part of speech and multi-words, as for Manulex.

We perform feature extraction and remove words for which no word embedding is available, which leaves us with 36,845 and 12,942 words in each dataset. Since words might occur in different levels, we assign each word in its first level of appearance, indicating its moment of acquisition, and we observe that the distribution of words is biased towards the third level in Manulex and the three first levels in FLElex. We believe this is because the third level in Manulex groups more ages than the first two, and in the case of FLElex the bias probably reflects the need of constructing a substantial lexicon during the first stages of learning. In order to learn from a more balanced dataset, we randomly remove words from the majority classes, obtaining a total of 26,424 words for Manulex and 9,194 for FLElex. The contents of the original and balanced resources are given in Table 1. 95% of the remaining data of each database is used for training, and 5% is used during the development stage to assess the correlation of the system's predictions with respect to the original classes the words belong to. To better estimate the complexity prediction capacity of the models on unseen data, we also remove from the training data words present in the synonym lists used for evaluation.

For evaluation, we use sets of synonyms manually ranked for complexity and made available by [START_REF] François | Bleu, contusion, ecchymose : tri automatique de synonymes en fonction de leur difficulté de lecture et compréhension[END_REF]. One example is osseux ! squelettique ! maigre, where words are ordered from more complex to simpler. Forty annotators were asked to order the synonyms according to their reading and comprehension difficulty without any given context. The resource consists of 36 groups of synonyms and 134 lexical units, with a few words being present in more than one group, ranked according to the majority ranking amongst all annotators. The reported Krippendorff's ↵ of 0.399 reflects the difficulty of this task even for humans. Because of the unavailability of word embeddings for multiword expressions and for some synonyms present in the dataset, 1 we remove 13 words and 3 groups of synonyms (for which there was none or only one synonym left after removing the words), which results in a dataset of 33 sets of synonyms and 121 words. 5 of these sets are used as a development set, which results in a test set of 28 synsets and 104 words.

We extract frequency and number of phonemes of lexical items to be used by our models from

Lexique3 [START_REF] New | Une base de données lexicales du français contemporain sur internet : LEXIQUE TM //a lexical database for contemporary french : LEXIQUE TM[END_REF], a large resource with various kinds of lexical information (phonetic transcription, syllabic structure, number of morphemes, etc.) for 142,728 French words.

Model description

We view complexity as a continuous, rather than categorical, notion with highly complex/simple words situated at the two extremes of the spectrum. In order to induce a continuous representation from the three Manulex levels and the six FLElex levels, we implement a simple feed-forward neural network that has as learning objective a number corresponding to the first level of appearance of a word in the corresponding resource. This serves as an indication of the age level or moment of acquisition of the word. The network takes as input features that represent the target word and outputs a real number that indicates its complexity. This way, we obtain a continuous representation of complexity from categorical data, which allows for a more fine-grained ranking than the one found in the original resources.

We experiment with different feature combinations, number of layers and layer sizes on a development set, and choose the best configuration for each feature combination for evaluation (see Section 5). We use three different architectures : one with no hidden layer, similar to logistic regression (no HL) ; one with a hidden layer of size 100 (1 HL) ; and one with two hidden layers of sizes 150 and 100 (2 HL). In all settings, rectified linear unit (ReLU) is used as an activation function and iterations are limited to 100.

For our experiments, we use the following lexical features which have been shown to be useful for predicting complexity in past work (Gala et al., 2014) :

-Number of characters. The hypothesis is that longer words tend to be more complex, and shorter words tend to be simpler. -Number of phonemes. This information is obtained from Lexique3, when available. For words not present in Lexique3, following Gala et al. (2014), we extract the information using eSpeak, an open source speech synthesizer. 2 The hypothesis is the same as for word length measured in terms of characters. We would expect the number of phonemes to be a better indication of word length for French given the abundance of many-to-one grapheme-phoneme correspondences. -Log frequency in a corpus of film subtitles, as encoded in Lexique3. Frequencies in Lexique3 were calculated by averaging the frequency of appearance of a word per million counts in 1. The lexical items with no embedding were : agent de police, droit de grâce, grâce presidentielle, mine de crayon, mine antichar, mine antipersonnel, descente en rappel, empourprer, cramoisir, vagir.

2 French subtitles of 4 types of films and series differing in their original language. Word frequency has been shown to be a very effective predictor of complexity in past work. -300-dimensional word embeddings 3 previously trained on Wikipedia with fastText using the Skip-Gram model [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF]. Word embeddings encode distributional information of words, and we expect them to be able to capture differences in complexity. Our hypothesis is that complex words appear in more complex texts.

Evaluation

We evaluate our systems on the manually ordered synonym lists described in Section 3 measuring the pair-based and rank-based accuracy of the predictions against human judgments. For the pair-based evaluation, we exhaustively create complex ! simple pairs of synonyms from every set of synonyms. For example, from the group forger ! formuler ! former ! inventer, ordered from complex to simple, we derive 6 pairs : (forger ! formuler, forger ! former, forger ! inventer, formuler ! former, formuler ! inventer, former ! inventer), preserving the directionality indicated by the annotators. We expect a good system to assign higher scores to more complex words, producing the correct ordering. We report the proportion of pairs that were correctly ordered by the system.

In the rank-based evaluation, the system has to order a full group of synonyms based on the score that it assigns to each word. We report the percentage of cases in which the system correctly predicts the exact position of a synonym. Following [START_REF] François | Bleu, contusion, ecchymose : tri automatique de synonymes en fonction de leur difficulté de lecture et compréhension[END_REF], we additionally report the proportion of cases where a synonym was placed in the original place or only one position away. For calculating the latter, we exclude groups of synonyms with only two words, which would otherwise always count as correct (there is only one such pair in the development set and one in the test set). In the case of ties between the scores of two synonyms, they are considered to be incorrectly placed, since this means the system is not able to detect the difference in complexity between the two words.

We test the three proposed architectures (with 1 or 2 HL(s), and without HL) on the development sets extracted from Manulex and FLElex and from the manually-ranked synonym resource. We retain as the best model for each feature combination the one that obtains the strongest correlation (Spearman's ⇢) with Manulex and FLElex original classes and the highest accuracy scores in the pair-based and rank-based evaluations on the synonym development set. In the case of two models performing equally or very similarly, we choose the simplest one (i.e. the one with fewer layers).

We compare the results of the best model for different feature combinations to the ones produced by François et al.'s model on the same test set. The size of the dataset used in our evaluation is slightly different from the one used in their original work, since lexical items that had no embeddings were left out. We recompute the accuracy of their system output on the reduced test set. As a baseline, we use the simple frequency feature with no training involved, which considers more frequent words to be simpler and orders them accordingly.

Results and discussion

Results are presented in Tables 2 (Manulex) and 3 (FLElex). The patterns observed and the scores obtained by the systems trained on Manulex and FLElex are, in general, similar. One could expect FLElex to result in a system producing better rankings because of its finer granularity with respect to Manulex, but the much bigger size of the latter probably counteracts this effect.

Among all feature combinations, frequency combined with number of characters or with phonemes seem to be the best predictors of complexity for both Manulex and FLElex. The number of phonemes is the worst performing feature when used on its own, and obtains the lowest score in all measures. This is somewhat surprising, since Gala et al. (2014) show that phonemes are slightly more strongly correlated to Manulex and FLElex classes than characters, which do a bit better in this setting but still are the next poorest performing feature. However, these features alone produce a considerable amount of ties which substantially lower their performance.

The combination of word embeddings with a strong feature such as frequency does not seem to improve the results of embeddings alone on FLElex. The system that relies only on embeddings obtains better results on this dataset than the system using all features. On Manulex, on the contrary, embeddings seem to benefit from being combined with other features. In Table 4 we give an example of a group of synonyms that is correctly ranked by one of the best systems -the one that combines frequency and length in characters (Freq+char) -in the FLElex evaluation, and which the embeddingsbased system ranked incorrectly. The lower part of the table shows a case where the opposite happened. We observe that the systems switch words that are situated next to each other in the rankings and which would be more difficult to rank than words situated in the two extremes, given that they might be similarly complex or simple. The difficulty of the task is highlighted by the low inter-annotator agreement reported by [START_REF] François | Bleu, contusion, ecchymose : tri automatique de synonymes en fonction de leur difficulté de lecture et compréhension[END_REF] which must be partly due to this type of hard-to-rank cases.

The frequency-based baseline is very powerful and is only beaten with a small margin in the rank-based accuracy evaluation by more complex models that have frequency among their features. François et al.

(2016)'s system is one of the best performing models, but whilst it employs 21 different features, its accuracy is slightly below that of simpler frequency-based models. This adds more evidence to the established idea that frequency is a crucial indicator of lexical complexity.

Conclusions

In this work, we have explored the use of word embeddings alone and combined with other features for lexical complexity prediction in French. Our system learns a complexity score from complexity levels based on age of acquisition (Manulex) and second language proficiency (FLElex). The evaluation on a synonym ordering task seems to indicate that whereas word embeddings obtain better results than other features in isolation, frequency-based systems -even a simple frequency baseline -are better suited for this task. The best performing systems use frequency or combine it with characters or phonemes, two features that have been proven to be useful in past work. In the future, we plan to experiment and compare with other kinds of word embeddings built from corpora with a higher level of stylistic variation.

  . When investigating the characteristics of lexical complexity for Spanish, Drndarević & Saggion (2012) conclude that combining frequency

			FLElex
	Manulex Class Original Balanced 1 9,384 9,384 2 8,040 8,040 3 19,421 9,000 Total 36,845 26,424	Class Original Balanced 1 3,617 2,000 2 2,470 2,000 3 3,661 2,000 4 1,178 1,178 5 1,561 1,561 6 455 455
		Total	12,942	9,194

TABLE 1 -

 1 Number of words available in the original Manulex and FLElex databases and in the balanced datasets. Words are classified per first class of appearance in each resource.

TABLE 2 -

 2 Results of the best systems trained on Manulex for each feature combination. Pairand rank-based accuracy is measured on the manually ranked set of synonyms used for evaluation. Correlation is calculated against the ranking of these words in Manulex. The two best scores for each measure are marked in bold.

TABLE 3 -

 3 Results of the best systems trained on FLElex for each feature combination. Pair-and rankbased accuracy is measured on the manually ranked set of synonyms used for evaluation. Correlation is calculated against the ranking of these words in FLElex. The two best scores for each measure are marked in bold.

	Freq + char (correct) dépouiller ! dérober ! piquer ! voler Embeddings dérober ! dépouiller ! piquer ! voler Freq + char mental ! spirituel ! fin Embeddings (correct) spirituel ! mental ! fin

TABLE 4 -

 4 Examples of synonyms that were correctly and incorrectly ranked by one of the best performing systems (Freq + char) and the embeddings-based system, both trained on FLElex.

Acknowledgements

We would like to thank the anonymous reviewers for their thoughtful and constructive comments. We particularly would like to thank Núria Gala and Thomas François for the interesting discussions on French lexical complexity resources, and for sharing with us the manually annotated synonyms dataset used for evaluation. This work has been supported by the French National Research Agency under project ANR-16-CE33-0013.

SPECIA L., JAUHAR S. K. & MIHALCEA R. (2012). Semeval-2012 task 1 : English lexical simplification. In Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1 : Proceedings of the main conference and the shared task, and Volume 2 : Proceedings of the Sixth International Workshop on Semantic Evaluation, p. 347-355.


Taoufik Mhammedi
email: taoufik.mhammedi@univ-lemans.fr

Lionel Camberlein

Frédéric Polet

Bruno Bêche

Etienne Gaviot

Fast enthalpy-sensing microsystem operating in continuous flow †

Keywords: heat flux transducer, planar thermopile, enthalpy of mixing, conjugated variables

A new microsystem, designed to detect and measure in real time the enthalpy of mixing of two fluid-constituents is presented. A preliminary approach to arrange miniaturized batch-cells allowing detecting enthalpy of dilution or mixing is first discussed. Then, a coherent rationale leading to structure devices operating in real time is formulated, considering the straightforward assessment of heat flux transducers (HFTs) capability. Basic thermodynamic observations regarding analogy between thermal and electrical systems are highlighted prior consideration of practical examples involving mixing of water and alcohols. Fundamentals about HFTs design are highlighted before presenting an adequate way to integrate both functions of mixing and measuring the entailed heat exchange as two continuously flowing fluids interact with each other. Then, a prototype of such a dedicated device is discussed with its relevant expected performance.

Introduction

As regard industrial processes, assessing the heat exchanges occurring as two fluid constituents are mixed together is a most common requirement, especially with miniaturized systems. Being a state variable and the intensive quantity associated with thermal energy, the absolute temperature is classically the only prominent quantity to be assessed by way of calorimetry. The extensive quantity conjugated with temperature is the entropy current that should be of most valuable interest insofar as dedicated sensors would be available [START_REF] Bejan | Entropy generation through heat and fluid flow[END_REF]. However, on a metrological standpoint, when small changes of temperature are involved, heat flow meters (HFMs) may supply quite relevant intelligence especially as regards detection of weak thermal changes [START_REF] Shirtliffe | Guarded Hot Plate and Heat Flow Meter Methodology[END_REF]. For instance, considering phase transitions with quasi-isothermal plateaus, the HFM approach have already proved substantial metrological advantages [START_REF] Gaviot | Metrological prospects for the assessment of transition plateaus[END_REF][START_REF] Gaviot | Towards a thermodynamic assessment of transition plateaus[END_REF]. In this paper we highlight a valuable way to take advantage of the planar thermopile constitutive of a generic HFM (fig. 1) so as to design a sensitive area fitted for measuring weak thermal exchanges otherwise hardly detectable in terms of ratio signal/noise. Then we describe a prototype (low impedance self-generating sensor) featuring a specific thermopile overlaid with micro-SU-8 channels [START_REF] Giordani | Design and experimental validation of SU-8 based Micro-psychrometers[END_REF] arranged on a glass substrate (fig. 2 & [START_REF] Gaviot | Metrological prospects for the assessment of transition plateaus[END_REF]. As examples of specifications, processes of mixing water + ethanol and water + propan-2-ol are considered [START_REF] Boyne | Enthalpies of mixing of Ethanol and Water at 25°C[END_REF][START_REF] Peeters | Endothermicity or exothermicity of water/alcohol mixtures[END_REF].

Materials and Methods

Considering calibrating, specifications for designing effective devices rely on data available in the literature, especially as regard fluid constituents to be mixed. As a mandatory requisite the constituents experimented with should be devoid of any kind of interaction with the materials constituting the systems. Prototypes involving Depron (=0.027 W/mK), Polylactic acid (PLA: =0,13 W/mK), Polyethylene terephthalate (PET: =0.14 W/mK), SU8 (=0.15 W/mK), Glass (=1 W/mK), proved rugged enough with mixings of water with alcohols. However, PLA was advantageously replaced with PET, both materials being easily shaped with a 3D printer.

Prerequisite for calibration: suitable fluids and ways to handle their mixing

Both Ethanol and Isopropyl-alcohol (IPA) mixed with water have been investigated, considering references 6and 7. Products used in this paper were all "for analysis" quality products provided by Prolabo and mixed with deionised water (10.6 M.cm). Taking advantage of the Peeters&Huyskens polynomial model, the excess enthalpies of mixing for any case of mixing may be calculated with regard to the following fitting equation where XW,A denote respective mole fraction of water and alcohol [with XW=(1-XA)]:

2 A w 1/2 A w 1/1 A 6 w 6/1 mix X X C X X C X X C H           (1)
Considering respective masses of water M added with alcohol m and the entailed ratio r=m/M, the mole fraction of alcohol may be written as: (

Then, with regard to specific heat to energy balance, either the effective exchanged heat Q (J) or its related steady state heat flow (W) may be given with:

           water alc MW MW M m H Q mix (3a),                         water water water alco alc alc MW t V MW t V / / H dt dQ mix (3b)
Indeed, as peristaltic pumps were used to introduce fluids within the prototypes, m and M values have to be inferred from their respective volumetric quantities with _IPA= 0.785 g/cm 3 and _Ethanol= 0.789 g/cm 3 .

Generic design for monitoring excess enthalpies

Heat flow meters (HFMs) are nowadays quite common self generating sensors [START_REF] Shirtliffe | Guarded Hot Plate and Heat Flow Meter Methodology[END_REF][START_REF] Gaviot | Metrological prospects for the assessment of transition plateaus[END_REF] allowing to measure heat transfers with low noise due to their reduced output impedance (Z<100). A generic structure, based on a planar thermopile, is depicted in Figure1. Then, we investigated mixings with a first family of devices fabricated with PET 3D printing units relying on such sensors. In any case an isothermal condition must be imposed on both liquids to be mixed. A mixing chamber (whose top is thermally insulated) was arranged on the upper face of a HFM whose other face was assembled on a heat sink.

Considering a planar thermopile structure involving N couples of thermojunctions, the amplified Seebeck output voltage representative of the heat current [mes (W)] proceeding through the acting surface is:

Vout = G. RS. = G.N.. , ( 4
)
where G is the gain of the amplifier (50<G<1000), Rs the responsivity of the HFM (V/W), the relative Seebeck coefficient (38µV/K) and  the local elementary differences in temperature available between two adjacent thermonctions as depicted in Figure 1a. Although quite liable, such systems based on HFMs featured a major drawback due to internal turbulence within the mixing chamber: then, a significant enhancement regarding the operating principle had to be considered.

Specific design with micro technology

Any extraneous noise due to uncontrolled turbulence may be viewed as a significant drawback while considering the most common control processes where the flow of a solute has to be adjusted for a given flow of the solvent, the enthalpy of mixing being determined with prior assessments. To overcome such an issue, it is possible to locally ensure the mixing process upon one thermojunction out of two of the thermopile as depicted in Figure 2, the other one being at the reference temperature that of both constituents before mixing. Then, we may now consider the thermopile output Seebeck e.m.f. given with the right term of Equation 4: it comes up as representative of a statistics on thermoelectric voltages localized on the numerous sets of thermojunctions: we verified that it could be considered as a valuable output signal practically associated with the mixing heat flow described with Equation 3b. Then, regarding a given mixing process in steady state with an expected theoretical heat flow rate  for a fixed water flow setpoint value, we may consider after amplification:

Vout = G.., (5)
with  (V/W) an apparatus-related constant depending on the design of the thermopile and its overlaying channels driving both constituents.

Results and brief discussion: Technological achievement and example of output signals

As depicted in Figure 3 the above mentioned thermopile is arranged on a glass substrate and walls made of SU8 allow an apt repartition of both constituents to be mixed together.

  ratio of molecular weights.

Figure 1 .

 1 Figure 1. HFM fundamentals: (a) Cross section view of a generic structure. (b) Arrangement of a specific golden plated planar thermopile deposited on a glass substrate.

Figure 2 .

 2 Figure 2.Main principles for constitutive parts of the self-generating microsystem ( Zout< 2k).

  Considering a controlled water flow fixed at 7.2  0.2 ml/min, we illustrate output signals volumetric flow values obtained with IPA and Ethanol in Figure4b. With such a configuration the  parameter binding the thermopile emf and the theoretical flow is:  = 2.083.10 -4 V/W. However, we must underline a slight -dependence on the water flow set point values.

Figure 3 .Figure 4 .

 34 Figure 3. (a) Layout of the prototype highlighting: the glass substrate (th = 1W/m.K), the Au-Constantan plated planar thermopile (N=360), the SU8 walls 400µm in height allowing to drive and mix both fluid-constituents, and the electrical terminals. (b) Global view of the device with both fluids inputs: the output being on the other side is here invisible.

Proceedings2018, 2, x


M Ar Ianna Ap

Dian Ak

Measuring the adequacy of cross-lingual paraphrases in a Machine Translation setting

Keywords: Cross-Lingual Word Sense Disambiguation, Cross-Lingual Lexical Substitution, paraphrasing, Machine Translation

Following the growing trend in the semantics community towards models adapted to specific applications, the SemEval-2 Cross-Lingual Lexical Substitution and Word Sense Disambiguation tasks address the disambiguation needs of Machine Translation (MT). The experiments conducted in this study aim at assessing whether the proposed evaluation protocol and methodology provide a fair estimate of the adequacy of cross-lingual predictions in translations. For this purpose, the gold SemEval paraphrases are fed into a state-of-the-art MT system and the obtained translations are compared to paraphrase quality judgments based on the source context. The results show the strong dependence of cross-lingual paraphrase adequacy on the translation context and cast doubt on the contribution that systems performing well in existing evaluation schemes would have on MT. These empirical findings highlight the importance of complementing the current evaluation schemes with translation information to allow a more accurate estimation of the systems impact on end-to-end applications.

Introduction

An important trend in computational semantics in recent years is the adaptation of inventories, models and evaluations to specific applications. In this vein, the Cross-Lingual Lexical Substitution (CLLS) and Word Sense Disambiguation (CL-WSD) tasks of SemEval-2 address the disambiguation needs of multilingual applications: what is being evaluated is the capacity of the participating systems to provide semantically correct translations for words in context that could, among others, constitute the input of Machine Translation (MT) systems [START_REF] Mihalcea | Semeval-2010 task 2: Cross-lingual lexical substitution[END_REF][START_REF] Lefever | Semeval-2010 task 3: Cross-lingual word sense disambiguation[END_REF]. 1 The underlying assumption is that the closer the output of a CLLS/CL-WSD system is to a manually built gold standard of cross-lingual paraphrases, the higher its contribution in a real application will be.

Paraphrasing is highly useful in MT as is shown by the substantial amount of research undertaken on the subject. 2 It permits to deal with out-of-vocabulary words [START_REF] Callison-Burch | Improved statistical machine translation using paraphrases[END_REF][START_REF] Marton | Improved Statistical Machine Translation Using Monolingually-Derived Paraphrases[END_REF], capture lexical variation during evaluation [START_REF] Zhou | Re-evaluating machine translation results with paraphrase support[END_REF][START_REF] Owczarzak | Contextual bitext-derived paraphrases in automatic mt evaluation[END_REF], expand the set of reference translations for minimum error rate training [START_REF] Madnani | Using Paraphrases for Parameter Tuning in Statistical Machine Translation[END_REF] and improve the general performance of MT systems [START_REF] Max | Example-based paraphrasing for improved phrase-based statistical machine translation[END_REF]. It is however interesting that in spite of the MT orientation of the CL SemEval-2 tasks, translation selection and evaluation are carried out by reference solely to the source language. The target language context which plays an important role in lexical selection in statistical MT systems, as highlighted by the strong influence of the language model on word choice, is not considered.

In this work, we explore the role of the target language in CLLS and CL-WSD by measuring the adequacy of CL paraphrases in translations. Our goal is not to estimate the impact of semantics in MT, as was the case in previous works on the subject [START_REF] Carpuat | Improving Statistical Machine Translation using Word Sense Disambiguation[END_REF][START_REF] Chan | Word sense disambiguation improves statistical machine translation[END_REF], but to empirically test the adequacy of the sense descriptions provided in the CL evaluation tasks in an MT setting. The paper is organized as follows. The CL SemEval-2 tasks are described in Section 2. The adopted experimental methodology and evaluation setup are presented in Section 3. The analysis of the obtained results, in Section 4, highlights the importance of the target language context for CLLS and CL-WSD, and the implications of these findings for CL semantic evaluations.

Translation context in cross-lingual semantic evaluations

The SemEval-2 Cross-Lingual tasks

In the CLLS and CL-WSD tasks of the SemEval-2 evaluation campaign, the participating systems had to predict semantically correct translations in different languages for English target words in context [START_REF] Mihalcea | Semeval-2010 task 2: Cross-lingual lexical substitution[END_REF][START_REF] Lefever | Semeval-2010 task 3: Cross-lingual word sense disambiguation[END_REF]. The performance of the systems was measured by comparing their output to a manually built gold standard (GS) of cross-lingual paraphrases. For example, the instance of the target word fresh in sentence #952 of the CLLS test set: "At first the user is impressed by the fresh clean smell coming out of the machine and how nice it makes their home smell.", was tagged by the following set of translations which express the sense of fresh in Spanish: fresco 4; puro 1; flamante 1; limpio 1; nuevo 1. GS translations are lemmatized and the frequency counts indicate the number of annotators that proposed each substitute.

The differences between the two tasks mainly lie in the targeted lexical samples and the involved language pairs. CLLS addresses words of all open-class parts of speech in one language pair (English-Spanish) while CL-WSD focuses on the translation of English nouns in five languages (French, Spanish, German, Dutch and Italian). 3 Another point of variation concerns the definition of senses. In CL-WSD, target word senses were described by means of clusters of their semantically similar translations [START_REF] Ide | Sense discrimination with parallel corpora[END_REF][START_REF] Apidianaki | Translation-oriented sense induction based on parallel corpora[END_REF]. More precisely, the translations of the target words in the Europarl corpus [START_REF] Koehn | Europarl: A Parallel Corpus for Statistical Machine Translation[END_REF] were manually clustered and the obtained clusters served for tagging. On the contrary, CLLS did not involve a clustering step and the annotators could propose translations found in any external resource. The CLLS test set was built from the English Internet Corpus [START_REF] Sharoff | Open-source corpora: Using the net to fish for linguistic data[END_REF] while CL-WSD test sentences were extracted from the BNC4 and the JRC-ACQUIS corpus [START_REF] Steinberger | The jrc-acquis: A multilingual aligned parallel corpus with 20+ languages[END_REF].

Translation context: a neglected parameter

Although the CL SemEval tasks are clearly oriented towards MT, annotator judgments and system suggestions are made on the basis of source language information. Translations are selected so as to express the meaning of the target words in the target language but the translation context in which they would be used has no influence on the selection process. This lack of target language information would have a minimal impact in settings where CLLS/CL-WSD systems serve to assist human users, but becomes more important in the context of MT where the proposed CL paraphrases have to be automatically filtered to select the most adequate translation. This selection is not straightforward for several reasons.

Words that seem interchangeable on the basis of formal criteria, such as distributional similarity, might not be substitutable in real texts because of other parameters preventing the substitution (e.g. syntactic structure, collocations). In a translation setting where the substitution is done cross-lingually, it is important that the paraphrases preserve both the sense of the original word (or phrase) and the fluency of the translated text. However, clustered translations are usually near-synonyms translating the same sense, but almost never absolute synonyms interchangeable in translations [START_REF] Edmonds | Near-synonymy and lexical choice[END_REF][START_REF] Apidianaki | Data-driven semantic analysis for multilingual WSD and lexical selection in translation[END_REF]. Consequently, although CLLS and CL-WSD could greatly contribute in MT by enhancing the semantic relevance of translations, the existing evaluations do not provide a fair estimate of the systems' capacity to propose translations that would also fit well in the translated texts.

We conduct a series of experiments to assess the adequacy of CL paraphrases in translations by exploiting the CLLS and CL-WSD test sets. As the two test sets were mainly built from monolingual corpora, no reference translations are available against which the quality of the CL paraphrases could be measured using standard MT evaluation metrics (BLEU, METEOR, etc.). So, we adopt a variation of the substitution-based approach used in works on paraphrasing [START_REF] Bannard | Paraphrasing with bilingual parallel corpora[END_REF] for validating candidate paraphrases, based on the assumption that items deemed to be paraphrases may behave as such only in some contexts and not in others. We translate the CLLS and CL-WSD test sets with a state-of-the-art MT system by exploiting the manually-defined GS paraphrases. Once the set of translations for each test sentence is produced, we measure the substitutability of the GS paraphrases using an automatic and a human ranking, as explained in the next section.

3 Experimental setup

Systems and data

The CLLS and CL-WSD test sets are translated into Spanish and French, respectively, using the baseline system of the WMT-2011 shared task (Moses) [START_REF] Koehn | Moses: Open source toolkit for statistical machine translation[END_REF]. The two MT systems are trained on the data released for WMT-2011 for the two language pairs, namely the French-English and Spanish-English parts of Europarl (version 6) [START_REF] Koehn | Europarl: A Parallel Corpus for Statistical Machine Translation[END_REF]. The language models used during decoding are trained on the monolingual Spanish and French parts of Europarl. For each test sentence, we constrain the decoder to produce translations by using all GS paraphrases. These are plugged into Moses using its 'XML Markup' feature which allows to specify translations for parts of the input sentence. The 'exclusive' mode is activated which forces the decoder to use the XML-specified translations and ignore any phrases from the phrase table that overlap with that span. 5 In total, 4,791 unique Spanish translations are produced for the CLLS test set and 4,220 French translations for the CL-WSD test set.

The GS paraphrases are lemmatized, so we first produce translations at the lemma level without dealing with inflections. At this stage, the test sentences are lemmatized and the MT systems are trained on lemmatized bi-texts. The CL-WSD test set is also translated into French using inflections. We gather all the inflectional variants of each paraphrase found in the training bi-text and provide them to Moses through the XML markup. For instance, to translate the test sentence: "Taking with determination this road leading to a dynamic European Union on the world scene will yield further substantial benefits to all parties involved in the EU and beyond." we provide all inflected forms of each GS paraphrase found in Europarl: scène/scènes, niveau/niveaux, marché/marchés, etc. The MT system then selects the best inflection depending on the surrounding context, as shown in Table 1.

Automatic ranking

The set of lemmatized translations produced by Moses for each test sentence is ranked by a target language model (lm). Language model scores reflect the probability of the sentences formed by substituting paraphrases and are useful for ranking candidate paraphrases in automatic paraphrasing tasks. [START_REF] Bannard | Paraphrasing with bilingual parallel corpora[END_REF], for example, combine a language model probability with a paraphrase probability to rank candidate paraphrases produced by the pivot method. 6 The use of a language model allows to account for the fact that the best paraphrase might vary depending on information about the sentence it appears in and lets the surrounding words in the sentence influence paraphrase ranking and selection.

We build two extended lms (in Spanish and French) using additional monolingual data compared to that used for training the lms used by Moses. The training data comprises Europarl, the News Commentary corpus and the 2009, 2010 and 2011 News Crawl data provided at the WMT-11 shared task for the two languages. We employ the SRILM toolkit [START_REF] Stolcke | SRILM -an extensible language modeling toolkit[END_REF] to compute two 5-gram language models and, subsequently, to score and rank the translations produced by Moses. As the use of different GS paraphrases may alter the context of the translated sentences normalized lm scores are used, defined as 1 n log(P), where n is the length of the translation prendre avec détermination cette voie conduisant à une dynamique de l' union européenne sur la scène mondiale enregistre encore des avantages substantiels pour toutes les parties concernées dans l' ue et au-delà .

2 niveau (2) ... menant à une union européenne dynamique au niveau mondial engendrera davantage des avantages substantiels pour toutes les parties concernées ...

2.13

vie (2) ... conduisant à une dynamique de l' union européenne sur la vie apportera des avantages substantiels pour toutes les parties impliquées ...

1.8

marché (1) ... conduisant à une dynamique de l' union européenne sur le marché mondial engendrera davantage des avantages substantiels pour toutes les parties concernées ...

1.96

plan (1) ... menant à une union européenne dynamique sur le plan mondial engendrera davantage des avantages substantiels pour toutes les parties concernées ...

2.09

Table 1: Ranking of Moses translations using GS paraphrases and lm scores.

and P the language model probability. Table 1 shows the normalized lm scores of the set of translations produced for the test sentence given in the previous section. 7 The lm ranking is compared to the GS one which reflects the semantic relevance of the paraphrases as estimated by reference to the source context. Our hypothesis is that a high correlation between the two rankings would indicate that translations privileged in the GS (i.e. with a high frequency) would serve to produce fluent translations (i.e. with better lm scores). Given the important role of lms in lexical selection, the low ranking of paraphrases could be interpreted as denoting their lower chances of being used in translations. However, this judgment cannot be absolute as the language model is one among other components that determine lexical choice in MT systems.

Human ranking

Although the lm scoring yields interesting results, we consider that it is not reliable enough to lead to safe conclusions as to the adequacy of CL paraphrases in translations. So, we also conduct a human evaluation. The annotators are asked to rank the set of Moses translations produced for each target word instance on a 3-point scale, according to the adequacy of the paraphrases and the fluency of the translated text.8 Good quality paraphrases (i.e. the highest ranked ones, assigned a '1' value) should preserve both the meaning of the source word and the grammaticality of the target sentence. This experiment can be viewed as a substitution test [START_REF] Callison-Burch | Syntactic Constraints on Paraphrases Extracted from Parallel Corpora[END_REF] with the difference that the paraphrases are not just substituted in the translated sentences but fed into the MT system which exploits them during translation. Consequently, the context surrounding the paraphrase might be altered as well, as shown in the examples given in Table 1.

The human ranking covers 538 instances of the CL-WSD test set with an average of 4.17 French paraphrases per instance. The 538 translation sets produced by Moses contain a total of 1821 unique translations and each translation is annotated twice. We calculate the inter-annotator agreement using Cohen's kappa coefficient for three different annotation configurations: the ranking performed using the 3-point scale and two coarser-grained rankings obtained by interpreting intermediate ('2') values as denoting good or low quality translations (i.e. converting them into '1's or '3's). As shown by the kappa values given in Table 2, agreement on the 3-point ranking is rather low (K = 0.35) but it gets higher when the intermediate values are interpreted as 'good' or 'bad'. In the first case kappa is 0.57, which is considered as substantial agreement, but it reaches its highest value (K = 0.72) when medium-ranked translations are considered as low quality ones (2!3). This practically means that in most cases both annotators perceive a problem in the translated texts but have a different estimate of its severity. The increase of the kappa value when a scale with fewer points is used is natural and has been observed in other works on paraphrasing.9 Examples of human-ranked translation sets are given in Table 3. We observe that mediumranked CL paraphrases, such as the translation charge of the target word strain (assigned values '2' and '3') or the translation parties of the noun side, do not fit well in the translated texts. However, the annotators give some credit to paraphrases that may seem awkward in the translated texts but still carry some of the semantic load of the source word, reserving the lowest values to erroneous translations from both points of view. Given the inadequacy of medium-ranked paraphrases in translations, we consider these judgments as low quality ones and distinguish between two categories. The K = 0.72 agreement obtained in this case is very high, especially for a semantics task like this one.

Results

Gold standard judgments vs language model scores

We calculate the correlation of the two rankings with the GS frequency ranking. We first compute the correlation between the semantic relevance of CL paraphrases, as reflected in the GS frequencies, and their adequacy in translated texts, as measured by the lm. We use the Spearman's rank order correlation coefficient (⇢), a non-parametric test, because the data does not seem to be normally distributed. The Spearman coefficient is defined as the Pearson correlation between ranked variables. To compute the correlation of two random variables X and Y, Pearson's coefficient divides their covariance by the product of their standard deviations.

⇢(X , Y ) = cov(X , Y ) X Y
(1)

To compute Spearman's ⇢, absolute values are transformed into ranks. 10 The correlation between the GS annotations in the French data set and the lm scores on the lemmatized translation dataset is ⇢ = 0.067 and highly significant with p = 1.361e 05 (< 0.05). Spearman correlation with the normalized lm scores is .014 with a p-value of .363. As the dataset with the normalized scores contains ties, we also calculate the Kendall's tau-b non-parametric correlation. Let (x 1 , y 1 ), (x 2 , y 2 ), ..., (x n , y n ) be a set of joint observations from two random variables X and Y , the Kendall's tau coefficient is defined as

⌧ = (|concordant pairs| |discordant pairs|) 1 2 n(n 1) (2)
Concordant is any pair of observations (x i , y i ) (x j , y j ) where the ranks for both elements agree (e.g. x i > x j and y i > y j ), otherwise it is discordant. Kendall's tau-a requires all the values of x i and y i to be unique for the p-value to be accurate, but Kendall's tau-b accounts for ties (i.e. pairs of observations where x i = x j or y i = y j ) . 11 The Kendall's tau-b correlation between the GS ranking and the normalized lm scores is low: .011 (p = .363). This lack of correlation could mean in practical terms that the best paraphrases from a semantics point of view would not lead to more fluent translations. To draw safer conclusions we present in the next section the results obtained by the human ranking.

The correlation between GS estimates and the unnormalized lm scores for Spanish is ⇢ = 0.0242, with lower significance than in French (p = 0.09). Given the similar size of the test sets in the two languages, this divergence might be due to the higher homogeneity of the French dataset which contains only nouns. Words of different parts of speech, found in the Spanish test set, are handled differently by the annotators and their paraphrases have a varying impact on translation fluency. The correlation computed between the Spanish GS scores and the normalized lm scores is low as well, with ⇢ = .005 (p = .726) and a Kendall's tau-b value of .004 (p = .723).

Gold standard vs target language human judgments

The dataset that consists of the GS frequency estimates and the human judgments of translation adequacy contains ties, so we calculate the Kendall's tau-b correlation. We use the values assigned in the first annotation pass. The obtained correlation is .271, for the 3-point scale (negative because the values in the two rankings are inverted), and .26 for the 2-point scale (conversion 2!3). Both correlations are significant at the 0.01 level. The 3-point scale judgments correlate slightly better with the GS ones because they are rated on the same scale. 12These results show that paraphrases privileged in the GS do not fit well in the translated texts, while translations ranked low in the GS might be preferred in translations.

We finally calculate the correlation between the human ranking and the normalized lm scores on unlemmatized translations. Kendall's tau-b correlation is .018 and .033, for the 3 and the 2-point scale respectively, but the p-values are quite high (.334 and .091). It would be interesting to repeat this correlation experiment once more annotated examples will be available. A detailed analysis of this discordance would provide valuable hints on the capacity of lms to measure fluency and paraphrase adequacy. We observe, for instance, that the annotators often base their judgments on the context surrounding the paraphrases although lm scores are computed on the entire sentences that might be altered during translation. Nevertheless, the fact that these correlation results are not yet safe does not influence the conclusions that can be drawn from the low correlation observed between the gold standard ranking and the human ranking of translation adequacy, which is highly significant.

Conclusion

The findings of this study reveal that the results of the CL SemEval-2 tasks are not indicative of the contribution that the participating systems would have in MT. It has been shown that although the proposed evaluation metrics address the semantic relevance of CL paraphrases, they do not account for their suitability in translations. These empirical results highlight the importance of integrating translation information in CL semantic evaluations by resorting either to simplified translation tasks [START_REF] Vickrey | Word-Sense Disambiguation for Machine Translation[END_REF] or to full-fledged MT systems. Evaluation metrics capable of rewarding semantically correct translations that do not distort the fluency of the translations are much needed in the field of MT for evaluating the output of MT systems and the contribution of disambiguation modules. Another perspective worth exploring is the set up of all-words CL evaluation tasks, in addition to the lexical sample ones, allowing to assess the global capacities of CLLS and CL-WSD systems and the coverage they can attain in real-life applications. This setting would also permit to explore the potential of collaboration between CL-WSD modules and MT systems for correct lexical selection.

  Inter-annotator agreement.

Table 3 :

 3 Manually ranked translations.

These systems can also help human translators in their work and assist language learners.

See[START_REF] Madnani | Generating Phrasal and Sentential Paraphrases: A Survey of Data-driven Methods[END_REF] for a comprehensive survey of data-driven methods for paraphrase generation.

The CLLS lexical sample is composed of 300 noun, 310 verb, 280 adjective and 110 adverb instances with approximately 5 Spanish substitutes per target word and a pairwise inter-annotator agreement of 0.2777. The CL-WSD test data contains 50 instances of 20 target nouns and their substitutes in five languages.

http://www.natcorp.ox.ac.uk/

The 'inclusive' mode allows phrase table entries to compete with the XML entry. This configuration permits to define probabilities for the provided translation choices and leave the final selection to the target language model.

In the pivot method, phrases in one language are considered to be potential paraphrases of each other if they share a translation in another language. The paraphrase probability is defined in terms of the translation model probabilities that the original phrase translates as a particular phrase in the other language.

Normalized scores are rounded to two decimal places. Translations with lower scores are considered as more fluent.

The annotators are native and highly proficient French speakers working on MT and paraphrasing.

[START_REF] Callison-Burch | Syntactic Constraints on Paraphrases Extracted from Parallel Corpora[END_REF] reports a kappa agreement of 0.33 when a 5-point scale is used and an agreement of 0.61 with a 2-point scale. The scale conversion is performed by measuring agreement in terms of how often the annotators assigned a value higher or lower than a pre-defined threshold.

The analysis is done using the R package: http://www.r-project.org

GS paraphrases were assigned frequencies from '1' to '3'; only two cases were assigned a '4' value.

Acknowledgments

We would like to thank the annotators for the time and effort they put into this task. This work was partly funded by the Cap Digital SAMAR project.


Lonneke Van Der Plas
email: lonneke.vanderplas@ims.uni-stuttgart.de

Marianna Apidianaki
email: marianna.apidianaki@limsi.fr

Cross-lingual Word Sense Disambiguation for Predicate Labelling of French

Keywords: transfert inter-langue, annotation sémantique automatique, prédicats, désambiguïsation lexicale, corpus parallèles cross-lingual transfer, automatic semantic annotation, predicates, Word Sense Disambiguation, parallel corpora

Nous abordons la question du transfert d'annotations sémantiques, et plus spécifiquement d'étiquettes sur les prédicats, d'une langue à l'autre sur la base de corpus parallèles. Des travaux antérieurs ont transféré ces annotations directement au niveau des tokens, conduisant à un faible rappel. Nous présentons une approche globale de transfert qui agrège des informations repérées dans l'ensemble du corpus parallèle. Nous montrons que la performance de la méthode globale est supérieure aux résultats antérieurs en termes de rappel sans trop affecter la précision.

Introduction

There has recently been a large interest in multilingual natural language processing. Several annotation efforts have been devoted to developing resources for different languages, needed for supervised learning (Hajič et al., 2009). However, there is a large number of languages that still lack linguistically annotated resources. For example for French, one of the most important European languages, there exists no corpus with predicate-argument annotations.

Predicate-argument annotations are the representation of the grammatically relevant aspects of a sentence meaning. This level of analysis provides a means of expressing a relation between syntactically different sentences, such as the sentence with the transitive verb in (1a) and the one with the intransitive verb in (1b). The semantic label theme expresses the fact that the object in (1a) has the same conceptual relation with the verb as the subject in (1b).

(1) a. Semantic parsing or semantic role labelling refers to the task of automatically labelling predicates and arguments with predicate-argument structure. This task can be divided in two parts. One part refers to the labelling of the predicates with a predicate sense, and the other to the labelling of the arguments with semantic roles. Semantic parsers and the results of cross-lingual annotation transfer are therefore evaluated on both tasks : predicate labelling and role labelling. We focus on predicate labelling in this paper.

Cross-linguistically, the predicate-argument structure of a sentence is considered to be more stable than its syntactic form. The English sentence in (2a) can be considered as equivalent to the French sentence in (2b), despite the fact that the position of their syntactic subject is occupied by different kinds of lexical elements and that the complements of the verbs differ both syntactically and semantically.

( Since manual annotation is a costly and time-consuming approach to resource development, cross-lingual annotation transfer offers an attractive alternative. In this approach, predicate-argument annotations on a source language, for which there exists semantic annotations, are transferred to a target language using parallel corpora [START_REF] Padó | Cross-lingual Annotation Projection Models for Role-Semantic Information[END_REF][START_REF] Basili R | Computational Linguistics and Intelligent Text Processing, chapter Cross-Language Frame Semantics Transfer in Bilingual Corpora[END_REF][START_REF] Annesi | Cross-lingual alignment of FrameNet annotations through Hidden Markov Models[END_REF][START_REF] Van Der Plas L | Scaling up cross-lingual semantic annotation transfer[END_REF]. The increased stability of predicate-argument structures across languages makes their cross-lingual transfer attractive when compared with, for example, syntactic structures.

Traditional methods for cross-lingual transfer rely on the semantic equivalence of the original and the translated sentences, and on correct and complete alignments between words or constituents in those sentences. Since the semantic annotations are transferred directly from token to token, we will refer to these traditional methods as direct cross-lingual transfer.

Alignment errors and translation shifts represent major sources of mistakes in the direct transfer approach which result in incorrect and incomplete annotations in the target language.

In this paper, we propose a different strategy to predicate labelling that is less sensitive to alignment errors and translation shifts. Instead of transferring semantic annotations on a token-to-token basis, we aggregate information across the whole parallel corpus to correct token-level mistakes resulting from direct cross-lingual transfer. Our approach consists of two steps : In the learning step, a global model is learned on the basis of source language (English) predicate annotations in a word-aligned English-French training corpus. In the labelling step, this model assigns predicate labels to verbs in target language texts (here, French). We model cross-lingual transfer of predicate labels as a cross-lingual word sense disambiguation (WSD) task, because this fits well with the lexical nature of the task : annotating French verbs with English predicate labels.

Our contributions are two-fold. First, we present a global approach to semantic annotation transfer that corrects tokenlevel mistakes as found in traditional direct transfer methods. Second, we show the strengths and limitations of global vs direct transfer.

In the next section, we present related work on cross-lingual annotation transfer. We then briefly discuss the semantic annotation framework we are using and explain why we decided to use an English semantic framework for annotating French. In Section 4, we briefly present the direct transfer method. In Section 5, we describe the tools and data we use and explain the adopted evaluation framework. In Section 6, we explain the global method proposed in this paper. The results are presented in Section 7, before concluding.

Related work

Transferring annotations from one language to another in order to train monolingual tools for new languages was first introduced by [START_REF] Yarowsky | Inducing multilingual pos taggers and np bracketers via robust projection across aligned corpora[END_REF]. In their approach, token-level part-of-speech (PoS) and noun phrase bracketing information was projected across word-aligned bitext and this partial annotation served to estimate the parameters of a model that generalized from the noisy projection in a robust way. In more recent work, [START_REF] Das | Unsupervised part-of-speech tagging with bilingual graph-based projections[END_REF] propose a graph-based framework for projecting syntactic information across language boundaries. They create type-level tag dictionaries by aggregating over projected token-level information extracted from bi-text and use label propagation on a similarity graph to smooth and expand the label distributions. A different approach to cross-lingual PoS tagging is proposed by [START_REF] Täckström | Token and type constraints for crosslingual part-of-speech tagging[END_REF] who couple token and type constraints in order to guide learning. These two types of information are viewed as complementary : token-level projections offer precise constraints for tagging in a particular context while broad coverage type-level dictionaries help to filter noise in token-level projections. Our approach to crosslingual predicate labelling follows this vein. Instead of solely relying on token-level information acquired from wordalignments, we combine this with type-level information captured by our global method which is trained on the entire corpus. We, however, are concerned with semantic annotations and not PoS tags.

Transfer of semantic annotation has started off with direct transfer of FrameNet semantic annotations [START_REF] Padó | Cross-lingual Annotation Projection Models for Role-Semantic Information[END_REF][START_REF] Basili R | Computational Linguistics and Intelligent Text Processing, chapter Cross-Language Frame Semantics Transfer in Bilingual Corpora[END_REF][START_REF] Annesi | Cross-lingual alignment of FrameNet annotations through Hidden Markov Models[END_REF]. With the addition of a learning step and the use of PropBank data, [START_REF] Van Der Plas L | Scaling up cross-lingual semantic annotation transfer[END_REF] have scaled up previous efforts. They show that a joint semantic-syntactic parser trained on the output of direct transfer and additional syntactic annotations produces better parses than the input it received by aggregating information across multiple examples. Our method is more resource-light, as we do not need syntactic annotations neither on the target nor on the source side.

The same emphasis on learning is found in cross-lingual model transfer, where source language models are adapted to work on the target language directly. For semantic role labelling, [START_REF] Kozhevnikov | Crosslingual transfer of semantic role models[END_REF] use shared feature representations (syntactic and lexical) to adapt a source model to a target-language model. They, however, do not consider the task of predicate labelling but only semantic role labelling.

In this work, we address predicate labelling in languages other than English as a cross-lingual WSD task. Word sense disambiguation is the task of automatically identifying the meaning of words in context [START_REF] Navigli R | Word Sense Disambiguation : a Survey[END_REF]. In its cross-lingual variant, the candidate senses are the words' translations in other languages and WSD aims at predicting semantically correct translations for instances of the words in context [START_REF] Resnik | Distinguishing Systems and Distinguishing Senses : New Evaluation Methods for Word Sense Disambiguation[END_REF][START_REF] Ng | Exploiting Parallel Texts for Word Sense Disambiguation : An Empirical Study[END_REF][START_REF] Carpuat | Improving Statistical Machine Translation using Word Sense Disambiguation[END_REF][START_REF] Apidianaki | Data-driven Semantic Analysis for Multilingual WSD and Lexical Selection in Translation[END_REF]. In our experiments, we apply the cross-lingual WSD method employed by [START_REF] Apidianaki | WSD for n-best reranking and local language modeling in SMT[END_REF] for improving the quality of Machine Translation, in a different setting : instead of assigning semantically appropriate translations to words in context, the WSD classifier serves for selecting the most adequate English predicate label for verbs in a new language. More details on the adaptation of the method to predicate labelling are given below, in Section 6 of the paper.

The semantic annotation framework

There exist three frameworks for annotating corpora with predicate-argument structure : FrameNet [START_REF] Fillmore | Background to FrameNet[END_REF], VerbNet [START_REF] Kipper | VerbNet : A broad-coverage, comprehensive verb lexicon[END_REF] and PropBank [START_REF] Palmer | The Proposition Bank : An annotated corpus of semantic roles[END_REF]. We chose PropBank for applying our predicate labelling method. In the following, we describe PropBank in more detail as well as the criteria that led to choosing this framework as well as how the English PropBank can be used to annotate French.

The Proposition Bank

The Proposition Bank (PropBank) is a linguistic resource that contains information on the semantic structure of sentences [START_REF] Palmer | The Proposition Bank : An annotated corpus of semantic roles[END_REF]. It consists of a one-million-word corpus of naturally occurring sentences annotated with semantic structures and a lexicon (the PropBank frame files) that lists all the predicates (verbs) that can be found in the annotated sentences and the sets of semantic roles they introduce.

Predicates are marked with labels that specify the sense of a verb in a particular context. Each lemma described in the frame files (3300 verbs) contains one or more lexemes (4500 verb senses), which are used as predicate labels. The PropBank frame files specify the interpretation of the roles for each verb in its different senses. The interpretation of the numbered roles is given for each lexeme separately. Table 1 illustrates the entry for the verb pay in the PropBank frame files.

The semantic role annotation is based on Dowty's theory of Proto-Roles [START_REF] Dowty | Thematic proto-roles and argument selection[END_REF]. Arguments are marked with the labels A0 to A5, which represent semantic roles of a very general kind. Only the labels A0 and A1 have approximately the same value with all verbs : they are used to mark instances of proto-agent (A0) and proto-patient (A1). The meaning of other numbered arguments is verb-dependent. It depends on the meaning of the verb, on the type of the constituent they are assigned to, and on the number of roles present in a particular sentence. A3, for example, can mark purpose as is the case in (3), or it can mark direction or some other role with other verbs. The indices are assigned according to the roles' prominence in the sentence. More prominent are the roles that are more closely related to the verb. The AM-* labels can be specified further as : location, cause, extent, time, discourse connectives, purpose, general purpose, manner, direction.

The labels for adjuncts are more specific but less verb-specific than the labels for arguments, and they do not depend on the presence of other roles in the sentence.

( Although PropBank is considered as the most language-specific of the three resources, Samardzić et al. (2010) motivate the use of PropBank for annotation transfer with two reasons. First, the lexicon in this resource is corpus-driven. It is built by extracting and describing all the predicates that occur in a predefined sample of naturally occurring sentences. Since the aim is to annotate a corpus of naturally occurring sentences exhaustively, we expect that such a lexicon can provide a better coverage than the lexicon in FrameNet and VerbNet, which are not corpus-driven. Second, the labels used in PropBank both for predicates and arguments involve fewer theoretical assumptions than the labels in FrameNet. While the FrameNet labels capture mostly linguistic intuition at the targeted level of lexical semantics and the relations between the lexical items, the PropBank labels rely strongly on the observable behaviour of words. The distinction between the different verb senses, for instance, is made taking into account the different sets of arguments and other observable differences, such as the presence of the particle that distinguishes pay.04 from pay.05 in Table 1. This approach can be expected to provide more tangible criteria for annotators in deciding how to annotate each instance of the predicate-argument structure found in the corpus, ensuring a more reliable and more consistent annotation. Also, it enables a more direct comparison of the structures across languages, since the representation of the structures does not include any hypothesized levels of abstraction.

Using the English PropBank to annotate French

Adapting a semantic framework to a new language is a time-consuming process. In order to generate broad-coverage annotations for a target language in limited time, we transfer semantic annotations from the source language directly to the target language without adapting the semantic annotation framework to the target language. This means that French verbs will be annotated with English predicate senses. For this to work, we need to show that PropBank is cross-lingually valid. [START_REF] Van Der | Cross-lingual validity of PropBank in the manual annotation of French[END_REF] did this in a manual annotation effort. For a complete description of the annotation procedure, that involved four evaluators and several stages, we point the reader to [START_REF] Van Der | Cross-lingual validity of PropBank in the manual annotation of French[END_REF]. In this section, we summarize the procedure and briefly discuss the main outcomes.

In this manual annotation study, annotators used the English PropBank frame files to annotate French sentences. This means that for every predicate found in a French sentence, they needed to translate it and find an English verb sense that was applicable to the French verb. If an appropriate entry could not be found in the frame files for a given predicate, the annotators were instructed to use the dummy label for the predicate and fill in the roles according to their own insights.

Some of the differences in annotation observed between annotators were due to lexical variation in English. For example, if one annotator put the label 'demonstrate.01' whereas the other used 'show.01' on an instance of the verb montrer, this should not be counted as a disagreement as the two senses are linked. Therefore, agreement scores were provided both on the basis of the predicate sense labels and on the basis of the verb class, using the verb classifications from VerbNet [START_REF] Kipper | VerbNet : A broad-coverage, comprehensive verb lexicon[END_REF] and the mapping to PropBank labels as given in the type mappings of the SemLink project 1 [START_REF] Loper | Combining lexical resources : Mapping between PropBank and VerbNet[END_REF]. VerbNet is a hierarchically organised verb lexicon of English verbs [START_REF] Kipper | VerbNet : A broad-coverage, comprehensive verb lexicon[END_REF]. It is organized into verb classes extending [START_REF] Levin | English Verb Classes and Alternations : A preliminary investigation[END_REF] classes through refinement and addition of subclasses to achieve syntactic and semantic coherence among members of a class. The senses 'demonstrate.01' and 'show.01' appear in the same verb class according to the type mappings of the SemLink project and were thus considered as correct.

The average inter-annotator agreement reported in these experiments was relatively low when the annotations on the PropBank verb sense level were compared : 59%. However, at the level of verb classes, the inter-annotator agreement 1. http://verbs.colorado.edu/semlink/ increased to 81%. The authors identify collocations and idiomatic expressions as the main sources of disagreement in predicate labellings among annotators, as is also shown in studies on other language pairs [START_REF] Burchardt | Multilingual FrameNets in Computational Lexicography : Methods and Applications, chapter FrameNet for the semantic analysis of German : Annotation, representation and automation[END_REF].

For a single annotator, the main measure of cross-lingual validity was the percentage of dummy predicates in the annotation. Van der Plas (2010) found 130 dummy annotations in 1000 sentences. A manual classification of the dummy labels showed that the dummy label was mainly used for French multi-word expressions (82%), most of which could be translated by a single English verb (47%) whereas others could not because they were translated by a combination that included a form of 'be' that was not annotated in PropBank (25%). The 47% of multi-word expressions that received the dummy label showed the annotator's reluctance to put a single verb label on a French multi-word expression. The annotation guidelines could however be adapted to instruct annotators not to hesitate in such cases. Based on these findings, the authors conclude that the annotation framework PropBank is cross-lingually valid.

Direct cross-lingual transfer

Before presenting our global method for predicate labelling, we would like to remind the reader of the method for direct cross-lingual transfer which is used in comparisons and combinations throughout this paper. It is taken from Van der Plas et al. ( 2011), but we give a short summary here for the readers convenience. The method is based on the Direct Correspondence Assumption for syntactic dependency trees by [START_REF] Hwa R | Bootstrapping parsers via syntactic projection accross parallel texts[END_REF].

Direct Semantic Transfer (DST) For any pair of sentences E and F that are translations of each other, we transfer the semantic relationship R(x E , y E ) to R(x F , y F ) if and only if there exists a word-alignment between x E and x F and between y E and y F , and we transfer the semantic property P (x E ) to P (x F ) if and only if there exists a word-alignment between x E and x F .

The properties that are transferred through DST are predicate senses. The relationships that are transferred are semantic role dependencies, but we are not concerned with them in this paper. The properties are transferred from the English side of a parallel corpus that is automatically annotated with syntactic-semantic analyses to the foreign language side, as described in the following section.

Tools and data

The parallel corpus used in our experiments is the English-French part of the Europarl corpus [START_REF] Koehn | Europarl : A Parallel Corpus for Statistical Machine Translation[END_REF]. The English part of the parallel corpus is annotated by a freely-available syntactic-semantic parser [START_REF] Henderson | A latent variable model of synchronous parsing for syntactic and semantic dependencies[END_REF][START_REF] Titov | Online graph planarisation for synchronous parsing of semantic and syntactic dependencies[END_REF] trained on the CoNLL 2009 training set (the Penn Treebank corpus [START_REF] Marcus | Building a large annotated corpus of English : the Penn Treebank[END_REF] merged with PropBank labels [START_REF] Palmer | The Proposition Bank : An annotated corpus of semantic roles[END_REF] and NomBank labels2 [START_REF] Meyers | Annotation guidelines for NomBank -noun argument structure for PropBank[END_REF]). In the experiments presented in this paper, we only use the predicate labels found in the English part of Europarl, omitting the labels assigned to the arguments.

For the direct transfer of semantic annotations from the English to the French side of the parallel corpus, we use the method described in Section 4. As is usual practice in pre-processing for automatic word alignment, both parts of the parallel corpus were tokenised and lowercased and only sentence pairs corresponding to a one-to-one sentence alignment, with lengths ranging from one to 40 tokens on both French and English sides, were considered. We subsequently word aligned the English and French sentences automatically using GIZA++ [START_REF] Och | A systematic comparison of various statistical alignment models[END_REF] in both translation directions and retained only intersecting alignments. Furthermore, because translation shifts are known to pose problems for the automatic projection of semantic annotation across languages [START_REF] Padó | Cross-lingual Annotation Projection Models for Role-Semantic Information[END_REF], we selected only those parallel sentences in Europarl that are direct translations from English to French, or vice versa. In the end, we obtained a word-aligned parallel corpus of 276-thousand sentence pairs.

For testing, we use the hand-annotated data described in [START_REF] Van Der | Cross-lingual validity of PropBank in the manual annotation of French[END_REF]. We randomly split those 1000 sentences into test and development set containing 500 sentences each. We use the development set for the current experiments, which contains 879 predicates.

Global cross-lingual predicate labelling

Traditional cross-lingual transfer methods are locally defined. Transfer takes place on a token-to-token basis and, as a consequence, missing or incorrect alignments lead immediately to missing and incorrect annotations in the target language. Our method for cross-lingual predicate labelling is globally defined and relies less on actual alignments.

Our aim is to put predicate labels that originate from the English side of the parallel corpus on the French verbs in the other side of the corpus. The predicate labels contain the English verb and its sense. For example, "give.01" stands for the first sense of the verb give. As the predicate label contains a lot of lexical information, assigning the correct English predicate label to a French verb is a task very close to word sense disambiguation (WSD), which aims at automatically identifying the meaning of words in context [START_REF] Navigli R | Word Sense Disambiguation : a Survey[END_REF]. In cross-lingual WSD, the candidate senses of words are their translations in other languages from which the most adequate has to be selected for contextualized instances of the words [START_REF] Carpuat | Improving Statistical Machine Translation using Word Sense Disambiguation[END_REF][START_REF] Apidianaki | Data-driven Semantic Analysis for Multilingual WSD and Lexical Selection in Translation[END_REF]. The main difference between cross-lingual WSD and our cross-lingual transfer of predicate labels is that we do not search for correct translations of French words but for the most appropriate predicate labels in context (i.e. verbs disambiguated with a predicate sense).

The global predicate labelling method that we propose consists of a learning step and a labelling step. During learning, we compute estimates for annotation transfer on the basis of the word alignments between English and French predicates over the entire parallel corpus. At labelling time, we label French verbs with English predicate labels without need for parallel data or alignments. The method is language-independent and only requires minimal linguistic resources.

In contrast to direct transfer, we provide a predicate label for all French verbs in the test set, not only aligned ones. We expect to augment the recall when using global estimates and hope that the affect on precision is not too negative.

Pre-processing

The WSD classifier is trained on the Europarl corpus tagged with PropBank information on the English side, as described in Section 5. To identify the sets of candidate predicate labels for each French verb, we replace the English verbs by the corresponding predicate label wherever this is available. Then we tag both parts of the corpus by part of speech (PoS) using the TreeTagger [START_REF] Schmid | Probabilistic part-of-speech tagging using decision trees[END_REF] and rebuild the parallel files (one sentence per line) by replacing words on both sides by the corresponding 'lemma_PoS tag' pair, and keeping the predicate labels in the place of English verbs. The corpus is then aligned at the word level in both directions using GIZA++ [START_REF] Och | A systematic comparison of various statistical alignment models[END_REF]) and a lexicon is built from intersecting alignments. Lexicon entries for French verbs contain the labels to which they were aligned in the training corpus. The entry for the verb encourager, for instance, contains seven predicate labels : {urge.01, foster.01, stimulate.01, promote.02, encourage.01, encourage.02, renew.01}, two of which correspond to the same English verb (encourage). We keep labels with a high alignment confidence score according to GIZA++ and experiment with two thresholds, 0.01 and 0.001. Naturally, the second threshold retains a higher number of candidate labels.

Learning

For each French verb (v) in the lexicon, we want to be able to identify its correct predicate label in a new context. A feature vector is built for each candidate label following the procedure described in [START_REF] Apidianaki | WSD for n-best reranking and local language modeling in SMT[END_REF]. For each candidate label L i of a French verb v, we extract the content word co-occurrences of v in the sentences where it translates an English verb tagged with the label L i . The retained French words constitute the features of the vector built for the label.

Let N be the number of features retained for each label L i of v from the corresponding French contexts. Each feature F j (1  j  N ) receives a total weight with the label tw(F j , L i ) learned from the data and defined as the product of the feature's global weight, gw(F j ), and its local weight with that label, lw(F j , L i ). The global weight of a feature F j is a function of the number n of labels (L i 's) to which F j is related, and of the probabilities (p ij ) that F j co-occurs with instances of v corresponding to each of the L i 's :

gw(F j ) = 1

P n i=1 p ij log(p ij ) n (1)
CROSS-LINGUAL WORD SENSE DISAMBIGUATION FOR PREDICATE LABELLING OF FRENCH Each p ij is computed as the ratio of the co-occurrence count of F j with v when it corresponds to a label L i to the total number of features (N ) seen with L i in the corpus :

p ij = cooc_count(F j , L i ) N (2)
The local weight lw(F j , L i ) between a feature F j and a label L i directly depends on their co-occurrence count :

lw(F j , L i ) = log(cooc_count(F j , L i )) (3)
The intuition underlying this weighting scheme is that if an interesting semantic relation exists between a feature F j and a specific predicate label L i of a verb v, then we expect the probability (p ij ) of the feature F j occurring in the contexts where v is translated by this label to be larger than if they were independent. In other words, a feature gets a high total weight (tw) with a label when it appears frequently in the corresponding French contexts and rarely in the contexts of the other labels.

Labelling

Predicate identification is done by selecting verbs based on the PoS labels provided by the tagger and subsequently filtering out modals and instances of the verb être. 3 The most suitable predicate labels are then assigned to the retained French verbs by our disambiguation classifier. The weighted feature vectors built for the candidate labels of a French verb as described in the previous section are compared to the context of a new instance of the verb and an association score is assigned to each candidate label. To facilitate comparison with the vectors, the new contexts (sentences) are lemmatized and PoS tagged on the fly (with TreeTagger) and the content word co-occurrences of the French verb are gathered in a bag of words. If common features are found between the new context and the vector of a label, their association score corresponds to the mean of the weights of their shared features with that label (i.e. found in its vector). In Equation 4,

(CF j ) |CF |
j=1 is the set of common features between the label vector V i and the new context C and tw is the weight of a CF with label L i , computed as explained in the previous section.

assoc_score(V i , C) = P |CF | j=1 tw(CF j , L i ) |CF | (4)
The label that receives the highest association score with the new context is returned and serves to annotate the corresponding French verb. For example, among the candidate labels for the verb encourager ({urge.01, foster.01, stimulate.01, promote.02, encourage.01, encourage.02, renew.01}), the classifier selects the predicate label encourage.02 for the following instance : D'ailleurs, le rapport von Wogau, que vient de voter le Parlement européen encourage [encourage.02] en ce sens.

The label selected in this case corrects the label [support.01] that was assigned through direct transfer.

Results and discussion

We run experiments using the global method for predicate labelling described in the previous section and compare the results to the ones obtained through direct transfer. The results are presented in Table 2 where they are also compared to upper bounds from manual annotations and previous work.

The first row of Table 2 shows the results from using the traditional direct transfer method. The second and third rows present the results obtained using the global method, where we use cross-lingual WSD to label predicates. In row 2, we present the results obtained when using an alignment confidence threshold of 0.01 (retaining labels with an alignment score above 0.01, according to GIZA++) and in row 3, the results obtained using a threshold of 0.001. For comparison, we show results when using a parser as in [START_REF] Van Der Plas L | Scaling up cross-lingual semantic annotation transfer[END_REF] who use a joint syntactic-semantic parser and syntactic annotations on French to do predicate and semantic role labelling. 4 We show an upper bound in the last row. This represents the inter-annotator agreement for manual annotation on a random set of 100 sentences taken from data provided by [START_REF] Van Der | Cross-lingual validity of PropBank in the manual annotation of French[END_REF]. Two evaluation settings were used in that work, in order to avoid penalizing synonymous verb senses assigned by the annotators : the inter-annotator agreement reached at the verb sense label was compared to the agreement reached using verb classes, as explained in Section 3.2. Because we do not want to penalize the predicate labelling system for selecting verb senses that are synonyms of the verb senses in the gold, we follow the same strategy and take verb classes into account during evaluation. More precisely, we calculate scores based on the exact correspondence between the label proposed by our system and the gold label found in the test data, and we also perform a coarser evaluation taking verb classes into account. The first evaluation is too strict as it penalizes the system when it selects a predicate sense that is synonymous to the gold sense or a predicate that belongs to the same word class. The results headed by verb classes evaluate on a more realistic basis, capturing semantic correspondences beyond surface variations. In this setting, predicate labels are correct if they belong to the same verb class as the predicate in the gold annotations.

When we look at the differences between the three automatic methods for the evaluation on predicate senses, we see that for the direct transfer method especially recall is very low, 29%. The global method (alignment score threshold = 0.01) has a much better recall, 39%. Precision is lower but the F-score increases by 5 percentage points. When the alignment confidence threshold is lowered to 0.001, which means that more candidate labels are retained, recall increases and precision goes down, as expected. As explained above, the parser from Van der Plas et al. ( 2011) (shown in row 4) has access to both PoS and syntactic information on the target side and uses a joint syntactic-semantic framework. When we take its dependence on two extra resources into consideration, its performance is not that impressive. However, we can learn from these results that structural information is beneficial. Nevertheless, our results show that the cross-lingual WSD method which relies on much less external knowledge, outperforms the direct method on both senses and verb classes. In future work, we plan to include word position information in our cross-lingual WSD method. This will give the method access to structural information while staying knowledge-lean.

When we look at the results using verb classes which permit to abstract from surface variations and capture semantic correspondences, the overall performance numbers are higher as expected. More importantly, the difference in performance between the cross-lingual WSD method using the more restrained set of labels (alignment score threshold = 0.01), which performs best, and the direct transfer method are now much larger (three times as important, from 5 to 17 percentage points) whereas the difference between our method and the parser is further reduced (from 10 to 5 percentage points). The differences between the parser and our method can be mainly attributed to arbitrary variations between predicate labels that belong to the same verb class. When we look at the precision and recall scores we see that the cross-lingual WSD method with the threshold set at '0.01' improves the recall of the direct transfer method by 23 percentage points, whereas precision only drops by 2 points. The cross-lingual WSD method that needs only a PoS tagger on the target and no syntactic annotation nor parsing frameworks, results in much better scores than direct transfer that is equally knowledge-light. All automatic methods are still quite far from the results from manual annotation.

In summary, these results show that the global cross-lingual WSD method for predicate labelling improves recall of direct transfer methods without sacrificing precision too much. In future work, we plan to combine the direct transfer method and the global cross-lingual WSD method, because the two are complementary in terms of recall and precision.

An example of predicate labelling might help the reader to get an idea of the contribution of the global method. In Table 3, we present an example where cross-lingual WSD annotates more verbs than the direct transfer : labels [stress.01] and [seem.01], assigned during disambiguation, are missing from the first sentence after transfer. Moreover, it would be impossible to get these labels through direct transfer from the English source sentence because they are simply not there, due to the non-literal translation. This example shows the limitations of token-to token, direct transfer and how the global method is able to compensate for that by using information aggregated across the whole parallel corpus.

Conclusion

In this paper, we present a knowledge-light global approach to the cross-lingual transfer of semantic annnotation that aggregates information across the whole parallel corpus. Previous work has transferred annotations directly from token to token in parallel sentences leading to low recall and token-level mistakes. We show how the global method, based on cross-lingual word sense disambiguation, improves recall by a large margin without sacrificing precision too much.

Given the knowledge-lean character of the proposed method, in future work we plan to apply it for cross-lingual predicate labelling in other language pairs. Furthermore, we would like to include target side structural information (e.g. word position information) in the cross-lingual WSD method. Last but not least, we intend to work towards global methods for role identification and labelling which will allow to propose a complete SRL annotation framework based on global information. In that respect, we would also like to try and combine global and direct methods because the two seem complementary in terms of recall and precision.

  [AGENT Mary] [REL-STOP.01 stopped] [THEME the car]. b. [THEME The car] [REL-STOP.01 stopped].

  2) a. [EXPERIENCER Mary] [REL-LIKE.01 liked] [CONTENT the idea]. (English) b. [CONTENT L'idée] a [REL-LIKE.01 plu] [EXPERIENCER à Marie]. (French)

  ) [ A0 The Latin American nation] has [ REL P AY.01 paid] [ A1 very little] [ A3 on its debt] [ AM T MP since early last year].

TABLE 1 -

 1 The PropBank lexicon entry for pay.

TABLE 2 -

 2 Percent recall, precision and F-measure for predicate labelling.

  English (automatic) :There is in particular one amendment, let[let.01] me point [point.02] out, concerning [concern.01] the energy sector, which, in my capacity as rapporteur, I see [see.01] as particularly important. Transfer : Il y a notamment un amendement, je le souligne, concernant [concern.01] le secteur de l'énergie, qui me paraît en tant que rapporteur particulièrement important. CLWSD : Il y a notamment un amendement, je le souligne [stress.01], concernant [concern.01] le secteur de l'énergie, qui me paraît [seem.01] en tant que rapporteur particulièrement important.

TABLE 3 -

 3 Predicate label addition and correction using CLWSD.

We limit our experiments to verbal predicates only because the semantic annotations on French test sentences are limited to verbal predicates. Even though verbal predicates in the target language can be expressed as non-verbal predicates in the source language, the transfer of Nombank labels to verbal predicates is not straightforward due to difficulties in mapping between the two annotation frameworks.

We exclude the verb être because its English counterpart (be) is not annotated in the CoNLL-2009 data used in our experiments.

The results are different from the results reported in[START_REF] Van Der Plas L | Scaling up cross-lingual semantic annotation transfer[END_REF] because we used the development set in our evaluations.