Andon Tchechmedjiev

Gilles Sérasset

Jérôme Goulian

Didier Schwab

Attaching Translations to Proper Lexical Senses in DBnary

Keywords: Wiktionary, Linked Open Data, Multilingual Resources

The DBnary project aims at providing high quality Lexical Linked Data extracted from different Wiktionary language editions. Data from 10 different languages is currently extracted for a total of over 3.16M translation links that connect lexical entries from the 10 extracted languages, to entries in more than one thousand languages. In Wiktionary, glosses are often associated with translations to help users understand to what sense they refer to, wether through a textual definition or a target sense number. In this article we aim at the extraction of as much of this information as possible and then the disambiguation of the corresponding translations for all languages available. We use an adaptation of various textual and semantic similarity techniques based on partial or fuzzy gloss overlaps to disambiguate the translation relations (to account for the lack of normalization, e.g. lemmatization and PoS tagging) and then extract some of the sense number information present to build a gold standard so as to evaluate our disambiguation as well as tune and optimize the parameters of the similarity measures. We obtain F1 score of the order of 80% (on par with similar work on English only), across the three languages where we could generate a gold standard (French, Portuguese, Finnish) and show that most of the disambiguation errors are due to inconsistencies in Wiktionary itself that cannot be detected during the generation of DBnary (shifted sense numbers, inconsistent glosses, etc.).

Introduction

Wiktionary is a lexical-semantic resource built collaboratively under the patronage of the Wikimedia Foundation (which also hosts the well known Wikipedia initiative). It is currently the biggest collaborative resource for lexical data. Wiktionary pages usually describe lexical entries by giving their part of speech, a set of definitions, examples, lexicosemantic relations and many translations in more than a thousand target languages.

The DBnary project [START_REF] Sérasset | DBnary: Wiktionary as a Lemon-Based Multilingual Lexical Resource in RDF[END_REF] aims at providing high quality Lexical Linked Data extracted from different Wiktionary language editions. It currently extracts data from 10 editions and gathers 3.16M translation links relating lexical entries from the 10 extracted languages to entries in more than a thousand languages. These numbers are steadily growing as the DBnary dataset is extracted as soon as Wikimedia releases new dumps of the data (around once every 10-15 days for each language edition).

The sources of these translation links are lexical entries. The purpose of this work is to attach these translations to the correct word sense and hence to increase the value and quality of the DBnary dataset. Comparable efforts have been carried out (mainly on the UBY dataset), but are limited to English and German. In this paper we worked on 10 language editions. Among them, we were faced with the various habits of the different Wiktionary communities. For example different languages editions exhibit different linguistic properties. After detailing related works, we present the structure of the DBnary dataset. Then, after showing how we built an endogenous golden standard used to evaluate this work, we detail the methods used to achieve our purpose. Finally we evaluate our method and discuss the results.

Related Work

Extracting Data from Wiktionary Language

Editions.

Since its inception in 2002, Wiktionary has steadily increased in size (both with collaborative work and with automatic insertions of available lexical data). Interest in Wiktionary as a source for lexical data for NLP applications has quickly risen. Studies like (Zesch et al., 2008b) or [START_REF] Navarro | Wiktionary and NLP: Improving synonymy networks[END_REF] show the richness and power of this resource.

Since then, efforts have mostly been focussed on the systematic extraction of Wiktionary data. Many of them, as resources for a specific project and thus merely snapshots of Wiktionary at a fixed point in time. As all Wiktionary language editions evolve regularly (and independently) in the way their data is represented, such efforts are not suitable to provide a sustainable access to Wiktionary data. Some efforts, however are maintained and allow access over time. One of the most mature project is the JWKTL API (Zesch et al., 2008a) giving access to the English, German and Russian language editions. It is used in the UBY project [START_REF] Gurevych | Uby: A large-scale unified lexicalsemantic resource based on LMF[END_REF] which provides an LMF based version of these editions. We should also mention the wikokit project [START_REF] Krizhanovsky | Transformation of Wiktionary entry structure into tables and relations in a relational database schema[END_REF]) that provides access to the English and Russian editions and that was used by JWKTL. [START_REF] Hellmann | Leveraging the Crowdsourcing of Lexical Resources for Bootstrapping a Linguistic Data Cloud[END_REF] presents another attempt under the umbrella of the DBpedia project [START_REF] Lehmann | DBpedia -A Largescale, Multilingual Knowledge Base Extracted from Wikipedia[END_REF], whose purpose is specifically to provide the Wiktionary data as Lexical Linked Open Data. The main reason this approach is interesting is the collaborative nature extraction template creation process (following the culture of the DBpedia project). Currently English, French, Russian and German Wiktionary editions are supported. This paper is part of the DBnary project [START_REF] Sérasset | DBnary: Wiktionary as a Lemon-Based Multilingual Lexical Resource in RDF[END_REF] that has a similar purpose to that of [START_REF] Hellmann | Leveraging the Crowdsourcing of Lexical Resources for Bootstrapping a Linguistic Data Cloud[END_REF].

Our goal is to provide LEMON (McCrae et al., 2012) based lexical databases that are structured like traditional lexica. Indeed, we extract data from Wiktionary, but we currently restrict ourselves to the "native" data of each language edition, e.g. the French data is extracted from the French language edition and we disregard French data contained in other editions. To the best of our knowledge DBnary is currently the most advanced extractor for Wiktionary with an active support of 10 languages. It is also the only initiative giving access to the whole extracted data history.

Disambiguation of the Source of Translations.

As far as attaching translations to the proper word sense (translation disambiguation) is concerned, the most similar work to ours is that of (Meyer and Gurevych, 2012b). Their intent matches our own, however their efforts only deal with the German and English editions. In their work, the gold standard was manually created and was significantly smaller than the endogenous gold standard we extracted from the resource itself. They use a backoff strategy (to the most frequent sense) when the heuristic based on similarity measures and the resource's structure fails. The other heuristics used with their similarity measure also imply a finer analysis of definitions and glosses so as to distinguish between linguistic labels (domain, register, title, etc.). Herein, we achieve similar scores on the languages we were able to evaluate with an endogenous gold standard, even though we only used string and token similarity measures in the context of languages with less common features (e.g. the agglutinative aspect of the Finnish language).

Similarity measures

Our method is based on the application of gloss overlap measures and their extension with ideas taken from Hybrid textual similarity measures that match sentences both at the character and at the token level. In the work mentioned above (Meyer and Gurevych, 2012b), a feature-based similarity is used (gloss overlap), while in some of their prior work [START_REF] Meyer | Worth its weight in gold or yet another resource -a comparative study of wiktionary, openthesaurus and germanet[END_REF], they use a textual similarity measure based on vector-spaces generated from corpora (Explicit Semantic Analysis).

We propose a simple similarity measure where we replace the exact word match of the overlap calculation with an approximate string distance measure and place ourselves in the general framework of the Tversky [START_REF] Tversky | Features of Similarity[END_REF] index (can be seen as a generalization of Lesk, the Dice coefficient , the Jaccard and Tatimono indexes, etc.)

The idea of "soft-cardinality" proposed by [START_REF] Jimenez | Text comparison using soft cardinality[END_REF][START_REF] Jimenez | Soft Cardinality: A Parameterized Similarity Function for Text Comparison[END_REF] is very similar in the sense that it exploits the Tversky index as a base and conjugates it with a textual similarity measure. That is, instead of incrementing the overlap count by 0 or 1, incrementing it by the value returned by the text similarity measure between the current pair of words being considered in the overlap calculation.

Their text similarity measure is based on an an empirical qgram model (character-grams that correspond to substrings) combined with point-wise mutual information weighting. However in our work, generating a language model for 10 languages would require considerable effort and with future additions of more languages, become a daunting task.

In the textual similarity tasks in SemEval, using approximate string matching for overlap calculation is not new and has been exploited by several system, including in 2013 a soft cardinality system by [START_REF] Jimenez | SOFTCARDINALITY: Hierarchical Text Overlap for Student Response Analysis[END_REF] or other systems such as that of [START_REF] Wu | MayoClinicNLP-CORE: Semantic representations for textual similarity[END_REF] who use longest common sub-strings and greedy string tiling .

As such, we chose to use a simple string distance measure for the approximate string match calculations. However, there are many such measure and it is necessary to select the right one for the task as will be detailed in Section 5. Moreover, there are existing so called "Level 2" or "Hybrid" similarity measures that already combine token overlap with token distance measures. Thus, we will need to evaluate our proposed method with some of the existing methods so as to evaluate their viability. The various measures and a detailed performance comparison in a name matching task are presented by [START_REF] Cohen | A comparison of string distance metrics for name-matching tasks[END_REF].

The DBnary Dataset

DBnary is a Lexical Linked Open Dataset extracted from 10 Wiktionary language editions (English, Finnish, French, German, Greek, Italian, Japanese, Portuguese, Russian and Turkish). It is available on-line at http://kaiko. getalp.org/about-dbnary. DBnary currently contains 35+M triples. This number is steadily growing as the dataset evolves in parallel with the original Wiktionary data. Indeed, the dataset is automatically updated as soon as Wikimedia releases new Wiktionary dumps, i.e. every 10-15 days per language edition. DBnary is structured according the LEMON ontology for lexical linked data [START_REF] Mccrae | Interchanging lexical resources on the Semantic Web[END_REF]. Table 1: Number of elements in the current DBnary dataset, detailing the number of entries and word senses, along with the number of translations. The table also details the number of Glosses attach to translations, among which the amount of textual glosses, of glosses giving the sense identifier and, finally, the number of glosses that contain both a textual description and a word sense identifier. a dbnary:Translation ; dbnary:gloss "amphibian"@en ; dbnary:isTranslationOf eng:frog__Noun__1 ; dbnary:targetLanguage lexvo:fra ; dbnary:usage "f" ; dbnary:writtenForm "grenouille"@fr .

The properties of this resource point to the source LexicalEntry, the language of the target (represented as a lexvo.org entity (de Melo and Weikum, 2008)), the target written form and optionally, a gloss and usage notes. Usage notes give information about the target of the translation (e.g. the gender or a transcription of the target). The gloss gives disambiguation information about the source of the translation. In the example given, it states that the given translation is valid for the word sense of frog that may be described by the hint "amphibian". Some of these glosses are textual and summarize or reprise the definition or part thereof for one or more specific sense to which the translation specifically applies to. As an example, the English LexicalEntry frog contains 8 word senses, defined as follows:

1. A small tailless amphibian of the order Anura that typically hops 2. The part of a violin bow (or that of other similar string instruments such as the viola, cello and contrabass) located at the end held by the player, to which the horsehair is attached 3. (Cockney rhyming slang) Road. Shorter, more common form of frog and toad 4. The depression in the upper face of a pressed or handmade clay brick

5. An organ on the bottom of a horse's hoof that assists in the circulation of blood 6. The part of a railway switch or turnout where the runningrails cross (from the resemblance to the frog in a horse's hoof)

7. An oblong cloak button, covered with netted thread, and fastening into a loop instead of a button hole.

8. The loop of the scabbard of a bayonet or sword.

Translations of this entry are divided in 4 groups corresponding to: "amphibian", "end of a string instrument's bow", "organ in a horse's foot" and "part of a railway".

Additionally among the glosses, some may contain sense numbers, indicated by users in an ad-hoc way (may or may not be present, and if they are no standard format is systematically followed or enforced). Furthermore, the presence of disambiguation information is very irregular and varies greatly between languages, both in terms of wiki structure and representation.

In the current state of the Wiktionary extraction process, we extract translation and when possible the associated glosses. However up to now, we have not exploited the information contained in the glosses to enrich and disambiguate the source senses of translation relations.

As mentioned above, the information contained in translation glosses and their format is very variable across languages, both quantitatively and qualitatively. Indeed, as shown in Table 1 some language like Italian, contain no gloss altogether, others, like English attaches textual glosses to translations almost systematically, but with no sense numbers. Others still, like German hardly contain textual glosses but give sense numbers to translations. In other cases, such as for Finnish, French and Portuguese, many translations have an attached (textual) gloss with associated sense numbers.

In order to evaluate our method we use mixed glosses that both contain a textual hint and a sense number, so as to create a endogenous gold standard.

Creation of a gold standard

False positives and variability are often present among available translation glosses that do contain textual information or sense numbers due the variety of structures employed in Wiktionary as well as artefacts resulting from the extraction process. Before we can proceed further we need to filter this information so as to keep only the relevant parts. However, no other preprocessing is performed. More concretely two steps must be followed if we are to successfully extract the information we need :

• Remove empty glosses, or glosses containing irrelevant textual content that often correspond to TO DO notes in various forms (e.g. translations to be checked)

• Extract sense numbers from the glosses when available using language dependent templates (e.g. "textual gloss (1)" or "1. textual gloss")

When enough glosses contained both a textual hint and sense numbers, we removed the sense numbers 2 from the gloss and used them to create a gold standard in the trec eval format. Only three of the ten language met the requirements as for many of the 10 languages there are no numbered glosses or no translation glosses altogether.

After successfully extracting as much information as possible from translation glosses, we disambiguatthe translation. While, the steps above are indeed language specific, our process is designed to be as generic and computationally efficient as possible. Indeed, we are required to periodically perform the disambiguation, whenever a new version of DBnary is extracted from the latest Wiktionary dumps.

Attaching Translations to Word Senses

Formalization of translation disambiguation

Let T be the set of all translation relations, L the set of all LexicalEntry in a given language edition of DBnary. Let T i ∈ T : Gloss(T i ) be a function that returns the gloss of any translation T i ∈ T and let Source(T i ) = L Ti be a function that returns a reference to the source LexicalEntry, L Ti of a translation T i . Let Senses(L i ) = S Li be the set of all the senses associated with LexicalEntry L i . Let S k Li be the k-th sense contained in S Li and let Def (S k Li ) be a function that returns the textual definition of a sense S k Li . Finally let Sim(A, B) be a function that returns a semantic similarity or relatedness score between A and B, where A, B are a pair of textual definitions or textual glosses. Then, we can express the disambiguation process as:

∀T i ∈ T, S = Senses(Source(T i )) : Source * (T i ) ← argmax S k ∈S {Score(Gloss(T i ), Def (S k ))}
This corresponds exactly to a standard semantic similarity maximisation and yields one disambiguated source sense per translation. However in many cases a translation corresponds to one or more senses. The solution adopted by (Meyer and Gurevych, 2012a) is to use a threshold k for their gloss overlap, however in our case, we want to be able to plug-in several different measures so as to find the most suitable one, thus, fixed and arbitrary value for k is not an option. Thus, we need to add one more constraint: that the values returned by our similarity function need to be normalized between 0 and 1.

Here, instead of taking a threshold k, we set a window δ around the best score in which the senses are accepted as a disambiguation of a given translation. We hypothesise that a relative threshold dependant on the maximal score will set a precedent and be more representative of the possible range of values. Of course, setting a fixed threshold has the effect of not assigning any senses if all the scores are low, thus increasing precision at the cost of lowering recall. While in a general setting, it is better to remove answers 2 Translation are ban be valid for several source senses that are more likely to be mistakes, as detecting errors a posteriori is difficult. However in the context of the experiment, we prefer to keep such low or null scores as we will then be able to pin-point errors more precisely with the help of the gold standard for the sake of our analysis. We can express this formally by modifying the argmax function as such:

∀T i ∈ T, S = Senses(Source(T i )) : M S = max S k ∈S (Score((Gloss(T i ), Def (S k ))), δ argmax Si∈S {Score(Gloss(T i ), Def (S k ))} = {S k ∈ S|M S > Score((Gloss(T i ), Def (S k )) > M S -δ}

Similarity Measure

In order to disambiguate the translation, we need to be able to compute some form of semantic similarity measure. Given that the only information available in the translations is the gloss that summarises the definition of the corresponding sense, we need a measure to capture the similarity by comparing the translation glosses and the sense definitions. The Lesk [START_REF] Lesk | Automatic Sense Disambiguation Using Machine Readable Dictionaries: How to Tell a Pine Cone from an Ice Cream Cone[END_REF] measure is a standard semantic similarity measure well suited for such tasks, as it computes a similarity based on the number of exact overlapping words between definitions. The Lesk similarity however, has several important issues that need to be addressed when its use is mandated:

• If the sizes of the glosses are not the same, the Lesk measure will always favor longer definitions.

• The size and the appropriateness of the words contained in the definitions is important, as one key word to the meaning of the definition missing (or the presence of a synonym for that matter) can lead to an incorrectly low similarity.

• The Lesk overlap is not in itself normalized, and the normalization process requires some though depending of the distinct problems at hand.

The issues of normalization and of the unequal length of definitions are actually related, as one way of compensating for unequal lengths is to divide by the length of the shortest definition, which also normalizes the score. Moreover, there is a striking similarity between Lesk and other overlap coefficients: the Dice Coefficient or the Jaccard or Tatimono indices. In fact, all of these measures are special forms of the Tversky index [START_REF] Tversky | Features of Similarity[END_REF]. The Tversky index can be defined as follows. Let s 1 ∈ Senses(L 1 ) and s 2 ∈ Senses(L 2 ) be the senses of two lexical entries L 1 and L 2 . Let d i = Def (s i ) be the definition of s i , represented as a set of words. The similarity between the senses Score(s 1 , s 2 ) can be expressed as

Score(s 1 , s 2 ) = |d 1 ∩ d 2 | |d 1 ∩ d 2 | + α|d 1 -d 2 | + β|d 2 -d 1 |
The measure can further be generalized following [START_REF] Pirrò | A Semantic Similarity Framework Exploiting Multiple Parts-of Speech[END_REF]) by replacing the cardinality function by any function F . Depending on the values of α and β, the Tversky index takes the particular form of other similar indexes. For (α = β = 0.5) for example it is equivalent to the dice coefficient, and for (α = β = 1) to the Tatimono index. More generally, the values of α and β express how much emphasis one wants to attribute to the commonality or differences of one or the other set. The Tversky index in itself is not a metric in the mathematical sense, as it is neither symmetric nor respects the triangular inequality, however, a symmetric variant has been proposed by [START_REF] Jimenez | Text comparison using soft cardinality[END_REF] for such cases where the symmetry property is important or required. However there are no indications that the space of overlap-based semantic similarity is actually a metric space where those properties are beneficial. We actually obtained better results with the non-symmetric variant. We motivate our choice of the Tversky index firstly because translation glosses are systematically composed of few words, whereas sense definitions are longer: the weights of the Tversky index allow for a good normalization in such situations. Furthermore, we are dealing with many languages so that building statistical similarity measures would require considerable efforts especially for lesser resourced languages. An overlap-based measure is a good choice for this situation.

Multilingual Setting & Partial overlaps

When working on a single language such as English or French, we have at our disposal tools such as a lemmatizer or a stemmer that may help to retrieve a canonical representation of the terms. Thus, we can hope to maximize the overlap and reduce the usual sparsity of glosses or sense definitions. For agglutinative languages like German or Finnish, highly inflective language (for example in the Bangla language, common stems are often composed of a single character, which makes stemming difficult to exploit) or languages with no clear segmentation, the preprocessing steps are paramount in order to make overlap based measures viable. If one is working on a single language, even if stemmers and lemmatizers do not exist, it is possible to build such a tool. However, in the context of this work we are currently dealing with 10 languages (and potentially in the future with all the languages present in Wiktionary) and thus, in order to propose a truly general method, we cannot expect as a prerequisite, the presence of such tools. How then, can we manage to compute overlaps effectively? When computing Lesk, if two words overlap, the score is increased by 1. Otherwise the overlap value does not change. What if we had a way to count meaningful partial overlaps between words? Instead of adding 1, we could add a value between 0 and 1 that represents a partial overlap. The simplest approach is to use a form of partial string matching to compute these partial overlaps: a seemingly trivial approach that can however, greatly improve the result [START_REF] Jimenez | Soft Cardinality: A Parameterized Similarity Function for Text Comparison[END_REF]. As mentioned in the Related Work section, there are many approximate string matching measures as reviewed by [START_REF] Cohen | A comparison of string distance metrics for name-matching tasks[END_REF]. We integrate these measures in the Teversky index by setting the F function that replaces the set cardinality function appropriately (a simplified version of soft cardinality):

A , a set : F (A) = ( Ai,Aj ∈A sim(A i , A j )) -1
In our case, sim will be an string distance measure.

Longest Common Substring Constraints

With this similarity measure, we are mainly interested in capturing word that have common stems, without the need for a stemmer: for example, we do not want to consider the overlap of prefixes or suffices, as they do not carry the main semantic information of the word. If two words only match by a common suffix that happens to be used very often in that particular language, we will have a non-zero overlap, but we will have captured no sematic information whatsoever. Thus, in this work we put a lower-bound of three characters on the longest common subsequence.

Experiments

We extracted a gold standards from the sense numbered textual glosses of translations (when we could). Then we strip all sense number information from the glosses, so we can disambiguate those same translation and then evaluate the results on the previously generated gold standard. We first describe how we generated the gold standard and the tools and measures used for the evaluation. We then proceed onto the empirical selection of the best parameters for our Tversky index as well as the most appropriate string distance measure to use for the fuzzy or soft cardinality. Then, we compare the results of the optimal Tversky index with other Level 2 similarity measures.

Evaluation

Let us first describe the gold standard generation process, then proceed on to describing how we represented the gold standard in Trec eval format, a scorer program from the query answering Trec Eval campaign. Let us then finish with the description of the evaluation measures we use.

Gold Standard

Only certain languages meet the requirements for the generation of a gold standard. To be more specific, we could only use languages where:

1. There are textual glosses (for the overlap measures) 2. There are numbers in said glosses indicating the right sense number 3. The above are available in a sufficient quantity (at least a few thousand)

Four languages could potentially meet the criteria (see the last column of Table 1): French, Portuguese, Finnish and Japanese.Due to the fact that the data available for Japanese was much smaller in size, we generated gold standards only for French, Portuguese and Finnish.

Trec eval, scoring as a query answering task

A query answering task is more generally a multiplelabelling problem, which is exactly equivalent to what we are producing when we use the threshold δ. Here, we can consider that each translation number is the query identifier and that each sense URI is a document identifier. We answer the "translation" queries by providing one or more senses and an associated weight. Thus, we can generate the gold standard and the results in the Trec eval format, the very complete scorer for an information retrieval evaluation campaign of the same name.

Measures

We will use the standard set-matching metrics used in Information Retrival and Word Sense Disambiguation, namely Recall, Precision and F 1 score.

Where,

P = |{Relevant}∩{Disambiguated}| |{Disambiguated}| , R = |{Relevant}∩{Disambiguated}| |{Relevant}|
, and F 1 = 2•P •R P +R , the harmonic mean of R and P . However, for the first step consisting in the estimation of the optimal parameters, we will only provide the F 1score, as we are interested in maximising both recall and precision in an equal fashion.

Similarity Measure Tuning

There are parameters to set in our Tversky index: the first step is to find the most suitable string distance measure.

Optimal String Distance Metric

The δ parameter influences performance independently of the similarity measure, so we can first operate with δ = 0, which restricts us to a single disambiguation per translation. Furthermore, the weights of the Tvsersky index are applied downstream from the string edit distance, and thus do not influence the relative performance of the different string distance metrics combined to our Tversky index. In simple terms, the ratio of the Tverski indices computed on different measures is constant, independently of α and β. Thus for this first experiment, we will set α = β = 0.5, in other words the index becomes the Dice coefficient. As for the selection of the string similarity measures to compare, we take the best performing measures from [START_REF] Cohen | A comparison of string distance metrics for name-matching tasks[END_REF], namely Jaro-Winkler, Monge-Elkan, Scaled Levenshtein Distance, to which we also add the longest common substring for reference. As a baseline measure, we will use the Tversky index with a standard overlap cardinality. We give the following short notations for the measures: Tversky Index -Ts; Jaro-Winkler -JW; Monge-Elkan -ME; Scaled Levenshtein -Ls; Longest Common Substring -Lcss; F -Fuzzy. For example standard Tversky index with classical cardinality shall be referred to as "Ti", while the fuzzy cardinality version with a Monge-Elkan string distance shall be referred to as "FTiME". Table 2 presents the results for each string similarity measure and each of the languages (Fr, Fi, Pt). As we can see, for all language, the best string similarity measure is clearly the scaled Levenstein measure as it systematically exhibits a score higher from +1% to +1.96%. Alpha/Beta-1 Finnish French Portuguese 0 0,9523 0,8123 0,8545 0,1 0,9584 0,8205 0,8622 0,2 0,9569 0,817 0,8579 0,3 0,9547 0,8034 0,8492 0,4 0,9539 0,7961 0,8401 0,5 0,9536 0,7853 0,8349 0,6 0,9521 0,775 0,8272 0,7 0,9512 0,7667 0,8203 0,8 0,9506 0,7643 0,8173 0,9 0,9498 0,7586 0,8147

Optimal α, β selection

Now that we have found the optimal string distance measure, we can look for the optimal ratio of α and β. We keep both values complementary, that is α = 1β so as to obtain balanced score (i.e. 0 to 1 range) Given that translation glosses are short (often a single word), it is likely that the optimum is around α = 1β = 0.1. What interests us is that the single word or few words in the translation gloss matches any of the definition words.

If we give equal importance to α and beta, then the overlap score will be very small even if it indicates an exact match. A smaller alpha will ensure that if all the words of the translation match, the score will be closer to 1. We chose, here, to evaluate the values of α and β in steps of 0.1. Figure 1 graphically shows the F 1 score for each pair of values of alpha and beta for all three languages. We can indeed confirm our hypothesis as the optimal value in all three cases is indeed α = 1β = 0.1 with a difference between +0.15% to +0.43% with the second best scores.

Optimal δ selection

Now that we have fixed the best values of α and β, we can search for the best value for δ. We make delta vary in steps of 0.05 between 0 and 0.3. The choice of the upper bound is based on the hypothesis that the optimal value is somewhere closer to 0, as a too large threshold essentially means that that most or all senses will be considered as disambiguation of the translation, as if, we had disambiguated nothing.

The δ heuristic affects the results of the disambiguation whether the measure is Tversky index or another Level 2 Textual similarity. Thus, in this experiment, we will also include Level 2 version of the three string distance measures that we used in the first experiment. Figure 2 graphically presents the F 1 scores for each value of delta and each language. The first apparent trend is that Level 2 measures systemically perform much worse (by up to 30%) than our own similarity measure. Depending on the language different values of delta are optimal, even though it is difficult to see a great difference. For French δ = 0.10, for Finnish δ = 0.15 and for Portuguese δ = 0.10. In all three previous experiments, it became apparent, that the same string similarity measure, the same values for alpha and beta as well as the same value for delta were optimal, which leads us to believe that their optimality will be conserved across all languages. However, especially for the string similarity measure, it is reasonable to believe that for languages such a Chinese or Japanese that lack segmentation, the optimal choice for the string distance measure may be entirely different.

Final Disambigation Results

Now that we estimated the optimal parameters, we can present the final results based on them in Table 3). We use the chance of random selection as well as the most frequent sense selection as baselines for this comparison. The first thing one can notice is that there is a stark difference between the scores of Finnish, and the rest. Indeed, first of all, the random baseline and most frequent sense baselines are an indication that the French and Portuguese DBNaries are highly polysemous, while Finnish contains a very large amount of monosemous entries, which artificially inflates the value of the score. Interestingly he random baseline is higher (up to 6.6%) than the most frequent sense baseline, which indicates that the first sense if often not the right sense to select to match the translation. This could be explained by the fact that translations in other language can often lead to different target words for every source sense and thus selecting the first sense will be correct of a most a small proportion of the translation relations leaving from the source word. We can see that for all three languages we achieve a good performance compared to what is presented in the literature, most notably in the fact that most of the errors, can easily be identified as such just by looking at whether they produced any overlap.

Error analysis

We did not perform a full fledged and systematic error analysis, but rather an informal manual sampling so as to have an idea of what the error can be and if there are ways to correct them by adapting the measures or the methodology. We looked at some of the errors and manually categorized them:

1. No overlap between the gloss and sense definitions (Random choice by our algorithm), this happens when the translation gloss is a paraphrase of the sense definition or simply a metaphor for it.

2. The overlap is with the domain category label or the example glosses, which we do not currently extract. This is a particular case of the first type of error.

3. New senses have been introduced in Wiktionary and shifted sense numbers, which were not subsequently updated in the resource. Such errors cannot be detected during the extraction process.

We can in fact easily find all the errors due to the lack of overlap and correct the errors of type 2 by enriching the extraction process of DBnary. Thus we can single-out errors that are due to inconsistencies in the resource and thus potentially use the disambiguation results to indicate to users where errors are located an need to be updated.

Conclusion

With our method, we were able to find an optimal similarity measure for translation disambiguation in DBnary. Similar results across three languages suggests that it is a general optimality that can be applied to all the languages currently present in DBnary, although for Asian Languages that have no segmentation, it is likely not the case. Then, we compared the results and concluded that our method is viable for the task of disambiguating glossed translation relations, especially considering the low random baselines and first sense baselines compared to the top score of our method.

For translation relations without glosses, the disambiguation process is more complex and is part of the Future Work that we plan on carrying out.

Figure 1 :

 1 Figure 1: F1 score for Finnish, French and Portuguese depending on the value of α and β.

Figure 2 :

 2 Figure 2: Graphical representation of the F1 score against delta for our measure and other Level 2 Measures.

Table 1

 1 The DBnary dataset represents translation relations in an ad-hoc manner: the LEMON model does not have a vocabulary for such information. A Translation is a RDF resource that gathers all extracted information pertaining to a translation relation. For instance, one of the translations of the lexical entry frog is represented as follows 1 :

	shows

Table 2 :

 2 Results comparing the performance in terms of F 1 score for French, Finnish and Portuguese (highest in bold).

		French Portuguese Finnish
		F1	F1	F1
	FTiJW	0.7853	0.8079	0.9479
	FTiLcss 0.7778	0.7697	0.9495
	FTiLs	0.7861	0.8176	0.9536
	FTiME	0.7684	0.7683	0.9495
	Ti	0.7088	0.7171	0.8806

Table 3 :

 3 Final results with optimal measure and parameter values. Precision, Recall, F1 score for all three languages compared against the MFS and Random Baselines.

Acknowledgements

This work was conducted as a part of the CHIST-ERA CAMOMILE project, which was funded by the ANR (Agence Nationale de la Recherche, France).


Gilles Sérasset

Andon Tchechmedjiev

Dbnary: Wiktionary as Linked Data for 12 Language Editions with Enhanced Translation Relations

Keywords: Wiktionary, Multilingual Lexical Resource, Lexical Networks, LEMON, RDF

This paper presents the current state of development of the DBnary dataset. DBnary is a RDF dataset, structured using the LEMON vocabulary, that is extracted from twelve different Wiktionary language editions. DBnary also contains additional relations from translation pairs to their source word senses. The extracted data is registered at http://thedatahub.org/dataset/dbnary.

Introduction

The GETALP (Study group for speech and language translation/processing) team of the LIG (Laboratoire d'Informatique de Grenoble) is in need for multilingual lexical resources that should include language correspondences (translations) and word sense definitions. In this regard, the set data included in the different Wiktionary language edition is a precious mine. Alas, many inconsistencies, errors, difference in usage do exist in the various Wiktionary language edition. Hence, we decided to provide an effort to extract precious data from this source and provide it to the community a Linked Data. This dataset won the Monnet Challenge in 2012, when it consisted of 6 language editions. The structure of this dataset, which is intensively based on the LEMON model [START_REF] Mccrae | Interchanging lexical resources on the semantic web[END_REF] is presented in [START_REF] Sérasset | DBnary: Wiktionary as a Lemon-Based Multilingual Lexical Resource in RDF[END_REF]. This short paper purpose is to present the current state of our dataset.

Extracting Data from Wiktionary

No Common Approach

Errors and incoherences are inherent to a contributive resource like Wiktionary. This has been heavily emphasized in related works by [START_REF] Hellmann | Leveraging the Crowdsourcing of Lexical Resources for Bootstrapping a Linguistic Data Cloud[END_REF] and (Meyer and Gurevych, 2012b). Still, we suceeded not only in extracting data from 12 different language editions, but we are maintaining these extractor on a regular basis. Indeed, our dataset evolves along with the original Wiktionary data. Each time a new Wiktionary dump is available (about once every 10/15 days for each language edition), the DBnary dataset is updated. This leads to a different dataset almost every day. Some language editions (like French and English) have many moderators that do limit the number of incoherence among entries of the same language. Moreover, those languages that contain the most data, use many templates that simplify the extraction process. For instance, the translation section of the French dictionary usually uses a template to identify each individual translation. This is not true however, with less developed Wiktionary language editions. For instance, in the Finnish edition, some translations are introduced by a template giving the language (e.g. {fr} precedes French translation) and others are introduced by the string "ranska" which is the Finnish translation for "French". In this case the translator needs to know the Finnish translation of all language names to cope with the second case and avoid losing almost half of the available translation data. Moreover, since 2012, we have added new languages that exhibits a different use of the Wikimedia syntax. For instance, translations in the Russian Wiktionary are entirely contained in one unique template, where target languages are a parameter. Moreover, in the Bulgarian Wiktionary, the full lexical entry is contained in one single template where sections are the parameters. In such language editions, templates can not be parsed using regular expressions, as they are inherently recursive (template calls are included in parameter values of other templates). This invalidates our initial approach which was based on regular expressions. In order to cope with these languages, we had to use an advanced parser of the Wikimedia syntax (called Bliki engine1 ) to deal with such data. Our extractors are written in Java and are open-source (LGPL licensed, available at http://dbnary.forge. imag.fr).

Tools to Help Maintenance

In this effort, we also had to develop tools to evaluate the extarctor's performance and to maintain it. Our first tool 2 compares extracted translations with interwiki links. Many of the translations in a Wiktionary language edition do point to entries in the Wiktionary edition of the target language. Such inter-wiki links are available through the Wiktionary API. By randomly sampling the extracted data, we are able to compare the extracted data with such links. This gives us an idea of the extractor performance. However, this relies on the availability of inter-wiki links, which is not the case in some language edition. When we maintain the extractor, we need to carefully check that the patches we added do not introduce regressions in the extractor. For this, we developped our own RDFdiff command line tool which computes the differences be- From time to time, a Wiktionary language edition drastically changes the way it encodes some data. Actively following the discussions on each Wiktionary edition to anticipate such changes is not an option with so many languages. Hence, with each language extraction update, we compute a set of statistics that gives detailed figures on the size of the data . These statistics are available live on the DBnary web site3 . Overall, the most useful statistics are the ones that capture the evolution of the extracted data over time.

For instance Figure 1 shows the evolution of the size of the extracted French datasets since its original extraction. This plot allowed us to detect that a major refactoring was happening on the French language edition. This allowed us to patch the extractor for this new organisation right away.

Extracted Data as a LEMON Lexical Resource

Extracted Entries

The main goal of our efforts is not to extensively reflect the specific structures and constructs of Wiktionary data, but to create a lexical resource that is structured as a set of monolingual dictionaries + bilingual translation information. Such data is already useful for several application, but most importantly it is a sound starting point for a future multilingual lexical database. Monolingual data is always extracted from its dedicated Wiktionary language edition. For instance, the French lexical data is extracted from the French language edition (the data is available on http://fr.wiktionary.org). However, we do not capture as of yet, any of the French data that may be found in other language editions. We also filtered out some parts of speech in order to produce a result that is closer to existing monolingual dictionaries. For instance, in French, we disregard abstract entries corresponding to prefixes, suffixes or flexions (e.g.: we do not extract data concerning in-or -al that are prefixes/suffixes and that have a dedicated page in the French language Edition).

Given that the scope and main focus of our work is the production of lexical data, we do not provide any reference or alignment to any ontology (toward top-level concepts for example).

LEMON and non-LEMON modelled Extracted Data

All of the extracted data could not be structured using solely the LEMON model. For instance, LEMON does not contain any mechanisms that allow to represent translations between languages, as the underlying assumption is that such translation will be handled by the ontology description. Moreover, LEMON further assumes that all data is well-formed and fully specified. As an example, the synonymy relation is a property linking a Lexical Sense to another Lexical Sense. While this is a correct assumption in principle, it does not account for the huge amount of legacy data that is available in dictionaries and lexical databases and that isn't disambiguated.

In order to cope with such legacy data, we introduced several classes and properties that are not LEMON entities. However, we make sure that whenever a piece of data can be represented as a LEMON entity, it is indeed represented as such. Most of these points have already been covered in [START_REF] Sérasset | DBnary: Wiktionary as a Lemon-Based Multilingual Lexical Resource in RDF[END_REF].

Links to other datasets

The DBnary dataset makes use of other datasets. Firstly, while all extracted lexical entries are associated with a language-specific part of speech that is given by its origi-nal Wiktionary language edition, we also add, when available a lexinfo:partOfSpeech relation to a standard value defined in the LexInfo ontology 4 [START_REF] Buitelaar | The Semantic Web: Research and Applications[END_REF]. Secondly, while the LEMON model uses a string value to represent languages, we additionally use the property dcterms:lang to point to a language entity defined in the Lexvo ontology (de Melo and Weikum, 2008).

Disambiguation of translation sources

Many of the translations present in Wiktionary are associated with a hint used by human users to identify the sense of the source of the translation. Depending on the language, this hint may take the form of a sense number (e.g. in German and Turkish), of a textual gloss (e.g. English) or of both a sense number and a textual gloss (e.g. French, Finnish). By using an adaptation of various textual and semantic similarity techniques based on partial or fuzzy gloss overlaps, we were able to disambiguate the translation relations. We obtained F-measures of the order of 80% (on par with similar work on English only, such as (Meyer and Gurevych, 2012a)), across the three languages where we could generate a gold standard (French, Portuguese, Finnish). We have shown that most of the disambiguation errors are due to inconsistencies in Wiktionary itself that cannot be detected at the generation of DBnary (shifted sense numbers, inconsistent glosses, etc.). The relations between translations and lexical senses has also been made part of this dataset.

Size of the involved data

Table 1 gives an overview of the number of main elements (Entries, Vocables, Senses and Translation), as extracted from the most up-to-date dumps at the time of writing. Table 2 details the number of lexico-semantic relations contained in each extracted languages.

Conclusion and Perspectives

The present article exhibits some preliminary results on what is essentially an open source tool to extract a LEMON 4 http://www.lexinfo.net/ontology/2.0/ lexinfo based lexical network from various Wiktionary language editions. Such a work is interesting for many users that will be able to use the extracted data in their own NLP systems. Moreover, as the extracted resource uses the Resource Description Framework (RDF) standard and the LEMON model, the extracted data is also directly usable for researchers in the field of the Semantic Web, where it could be used to ease the construction of ontology alignment systems when terms in different languages are used to describe the ontologies of a domain. Current work consists in extending the set of extracted languages, generalizing the extraction engine so that maitenance and definition of extractors will be easier, and adding more semantics to the dataset by providing internal and external links to LexicalSenses (as we started with translations). We are currently working on cross-lingual string similarity measures that will be used to establish such links. Also, we believe that the different initiatives aiming the extraction of Lexical Data from Wiktionary (e.g. UBY (Meyer and Gurevych, 2012b) or [START_REF] Hellmann | Leveraging the Crowdsourcing of Lexical Resources for Bootstrapping a Linguistic Data Cloud[END_REF]), should meet and work conjointely to produce even better and larger Lexical Linked Data.

Figure 1 :

 1 Figure 1: Some of the statistics available about the French extracted data.

Table 1 :

 1 Number of lexical elements in the graphs.

	Language	Entries	Vocables	Senses Translations	Total
	Bulgarian	18, 831	27, 071	18, 798	13, 888	78, 588
	English	553, 499	528, 341	447, 073	1, 332, 332 2, 861, 245
	Finnish	50, 813	50, 488	59, 612	122, 724	283, 637
	French	318, 101	304, 465	383, 242	507, 359 1, 513, 167
	German	211, 564	282, 902	102, 468	390, 938	987, 872
	Italian	34, 363	101, 525	45, 022	62, 305	243, 215
	Japanese	25, 492	25, 637	29, 679	87, 906	168, 714
	Modern Greek (1453-)	246, 211	241, 845	137, 072	57, 615	682, 743
	Portuguese	45, 788	45, 968	81, 807	267, 801	441, 364
	Russian	130, 879	143, 653	116, 925	365, 389	756, 846
	Spanish	58, 679	65, 854	85, 852	114, 951	325, 336
	Turkish	64, 899	69, 383	91, 418	66, 928	292, 628
	Total	1, 759, 119 1, 887, 132 1, 598, 968 3, 390, 136 8, 635, 355
	tween 2 RDF dumps. Such a command is already pro-
	vided in the JENA toolbox, however, the JENA implemen-
	tation does not correctly deal with anonymous nodes. In-
	deed, anonymous nodes are always considered as different
	by the JENA implementation when the RDF specification
	states that 2 anonymous nodes that share the same proper-
	ties should be considered equal. Our version of RDFDiff
	correctly handles such anonymous node (that are heavily
	used in the LEMON model). With this implementation, it
	is now easy to compute the difference between the original
	extraction and the new one and to decide, based on these
	differences, if the new version is good enough for produc-
	tion.

Table 2 :

 2 Number of lexico-semantic relations in the graphs.

https://code.google.com/p/gwtwiki/

http://kaiko.getalp.org/about-dbnary

Acknowledgements

This work was conducted as a part of the CHIST-ERA CAMOMILE project, which was funded by the ANR (Agence Nationale de la Recherche, France).