Natalia Grabar
email: natalia.grabar@univ-lille3.fr

Iris Eshkol-Taravela

Detection of Reformulations in Spoken French

Keywords: Spoken Corpora, Reformulation, Reformulation Marker, Paraphrase, Supervised Machine Learning

Our work addresses automatic detection of enunciations and segments with reformulations in French spoken corpora. The proposed approach is syntagmatic. It is based on reformulation markers and specificities of spoken language. The reference data are built manually and have gone through consensus. Automatic methods, based on rules and CRF machine learning, are proposed in order to detect the enunciations and segments that contain reformulations. With the CRF models, different features are exploited within a window of various sizes. Detection of enunciations with reformulations shows up to 0.66 precision. The tests performed for the detection of reformulated segments indicate that the task remains difficult. The best average performance values reach up to 0.65 F-measure, 0.75 precision, and 0.63 recall. We have several perspectives to this work for improving the detection of reformulated segments and for studying the data from other points of view.

Introduction

Reformulations may occur in written and spoken languages, in which they show different functions [START_REF] Flottum | Dire et redire. La reformulation introduite par "c'est-à-dire[END_REF][START_REF] Rossari | De l'exploitation de quelques connecteurs reformulatifs dans la gestion des articulations discursives[END_REF]: in spoken language, they mark the elaboration of ideas, and are punctuated by hesitations, false starts, and repetitions [START_REF] Blanche-Benveniste | Le franc ¸ais parlé[END_REF], in written documents, we usually find the final result of the reformulation process [START_REF] Lang | L'homme de paroles[END_REF]. It is considered that reformulation is the activity of speakers built on their own linguistic production or on the one of their interlocutor, with or without specific markers. The objective is then to modify some aspects (lexical, syntactic, semantic, pragmatic) but to keep the semantic content constant [START_REF] Gülich | Les actes de refor-mulation dans la consultation. La dame de Caluire[END_REF][START_REF] Kanaan | Reformulations, contacts de langues et compétence de communication: analyse linguistique et interactionnelle dans des discussions entre jeunes Libanais francophones[END_REF]. Specific reformulation markers may provide the formal mark-up of reformulations. Reformulation is closely related to paraphrases, in that way that reformulated sequences can produce the paraphrases [START_REF] Neveu | Dictionnaire des sciences du langage[END_REF]. Reformulation and paraphrase play an important role in languages:

• When studying languages, a common exercise consists of paraphrasing expressions in order to control their understanding by students;

• In the same way, it is possible to control the understanding of ideas. The first exercises of the kind have appeared with the exegesis of ancient texts: sacred texts (Bible, Koran, Torah) first, and then theological, philosophical and scientific texts;

• More naturally, speakers use the reformulation and paraphrase in order to precise and to better transmit their thoughts. It is also common to find reformulations in written language: between various versions of the same literary piece of work [START_REF] Fuchs | La paraphrase[END_REF], of the Wikipedia articles [START_REF] Vila | Relational paraphrase acquisition from wikipedia: The WRPA method and corpus[END_REF], or of scientific articles. The authors can thus rewrite several times their text until they produce the one that suits them at last.

Reformulation and paraphrase also play an important role in different NLP applications [START_REF] Madnani | Generating phrasal and sentential paraphrases: A survey of data-driven methods[END_REF][START_REF] Androutsopoulos | A survey of paraphrasing and textual entailment methods[END_REF][START_REF] Bouamor | Étude bilingue de l'acquisition et de la validation automatiques de paraphrases sous-phrastiques[END_REF]. The objective is to detect linguistic expressions that differ by their form but convey the same or similar meaning:

• In information retrieval and extraction, paraphrases permit to increase the coverage of the found or extracted results. For instance, pairs like {myocardial infarction, heart attack} and {Alzheimer's disease, neurodegenerative disease} contain different expressions that convey identical or close semantics;

• In machine translation, paraphrases permit to avoid lexical repetitions [START_REF] Scarpa | La Traduction spécialisée. Une approche professionnelle à l'enseignement de la traduction[END_REF];

• Textual entailment [START_REF] Dagan | Recognizing Textual Entailment[END_REF] consists of creating relation between two textual segments, called Text and Hypothesis. Entailment is a directional relation, in which the truth of the Hypothesis must be inferred through the analysis of the Text. For instance, the Text The drugs that slow down or halt Alzheimer's disease work best the earlier you administer them allows inferring that the Hypothesis Alzheimer's disease is treated by drugs is true; while the Hypothesis Alzheimer's disease is cured by drugs cannot be inferred from this Text. In this example, the paraphrases {administer drugs, treated by drugs} permit to establish the right link between the Text and the Hypothesis.

As these few examples indicate, reformulation and paraphrase may cover various linguistic phenomena. The corresponding classifications may be more or less complex: from 25 [START_REF] Bhagat | What is a paraphrase?[END_REF] to 67 [START_REF] Melčuk | Paraphrase et lexique dans la théorie linguistique sens-texte[END_REF] categories. Most often, these classifications address one given aspect, such as linguistic characteristics [START_REF] Melčuk | Paraphrase et lexique dans la théorie linguistique sens-texte[END_REF][START_REF] Vila | Paraphrase concept and typology. a linguistically based and computationally oriented approach[END_REF][START_REF] Bhagat | What is a paraphrase?[END_REF], size of the paraphrased units [START_REF] Flottum | Dire et redire. La reformulation introduite par "c'est-à-dire[END_REF][START_REF] Fujita | Typology of paraphrases and approaches to compute them[END_REF][START_REF] Bouamor | Étude de la paraphrase sousphrastique en traitement automatique des langues[END_REF], knowledge required for understanding the paraphrastic relation [START_REF] Milicevic | La paraphrase : Modélisation de la paraphrase langagière[END_REF], language register. To our knowledge, there is only one multidimensional classification of paraphrase [START_REF] Milicevic | La paraphrase : Modélisation de la paraphrase langagière[END_REF]. In our work, we also propose to use a multidimensional classification, that covers the following dimensions, some of which are inspired by the previous works [START_REF] Gulich | Les marqueurs de la reformulation paraphrastique[END_REF][START_REF] Beeching | La co-variation des marqueurs discursifs[END_REF][START_REF] Vila | Paraphrase concept and typology. a linguistically based and computationally oriented approach[END_REF]:

• syntactic category of the reformulated segments,

• type of lexical relation between the segments (e.g. hyperonymy, synonymy, antonymy, instance, meronymy),

• type of lexical modification (e.g. replacement, removal, insertion),

• type of morphological modification (i.e. inflection, derivation, compounding),

• type of syntactic modification (e.g. passive/active way),

• type of pragmatic relation between the reformulated segments (e.g. definition, explanation, precision, result, linguistic correction, referential correction, equivalence).

Existing works in the automatic acquisition of paraphrases

Several approaches have been proposed for the automatic detection of paraphrases. As explained above, in our work, we associate reformulation and paraphrase, which can be seen as the result of reformulation. Usually, the existing approaches exploit paradigmatic properties of words and their capacity to replace each other in a given context. These approaches depend on the corpora exploited. Four types of corpora are usually distinguished: monolingual, monolingual parallel, monolingual comparable, and bilingual parallel.

Monolingual corpora. Two kinds of approaches may be used with monolingual corpora:

• computing the similarity of strings permits to detect linguistic units (words, expressions, etc.) that show common surface features such as with {When did Charle de Gaulle die?, Charles de Gaulle died in 1970} [START_REF] Malakasiotis | Learning textual entailment using SVMs and string similarity measures[END_REF],

• distributional methods allow to detect units that occur in similar contexts. Such units have similar contextual or syntactic vectors, and may be good candidates for the paraphrase (e.g. {Y is solved by X, Y is resolved in X}) [START_REF] Lin | Dirt -discovery of inference rules from text[END_REF][START_REF] Pasc ¸a | Aligning needles in a haystack: Paraphrase acquisition across the Web[END_REF].

Monolingual parallel corpora. When a given text is translated more than once in another language, these translations allow to build monolingual parallel corpora. One of the most used corpora is Jules Verne's 20 000 lieux sous la mer that has been translated twice in English. Once these corpora are aligned at the sentence level, it is possible to exploit them with word alignment tools [START_REF] Och | Improved statistical alignment models[END_REF]. Various methods have been proposed for such exploitation [START_REF] Ibrahim | Extracting structural paraphrases from aligned monolingual corpora[END_REF][START_REF] Quirk | Monolingual machine translation for paraphrase generation[END_REF][START_REF] Barzilay | Extracting paraphrases from a parallel corpus[END_REF]. They allow to extract paraphrases such as {countless, lots of}, {undertone, low voice}, {shrubs, bushes}, {refuse, say no}, {dull tone, gloom} [START_REF] Barzilay | Extracting paraphrases from a parallel corpus[END_REF]. Monolingual comparable corpora. Monolingual comparable corpora typically contain texts on the same event but created independently, such as news articles. The thematic coherence of these texts and the distributional methods or alignment of comparable sentences may lead to the detection of paraphrases [START_REF] Shinyama | Automatic paraphrase acquisition from news articles[END_REF][START_REF] Sekine | Automatic paraphrase discovery based on context and keywords between NE pairs[END_REF][START_REF] Shen | Adding syntax to dynamic programming for aligning comparable texts for the generation of paraphrases[END_REF]. More particularly, named entities and numbers are part of the clues used for the extraction of paraphrases, such as in {PERS1 killed PERS2, PERS1 let PERS2 die from loss of blood} or {PERS1 shadowed PERS2, PERS1 kept his eyes on PERS2} [START_REF] Shinyama | Automatic paraphrase acquisition from news articles[END_REF]. Bilingual parallel corpora. Bilingual parallel corpora typically contain translations of a given text in another language. Once they are aligned at the level of sentences, they can also be used for the detection of paraphrases. Different translations of a given linguistic unit can provide paraphrases [START_REF] Bannard | Paraphrasing with bilingual parallel corpora[END_REF][START_REF] Callison-Burch | Parametric: An automatic evaluation metric for paraphrasing[END_REF][START_REF] Kok | Hitting the right paraphrases in good time[END_REF]. For instance, the paraphrases {under control, in check} can be extracted because they are translations of unter kontrolle [START_REF] Bannard | Paraphrasing with bilingual parallel corpora[END_REF].

Objectives

We have multi-fold objectives:

1. propose annotation guidelines for reformulations and to test them when annotating enunciations from French spoken corpora. These guidelines are presented in a previous work (Eshkol-Taravella and Grabar, 2014) and outlined at the end of Section 1.

They allow creating the reference data;

2. study three specific reformulation markers: c'est-àdire (in other words), je veux dire (that is to say / I mean), and disons (let's say). Several other reformulation markers exist (notamment, en d'autres mots, en d'autres termes...), but we propose to concentrate here on these three markers, coined on the verb dire (to say);

3. work with specific structure: source-entity marker target-entity, such as in example (1), in which the two entities démocratiser l'enseignement (democratize the education) and permettre à tout le monde de rentrer en faculté (allow everybody to enter the university) show the reformulation relation expressed syntagmatically;

4. distinguish enunciations that contain reformulations around the markers studied. This aspect is processed by the method proposed in Section 5.2.;

5. detect the segments that are reformulated: {démocratiser l'enseignement, permettre à tout le monde de rentrer en faculté} ({democratize the education, allow everybody to enter the university}). This aspect is processed by the method proposed in Section 5.3. We use three reformulation markers (RM): c'est-à-dire [START_REF] Gulich | Les marqueurs de la reformulation paraphrastique[END_REF][START_REF] Hölker | Zur Analyse von Markern[END_REF][START_REF] Beeching | La co-variation des marqueurs discursifs[END_REF], je veux dire (Teston-Bonnard, 2008) and disons [START_REF] Hwang | alors", "enfin" et "disons" en franc ¸ais parlé contemporain[END_REF][START_REF] Petit | Discrimination prosodique et représentation du lexique : application aux emplois des connecteurs discursifs[END_REF][START_REF] Saunier | Disons: un impératif de dire? Remarques sur les propriétés du marqueur et son comportement dans les reformulations[END_REF]. The common point between them is that they are coined on the basis of the verb dire (to say). In the existing work, these markers are recognized for their capacity to introduce reformulations, although they can play other roles, such as argumentation and disfluencies.

Corpora processed

We

Disfluency markers

We use a set of disfluency markers: allez, allons, alors, là, enfin, euh, heu, bah, ben, hm, hum, hein, quoi, ah, oh, donc, bon, bè, eh.

Reference data

In the reference data, the reformulations are annotated through a multidimensional classification: syntactic, lexical, morphological and functional properties (see example (2)) annotated using dedicated guidelines.

(2) euh <VP1>démocratiser l'enseignement </VP1> <RM>c'est-à-dire</RM> <VP2 rel-lex="syno(démocratiser/permettre à tout le monde) mero(enseignement/faculté)" modif-lex="ajout(rentrer à)" rel-pragm= "explic"> permettre à tout le monde de rentrer en fac-ulté</VP2> modif-lex="insertion(enter)" rel-pragm="explic"> allow everybody to enter the university</VP2>) Three annotator pairs have participated in the creation of the reference data. After the independent annotation, consensus has been reached. The inter-annotator agreement is computed on the judgment of whether a given enunciation contains a reformulation or not (in which case, the marker introduces disfluency, argumentation...). The interannotator agreement [START_REF] Cohen | A coefficient of agreement for nominal scales[END_REF]) is substantial 0.617 in ESLO1 and moderate 0.526 in ESLO2 [START_REF] Landis | The measurement of observer agreement for categorical data[END_REF]. On the whole, 611 enunciations in ESLO1 and 498 enunciations in ESLO2 are annotated. Currently, 168 reformulations are provided by ESLO1 and 186 by ESLO2 (59 and 37 recordings, respectively). The rate of reformulations is 28% in ESLO1 and 37% in ESLO2. These reference data are used for making observations, for training the system and for evaluating the automatically obtained results.

Approaches for the processing and detection of reformulations

In Figure 1, we present the general schema of the method composed of several steps: preprocessing of data (Section 5.1.), detection of enunciations with reformulations (Section 5.2.). Only those that contain reformulations are processed further. We then perform the detection of reformulated segments (Section 5.3.) and evaluate the results (Section 5.4.).

Preprocessing of data

Transcriptions from the ESLO corpora have adopted standard French spelling and non-use of punctuation. The original segmentation is done with the intuitive unit breath group detected by human transcribers, or with the speech turn detected with the change of speakers. We have rebuilt the enunciations using the speech turns with the change of speakers, and the overlappings when two or more speakers speak at the same time. With overlappings, the corresponding segments are associate to enunciations of each of the involved speakers. The processed unit corresponds to one enunciation.

Enunciations are processed with the SEM chunker [START_REF] Tellier | [END_REF] adapted to French spoken language in order to detect the minimal chunks.

Detection of enunciations with reformulations

We consider that there is no reformulation if:

1. the context is not sufficient (beginning or end of enunciation);

2. markers occur with disfluencies, primes (s-), etc., [START_REF] Blanche-Benveniste | Le franc ¸ais parlé[END_REF];

3. markers occur in specific contexts (use of nous (we) with disons then meaning (we say));

4. markers are found within existing expressions, like indépendamment de (independently on) (example (3)).

The last test is done with the chunker output without markers and disfluencies: we record the frequencies obtained for the tested segments. Several thresholds of frequency are tested (e.g. 60, 80, 100). If the observed frequency is higher than the thresholds, we consider that the segment contains an existing expression with disfluency.

Extraction of segments

Extraction of reformulated segments

The boundaries of the reformulated segments are detected with a CRF machine learning algorithm [START_REF] Lavergne | Practical very large scale CRFs[END_REF]. The reference data are divided in training and test sets with 60% and 40% of the enunciations, respectively.

Categories to be detected

The objective is to detect the two reformulated segments: the source segment, that is reformulated later in the text, and the target segment, that proposes new linguistic expression of the idea already expressed by the source segment. The categories to be detected are the following:

1. M: reformulation marker, 2. SEG1: source segment, which occurs before the marker, 3. SEG2: target segment, which occurs after the marker, 4. O: other tokens (out position).

Set of features

Each word is described with a set of features (Table 1):

• form: form of each word as it occurs in text;

• POS: POS tag computed by SEM;

• chunksBI: SEM chunks with beginning and end markup;

• chunks: SEM chunks without beginning and end mark-up;

• heu: we check out whether the word is part of the list with the disfluency markers (Section 4.3.);

• num: number of each word;

• début/milieu/fin: relative position of each word. Possible values: beginning (first 20% of words), end (last 20% of words), and middle M IL (all the rest);

• stem: words stemmed by Snowball [START_REF] Porter | An algorithm for suffix stripping[END_REF];

• RM: mark-up of the RM.

CRF patterns

The CRF patterns provide the possibility to indicate how the features must be exploited: their combinations and the context they must be studied within. We propose two sets of experiments:

1. within context from 2*3 to 2*12 words before and after a given token, we use the form and the combination form/RM;

2. combinations of various features in a 2*7-word window size around a given token.

Evaluation

The evaluation is done through the comparison with the reference data. We compute precision, recall and F-measure at the level of categories [START_REF] Sebastiani | Machine learning in automated text categorization[END_REF]. The baseline corresponds to the use of the form and combination form/RM in the 2*7-word window. This is the basic information directly available in the text, within the average size context.

Results

Detection of enunciations with reformulations

For the detection of enunciations with reformulations, filters 1, 3 and 4 provide the best combination. Second rule, stating that when markers occur with disfluencies and primes there is no reformulation, is not efficient. Indeed, in spoken language, disfluencies can occur around reformulated segments. With filters 1, 3 and 4, the best precision obtained is 0.66%, which is higher than the inter-annotator agreement (0.617 in ESLO1 and 0.526 in ESLO2).

Extraction of reformulated segments

Tables 2 and3 present the results obtained during the step dedicated to the extraction of reformulated segments: we indicate precision, recall and F-measure for categories O, SEG1 and SEG2. 0,81 0,97 0,88 0,61 0,13 0,21 0,50 0,13 0,20 E2 (2*4 words) 0,81 0,95 0,88 0,51 0,18 0,27 0,39 0,13 0,20 E3 (2*5 words) 0,81 0,97 0,88 0,57 0,13 0,21 0,45 0,12 0,19 E4 (2*6 words) 0,81 0,97 0,88 0,59 0,11 0,19 0,46 0,12 0,19 E5 (2*7 words) -Baseline 0,82 0,94 0,88 0,47 0,13 0,21 0,37 0,18 0,24 E6 (2*8 words) 0,81 0,98 0,88 0,62 0,12 0,20 0,51 0,11 0,18 E7 (2*9 words) 0,81 0,97 0,88 0,59 0,11 0,19 0,49 0,14 0,22 E8 (2*10 words) 0,81 0,97 0,88 0,61 0,13 0,21 0,48 0,15 0,22 E9 (2*11 words) 0,82 0,96 0,88 0,61 0,20 0,30 0,51 0,16 0,24 E10 (2*12 words) 0,81 0,97 0,88 0,61 0,13 0,21 0,46 0,15 0,22 We can propose several additional observations:

• the O positions are well detected,

• detection of the source and target segments remains difficult and shows variable performance,

• among the best experiments, we count the baseline (use of forms within 2*7-word context and of combination form/RM) and the experiments based on various combinations of features (Table 3),

• 2*4, 2*7 and 2*11 words are among the optimal context sizes.

Discussion

We also processed separately the ESLO1 and ESLO2 corpora, and the non-consensual annotations (annotators A1 and A2). The models created on each dataset (ESLO1/A1, ESLO1/A2, ESLO2/A1, ESLO2/A2) have been applied on other datasets in order to study the portability of these models. These experiments indicate that:

• it is easier to detect reformulated segments in ESLO1, O SEG1 SEG2 Various combinations P R F P R F P R F Baseline
0,82 0,94 0,88 0,47 0,13 0,21 0,37 0,18 0,24 E1 0.84 0.94 0.89 0.47 0.42 0.44 0.70 0.17 0.27 E2 (E1 + 2*7 words(chunk/RM)) 0.83 0.36 0.50 0.12 0.59 0.20 0.27 0.52 0.36 E3 (E2 + 2*7 words(POS/RM))

0 whatever the model (corpus or annotator). Indeed, ESLO2 contains much longer enunciations, which makes the segment detection more difficult;

• models related to annotators also show various performance, although less important than those related to the influence of corpora;

• quite frequently, detection of the target segments SEG2 is easier than detection of the source segments SEG1.

Experiments with consensual reference data and merged corpora slightly improve the results.

Analysis of errors

An analysis of the results indicates that often segments can be detected but with frontiers different from those expected by the reference data. In Figure 2, we present two examples: in A we can find the manual reference annotations and in B the automatically detected segments. Reformulated segments are in blue and underlined. We observe that in these examples, the automatically detected segments are larger than the reference annotations. Notice that several of the automatically proposed solutions are acceptable: similarly, during the manual annotation process, the annotators had several hesitations as for the right size of the segments. We assume that the automatically computed output can be used as basis for the manual annotation.

As indicated above, it can be easier to detect target segment that source segment (example (4)). Another type of errors is observed when RMs are merged with one of the segments. Notice that this mainly happens with the marker disons, which is the less grammaticalized in the function of reformulation.

Significance

Going beyond the theoretical contribution to the studies on reformulations and paraphrases, there is also a practical significance of the proposed work. The methods proposed and the results provided by these methods can be useful for studying and comparing various corpora, both spoken and written, from the point of view of discursive and reformulation structures they use. For instance, in spoken language, it can be interesting to know in which discusison conditions speakers need to better explain thier ideas and to produce more reformulations. In addition, reformulations and paraphrases are related in the way that reformulations may provide paraphrases. As we already noticed in Section 1., paraphrases are very useful in several NLP applications such as information retrieval and extraction, textual entailment, machine translation, etc.

Conclusion

We have proposed a work on automatic filtering of enunciations with reformulations and on extraction of reformulated segments. The work shows two main original points:

• the reformulations are studied in syntagmatic structures, and more specifically, they can be found within the following structure: SEG1 RM SEG2

• the work is done with spoken corpora, in which reformulations are frequent and can be observed thanks to specific markers.

We proposed to use a rule-based system for filtering out enunciations without reformulations, and a CRF-based system for the automatic detection and extraction of reformulated segments. Various rules, features, their combinations and context sizes are tested. When filtering out the enunciations without reformulations, we reach up to 0.66% precision, which is higher than the inter-annotator agreement. Among the best experiments proposed for the extraction of reformulated segments, we find the baseline (use of forms and of combination form/RM), and of various combinations of features (e.g. chunk/RM, pos/RM, stem/RM) in 2*7-word context. The best average results reach up to 0.65 F-measure, 0.75 precision and 0.63 recall. We observe that the reformulated segments remain difficult to be detected correctly: Fmeasure is seldom higher than 0.30 for both segments.

Future Work

We have several directions for future work. We can test other classifiers, such as Long Short Term Memory (LSTM) [START_REF] Hochreiterand | Long shortterm memory[END_REF], and other features and their combinations. Once stabilized, the models can be used for pre-annotating new data and preparing data for human annotation. We assume this may help the manual annotation. We plan also to use the models generated with these three markers on enunciations with other RMs (e.g. c ¸a veut dire, j'allais dire, notamment, autrement dit, en d'autres termes, en d'autres mots) and other corpora. This will enable to study whether reformulations introduced by different markers have common regularities.

Other directions for future work consist of analyzing these data from other points of view. For instance, we can also study prosodic and acoustic features for improving the distinction between enunciations with and without reformulations (Section 5.2.). The hypothesis is that enunciations with reformulations may also show phonetic specificities. This step may involve a machine learning-based system as well, instead of the rule-based system. Study of reformulations in written corpora (discussion fora and journalistic news) is yet another perspective. Since the two spoken corpora exploited have been built with similar principles but with 40 year difference, this offers the possibility to perform a diachronic study of PMs. Finally, we plan to exploit the paraphrases generated in this work in other NLP applications, such as information retrieval and extraction.

  enseignement c'est-à-dire permettre à tout le monde de rentrer en faculté (democratize the education in other words allow everybody to enter the university) [ESLO1 ENT 121 C]

Figure 1 :

 1 Figure 1: General schema of the method.

  enfin par industriel je veux dire euh j'ai le côté commercial B. oui enfin par industriel je veux dire euh j'ai le côté commercial [ESLO1 ENT 002 C] (oh well by industrial I mean euh I have commercial dispositions)

Figure 2 :

 2 Figure 2: Analysis of the automatic detection of reformulated segments: difference in the detection of frontiers of the segments.

  work with the ESLO (Enquêtes Sociolinguistiques à Orléans) corpora (Eshkol-Taravella et al., 2012): ESLO1 and ESLO2. ESLO1 has been created in 1968-1971 by French teachers from the Essex university, Language Centre, Colchester, UK, in collaboration with the B.E.L.C. lab (Bureau pour l'étude de l'enseignement de la langue et de la civilisation franc ¸aises de Paris). ESLO1 contains 300 hours of speech (4,500,000 words approximately) and includes several types of recordings. Building of ESLO2 started in 2008. It will contain over 350 hours of speech. ESLO1 and ESLO2 are accessible online (http://eslo.tge-adonis.fr/).

Table 2

 2

	indicates the results when only forms and combina-
	tion form/RM are used. The difference between the exper-
	iments is due to the variation of the context size from 2*3
	to 2*12 tokens. This set of experiments gives the average
	value around 0.60: the average precision is around 0.70,

while the average recall is around 0.58. Reformulated segments are detected with precision between 0.40 and 0.62, although the recall values are low (less than 0.20). The baseline is competitive. The best average values are observed with the context size with 2*11 and 2*4 words. Table

3

indicates the results obtained with various combinations of features in the 2*7-word context. The experiments

Table 1 :

 1 Excerpt from the annotated data, with reformulation.

			O			SEG1			SEG2
	Context size	P	R	F	P	R	F	P	R	F
	E1 (2*3 words)

Table 2 :

 2 Use of the form and of the combination form/RM within 2*3 to 2*12 context size. It seems that use of additional features, and removal of the POS and heu features are beneficial.

	outperform the baseline. E1 corresponds to the use of sev-
	eral features (POS, chunk, heu, stem, RM) within 2*7-word
	context, and of several combinations (chunk/RM, POS/RM,
	stem/RM) within 2*1-word context. Other experiments de-
	rive from E1. The experiments show that the best detection
	of reformulated segments (F-measure over 0.30) are E4, E6
	and E8.

Table 3 :

 3 Various combinations of features within 2*7-word context. The experiment E1 corresponds to the combination of several features (POS, chunk, heu, stem, RM) within 2*7-word context, and of chunk/RM, POS/RM, stem/RM within 2*1-word context. Other experiments derive from E1.

		.85 0.19 0.32 0.36 0.43 0.39 0.15 0.93 0.26
	E4 (E3 + 2*7 words(stem/RM))	0.87 0.36 0.51 0.20 0.65 0.31 0.21 0.69 0.32
	E5 (E4 + 2*7 words(début))	0.83 0.46 0.60 0.27 0.50 0.35 0.16 0.55 0.25
	E6 (E5 -2*7 words(POS))	0.84 0.48 0.61 0.23 0.52 0.32 0.22 0.66 0.33
	E7 (E5 -2*7 words(chunk))	0.84 0.26 0.40 0.20 0.60 0.30 0.18 0.78 0.30
	E8 (E5 -2*7 words(heu))	0.85 0.39 0.53 0.24 0.45 0.32 0.20 0.81 0.32
	E9 (E5 -2*7 words(stem))	0.85 0.32 0.47 0.31 0.28 0.29 0.15 0.82 0.26
	E10 (E5 -2*7 words(RM))	0.83 0.42 0.56 0.17 0.47 0.25 0.22 0.69 0.34

  A. et cinq kilomètres c'est-à-dire j'avais quatre kilomètres à faire quatre et quatre huit je faisais huit kilomètres tous les jours et à pieds ah oui B. et cinq kilomètres c'est-à-dire j'avais quatre kilomètres à faire quatre et quatre huit je faisais huit kilomètres tous les jours et à pieds ah oui [ESLO1 ENT 011 C] (and five kilometers in other words I had four kilometers to make four and four gives eight I was making height kilometers every day and by foot oh yes)A. et et vous par exemple approximativement vous combien de fois euh quelle est la fréquence avec laquelle vous regardez le dictionnaire c'est à dire une fois par mois une fois par an une fois par euh oh B. et et vous par exemple approximativement vous combien de fois euh quelle est la fréquence avec laquelle vous regardez le dictionnaire c'est à dire une fois par mois une fois par an une fois par euh oh [ESLO1 ENT 047 C] (and and you for instance approximately how many times euh what is the frequency with which you use the dictionary in other words once a moth once a year once a euh oh)


Natalia Grabar
email: natalia.grabar@univ-lille3.fr

Thierry Hamon
email: hamon@limsi.fr

A Large Rated Lexicon with French Medical Words

Keywords: Medical area, Medical terminology, Readability, Annotation, Inter-annotator agreement, French

Patients are often exposed to medical terms, such as anosognosia, myelodysplastic, or hepatojejunostomy, that can be semantically complex and hardly understandable by non-experts in medicine. Hence, it is important to assess which words are potentially non-understandable and require further explanations. The purpose of our work is to build specific lexicon in which the words are rated according to whether they are understandable or non-understandable. We propose to work with medical words in French such as provided by an international medical terminology. The terms are segmented in single words and then each word is manually processed by three annotators. The objective is to assign each word into one of the three categories: I can understand, I am not sure, I cannot understand. The annotators do not have medical training nor they present specific medical problems. They are supposed to represent an average patient. The inter-annotator agreement is then computed. The content of the categories is analyzed. Possible applications in which this lexicon can be helpful are proposed and discussed.

Introduction

The medical field has a very specific terminology (anosognosia, myelodysplastic, hepatojejunostomy or anastomosis) commonly used by medical professionals. For its correct understanding, an extensive knowledge is required, as indeed it appears difficult to understand information contained in drug package inserts [START_REF] Patel | Errors in interpreting quantities as procedures : The case of pharmaceutical labels[END_REF], in websites [START_REF] Rudd | Annual Review of Adult Learning and Literacy[END_REF][START_REF] Berland | Health information on the internet. accessibility, quality, and readability in english ans spanish[END_REF][START_REF] Mccray | Promoting health literacy[END_REF]Oregon Practice Center, 2008), and more generally in patients and medical doctors communication [START_REF] Mccray | Promoting health literacy[END_REF][START_REF] Jucks | Choice of words in doctor-patient communication: an analysis of healthrelated internet sites[END_REF][START_REF] Tran | Internet et soins : un tiers invisible dans la relation médecine/patient ?[END_REF]. In this work, we present a lexicon with French medical words, which have been rated by human annotators on a scale with three positions: I can understand, I am not sure, I cannot understand. Study of the understanding of words used in medical field is the first step towards the simplification of medical texts. Indeed, before the simplification can be performed, it is necessary to know which textual units may show understanding difficulty and should be simplified.

Related Work

The understanding of words is a complex notion closely linked to different research fields, such as linguistics (Section 2.1.), psycholinguistics (Section 2.2.) and Natural Language Processing (NLP) (Section 2.3.).

Linguistics

In linguistics, the question is closely related to the lexical complexity and composition. It has been indeed observed that several factors may be involved in semantic complexity:

1. Knowledge of the components of complex words, which permits to correctly segment the words and then to understand them (e.g. appendicitis, otitis, reticulitis);

2. Variety of morphological patterns and relations among components, [START_REF] Booij | Construction Morphology[END_REF] which permits to apply patterns such as [N1N2] in erythrocyte and ovocyte, and then to induce the relation between the components and the semantics of the whole lexeme;

3. Polysemous components [START_REF] Iacobini | Distinguishing derivational prefixes from initial combining forms[END_REF][START_REF] Amiot | Integrating combining forms into a lexeme-based morphology[END_REF], which may make the understanding more complicated;

4. Difference in the order of components [START_REF] Iacobini | Composizione con elementi neoclassici[END_REF], which gives the clues for the interpretation of compounds (e.g. snow tyre is a kind of tyre, erythrocyte is a kind of cyte/cell).

Psycholinguistics

In psycholinguistics, specific research questions are studied and some of them can be related to those studied in linguistics:

• Knowledge of components within complex words and their decomposition, which main purpose is to study how the complex words are processed and recorded in brain and then to make hypotheses on how these words are decoded and produced by speakers. Thus, it has been demonstrated that several factors can facilitate reading and production of complex words: hyphenation [START_REF] Bertram | The hyphen as a segmentation cue in triconstituent compound processing: It's getting better all the time[END_REF], space character [START_REF] Frisson | The role of semantic transparency in the processing of english compound words[END_REF], presence of other morphologically related words [START_REF] Lüttmann | Sharing morphemes without sharing meaning: Production and comprehension of german verbs in the context of morphological relatives[END_REF], use of primes [START_REF] Bozic | Differentiating morphology, form, and meaning: Neural correlates of morphological complexity[END_REF][START_REF] Beyersmann | Parallel processing of whole words and morphemes in visual word recognition[END_REF], of pictures [START_REF] Dohmes | The impact of semantic transparency of morphologically complex words on picture naming[END_REF][START_REF] Koester | The functional neuroanatomy of morphology in language production[END_REF] and of supportive contexts [START_REF] Cain | The development of idiom comprehension: An investigation of semantic and contextual processing skills[END_REF];

• Order of components and variety of morphological patterns, which focus on the position of the components. It has been observed that these factors show a stable influence for the recognition of complex words [START_REF] Libben | Compound fracture: The role of semantic transparency and morphological headedness[END_REF][START_REF] Holle | The time course of lexical access in morphologically complex words[END_REF][START_REF] Feldman | Morphological priming: The role of prime duration, semantic transparency, and affix position[END_REF]. The notion of the morphological headedness has been isolated [START_REF] Jarema | Processing compounds: A crosslinguistic study[END_REF][START_REF] Libben | Compound fracture: The role of semantic transparency and morphological headedness[END_REF]: the related work indicates that the morphological head plays an important role in decomposition of complex words, in detection of decomposition patterns, and more generally in lexical activity.

• Impact of the word length and of the types of affixes [START_REF] Meinzer | Opaque for the reader but transparent for the brain: Neural signatures of morphological complexity[END_REF] and the frequency of bases [START_REF] Feldman | What do graded effects of semantic transparency reveal about morphological processing?[END_REF].

Natural Language Processing

In the NLP area, a variety of work is dedicated to the word understanding and readability studies. The purpose is to decide whether given documents are accessible for a given reader. The readability measures are widely used for evaluating complexity of documents. Among the existing measures, it is possible to distinguish classical and computational measures:

• Classical measures usually exploit information on number of characters and/or syllables within words, and on linear regression models [START_REF] Flesch | A new readability yardstick[END_REF][START_REF] Gunning | The art of clear writing[END_REF]);

• Computational measures can use vector models and a great variety of descriptors, among which the following have been utilized: combination of classical readability measures with medical terminologies [START_REF] Kokkinakis | Comparing lay and professional language in cardiovascular disorders corpora[END_REF]; n-grams of characters [START_REF] Poprat | A language classifier that automatically divides medical documents for experts and health care consumers[END_REF]; stylistic [START_REF] Grabar | Classification of health webpages as expert and non expert with a reduced set of cross-language features[END_REF] and discursive [START_REF] Goeuriot | Caractérisation des discours scientifique et vulgarisé en franc ¸ais, japonais et russe[END_REF] descriptors; lexicon [START_REF] Miller | A classifier to evaluate language specificity of medical documents[END_REF]; morphological information [START_REF] Chmielik | Détection de la spécialisation scientifique et technique des documents biomédicaux grâce aux informations morphologiques[END_REF]; combination of different kinds of descriptors [START_REF] Wang | Automatic recognition of text difficulty from consumers health information[END_REF][START_REF] Zeng-Treiler | Text characteristics of clinical reports and their implications for the readability of personal health records[END_REF][START_REF] Leroy | Evaluating online health information: Beyond readability formulas[END_REF][START_REF] Franc ¸ois | Les apports du TAL à la lisibilité du franc ¸ais langue étrangère[END_REF].

Description of the Source Terminology

The source terms are obtained from the medical terminology Snomed International [START_REF] Côté | Répertoire d'anatomopathologie de la SNOMED internationale[END_REF] in French, available from the ASIP SANTE website1 . The purpose of this terminology is to provide an extensive description of the medical field. Snomed contains 151,104 medical terms structured into eleven semantic axes such as disorders and abnormalities, procedures, chemical products, living organisms, anatomy, social status, etc. We keep here five axes related to the main medical notions (disorders, abnormalities, procedures, functions, anatomy), which are the categories the most often used in medical texts. The objective is not to consider axes such as chemical products (hydrogen sulfide) and living organisms (Sapromyces, Acholeplasma laidlawii) that group very specific terms, hardly

Pre-processing of the Source Lexicon

The 104,649 selected terms are tokenized in words (or tokens), POS-tagged and lemmatized by TreeTagger [START_REF] Schmid | Probabilistic part-of-speech tagging using decision trees[END_REF] and then corrected by Flemm [START_REF] Namer | FLEMM : un analyseur flexionnel du franc ¸ais à base de règles[END_REF]. The syntactic categories are assigned to words within the context of their terms. In order to make syntactically acceptable structures, the terms are transformed in sentence. For instance, infarctus du myocarde becomes C'est un infarctus du myocarde ((This is a myocardial infarction)). When hyphenated, the prefixes are not separated from their bases (e.g. anti-virus), the compounds are not tokenized either (e.g. canaliculo-rhinostomie).

Similarly, the names of chemicals remain non-tokenized acétylgalactosaminyl-O-glycosyl-glycoprotéine.

In this way, we obtain 29,641 unique words. For instance, infarctus du myocarde (myocardial infarction) gives three words in French (infarctus, du, myocarde). ¿From the morphological point of view, this dataset contains three kinds of words:

• compound words which contain several bases: abdominoplastie (abdominoplasty), dermabrasion (dermabrasion);

• constructed (or derived) words which contain one base and at least one affix: cardiaque (cardiac), acineux (acinic), lipoïde (lipoid);

• simple words which contain one base, no affixes and possibly inflections (when the lemmatization fails): acné (acne), fragment (fragment).

In addition to French words, this dataset contains also borrowings from other languages (Latin, Greek, English), and abbreviations.

The stopwords are also removed during the pre-processing step.

Annotation Process

The 29,641 words from the original set are annotated by three French speakers, 25-40 year-old, without medical training, without specific medical problems, but with linguistic background. To our opinion, these annotators represent the average knowledge of medical words among the population as a whole. One of the authors participated in the annotation process.

The annotators are presented with a list of terms and asked to assign each word to one of the three categories:

1. I can understand the word;

2. I am not sure about the meaning of the word;

3. I cannot understand the word.

The assumption is that the words, which are not understandable by the annotators, are also difficult to understand by patients.

The annotators were asked not to use dictionaries during this annotation. Given the large number of words to process, the annotation process needed one to two months.

Categories

Description and Discussion of the Rated Lexicon with Medical Words

Description and discussion of the rated lexicon is done following several points: presentation of the annotation output (Section 6.1.), discussion of the annotation process (Section 6.2.), analysis and discussion of annotations (Section 6.3.), presentation of possible applications and expected impact of the lexicon (Section 6.4.).

Annotation Output

The results of the manual annotations are presented in Table 1. On the basis of the annotations, we can create five sets of data: one for each annotator (A1, A2 and A3), the Unanimity set in which all the annotators agree, and the Majority set in which at least two annotators agree. We can see that the datasets corresponding to the three annotators provide very similar distribution of words among the three categories. The less frequent category is I am not sure (between 4 and 6% of the whole set of words), which means that the decision on the understanding of words from specialized areas is quite easy to perform for non-expert annotators. The most frequent category is I cannot understand: it gathers 66 to 70% of words. This means that highly specialized areas, such as medicine, contain a large number of specialized expert words which may require explanations in order to be correctly understood and used by patients.

Discussion of the Annotation Process

The annotation process implies only that the annotators give their opinion on their understanding of words and that they assign each word to one of three categories. Given the large size of the set of words (almost 30,000 words), it is impossible to verify whether this understanding is real or not. For collecting a more correct judgment on the understanding of words, it would be necessary to require that annotators also provide an explanation to the processed words, which should be then checked out [START_REF] Zeng | A text corpora-based estimation of the familiarity of health terminology[END_REF] by experts. As we indicated, such an approach is hardly possible when processing such a large set of words.

The categories are defined is such a way that they may include both (un)known words and words felt as such by the annotators. Several situations can occur, such as:

• when a word is known and understood, then this word is to be assigned to the I can understand category;

• when a word has already been read or heard by the annotator, but if its meaning is not known, if the word itself cannot be parsed, interpreted, and if the global meaning cannot be deduced by the annotator, then this word is to be assigned to the I cannot understand category;

• when a word is unknown but if its components (bases, affixes) can be parsed, interpreted, and the global meaning can be then deduced by the annotator, then this word can be assigned to the I can understand category;

• when a word is unknown and if its components (bases, affixes) cannot be parsed, interpreted, and the global meaning cannot be deduced by the annotator, then this word can be assigned to the I cannot understand category.

Analysis and Discussion of Annotations

The inter-annotator agreement is computed with the Cohen's Kappa [START_REF] Cohen | A coefficient of agreement for nominal scales[END_REF], applied to pairs of annotators, which values are then leveraged to obtain the unique average value; and Fleiss' Kappa [START_REF] Fleiss | The equivalence of weighted Kappa and the intraclass correlation coefficient as measures of reliability[END_REF], suitable for the processing of data provided by more than two annotators. The standard interpretation of the scores are for instance [START_REF] Landis | The measurement of observer agreement for categorical data[END_REF]: substantial agreement between 0.61 and 0.80, almost perfect agreement between 0.81 and 1.00. On our dataset, the inter-annotator agreement shows substantial agreement: Fleiss' Kappa 0.735 and Cohen's Kappa 0.736.

Regarding the Majority and Unanimity sets, they indicate that 878 words do not have the majority agreement, while up to 6,716 words do not have the unanimity agreement. An analysis of the content of these categories indicates that I cannot understand category contains several types of words, some of which have already been noticed in the related work (Section 2.):

• neoclassical compounds (e.g. coproporphyrie, abiotrophie, dermacentorose, dysurie, abomasopexie, angiomyoliposarcome, fistulo-végétant). As indicated in Section 2., compounds are also addressed in studies in linguistics and psycholinguistics because such words show complex morphological structure and are, for this reason, difficult to understand. Indeed, several factors induce this complexity (e.g. knowledge of the components, morphological patterns and relations, polysemy of components);

• abbreviations (e.g. ADPase, Pro-leu, Fyx, Gln-glu, Hga). Abbreviations correspond to specific representations of usually complex terms. Unless already known by a reader, the abbreviations provide very little information on their content and meaning. We assume that this is the main reason why abbreviations are systematically categorized as non-understandable;

• Latin borrowings (e.g. Chrysocoma, Adiaspiromycoses, aborta, urtica, aberrans, abdominalis, dolens). Since Latin borrowings are not commonly used in the modern French language, the current usage provides no or little examples with their occurrences and, for this reason, Latin borrowings may remain opaque for the speakers. Even if some of them show similar surface form with the corresponding French words, such as {abdominalis, abdomen}, the Latin grammatical and casual system may prevent their parsing and understanding;

• proper names (e.g. Christiansborg, Malacarne, Glasgow, Anton-Babinski). The difficulty with the proper names is that they are often used within specific contexts (e.g. disorder or laboratory test names). Besides, they are closely connected to the research work performed by the researchers who gave their name to a given disease or examination, to the place in which this research or discovery have been performed, etc.

Unless already known, it remains difficult to understand such words and terms properly;

• anatomical terms (e.g. coracohuméral, abdominalis, diaphragmatique, acral, endaurale). The main difficulty with the anatomical terms and words is that they convey very specific and precise meaning, are seldom used by non-experts in medicine and keep close links and resemblance with the corresponding Latin or Greek words;

• chemicals (e.g. N-acétylgalactosaminyltransférase, abequosyltransférase, P-crésol, aminocaproate, diméthylsulfoxide, UDP-N-acétylmuramoyl-L-alanyl-D-glutamyl-L-lysine). As for the chemical names, these are artificially coined words and terms [START_REF] Klinger | Detection of iupac and iupaclike chemical names[END_REF]. Usually, their objective is to describe or represent the structure of the corresponding chemical products and molecule. As noticed above for other categories of words, the names of the components of chemicals may also be borrowed from Latin.

Besides, the names of chemicals obey to specific morphological rules (World Health Organization, 2006), which remain opaque to non-experts.

Possible Applications and Impact of the Lexicon

This graded lexicon is freely available for the research purposes. It is accessible online2 . The proposed lexicon can be used in different ways:

• One of the questions asked when working with specialized areas is How many specialized words the medical area contains? Our work and the lexicon built can answer such questions precisely, because the I cannot understand category may be associated with specialized words, while the I can understand category may be associated with non-specialized words. Hence, the rate of specialized and technical words within the medical area reaches up to 70% (around 20,000 words), while up to 30% of words (around 8,000 words) may be considered as general-language words. Of course, the number of technical words and terms will increase if more medication and chemical names are considered;

• This lexicon provides a good material for training and fitting supervised or non-supervised machine-learning algorithms [START_REF] Gala | Towards a french lexicon with difficulty measures: NLP helping to bridge the gap between traditional dictionaries and specialized lexicons[END_REF][START_REF] Grabar | Automatic diagnosis of understanding of medical words[END_REF];

• This lexicon, build with material from specialized medical area, can be compared with similar generallanguage lexicon [START_REF] Gala | Towards a french lexicon with difficulty measures: NLP helping to bridge the gap between traditional dictionaries and specialized lexicons[END_REF] and with similar lexica from other specialized areas if available. Similarly, cross-language comparison may be performed if similar rated lexica exist in other languages. This may help in making contrastive studies across languages, discourses and domains;

• In relation with the readability task, the lexicon can be used for making the diagnosis of difficulty and understanding of a given piece of text [START_REF] Wang | Automatic recognition of text difficulty from consumers health information[END_REF][START_REF] Zeng-Treiler | Text characteristics of clinical reports and their implications for the readability of personal health records[END_REF][START_REF] Borst | Lexically based distinction of readability levels of health documents[END_REF][START_REF] Leroy | Evaluating online health information: Beyond readability formulas[END_REF];

• Going beyond the diagnosis of difficulty and understanding of a given piece of text, the lexicon can be used for the detection of zones in documents which contain complex and non-understandable terms and words. An example of this utilization is shown in Figure 1, in which potentially difficult words are marked in red. The recognition of the zones can be done by direct projection of the rated lexicon or through the application of a machine-learning algorithm trained with the rated lexicon and then applied to raw text [START_REF] Grabar | Automatic diagnosis of understanding of medical words[END_REF];

• In relation with the simplification task, we assume the words from the I cannot understand category must be provided with explanations and definitions in order to make them understandable by non-expert users of the medical and health texts. In this way, this lexicon may be used prior to the simplification tasks in order to guide the building of vocabulary with terms and words which meaning should be explained;

• In the patient-medical doctor communication, this lexicon may focus attention of the medical staff on notions which should be explained to patients in order to make the communication more successful and easy.

Conclusion

In this work, we propose to build a lexicon with French medical words, which are assigned to three categories (I can understand, I am not sure, I cannot understand). The The content of the obtained lexicon is then presented and discussed. Besides, we also outline some possible usages and applications for which the lexicon can be helpful. The lexicon is freely available for the research purposes.

It is accessible online at http://natalia.grabar. perso.sfr.fr/rated-lexicon.html.

Figure 1 :

 1 Figure 1: Detection of non-understandable words within clinical discharge summaries.

Table 1 :

 1 Number (and percentage) of words assigned to reference categories by human annotators (A1, A2 and A3).

		A1	A2	A3	Una.	Maj.
		Nb. (%)	Nb. (%)	Nb. (%)	Nb. (%)	Nb. (%)
	1. I can understand	8,099 (28,0%) 8,625 (29,0%) 7,529 (25,0%) 5,960 (26,0%) 7,655 (27,0%)
	2. I am not sure	1,895 (6,0%) 1,062 (4,0%) 1,431 (5,0%)	61 (0,3%)	597 (2,0%)
	3. I cannot understand 19,647 (66,0%) 19,954 (67,0%) 20,681 (70,0%) 16,904 (73,7%) 20,511 (71,0%)
	Total annotated words 29,641	29,641	29,641	22,925	28,763

http://esante.gouv.fr/services/referentiels/referentiels-dinteroperabilite/snomed-35vf known by laymen, and that can be easily categorized as non-understandable by patients.

http://natalia.grabar.perso.sfr.fr/ rated-lexicon.html


Thierry Hamon

Natalia Grabar
email: natalia.grabar@univ-lille3.frd

Adaptation of Cross-Lingual Transfer Methods for the Building of Medical Terminology in Ukrainian

Keywords:

teaching and research institutions in France or abroad, or from public or private research centers.

Introduction

Automatic acquisition of terminological resources has gone through a very active period and provides nowadays several automatic tools and methods [10,5,21] for several European languages and for Japanese. Nevertheless, other languages remain low-resourced and may require specific Natural Language Processing (NLP) developments, which must take into account morphological specificities of such languages [13,22], for instance. In the existing studies, statistical methods for extracting collocations and repeated segments are often exploited [26,6,9,22] and allow to extract results with reliable recall but low precision.

We propose to take advantage of the advanced research work done in languages like English or French, and to transpose it on low-resourced languages. For such objectives, we propose to exploit the transfer methodology together with parallel and aligned corpora. We consider that the transfer methodology can be suitable for the objectives related to terminology extraction. The principle is the following. Suppose we have parallel and aligned corpora with two languages L1 and L2, and we have several types of syntactic or semantic annotations and information associated to L1. The transfer approach permits to transpose these annotations or information from L1 to L2, and to obtain in this way the corresponding annotations and information in the L2 text. From this point of view, L1 is considered as the source language while L2 is considered as the target language. This kind of approach is particularly interesting when working with low-resourced languages for which less tools and semantic resources are available. An increasing availability of parallel bilingual corpora, and of automatic methods and tools for their processing makes it possible to build linguistic and terminological resources using the transfer methodology [29,15]. Very few works have been done in this direction, and we assume they open novel and efficient ways for the processing of multilingual texts in particular from lowresourced languages [30,16]. Notice that the modeling of cross-language features aims at using language-independent features to create various types of annotations. Among such features, we can mention part-of-speech, semantic categories or even acoustic and prosodic features.

In the following of this paper, we start with the presentation of the motivation for this study (section 2). We then present the material used for the acquisition of terminology (section 3), and the methods designed for achieving this objective (section 4). We discuss the results we obtain (section 5), and conclude with directions for future work (section 6).

Motivation and Rationale

The motivation of our work is double. We want (1) to design specific methods for the acquisition of terminological resources for low-resourced languages such as Ukrainian, and (2) to automatically build medical terminology for Ukrainian.

If little digitalized resources are currently available for Ukrainian, terminological work is an active research area there, although it mainly is concerned with theoretical and linguistic issues. For instance, terminological descriptions are available for several specialized domains and languages: physics [35]; law [40]; computer science [34,7]; religion [39]; literature [24]; Crimean tatar language [1,17]. Then, following recent research orientations, work on construction of electronic corpora [11,33], on their use for building of terminologies and dictionaries [38,31,32], and on transformation of traditional dictionaries in electronic format [37] appear. Still, little work is oriented on automatic building and utilization of terminologies. Since terminology extraction tools require Part-of-Speech (POS) tagging of texts and the two POS taggers developed for Ukrainian are not easily available or usable (the UGtag tagger [12] does not perform the syntactic and morphological disambiguation, and a module for the TNT POS tagger [3] is difficult to obtain), it remains difficult to use such tools for the pre-processing of corpora in order to prepare traditional terminology acquisition process. As for the utilization of terminologies, we can cite for instance localization of tools [25] and indexing of a language therapy terminology [36].

As we indicated above, we propose to take advantage of the advanced research work done in languages such as English or French, and to transpose it on lower-resourced languages using cross-lingual transfer methodology. We work with medical data and in three languages: Ukrainian, French, and English.

The general rationale of our approach is the following. Our work is based on exploitation of two kinds of corpora: Wikipedia in Ukrainian which provides several useful kinds of information (such as term labels and their codes) with a high level of quality, and parallel corpus MedlinePlus. Each corpus is exploited through dedicated methods. The MedlinePlus corpus provides the basis for the building of the terminology, while the Wikipedia corpus permits to enrich this information and helps the word-level alignment of the MedlinePlus corpus. -MedlinePlus: parallel medical corpus from MedlinePlus3 . The source data are built by MedlinePlus from the National Library of Medicine. They contain patient-oriented brochures on several medical topics (body systems, disorders and conditions, diagnosis and therapy, demographic groups, health and wellness). These brochures have been created in English and then translated in several other languages, among which French and Ukrainian. In Figure 2, we present an excerpt from this corpus;

Material

Corpora

-Wikipedia: medicine-related articles from Wikipedia. This corpus is extracted from the Ukrainian part of Wikipedia using medicine-related categories, such as Медицина (medicine) or Захворювання (disorders). The corpus potentially covers a wide range of medical notions. In Figure 1, we indicate an example of source page composed of three parts: the navigation frame on the left, the text and explanations in the center, and the infobox with illustrations and codings on the right.

In The UMLS (Unified Medical Language System) [14] merges over 100 biomedical terminologies, such as international terminologies MeSH [19] and ICD [4]. Such terminologies may exist in several languages. For instance, French and English versions of MeSH are included in the UMLS. No terminologies in Ukrainian are part of the UMLS. Each UMLS term is provided with unique identifier, which allows finding the corresponding terms in other terminologies or languages.

Stopwords in Ukrainian

We use a list with 385 stopwords in Ukrainian (на (on), або (or), etc.) issued from an existing resource dedicated to the localisation of graphical interfaces4 .

Methods

The methods we propose for the extraction of bilingual terminology are adapted to each kind of corpora and of data they contain: the MedlinePlus corpus (section 4.1) and the Wikipedia corpus (section 4.2). We then present their crossfertilization (Section 4.3), and evaluation of the results (Section 4.4).

Extraction of bilingual terminology from the MedlinePlus corpus

Prior to the exploitation of the MedlinePlus data, the documents are first transformed in a suitable format: (1) the source pdf documents are converted in text format;

(2) in each language, documents are segmented in paragraphs;

(3) French/Ukrainian and English/Ukrainian alignments are generated, in which n th paragraph from one language is put in front of the n th paragraph from the other language; (4) alignments within pairs of languages are then verified manually. In Figure 2, we present an excerpt from the English/Ukrainian aligned corpus.

English Ukrainian

ixmple of the prgrphEligned wedlinelus orpus @inglishGkrininAF

In French and English, we can use terminology extraction tools for bootstraping the acquisition of terminology. Hence, we use the Y A T E Aterm extractor [2], that is applied to POS-tagged documents (Treetagger [23], accompanied by Flemm morphological analyzer [18] in French). On the left of Figure 2, we show in bold the extracted candidate terms in English. Then, exploitation of the Med-linePlus parallel and aligned corpus is performed in several ways (Figure 3).

Transfer 1. The simplest situation is when the two aligned lines contain one candidate term in each language: these terms are recorded as candidates for the alignment. For instance, in Figure 2, the pairs {Tiredness, Втома} and {Pain, Бiль} are issued from this kind of alignment.

Transfer 2. When paragraphs contain complex expressions or sentences, the processing is done as follows (Figure 3):

1. paragraph-aligned corpora are aligned at the word level using GIZA++ [20], 2. in each paragraph pair (French/Ukrainian and English/Ukrainian) of the word-aligned corpora, terms recognized in French and English are transfered on Ukrainian paragraph; 3. extracted alignments are recorded as candidates for building the terminology.

For instance, in Figure 2, the term Cancer cells is automatically extracted from the English corpus. GIZA++ proposes that Cancer cells is aligned with Раковi клiтини. Thus, through the word-aligned text, we can propose that Cancer cells is the translation of Раковi клiтини. This processing is performed on two pairs of languages (French/Ukrainian and English/Ukrainian).

As indicated in Table 1, our corpora are rather small for statistical alignment performed by GIZA++. For this reason, we provide GIZA++ with a bilingual dictionary in order to help alignment at the word level. Besides, we remove term pairs in which Ukrainian unit, corresponding to stopwords, is aligned as candidate term with French or English terms.

Extraction of bilingual terminology from the Wikipedia corpus

The Wikipedia corpus is used to complete and to help the method applied to the MedlinePlus corpus. The exploited content is extracted from infoboxes (on the right in Figure 1) and is reachable through the MediaWiki source code of Wikipedia. This provides labels of medical terms in Ukrainian and their MeSH codes. The process is the following (Figure 4):

1. the infobox content is parsed 5 which provides label and MeSH code of terms, 2. the MeSH code is used to query the UMLS, and to get the corresponding French and English terms, krinin ikipedi medil prt roessing of the snfofoxes wedil terms with wer odes wv uerying wv irs of medil terms @uGp nd uGixA Fig. 4. ixtrtion of medil terms from ikipedi @krininauD prenhapD inE glishaixAF 3. the term pairs French/Ukrainian and English/Ukrainian are associated and provide good candidates for the bilingual terminology building.

For instance, in Figure 1, the term нанiзм is extracted, as well as its MeSH code D004392. In the UMLS, the corresponding English terms are dwarfism and nanism, while the corresponding French term is nanisme. Notice that similar method has been used for building of medical terminology in the Arabic language [28]. This part of the method exploits specific and intentionally created content and provides reliable information.

Cross-fertilization

Two kinds of cross-fertilization between the two methods (Sections 4.1 and 4.2) are performed:

-Wikipedia terms are used to enrich the extracted terminology, single-word terms extracted from Wikipedia and/or the Transfer 1 method in MedlinePlus are provided to GIZA++, as additional bilingual dictionary, in order to help the alignment of MedlinePlus at the word level. In our best setting, both resources of single-word term pairs are used.

Evaluation

Evaluation is handled manually in order to check whether candidate terms extracted for building bilingual terminologies are correct. The evaluation is performed by a native Ukrainian speaker, non expert in medicine but with several years experience in medical informatics. Terms are validated independently in each language. Besides, bilingual and trilingual relations between the Ukrainian, English and French terms are also evaluated. Thus, precision of results can be computed, i.e. the ratio between correct answers and all the answers.

Table 2 presents results and precision of the terms extracted by the three methods. As the precision values suggest, the Transfer 1 method generates few errors. Their analysis indicates that they are mainly due to partial or non-literal translations: {появу виразок у ротi, mouth sores} -lit. (appearance of ) mouth sores, {ви можете спати, dormir/sleep} -lit. you can sleep. The silence of the method can be explained by two reasons: (1) Again translation can prevent the Transfer 1 method from extracting terms in French or English. For instance, Soins is the French translation of Your care, which gives simple term in one language and complex term in another language and makes impossible their extraction by the Transfer 1 method. The Transfer 2 method will solve this problem; (2) However, the main reason of the silence is the incapacity of the term extractor to identify French or English terms because of errors in the POS tagging.

As for the Transfer 2 method, we present the results obtained when the pairs of single-word terms from the MedlinePlus corpus and Wikipedia are used to help the GIZA++ alignment. Then, the Transfer 2 method allows to extract 9,040 Ukrainian terms with 0.454 precision (exact match). Precision of French and English terms is higher: 0.674 and 0.761 respectively (exact match). Moreover, number of French and English terms is dramatically lower (about -45% and -40%) than in Ukrainian: the rich morphology of Ukrainian provides indeed several inflected forms for a given term ({напад, нападу} -attack, {припадкiв, припадки} -seizure, {костей, кiстки} -bones). Besides, we can also extract synonymous terms ({приступам, припадкiв} -attacks/seizures, {биття, удару} -beats). Precision of inexact match, when the correct answer is included or includes the candidate term, is much higher and gains up to 0.40 points in Ukrainian and 0.05 in French and English. We assume this difference on Ukrainian is mainly due to the improvement of the alignment quality. As for the interlingual relations, the Transfer 2 method collects 3,724 pairs of Ukrainian/French terms with 0.309 precision, 4,745 pairs of Ukrainian/English terms with 0.401 precision and 4,724 triples with 0.419 precision.

An analysis of the results shows that most of the errors are due to the alignment problems. Indeed, we observe that when the alignment is correct, the Ukrainian terms are correctly extracted by the transfer methods. Otherwise, the errors occur.

As we indicated, the MedlinePlus corpus contains patient-oriented brochures, which are not highly specialized. Yet, most of the extracted terms are specific to the medical domain ({трахеотомiєю, tracheostomy}), {фактори ризику, risk factors}, {шприца, syringe}, {холестерину, cholesterol}). The extracted terms can also refer to close and approximating notions which reflect this type of documents: {дiти, children}, {здорову їжу, healthy diet}, {серцевий напад, heart attack}, {склянок рiдини, glasses of liquid}. Another interesting observation is that some French and English terms correspond to propositions in Ukrainian: {не до кiнця приготовлену їжу, undercooked foods} (lit. food which is not fully cooked), {При цьому обстеженнi Ви не вiдчуєте жодного болю, indolore (painless)} (lit. With this exam you will feel no pain).

Finally, all the methods combined allow to build a terminological resource containing 4,588 Ukrainian medical terms and their 34,267 relations with French and English terms.

Conclusion and Future Work

In this work, we propose to exploit two kinds of freely available multilingual corpora in French, English and Ukrainian. Each corpus is exploited with appropriate methods which allow extracting candidate terms and creating Ukrainian/French and Ukrainian/English term pairs. In particularly, French and English corpora are processed with NLP and term extraction tools. Then, thanks to the transfer methods these terms are transposed on Ukrainian. We also propose to use existing terminologies and to exploit simple terms for improving the alignment performed at the word level with GIZA++.

Our future work will address enrichment of the created resource with terms from other corpora. Besides, in the Wikipedia corpus, we can use other codes, such as those from МКХ-10 (ICD10) or MedlinePlus. This will also augment the coverage of the term pairs extracted in the current work. Another perspective of this work is the improvement of the bilingual alignment of documents at the word level, which is currently a major source of errors. In that respect, we plan to investigate the use of other alignment algorithms, such as FastAlign [8] or the Lingua::Align toolbox [27].

IF elievD FX ynomtopoei words in the grimen tr lngugeF henye zpiski IV@SUAD V!II @PHHSA PF euinD FD rmonD FX smproving term extrtion with terminologil resouresF snX pinev PHHTF ppF QVH!QVUF xoF RIQW in vxesD pringer @PHHTA QF fyhD FD ierleD uFD fyhD fFX hevelopment of hyrid mhine trnsltion systems for underEresoured lngugesX eutomted retion of lexil nd morE phologil resoures for mtF snX epplied nd viterry rnsltion nd snterpretingX heoryD wethodologyD rtieF pF SppF uyivD krine @epril PHIQA RF fr¤ merD qFX snterntionl sttistil lssi(tion of diseses nd relted helth prolemsF tenth revisionF orld relth tt RI@IAD QP!T @IWVVA SF gr¡ eD wFD istop D FD ivldiD tFX eutomti term detetionX review of urrent systemsD ppF SQ!VVF tohn fenjmins @PHHIA TF hel § D hFD urle § zD FD hlelo f § si¡ D fFD § njderD tFD § ri¡ D pFX ermeX e tool for ollotion extrtionF snX gsgvingD vxg SRRWF ppF IRW!ISU @PHHWA UF hmytrukD FX ypologil fetures of wordEformtion in omputingD the internet nd progrmming in the (rst dede îf the s enturyF snX ÓÄÊF ppF I!II @PHHWA VF hyerD gFD ghhuneuD FD mithD xFeFX e simpleD fstD nd e'etive reprmeteriE ztion of im model PF snX xeegvGrvF ppF TRR!TRV @PHIQA WF qrigonyteD qFD imkuteD iFD tkD eFD foizouD vFX ixperiments on vithunin term extrtionF snX xyhevshe PHIIF ppF VP!VW @PHIIA IHF ugeurD uFD minoD fFX wethods of utomti term reognitionF snX xtionl genter for iene snformtion ystemsF ppF I!PP @IWWTA IIF uelihD iFD fukD FD qrzyekD FD ovenhkD eFX rojet desriptionX designing nd onstruting typologilly lned ukrinin text dtseF snX Ìåòîäè àíàëiçó òåêñòóF ppF IPS!IQP @PHHWA IPF uotsyD xFD wykulykD eFD hevhenkoD sFFX qgX morphologil nlyzer nd tgger for the krinin lngugeF snX roeedings of the interntionl onferene rtil epplitions in vnguge nd gomputers @evg PHHWA @PHHWA IQF uruglevskisD FD nneD sFX erm extrtion from legl texts in ltvinF snX eond flti gonferene on rumn vnguge ehnologies @PHHSA IRF vindergD hFD rumphreysD fFD wgryD eFX he uni(ed medil lnguge systemF wethods snf wed QP@RAD PVI!PWI @IWWQA ISF vopezD eFD xosslD wFD rwD FD esnikD FX ordElevel lignment for multilingul resoure quisitionF snX vig orkshop on vinguisti unowledge equisition nd epresenttionX footstrpping ennotted htF vs lmsD pin @PHHPA ITF whonldD FD etrovD FD rllD uFX wultiEsoure trnsfer of delexilized depenE deny prsersF snX iwxv @PHIIA IUF wemetovD iFX vexiophrseologil expressive mens of the grimen tr lnE gugeF henye zpiski IV@SUAD QU!QW @PHHUA IVF xmerD pFX pviww X un nlyseur )exionnel du frn is se de r eglesF riteE ment utomtique des lngues @evA RI@PAD SPQ!SRU @PHHHA IWF xtionl virry of wediineD fethesdD wrylndX wedil ujet redings @PHHIAD www.nlm.nih.gov/mesh/meshhome.html PHF yhD pFD xeyD rFX smproved sttistil lignment modelsF snX egvF ppF RRH!RRU @PHHHA PIF zienzD wFFD ennhiottiD wFD nzottoD pFX erminology extrtionX en nlE ysis of linguisti nd sttistil pprohesF snX irmkessisD F @edFA unowledge winingD tudies in puzziness nd oft gomputingD volF IVSD ppF PSS!PUWF pringer ferlin reidelerg @PHHSA PPF innisD wFD vjue § si¡ D xFD tefnesuD hFD kdi nD sFD di¡ D wFD qornostyD FX erm extrtionD tggingD nd mpping tools for underEresoured lngugesF snX ui PHIPF ppF IWQ!PHV @PHIPA PQF hmidD rFX roilisti prtEofEspeeh tgging using deision treesF snX snternE tionl gonferene on xew wethods in vnguge roessingF ppF RR!RW @IWWRA PRF htlinD yFX viterture terminology of the yld krinin literture of the IVth enturyF henye zpiski IV@SUAD S!U @PHHSA PSF hyshkinD xFD orkoD qFD veskoD vFX erminology work nd softwre loliztion in krineF snX rolems of gyernetis nd snformtisF ppF IU!PH @PHIHA PTF di¡ D wFD § ojtD uFX pinding multiword term ndidtes in grotinF snX siv orkshopD exv gonfereneF ppF IHP!IHU @PHHQA PUF iedemnnD tFD uotz¡ eD qFX e disrimintive pproh to tree lignmentF snX sliseiD sFD ekrD FD fernrdiniD F @edsFA snterntionl orkshop on xturl vnguge roE essing wethods nd gorpor in rnsltionD vexiogrphy nd vnguge vernE ingF ppF QQ!QW @PHHWA PVF ivldiD tFD odr ¡ iguezD rFX eri medil term ompiltion from ikipediF snX ro of gs PHIR @PHIRA PWF rowskyD hFD xgiD qFD ientowskiD FX snduing multilingul text nlysis tools vi roust projetion ross ligned orporF snX rv @PHHIA QHF emnD hFD esnikD FX grossElnguge prser dpttion etween relted lnE gugesF snX xv for vess rivileged vnguges @PHHVA QIF ÁóãàêîâD yFX Ñîçäàíèå ñåìàíòè÷åñêîãî ñëîâàðÿ ïðåäëîaeíûõ êîíñòðóêöèé íà îñíîâå óêðàèíñêîãî íàöèîíàëüíîãî ëèíãâèñòè÷åñêîãî êîðïóñàF ehF repFD Óêðàèíñêèé ÿçûêîâîEèíôîðìàöèîííûé ôîíä ÍÀÍ ÓêðàèíûD ÊèåâD Óêðàèíà @PHHTA QPF ÃëèáîâåöD eFD ÐåøåòíåâD sFX Ìåòîä èòåðàòèâíîãî ïîñòðîåíèÿ òåðìèíîëîãèè â êîëëåêöèÿõ íàó÷íûõ òåêñòîâ íà óêðàèíñêîì ÿçûêåF Êèáåðíåòèêà è ñèñòåìíèé àíàëèç SH@TAD SQ!TP @PHIRA QQF ÄåìñüêàD yFX Òåêñòîâèé êîðïóñX iäåÿ iíøî¨ôîðìèF ÂÏÖ ÍàÓÊÌÀD Êè¨âD Óêðà¨íà @PHIIA QRF ÊîññàêD yFX Óêðà¨íñüêà êîìï9þòåðíà òåðìiíîëîãiÿF snX Ñó÷àñíi ïðîáëåìè â êîìï9þòåðíèõ íàóêàõF ppF QW!RP @PHHHA QSF Êî÷åðãàD yFD Ìåéíàðîâè÷D iFX ÀíãëiéñüêîEÓêðà¨íñüêîEÀíãëiéñüêèé ñëîâíèê íàóêîâî¨ìîâèF Ôiçèêà òà ñïîðiäíåíi íàóêèF Íîâà êíèãàD ÂiííèöÿD Óêðà¨íà @PHIHA QTF Ëàëà¹âàD FD ÑóðîâàíåöüD [U+FFFD]D ÒèùåíêîD fFX síäåêñàöiÿ ïîëüñüêîED ðîñiéñüêîE òà óêðà¨íñüêîìîâíî¨ëîãîïåäè÷íî¨òåðìiíîëîãi¨F Ëåêñèêîãðàôi÷íèé áþëåòåíü IHD PW!QT @PHHRA QUF Ëåâ÷åíêîD yFD Êóëü÷èöüêèéD sFX Òåõíîëîãiÿ ïåðåòâîðåííÿ ï9ÿòèìîâíîãî ñëîâíèêà ïîðiâíÿíü â åëåêòðîííó ôîðìóF snX síôîðìàöiéíi ñèñòåìè òà ìåðåaeiF ppF IPW!IQV @PHIQA QVF ÌîíàõîâàD FX Çàñòîñóâàííÿ ïðèéîìiâ êîðïóñíî¨ëiíãâiñòèêè â ëåêñèêîãðàôi¨F Íàóêîâi ïðàöi WV@VSAD SS!TH @PHHWA QWF Ïóðÿ¹âàD rFX Äåùî ïðî ìîâó áîãîñëóaeiííÿ âçàãàëi òà ïðî ñëîâíèê ìîâè áîãîñëóaeiííÿ çîêðåìàF Ëåêñèêîãðàôi÷íèé áþëåòåíü IHD QT!RP @PHHRA RHF ÒèìåíêîD [U+FFFD]X ËåêñèêîEòåìàòè÷íi ãðóïè óêðà¨íñüêî¨þðèäè÷íîò åðìiíîëîãi¨ïî÷àòêó xx ñòîëiòòÿF Ëåêñèêîãðàôi÷íèé áþëåòåíü IHD TS!UH @PHHRA

Fig. 1 .

 1 Fig. 1. ixmple of krinin ikipedi soure pge @for DwarsmAF he infoox with the oding is on the rightF

Fig. 3 .

 3 Fig. 3. ixtrtion of medil terms from the MedlinePlus orpor @krininauD prenhapD inglishaixAF

  Table1, we indicate the size of the corpora. Not surprisingly, the Wikipedia corpus is much larger although only part of its information is exploited, as we explain it in Section 4.

	Corpus	Size (occ of words)
	Wikipedia/UKmed	PRTDQTVDRII
	MedlinePlus/UK	RQDIVR
	MedlinePlus/FR	SQDHTU
	MedlinePlus/EN	RTDSRR
	Table 1. ize of the exploited orpor @krininauD prenhapD inglishaixAF
	3.2 UMLS: Unified Medical Language System

  Table3presents results and precision concerning the pairs and triples of terms.Use of the Transfer 1 method allows to extract 436 Ukrainian terms with a high precision (0.966). These terms are associated with 316 French and 354 English terms, within 282 triples between Ukrainian, French and English terms, with 0.954, 0.937 and 0.965 precision, respectively. Within this set, 63 pairs exist only between Ukrainian and French terms, and 115 pairs only between Ukrainian and English terms. Then, the Transfer 2 method allows to collect 334 Ukrainian/French term pairs (including 108 pairs with single-word terms) and 380 Ukrainian/English term pairs (including 135 pairs with single-word terms). We observe that these relations can involve synonyms in either lan-

	term is associated with at least one French and English term, it allows to build
	28,840 triples. We assume that precision of this terminology is 1 because the
	source information is created manually and is highly reliable.
	5.2 Extraction of bilingual terminology from the MedlinePlus
	corpus
	Source		UK		FR	EN
		5terms reF 5terms reF 5terms reF
	Wikipedia		QSU I	IDRPV I	QDTPS I
	MedlinePlus T ransf er1	RQT HFWTT	QIT HFWUI	QSR HFWVW
	inexact match	HFWWV	HFWVU	HFWWU
	MedlinePlus T ransf er2 WDHRH HFRSR QDTUI HFTUR QDSWU HFUTI
	inexact match	HFVRH	HFUPT	HFUWW
	otl	WDSPW HFRVI SDPHH HFUTW UDQQS HFVVQ
	otl of orret terms RDSVV	QDWWV	TDRUT
	Table 2. ivlution of terms extrted @krininauD prenhapD inglishaixAF
	Source	UK/FR	UK/EN UK/FR/EN	otl
		5relF reF 5relF reF 5trplF reF 5trplF reF
	Wikipedia	IDSIS I	QDUVW I	PVDVRH I	PVDVRH I
	MedlinePlus T ransf er1	TQ HFWQU IIS HFWTS	PVP HFWSR	RTH HFWSR
	inexact match		HFWVR	I		HFWVP	HFWVU
	MedlinePlus T ransf er2	QDUPR HFQHW RDURS HFRHI RDUPR HFRIW IQDPIV HFQVI
	inexact match		HFUSI	HFVRH	HFSVT	HFUPR
	otl	QDUWV HFQIV RDVIW HFRI QQDVRS HFWIV RPDRTP HFVHU
	otl of orret reltions IDPHU		IDWUR	QIDHVT	QRDPTU
	Table 3. ivlution of pirs nd triples @krininauD prenhapD inglishaixAF
	5.1 Extraction of bilingual terminology from the Wikipedia corpus
	Exploitation of Wikipedia infoboxes allows to collect 357 Ukrainian medical
	terms among which 177 are single-word terms. By querying the UMLS with
	MeSH codes, those terms are associated with 1,428 French terms (including 339
	single-word terms) and 3,625 English terms (including 448 single-word terms).
	Higher number of French and English terms, compared to the number of Ukrainian
	terms, is due to synonyms proposed by MeSH. As for the pairs of terms, we ob-
	tain 1,515 Ukrainian/French and 3,789 Ukrainian/English term pairs, including,
	270 and 405 pairs between single-word terms, respectively. Since each Ukrainian

guage: {фаллопiєва труба, trompes de fallope/trompe utŕine} (fallopian tube), {втрата слуху/втрачається слух, hearing loss}, {втома, fatigue/tiredness}. Besides, in Ukrainian, several inflectional forms can be associated to a given English or French form: {вагiтнiсть, pregnancy} and {вагiтностi, pregnancy}.

www.nlm.nih.gov/medlineplus/healthtopics.html

httpsXGGgithuFomG)uxGlngsGloGmsterGkrininGstopwordsFtxt

Text::MediawikiFormat @httpXGGserhFpnForgG£ szgGextEwediwikipormtA

Acknowledgments projet TH