domain: info shs

text:
Rémi Cardon
email: remi.cardon@univ-lille.fr

Natalia Grabar
email: natalia.grabar@univ-lille.fr

A French Corpus for Semantic Similarity

Keywords: semantic similarity, manual annotation, French language, regression

Semantic similarity is an area of Natural Language Processing that is useful for several downstream applications, such as machine translation, natural language generation, information retrieval, or question answering. The task consists in assessing the extent to which two sentences express or do not express the same meaning. To do so, corpora with graded pairs of sentences are required. The grade is positioned on a given scale, usually going from 0 (completely unrelated) to 5 (equivalent semantics). In this work, we introduce such a corpus for French, the first that we know of. It is comprised of 1,010 sentence pairs with grades from five annotators. We describe the annotation process, analyse these data, and perform a few experiments for the automatic grading of semantic similarity.

Introduction

Semantic textual similarity is a subtask of Natural Language Processing. At the level of sentences, the task consists in evaluating to what extent two sentences express the same meaning. This task is useful for several applications, such as machine translation, text summarization, information retrieval, natural language generation, or text simplification [START_REF] Wieting | Beyond BLEU:training neural machine translation with semantic similarity[END_REF][START_REF] Vadapalli | SSAS: Semantic similarity for abstractive summarization[END_REF][START_REF] Yasui | Using semantic similarity as reward for reinforcement learning in sentence generation[END_REF][START_REF] Kajiwara | Building a monolingual parallel corpus for text simplification using sentence similarity based on alignment between word embeddings[END_REF]. The computing of the semantic textual similarity requires corpora with annotated pairs of sentences. The annotation is most of the time performed on a continuous scale where scores range from 0 (the sentences express completely unrelated meanings) to 5 (the meaning is exactly the same in both sentences). Several challenges dedicated to semantic textual similarity (STS) have been held within the SemEval evaluation campaign between 2012 and 2017. STS provides the research community with bilingual and monolingual data. In our work, we are interested in monolingual semantic similarity. In relation with the monolingual semantic similarity, data from a few languages (English, Spanish and Arabic) have been exploited [START_REF] Cer | SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation[END_REF] and made available for the research community. The overall STS benchmark data for English 1 , with data taken from editions held from 2012 to 2017, contains 8,628 sentence pairs, while only 250 sentence pairs were proposed for Spanish and for Arabic. Besides, similar data are also proposed for Portuguese through the ASSIN workshop [START_REF] Feitosa | Análise de medidas de similaridade semântica na tarefa de reconhecimento de implicac ¸ão textual (analysis of semantic similarity measures in the recognition of textual entailment task)[END_REF] dataset, which is composed of 10,000 pairs -5,000 for Brazilian Portuguese and 5,000 for European Portuguese. All those datasets are taken from general language and various sources : news articles, forum posts and video subtitles. Yet, there is no similar data in French. In our work, we introduce a semantic textual similarity corpus for French. We first describe the data that have been used and the annotation process, then we present the resulting resource. We also describe an experiment that shows an 1 http://ixa2.si.ehu.es/stswiki/index.php/ STSbenchmark attempt at reproducing the annotation automatically.

Corpus and Annotation Process

In this section, we first present the data provided to the annotators. We then describe the annotation process and analyse the annotation criteria defined by the annotators.

Data Processed

The same batch with 1,010 sentence pairs was provided to five annotators. The sentence pairs are issued from a general language corpus containing sentences extracted from Wikipedia2 and Vikidia3 articles, and from texts related to the medical field. In this last case, the sentences are extracted from the CLEAR corpus [START_REF] Grabar | CLEAR -Simple Corpus for Medical French[END_REF], which includes information about drugs, medical literature reviews, and medicine-related articles from Wikipedia and Vikidia. The purpose of this corpus is to propose comparable contents which are distinguished by their technicality: technical and difficult to understand texts are paired with the corresponding simple or simplified texts. This is another factor that distinguishes our dataset from the existing datasets in other languages mentioned in section 1.. The candidate pairs of sentences were generated automatically while building a classification method [START_REF] Cardon | Parallel sentence retrieval from comparable corpora for biomedical text simplification[END_REF] and then validated and selected manually. That method is similar to the one described in section 4.1.. The main difference is that it is based on the Random Forest classifier algorithm, whereas below we use it as a Regressor. In the work presented in this paper, the goal is to retain sentence pairs pertaining to various degrees of similarity in order to be able to train a model to assign values on a continuous scale instead of binary values (aligned or not aligned). Hence, the semantic similarity between sentences within a given pair is due to their technicality and to the complexity of their contents, which can be lexical, syntactic or semantic. Here is an example from the CLEAR corpus, with an English translation :

Annotation Process

The five annotators involved have received higher education: two of them are trained in Natural Language Processing, one is a medical practitioner. Except one, all annotators are native French speakers. The authors were not part of the annotators. The annotation guidelines provided to the annotators were very simple and short:

• to assign a score of 0 when the sentences are completely unrelated,

• to assign a score of 5 when the sentences mean the same,

• to come up with their own scale and criteria for the intermediate values,

• to define a short description of the annotation criteria.

We prefered not to bias the manual annotations with some a priori criteria, such as 1. use the score n for sentence pairs with syntactic modifications,

2. use the score m for sentence pairs with lexical modifications, etc.

Indeed, our motivation was to exploit the linguistic competence of the annotators and to compare their semantic sensitivity and judgements. We assume also that, in this way, the overall semantic scores should better represent the semantic similarity between the sentences. The annotators estimated that the annotation of the 1,010 pairs of sentences took between seven and fifteen hours.

Scales and Annotation Criteria according to the Annotators

The scales and criteria that were used by the annotators can be seen in Table 1. We can observe differences and similarities between the various annotation principles provided by the annotators:

• Except one, all the annotators assigned integer scores [0, 1, 2, 3, 4, 5] to the pairs of sentences. One annotator also used intermediary scores [0.5, 1.5, 2.5, 3.5, 4.5];

• The A3 annotator considered that he took the sentences strictly as they were given, which means that the unknown context was considered as non-existent.

That implies for example that pronouns in one sentence were never assumed to be referring to an element explicitly mentioned in the other sentence, increasing the likelihood of dissimilarity;

• The scales from A2 and A3 are much more conservative than the other three. Yet they greatly differ from one another. A2 is the only annotator who focused on phrasing. In order to assign the highest score according to their scale, the two sentences have to be identical. The scale given by A3 is more similar to the other ones but it is conservative because of the strict view related to context not being assumed;

• For specific grades, 2, 3 and 4 are quite similar for all the annotators but A2: 2 involves that the sentences have something that differentiates them, but they deal with the same subject. 3 implies each time that there is shared information but that one sentence expresses information that is not found in the other one, and 4 implies that the information is "almost" or "slightly" the same;

• It is more difficult to analyse the relationship between the descriptions for grade 1. A1 and A4 both mention something in common, the domain, or grounds for inference, but also state that nothing more can reinforce the link between the two. A3 and A5 focus on the lack of relation between the sentences.

To summarize, we can see that the annotators paid attention to several criteria when deciding about the semantic relatedness of the sentences:

• intersection of the meaning, such as missing information, incomplete information or extra words on either side,

• use of paraphrases and different expressions,

• possibility to do textual inference.

We observe also that the completeness of information is the most frequently used criteria by all the annotators.

Global scores

Using all the scores from the five annotators, we computed two more values :

• The average score for each pair, rounded ("Avg" further down);

• The most frequent score out of the five for each pair ("Vote" further down).

Analysis of the Annotations

In this section, we further analyse the annotations: their breakdown by score and the correlation of the scores from the five annotators. We can observe that the 0 score is the most used by every annotator but one (A1). The annotator A3 assigned the 0 score to almost half the pairs, which is coherent with the annotation criteria of this annotator, who did not assume context for coreference and thus had the most conservative approach.

Breakdown by Score

We can also see that every annotator but one (A1) used 4 more often than 5. This can be explained by the nature of the sentence pairs. As stated in section 2.1., the source corpus is aimed towards simplification and the sentence pairs come from document pairs where one is more technical than the other one. In consequence, it can be expected that there are more almost identical sentences than entirely identical ones, as the texts are not written for the same audience and thus do not deliver the exact same information in the exact same way.

Looking at Avg and Vote, we observe that grades 3 and 4 are the most consistent overall. Grade 2 seems to be the most inconsistent, with an average that is way above the individual counts, and a vote that is low.

Correlation Coefficients

We computed the Krippendorff's α [START_REF] Krippendorff | Estimating the reliability, systematic error and random error of interval data[END_REF] to evaluate the global correlation coefficient of the annotations. The α value for the five annotators is 0.69. This value is above the generally observed threshold which is considered as reliable (α = 0.67). Yet, this score is quite low. When we take the average and the vote scores into account for the computation, the α value goes up to 0.77, which is a sign that putting all the annotations together significantly improves the data reliability. In order to explore those results more deeply, we computed the correlation between pairs of annotators. Table 2 shows the Pearson correlation (Kirch, 2008) for every combination of two annotators. The observations we can make are consistent with figure 1 and the criteria described in section 2.3..:

• The lowest correlation coefficient (0.64) occurs between A2 and A3: A2 is the annotator who used steps of .5 in his scale and A3 relied on annotation principles that had him assign 0 to almost half the pairs. Hence, those two annotators applied annotation scales and criteria that differ the most from the other ones.

• The correlation coefficients between the other three annotators (A1, A4 and A5) are the highest: 0.84 for A1 and A4, 0.81 for A1 and A5 and 0.80 for A4 and A5.

• The other associations range between 0.70 and 0.77.

Globally, the correlation coefficients show a satisfying reliability for the dataset, with variations according to the different scales that were used. We see that the two scales that stand out have the lowest correlation coefficient with each other, but at the same time they have a good correlation coefficient with the other three. Those other three have strong coefficients with one another.

Experiments

In order to study how the resulting corpus can be exploited, we ran an experiment to check how accurately we could automatically reproduce the annotations. In this section, we first describe the automatic approach for scoring the pairs of sentences and then the results obtained.

Automatic Approach for Scoring the Pairs of Sentences

We exploited a previously proposed method dedicated to the detection of parallel sentences in comparable corpora [START_REF] Cardon | Parallel sentence retrieval from comparable corpora for biomedical text simplification[END_REF]). Yet, in order to predict values on a continuous scale, the Random Forest Regressor is exploited instead of the classifier. We compute and use several sets of features, mainly obtained from the lexical and sublexical content of the sentences, their word-based similarity, and the corpus-suggested similarity from word embeddings:

1. Number of common non-stopwords. This feature permits to compute the basic lexical overlap between specialized and simplified versions of sentences [START_REF] Barzilay | Sentence alignment for monolingual comparable corpora[END_REF]. It concentrates on non-lexical content of sentences;

2. Percentage of words from one sentence included in the other sentence, computed in both directions. This features represents possible lexical and semantic inclusion relations between the sentences;

3. Sentence length difference between specialized and simplified sentences. This feature assumes that simplification may imply stable association with the sentence length;

4. Average length difference in words between specialized and simplified sentences. This feature is similar to the previous one but takes into account average difference in sentence length;

5. Total number of common bigrams and trigrams. This feature is computed on character ngrams. The assumption is that, at the sub-word level, some sequences of characters may be meaningful for the alignment of sentences if they are shared by them;

6. Word-based similarity measure exploits three scores (cosine, Dice and Jaccard). This feature provides a more sophisticated indication on word overlap between two sentences. Weight assigned to each word is set to 1; 7. Character-based minimal edit distance [START_REF] Levenshtein | Binary codes capable of correcting deletions, insertions and reversals[END_REF]. This is a classical computation of edit distance.

It takes into account basic edit operations (insertion, deletion and substitution) at the level of characters. The cost of each operation is set to 1; 8. Word-based minimal edit distance [START_REF] Levenshtein | Binary codes capable of correcting deletions, insertions and reversals[END_REF]. This feature is computed with words as units within sentence. It takes into account the same three edit operations with the same cost set to 1. This feature permits to compute the cost of lexical transformation of one sentence into another; 9. WAVG. This features uses word embeddings. The word vectors of each sentence are averaged, and the similarity score is calculated by comparing the two resulting sentence vectors [START_REF] Stajner | Cats: A tool for customised alignment of text simplification corpora[END_REF]; 10. CWASA. This feature is the continuous word alignment-based similarity analysis, as described in [START_REF] Franco-Salvador | Cross-language plagiarism detection over continuous-space and knowledge graph-based representations of language[END_REF].

For the last two features, we trained the embeddings on the CLEAR corpus using word2vec [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF], and the scores are computed using the CATS tool [START_REF] Stajner | Cats: A tool for customised alignment of text simplification corpora[END_REF].

Results

We ran the experiment for every annotator. We also ran the experiment for Avg and Vote. We randomly split the data into 90% for training and 10% for testing. As there are small variations on each run due to random splitting, each reported score represents the average over twenty runs. ). This shows that the various scales can be automatically reproduced, and even if there are important difference between them, the annotations can be considered to be coherent. The most engaging observation is that the best results (0.87) are obtained on the average scores. This may mean that the average scores and collective perception of the semantic similarity remain coherent despite the differences observed during the annotation process. Interestingly, the result for Vote is the mean of the scores for the five annotators individually.

Conclusion

We introduced a corpus annotated for semantic textual similarity for French. Currently, this kind of data is indeed missing in French. The corpus is composed of 1,010 sentence pairs that come from comparable corpora aimed towards text simplification. More precisely, the original texts come from the CLEAR corpus and from Wikipedia and Vikidia articles. The corpus comes with grades manually assigned by five annotators. Together with the scores, the annotators provided the annotation scheme they adopted. We performed an analysis of the resulting data and showed that there are discrepancies in the scores that have been assigned. Those discrepancies can be explained with different annotation factors. We then used these data to automatically predict the scores of the pairs of sentences. This set of experiments shows that the scores can be quite well reproduced with automatic approaches. This indicates that the manually created data are reliable and can be used for a variety of experiments where semantic textual similarity is of interest. At the time of publication, the dataset is being used in an NLP challenge and will be made available for the research community.

Figure 1 :

 1 Figure 1: Breakdown by category and annotator

  Pearson's correlation coefficients between the annotators

Table 1 :

 1 Annotation criteria defined by the annotators

		A1	A2			A3	A4	A5
	0.5		A few identical
			segments
	1	Same topic, loose	One summarizes	Little shared infor-	Inference can be	Almost unrelated
		relation	the other		mation	drawn	meaning
	1.5		Incomplete main
			information	on
			one side and ex-
			tra	information
			missing
	2	Same topic, differ-	Incomplete main	Same function, lit-	Intermediate level	Same subject, dif-
		ent information	information on one	tle shared informa-		ferent information
			side			tion
	2.5		Same	meaning,
			radically different
			expression
	3	Same topic, loosely	Same meaning, dif-	Extra information	Main concept of	Extra information
		shared information	ferent expression	on one side	one sentence is	on one side
							missing in the other
							one
	3.5		Same	meaning,
			paraphrases	are
			found
	4	Almost same con-	Same	meaning,	Same function and	Additional infor-	One slight differ-
		tent, additional in-	slight rephrasing	almost same infor-	mation on one	ence in the deliv-
		formation one				mation	side	ered information
		side
	4.5		Same	meaning,
			slight	syntactic
			difference

1. Les effets graves intéressant les systèmes hépatique et/ou dermatologique ainsi que les réactions d'hypersensibilité imposent l'arrêt du traitement.

(Severe effects affecting the liver and/or dermatological systems and hypersensitivity reactions require discontinuation of treatment.)

Table 3

 3

	A1	A2	A3	A4	A5	Avg Vote
	0.82 0.73 0.80 0.79 0.78 0.87 0.78
	Table 3: Pearson's correlation coefficient on regression ex-
	periments
	0.82 (A1

shows the results obtained when scoring the pairs of sentences tackled as the regression task. For the annotators, the correlation coefficients range from 0.73 (A2) to

https://fr.wikipedia.org/

https://fr.vikidia.org

Le traitement doit être arrêté en cas de réaction allergique généralisée, éruption cutanée ou altérations de la fonction du foie. (Treatment should be discontinued in the event of a generalized allergic reaction, rash or impaired liver function.)

Acknowledgements

We are grateful to our annotators for their valuable work. We would also like to thank the reviewers for their helpful comments. This work was funded by the French National Agency for Research (ANR) as part of the CLEAR project (Communication, Literacy, Education, Accessibility, Readability), ANR-17-CE19-0016-01.


text:
Rémi Cardon

Natalia Grabar

Parallel sentence alignment from biomedical comparable corpora

Keywords: sentence alignment, text simplification, classification

Parallel sentences provide semantically similar information which can vary on a given dimension, such as language or register. Parallel sentences with register variation (like expert and non-expert documents) can be exploited for the automatic text simplification. The aim of automatic text simplification is to better access and understand a given information. In the biomedical field, simplification may permit patients to understand medical and health texts. Yet, there is currently no such available resources. We propose to exploit comparable corpora which are distinguished by their registers (specialized and simplified versions) to detect and align parallel sentences. These corpora are in French and are related to the biomedical area. We treat this task as binary classification (alignment/non-alignment). Our results show that the method we present here can be used to automatically generate a corpus of parallel sentences from our comparable corpus.

Introduction

Parallel sentences provide semantically similar information which can vary on a given dimension. The dimension on which the parallelism is positioned can come from many levels, here we are concerned with expert and non-expert register of language. The following pair of sentences illustrates this:

-Expert : Drugs that inhibit the peristalsis are contraindicated in that situation -Non-expert : In that case, do not take drugs intended for blocking or slowing down the intestinal transit Pairs differentiated by their degree of technicality can be used for text simplification. The purpose of text simplification is to provide simplified versions of texts, in order to remove or replace difficult words or information.

Automatic text simplification can be used as a preprocessing step for NLP applications or for producing suitable versions of texts for humans. In this second case, simplified documents are typically created for children [START_REF] Tu | Learning to simplify children stories with limited data[END_REF], or for people with mental or neurodegenerative disorders [START_REF] Chen | Automatic text simplification for people with intellectual disabilities[END_REF].

Helping patients to better understand medical and health information is an important issue, which motivates our work [START_REF] Ama | Health literacy: report of the council on scientific affairs. Ad hoc committee on health literacy for the council on scientific affairs[END_REF].

In order to perform biomedical text simplification, we propose to collect parallel sentences, which align difficult and simple information. We can exploit an existing monolingual comparable corpus with medical documents in French [START_REF] Grabar | CLEAR -Simple Corpus for Medical French[END_REF]. The corpus is composed of documents created for medical professionals and documents created for patients. The purpose of our work is to detect and align parallel sentences from this comparable corpus. We also propose to test what the impact of imbalance on categorization results is: imbalance of categories is indeed a natural characteristic in textual data.

The existing work on searching parallel sentences in monolingual comparable corpora indicates that the main difficulty is that such sentences may show low lexical overlap but be nevertheless parallel. This task is usually explored in general-language corpora and performed as assigning a similarity score from 0 to 5 [START_REF] Agirre | SemEval-2016 task 1: Semantic textual similarity, monolingual and crosslingual evaluation[END_REF]. Among the exploited methods, we can notice lexicon-based methods which rely on similarity of subwords or words from the processed texts or on machine translation [START_REF] Madnani | Re-examining machine translation metrics for paraphrase identification[END_REF]; knowledge-based methods which exploit external resources, such as WordNet [START_REF] Miller | Introduction to wordnet: An on-line lexical database[END_REF];syntax-based methods which exploit the syntactic modelling of sentences. [START_REF] Tsubaki | Non-linear similarity learning for compositionality[END_REF]; or corpus-based methods [START_REF] Mueller | Siamese recurrent architectures for learning sentence similarity[END_REF].There is no existing work on building a corpus for text simplification in the biomedical domain.

Method

We use the CLEAR comparable medical corpus [START_REF] Grabar | CLEAR -Simple Corpus for Medical French[END_REF] available online1 which contains three comparable sub-corpora in French. Documents within these sub-corpora are contrasted by the degree of technicality of the information they contain with typically specialized and simplified versions of a given text. These corpora cover three genres: drug information, summaries of scientific articles, and encyclopedia articles. The Drugs corpus contains drug information such as provided to health professionals and patients. This corpus is built from the public drug database2 of the French Health ministry. The Scientific corpus contains summaries of meta-reviews of high evidence health-related articles, such as proposed by the Cochrane collaboration and the simplified versions that they provide [START_REF] Sackett | Evidence based medicine: what it is and what it isn't[END_REF]. This corpus has been built from the online library of the Cochrane collaboration 3 . The Encyclopedia corpus contains encyclopedia articles from Wikipedia4 and Vikidia 5 . Wikipedia articles are considered as technical texts while Vikidia articles are considered as their simplified versions (they are created for children from 8 to 13 year old). Only articles indexed in the medical portal are exploited in this work.

We exploit a reference dataset with sentences manually aligned by two annotators within this corpus.

Automatic Detection and Alignment of Parallel Sentences

Automatic detection and alignment of parallel sentences is the main step of our work. The objective is to assess whether two given sentences have the same meaning.

The reference data with aligned sentence pairs, which associate technical and simplified contents, are created manually. We have randomly selected 39 documents from the corpus. The sentence alignment is done by two annotators. This alignment process provides a set of 238 equivalent sentence pairs. The inter-annotator agreement is 0.76 [START_REF] Cohen | A coefficient of agreement for nominal scales[END_REF], before consensus. In order to perform the automatic categorization, we also need negative examples, which are obtained by randomly pairing all sentences from all the document pairs except the sentence pairs that are already found to be parallel. Approximately 590,000 non-parallel sentences pairs are created in this way. That high degree of imbalance is the main challenge in our work and we address it in the experimental design (sec. 2.2).

For the automatic alignment of parallel sentences, we use a binary classification model that relies on the random forest algorithm [START_REF] Breiman | Random forests[END_REF]. The implementation we use is the one that is available in scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF]. The features we use are the following: number of common non-stopwords; percentage of words from one sentence included in the other sentence, computed in both directions; sentence length ratio; average word length ratio, total number of common character bigrams and trigrams, word-based similarity measure exploits three scores (cosine, Dice and Jaccard) character-based minimal edit distance, word-based minimal edit distance [START_REF] Levenshtein | Binary codes capable of correcting deletions, insertions and reversals[END_REF], syntactic token-based minimal edit distance; number of common CUI (we generate all the possible subsequences of words in both sentences and check whether they have a corresponding CUI in the UMLS/SNOMED lexicon [START_REF] Stajner | Cats: A tool for customised alignment of text simplification corpora[END_REF]. We count the number of times that a same CUI appears in both sentences); WAVG and CWASA [17].

Experimental Design

We work on sentences that are equivalent (they both express the same meaning). 238 equivalent pairs were obtained.

We performed two sets of experiments:

1. We train and test the model with balanced data (we randomly select as many nonaligned pairs as aligned pairs), and then we progressively increase the number of non-aligned pairs until we reach a ratio of 3000:1, which is close to the real data (∼4000:1). 2. Then, for each ratio, we apply the obtained model to the whole dataset and evaluate the results. Note that the training data is included in the whole dataset, we proceed this way because of the low volume of available data.

This means that each model is evaluated twice : once on the test set, and once on the whole dataset. We only report scores for the aligned category. The results that are presented correspond to the mean values over the fifty runs. Finally, we apply the best model on another 30 randomly selected documents and manually evaluate the output.

Results

We present the results in Figures 1 and2: The x axis represents the growing of imbalance (the first position is 1 and corresponds to balanced data), while the y axis represents the values of Precision, Recall and F-measure. Figure 1 presents the results for the experiments where we evaluate the model on the test set, Figure 2 presents the results for the experiments where we evaluate the model on the whole dataset. We can observe that the use of balanced data provides very high results, both for Precision and Recall, which are very close to the reference data (> 0.90 performance). These good results in an artifical setting cannot be applied to the real dataset, as is indicated by the starting point in Figure 2

Discussion

When the model is learned on a substantial degree of imbalance, the Precision score is high when that model is applied to the real data, which has a ratio of about 4,000:1. The recall value is also high, but since two thirds of the aligned sentences have been used for training, that good score should be considered cautiously.

For further evaluation, we randomly selected 30 pairs of documents to evaluate the performances of the models. We used the model that was trained at a ratio of 1200:1. In terms of precision, the model shows 98.75% on all the sentence pairs aligned (80 sentence pairs), including equivalence, inclusions and intersection. Only one result showed two unrelated sentences. Those results show that we have a model that can be used to automatically generate a parallel corpus with reduced noise, from highly imbalanced comparable corpora, for text simplification purposes.

Conclusion

We addressed the task of detection and alignment of parallel sentences from a monolingual comparable French corpus. We use the CLEAR corpus, that is related to the biomedical area.

We made observations on the effect of imbalance during training on the performance on the real data. We show that increasing the imbalance during training increases the Precision of the model while still maintaining a stable value for Recall.

We will use that model to generate a corpus of parallel sentences in order to work on the development of methods for biomedical text simplification in French.

Figure 1 .

 1 Figure 1. Evaluation on the test set.

Figure 2 .

 2 Figure 2. Evaluation on the whole dataset.

http://natalia.grabar.free.fr/resources.php#clear

http://base-donnees-publique.medicaments.gouv.fr/

http://www.cochranelibrary.com/

https://fr.wikipedia.org

https://fr.vikidia.org


B Marcote
email: marcote@jive.eu

M Ribó

J M Paredes

C H Ishwara-Chandra

John D Swinbank

J W Broderick

S Markoff

Robert Fender

R A M J Wijers

G G Pooley

A J Stewart

M E Bell

R P Breton

D Carbone

S Corbel

J Eislöffel

H Falcke

J.-M Grießmeier

M Kuniyoshi

M Pietka

A Rowlinson

M Serylak

A J Van Der Horst

J Van Leeuwen

M W Wise

P Zarka

Measuring the expansion velocity of the outflows of LS I +61 303 through low-frequency radio observations

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

Gamma-ray binaries are binary systems composed of a young massive star and a compact object. These systems exhibit non-thermal emission from radio to TeV gamma-rays, with their spectral energy distribution clearly dominated by the MeV-GeV photons. Up to now only five systems have been discovered: PSR B1259-63, LS 5039, LS I 61 303, HESS J0634+057, and 1FGL J1018.6-5856. A detailed review of all these systems can be found in [1].

LS I 61 303 is composed of a young B0 Ve star and a compact object orbiting it every 26.4960 d [2] in an orbit with an eccentricity of ∼0.72 [3]. The system exhibits a 1-10 GHz radio light-curve orbitally modulated. The source exhibits a clear outburst per orbital cycle [4,5,6]. A modulation with a period of 1667 d is also observed at all wavelengths, the so-called superorbital modulation [2]. This modulation affects the amplitude of the non-thermal outbursts and the orbital phases at which the peak emission takes place.

OBSERVATIONS

To perform a detailed study on the variability of LS I +61 303 below 1 GHz we have analyzed all the archival observations from the Giant Metrewave Radio Telescope (GMRT) at 150, 235 and 610 MHz. The last two frequencies were recorded simultaneously during the same observations. Most of these observations were taken during a monitoring of the source performed during three consecutive orbital cycles in 2005-2006. We have also conducted and analyzed observations at 150 MHz with the Low Frequency Array (LOFAR) in 2013. All these data have been compared with contemporaneous observations at 15 GHz with the Ryle Telescope and the Owens Valley Radio Observatory 40-m telescope (OVRO), which show the known behavior of LS I +61 303.

Whereas the GMRT and RT observations were taken at a superorbital phase of ∼0.2, the LOFAR and OVRO observations were taken at a superorbital phase of ∼0.8-0.0. The behavior of the source at these two different superorbital phases is expected to be completely different. A comparison between the three low frequencies is thus impossible and both datasets will be discussed separately in the following.

RESULTS

Light-curves at 235 and 610 MHz

The monitoring performed by the GMRT at 235 and 610 MHz in 2005-2006 allowed us to determine the behavior of LS I +61 303 during three consecutive orbital cycles. Despite each cycle is poorly sampled, a folding of the data with the orbital phase show the light-curves of the source at the two mentioned frequencies, see Figure 1. The 610-MHz light-curve shows a quasi-sinusoidal emission, with the maximum taking place at a orbital phase between 0.8 and 1.1. Whereas the increase of the emission starts at similar orbital phases than the one at 15-GHz, the maximum is much more wider and the decay takes place around 0.3 orbital phases after the one at 15 GHz.

The spectral indexes are basically modulated by the 610-MHz emission (see Figure 1, as the source exhibits a smaller variability at 235 MHz.

Light-curve at 150 MHz

At 150 MHz only six observations have been conducted at the moment. However, the presence of variability in the emission of LS I +61 303 at this frequency has already pointed out (Figure 2. We observe a flux density modulation with the maximum emission taking place at orbital phases around 1.0. A comparison of these data with contemporaneous OVRO data at 15 GHz shows that the peak of the emission is delayed ∼0.5 orbital phases.

DISCUSSION

We have obtained for the first time the light-curves of LS I +61 303 at frequencies below 1 GHz. These light-curves show significant differences with respect to the known behavior of the source in the frequency range of 1-15 GHz (see e.g. [6]). Additionally, the observed flux densities support the presence of a turnover between 235 and 1 GHz (the emission at larger frequencies is known to be consistent with a power-law [6]).

The most plausible scenario at this point is that the absorption mechanisms responsible for the turnover could also be responsible for the observed differences in the light-curves. The available data do not allow us to infer the dominant absorption mechanism at frequencies below 1 GHz. However, in this kind of systems we expect that either free-free absorption (FFA) or synchrotron self-absorption (SSA) dominates at around the turnover frequency (see e.g. [7]. In the following we model the radio emitting region of LS I +61 303 assuming a region dominated by free-free absorption or by synchrotron self-absorption. With these models and the available data we infer the properties of this region.

In both cases we have assumed a one-zone model with an homogeneous synchrotron-emitting region. A detailed explanation of the considered models can be seen in [8]. We assume that the peak emission corresponds to a change in the opacity of the emitting region, becoming optically thick at a particular frequency. Given that the transition between an optically thin to an optically thick region would take place at different moments different frequencies, this assumption can easily explain why the peak emission takes place at different orbital phases. The delay between the different frequencies can thus constrain the expansion velocity of this region.

A FFA dominated region would require an expansion velocity of ∼ 700 km s -1 (see Figure 3) to explain the observed delays at 235 and 610 MHz for an assumed mass-loss rate of 5 × 10 -8 M yr -1 [9].

In the case of a SSA dominated region, an additional dependency of the magnetic field, B with the angular size of the region, is required. In this case we can assume a decay inversely proportional to the size (as in a conical or toroidal expansion) or to the square of the size (as expected in a spherical expansion). Figure 4 shows the effect of different expansion velocities for both cases. An expansion velocity of ∼ 1 000 km s -1 and of ∼ 17 000 km s -1 is derived from this model for a dependency of B ∼ -2 or B ∼ -1 , respectively.

CONCLUSIONS

We have studied for the first time the emission of LS I +61 303 at frequencies below 1 GHz. The presence of a turnover at frequencies between 235 MHz and 1 GHz is inferred from these observations. Moreover, we have detected variability at 150, 235, and 610 MHz, seen significant differences with respect to the light-curves observed at 1-15 GHz. These differences can be explained by the presence of transition to an optically thick region. Whereas the dominant absorption mechanism remains unclear, we have modelled this emitting region with the two most plausible scenarios: a region dominated by free-free absorption and by synchrotron self-absorption. In both cases we have derived expansion velocities for the emitting region which are clearly sub-relativistic. Additionally, in some cases these velocities are close to the stellar wind velocity (∼ 1 000 km s -1 ). This fact would give further support to the scenario of a young non-accreting pulsar with a collision between the stellar wind and the relativistic pulsar wind. A micro quasar scenario would provide much higher expansion velocities, due to the presence of a relativistic jet.

FIGURE 1 .

 1 FIGURE 1. Top: Folded light-curve of LS I +61 303 with the orbital period at 235, 610 MHz (GMRT data), and 15 GHz (RT data).Whereas the source exhibits a quasi-sinusoidal modulation at 610 MHz with the maximum located at orbital phases between 0.8 and 1.1, we observe a delay of about 0.5 orbital phases in the light-curve at 235 MHz. The variability is clearly different form the one observed at 15 GHz. Bottom: Spectral index derived from the 235 and 610 MHz data.

FIGURE 2 .

 2 FIGURE 2. Folded light-curve of LS I +61 303 with the orbital period at 150 MHz (LOFAR and GMRT data) compared with contemporaneous 15-GHz OVRO data. We observe a similar behavior, although a delay of about 0.5 orbital phases could be supported from these data.

FIGURE 3 .

 3 FIGURE 3. Expected shift in orbital phase for the peak emission of LS I +61 303 at a frequency ν with respect to 15 GHz for different expansion velocities, v FFA . In this model we have assumed a region dominated by free-free absorption (FFA). The black circles represent the shifts derived from Figure1.

FIGURE 4 .

 4 FIGURE 4. Expected shift in orbital phase for the peak emission of LS I +61 303 at a frequency ν with respect to 15 GHz for different expansion velocities, v SSA In this model we have assumed a region dominated by synchrotron self-absorption (SSA) and two possible dependencies of the magnetic field, B, with respect to the size of the region, . The black circles represent the shifts derived from Figure1.

ACKNOWLEDGMENTS

We thank the staff of the GMRT and LOFAR who made these observations possible. GMRT is run by the National Centre for Radio Astrophysics of the Tata Institute of Fundamental Research. LOFAR, the Low Frequency Array designed and constructed by ASTRON, has facilities in several countries, that are owned by various parties (each with their own funding sources), and that are collectively operated by the International LOFAR Telescope (ILT) foundation under a joint scientific policy. The Ryle Telescope is operated by the University of Cambridge and supported by STFC. This research has made use of data from the OVRO 40-m monitoring program which is supported in part by NASA grants NNX08AW31G and NNX11A043G, and NSF grants AST-0808050 and AST-1109911. We thank T. Hovatta for providing the OVRO 40-m dish data. B.M., M.R. and J.M.P. acknowledge support by the Spanish Ministerio de Economía y Competitividad (MINECO) under grants AYA2013-47447-C3-1-P, FPA2013-48381-C6-6-P, MDM-2014-0369 of ICCUB (Unidad de Excelencia 'María de Maeztu'), and the Catalan DEC grant 2014 SGR 86. B.M. acknowledges financial support from MINECO under grant BES-2011-049886. J.M.P. acknowledges financial support from ICREA Academia. J.W.B. acknowledges support from European Research Council Advanced Grant 267697 '4 Pi Sky: Extreme Astrophysics with Revolutionary Radio Telescopes'. R.P.B. has received funding from the European Union Seventh Framework Programme under grant agreement PIIF-GA-2012-332393. S.C. acknowledges financial support from the UnivEarthS Labex program of Sorbonne Paris Cité (ANR-10-LABX-0023 and ANR-11-IDEX-0005-02). The financial assistance of the South African SKA Project (SKA SA) towards this research is hereby acknowledged. Opinions expressed and conclusions arrived at are those of the authors and are not necessarily to be attributed to the SKA SA.


text:
Rémi Cardon
email: remi.cardon@univ-lille.fr

Natalia Grabar
email: natalia.grabar@univ-lille.fr

Reducing the Search Space for Parallel Sentences in Comparable Corpora

Keywords: Parallel corpus creation, syntax, French

This paper describes and evaluates three methods for reducing the research space for parallel sentences in monolingual comparable corpora. Basically, when searching for parallel sentences between two comparable documents, all the possible sentence pairs between the documents have to be considered, which introduces a great degree of imbalance between parallel pairs and non-parallel pairs. This is a problem because, even with a highly performing algorithm, a lot of noise will be present in the extracted results, thus introducing a need for an extensive and costly manual check phase. We propose to study how we can drastically reduce the number of sentence pairs that have to be fed to a classifier so that the results can be manually handled. We work on a manually annotated subset obtained from a French comparable corpus.

Introduction

Monolingual parallel corpora are useful for a variety of sequence-to-sequence tasks in natural language processing, such as text simplification [START_REF] Xu | Problems in current text simplification research: New data can help[END_REF], paraphrase acquisition [START_REF] Deléger | Extracting lay paraphrases of specialized expressions from monolingual comparable medical corpora[END_REF] or style transfer [START_REF] Jhamtani | Shakespearizing modern language using copy-enriched sequence to sequence models[END_REF]. In order to build such parallel corpora, the typical approach is to start from comparable corpora and extract sentence pairs that share the same meaning. For instance, the participants of the BUCC 2017 shared task had to address this problem using bilingual corpora [START_REF] Zweigenbaum | Overview of the second BUCC shared task: Spotting parallel sentences in comparable corpora[END_REF]. One major obstacle is that, when considering two documents A and B, every single sentence from A has to be evaluated against every single sentence of B, when document metadata cannot be used to make assumptions as to where to look for corresponding sentences. This produces a large amount of noise, and even with highly performing algorithms, the result of the extraction has to be manually checked for quality. With large volumes of data, this can be extremely costly. This is a known issue when working with comparable corpora [START_REF] Zhang | zNLP: Identifying parallel sentences in Chinese-English comparable corpora[END_REF]). Yet, the issue is either not mentioned in works on parallel corpora creation from comparable corpora, or external information is used, such as metadata [START_REF] Smith | Extracting parallel sentences from comparable corpora using document level alignment[END_REF], which helps a lot the task. In our work, we propose and evaluate methods for filtering out sentences and sentence pairs that have no chance of being of interest for the building of a parallel corpus. Hence, the purpose is to reduce the amount of manual check that needs to be performed on the output of a classifier.

Data collection and pre-processing

To perform our experiments, we work with a French comparable corpus containing biomedical documents with technical and simplified contents [START_REF] Grabar | CLEAR -Simple Corpus for Medical French[END_REF]. The corpus is composed of three subcorpora: drug information for medical practitioners and patients released by the French Ministry of Health 1 , medical literature reviews and 1 http://base-donnees-publique. medicaments.gouv.fr/ their manual simplification released by the Cochrane foundation2 , and encyclopedia articles from Wikipedia3 and Vikidia4 . The documents are organised in pairs where the texts address the same topic for different audiences, so that the delivered information and the phrasing are not identical. More importantly, the order in which the information is delivered is not the same, which means that the document structure cannot be used for assuming where to look for parallel sentences.

For our experiments, we took 39 randomly selected document pairs from that corpus and manually annotated them for two types of sentence pairs :

• Equivalence : the sentences mean the same, but they are not identical;

• Inclusion : the meaning of one sentence is included in the other one, where additional information can also be found. This retains information about sentence splitting or merging and about information deletion or addition.

The documents are pre-processed for syntactic POStagging and syntactic analysis into constituents [START_REF] Kitaev | Constituency parsing with a self-attentive encoder[END_REF]. In the manually annotated set, only sentences that have a verb are kept. This yields 266 sentence pairs: 136 equivalent pairs, and 130 inclusion pairs (56 in one direction, 74 in the other one).

For the automatic processing, we produced the whole possible combinations of sentences within each of the 39 document pairs, and ended up with 1,164,407 sentence pairs. Thus, given that, out of more than one million possible pairs, only 266 sentence pairs are considered as useful for the parallel corpus creation, we observe a high degree of imbalance: little less than 4,400:1. Our purpose is to reduce this imbalance for facilitating the search of parallel sentences and improving the overall quality of the results.

Method

In order to address that extremely high degree of imbalance, we propose to investigate three methods using formal and syntactic indicators:

• First method is based on the number of tokens in sentences. Hence, each candidate sentence must contain at least five tokens. This permits to consider sentences that are grammatically complete and convey some semantics. We set that value to five because that is the length of the shortest sentence in the set with the manual annotations;

• Second method prevents from producing pairs with identical sentences;

• Third method relies on syntactic information. We base our work on a method that uses constituency parsing for measuring similarity between sentences in a monolingual setting [START_REF] Duran | Similarity of sentences through comparison of syntactic trees with pairs of similar words[END_REF]. In the original work, the authors detect similar words in sentences and assign a similarity score that is computed by looking at similar labels of nodes that contain similar words. The process is described in Figure 1. It is difficult to adapt that method as it is described in the paper. The main reason is that it relies heavily on a table that establishes which grammatical categories for constituents are similar to one another. It is made for English and there is no indication as to how it was built. Nonetheless, we make the assumption that adopting a similar approach could help in the process of weeding out undesired pairs for building a parallel corpus. Hence, instead of calculating a similarity score, we just choose between keeping the sentence pair as a candidate for a classifier, or rejecting it.

For a given pair, we produce a syntactic tree for each of the two sentences. Then, if both sentences contain a verb, we compare all the leaves (i.e. words) of the trees, except the ones that are part of the stop words list. The list contains 83 items that are grammatical words, such as determiners or prepositions for example. If we find two identical words, we look at their parents nodes' labels. If those are identical, we keep the sentence in the candidates list. That process is illustrated in Algorithm 1 below. We also perform the same approach but instead of stopping if the parents nodes' labels are not identical, we go up a level to perform the same comparison, and up another level if the previous comparison was not successful. As soon as one comparison succeeds, we keep the sentence pair in the candidates list. This other approach is illustrated in Algorithm 2. That movement to the third parent of the leaves is what is chosen in the method which inspires this work, we chose to implement it to learn how the depth of exploration influences our filtering.

To parse the sentences in order to obtain their syntactic tree with constituents, we use the Berkeley Neural Parser and the language model that is provided with it for French, with the benepar Python library [START_REF] Kitaev | Constituency parsing with a self-attentive encoder[END_REF]. The, we use the NLTK's Tree library [START_REF] Bird | Natural Language Processing with Python[END_REF] for tree manipulation and exploration.

Evaluation

We evaluate the results obtained in three different ways:

• we compare the number of initial sentence pairs to the number of remaining sentence pairs after the filtering,

• we check whether the removed pairs are manually an- Algorithm 2: Filtering method looking up to the third parent node of the leaves notated as parallel, be it equivalence or inclusion relation, in the reference dataset,

• we give the remaining data to a random forest classifier algorithm, such as done in a previous work [START_REF] Cardon | Parallel sentence retrieval from comparable corpora for biomedical text sim-plification[END_REF], and evaluate recall and precision of the output.

The overall goal is to remove as many negative examples as possible, while preserving the positive examples.

Results and Discussion

We first look at how the volume of data is reduced further to the filtering operations. The first column in Table 1 shows the number of raw sentence pairs, the second colum indicates the number of pairs after using the formal indicators (FI), the third and fourth columns show the number of pairs remaining when using the syntactic filter, respectively with looking at the first syntactic parent node and up to the third parent node. The formal indicators are applied before the syntactic filters. The syntactic filters are used independently from one another. We can see that the simple formal indicators reduce the total number of sentence pairs by 65% (from 1,164,407 to 409,530 sentence pairs). These two indicators were defined on the basis of observation of our data. They are very straightforward and we expected that no positive example (equivalent and inclusion pairs) would be lost in the process. This hypothesis is verified indeed: all the good candidates for parallel pairs are kept at this step.

Starting from the 409,530 pairs obtained after this first filter, we can see that both syntactic filters lead to a huge reduction of the volume of remaining sentence pairs:

• when using depth 1 leaves 16,879 pairs (∼96% reduction) remain,

• when using depth 3 leaves 21,428 pairs (∼95% reduction) remain.

The downside is that a substantial amount of positive examples is also lost in the process:

• 42 out of 136 (∼30%) for equivalent pairs with both depths used,

• 36 out of 130 (∼27%) for inclusion pairs with depth 1, 32 out of 130 (∼24%) for inclusion pairs with depth 3.

The over 95% reduction with the syntax filter on data that were already greatly reduced complies with our initial goal. Yet, we lose several good candidates for parallel sentences. Hence, we look at the positive examples that were rejected by the syntactic filter in order to understand why it is the case and how we can address this issue. For instance, consider the following sentence pair:

• Dans le cas où le patient devrait arrêter le traitement, il est recommandé de réduire progressivement la posologie. (In case the patient should stop the treatment, it is recommended to decrease the dose progressively.)

• L'arrêt du traitement doit se faire de manière progressive. (The cessation of treatment must be done progressively.)

Unfiltered FI Syntax Depth 1 Syntax Depth 3 Set P R F1 P R F1 P R F1 P R F1 Equivalent
Neg. 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 Equivalent Pos. 0.79 0.43 0.55 0.82 0.32 0.46 0.75 0.39 0.51 0.84 0.40 0.54 Inclusion Neg.

1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 Inclusion Pos. 0.71 0.09 0.17 0.50 0.16 0.24 0.71 0.15 0.24 0.56 0.15 0.24 Table 2: Precision, Recall and F1 scores on the different sets of sentence pairs with classification.

The reason why this kind of sentence pairs is rejected is because the labels of parent nodes for identical words (such as traitement (treatment) in this example) differ in the trees produced by the syntactic parser. Indeed, in the first sentence, le traitement (the treatment) is labelled as an NP-OBJ, while it is labelled as an NP in the second sentence. The error is caused by the fact that le traitement from the second sentence (in du traitement, which is correctly analyzed as de le traitement) is an NP in a PP that depends on the noun arrêt. The parser that we use sometimes adds the information about the function of a phrase, this is the case in the first sentence here where le traitement is the object of the verb arrêter. This kind of examples suggests to put together similar node labels, such as NP and NP-OBJ.

It would also be interesting to see whether some nodes are consistently similar in the parallel pairs, and hopefully find that those consistencies do not appear in pairs that should not be retained in a parallel corpus. Let's analyze another typical example:

• La prudence est recommandée chez les sujets atteints d'ulcères gastroduodénaux. (The vigilance is recommended in subjects suffering from gastroduodenal ulcers.)

• Ce médicament doit être utilisé avec prudence en cas d'ulcère de l'estomac ou du duodénum. (This medication must be used with vigilance in case of ulcers of the stomach and duodenum)

There is only one pair of identical words here : prudence (vigilance). This work is labelled as an NP in the first sentence and as a PP in the second sentence. The presence of ulcère (ulcer) in both sentences is not detected: the filter is currently looking for strictly identical words, while in these two sentences, ulcère (ulcer) occurs in its plural form in the first sentence and in its singular form in the second sentence. Hence, the filter must be more permissive in order to detect such occurrences. One solution is to work with a lemmatizer, another solution is to propose a more sophisticated word comparison function. This is a task where word embeddings could also be useful. We intend to test this possibility in future works.

Table 2 shows the results of classification with the different sentence pairs sets. For each experiment, the data were divided in two thirds for training and one third for testing. The results are reported by class (negative and positive) and positive class type (either equivalence or inclusion). The negative class has a perfect score in every metric because of the high degree of imbalance, the false negatives are not numerous enough to have an influence on the score. We can see that the syntactic method with a depth of exploration of three levels has a positive influence on precision, compared to unfiltered data, and recall is negatively impacted. We believe that being deprived of one third of such a small set of positive examples has a strong negative impact on performance. We should be able to improve recall if we prevent the positive examples from being filtered out, as we mentioned in the error analysis above. The results for inclusion show that this type of sentence pair is hard to recognize automatically. There is some improvement with filtered data, but the scores are low, especially recall. What we draw from those results is that the different sentence pairs types should be handled differently. It seems that we cannot expect to extract inclusion pairs in the same way as we extract equivalent pairs.

Conclusion

In this work, we proposed to address the problem of imbalance in the process of extracting parallel sentences from comparable corpora. We worked on a French comparable corpus made for biomedical text simplification. We showed that we could drastically reduce the number of negative examples (>98%) with simple heuristics and a syntactic comparison of sentence pairs, at the cost of losing some positive examples. Analyzing the errors, we showed that there were consistencies in what was left out and that should be kept, that can be addressed with improvements to the method, such as a better word comparison function and a more careful work on syntactic node label similarity. Even with those issues, we reduce the imbalance and improve precision on a classification task for equivalent sentences, thus reducing the manual work needed to check the output, which was the main objective. We also showed that inclusion pairs are much harder to process and that another method should be used for extracting that type.

Figure 1 :

 1 Figure1: The similarity method described in[START_REF] Duran | Similarity of sentences through comparison of syntactic trees with pairs of similar words[END_REF]

:

  Data: A pair of syntactic trees (T 1 and T 2 ), a list of stop words (SW) Result: Boolean Boolean ← False; if one verb is found in both sentences then foreach leaf in T 1 (L 1 ) not found in SW do foreach leaf in T 2 (L 2 ) not found in SW do if L 1 is identical to L 2 then if L 1 's parent node's label is identical to L 2 's parent node's label then Filtering method only looking at the immediate parent nodes of the leaves

	Boolean ← True;
	else
	nothing;
	end
	else
	nothing;
	end
	end
	end
	else
	nothing;
	end
	return Boolean;
	Algorithm 1

Table 1 :

 1 Pairs remaining after the various filtering methods. False; if one verb is found in both sentences then foreach leaf in T 1 (L 1 ) not found in SW do foreach leaf in T 2 (L 2 ) not found in SW do if L 1 is identical to L 2 then if L 1 's parent node's label (P 1 ) is identical to L 2 's parent node's label (P 2 ) then

	Remaining Pairs Unfiltered	FI	Syntax Depth 1 Syntax Depth 3
	Total	1,164,407 409,530	16,879	21,428
	Equivalent	136	136	94	94
	Inclusion	130	130	94	100
	Data: A pair of syntactic trees (T 1 and T 2 ), a list of stop
	words (SW)
	Result: Boolean
	Boolean ← Boolean ← True;
	else
	if P 1 's parent node's label (PP 1 ) is
	identical to P 2 's parent node's label
	(PP 2 ) then
	Boolean ← True;
	else
	if PP 1 's parent node's label is
	identical to PP 2 's parent node's
	label then
	Boolean ← True;
	else
	nothing;
	end
	end
	end
	else
	nothing;
	end
	end
	end
	else
	nothing;
	end
	return Boolean;

https://france.cochrane.org/ revues-cochrane

https://fr.wikipedia.org/

https://fr.vikidia.org/

Acknowledgements

We would like to thank the reviewers for their comments. This work was funded by the French National Agency for Research (ANR) as part of the CLEAR project (Communication, Literacy, Education, Accessibility, Readability), ANR-17-CE19-0016-01.