Hanna Pylieva

Artem Chernodub

Natalia Grabar

Thierry Hamon

Improving Automatic Categorization of Technical vs. Laymen Medical Words using FastText Word Embeddings

Keywords: text simplification, difficulty detection, word embeddings

Detection of difficult for understanding words is a crucial task for ensuring the proper understanding of medical texts such as diagnoses and drug instructions. In this paper, we study usage of recently developed word embeddings, which contain context information for words together with other linguistic and non-linguistic features, for improving the detection of difficult medical words. We propose new cross-validation scenarios in order to test the generalization ability of the medical words difficulty detection from different perspectives and provide the experimental study of previously used methods for feature extraction together with recently proposed FastText embeddings. We found that for known words and unknown users FastText embeddings surely improves the detection of word understandability reaching 85.9 F-score (up to 2.9 F-score improvement).

Introduction

Specialized areas, such as medical area, convey and use technical words, or terms, which are typically related to knowledge developed within these areas. In the medical area, this specific knowledge often corresponds to fundamental medical notions related to disorders, procedures, treatments, human anatomy, etc. For instance, technical terms like blepharospasm (abnormal contraction or twitch of the eyelid), alexithymia (inability to identify and describe emotions in the self), appendicectomy (surgical removal of the vermiform appendix from intestine), or lombalgia (low back pain) are frequently used in the medical area texts.

As in any specialized areas, two main kinds of users exist in the medical area:

 medical doctors, both researchers of practitioners, are experts of the domain. They contribute to the creation and development of biomedical knowledge and its exploitation for the healthcare process of patients;

 patients and their relatives are consumers of the healthcare process. Usually, they do not have expert knowledge, while it is important that they understand the purpose and issues of their healthcare process.

If the understanding of technical medical terms is easy for the medical staff, patients and their relatives may present some difficulties in the understanding and using of such terms: they show indeed poor health literacy.

Hence, the existing literature provides several studies dedicated to the understanding of medical notions and terms by non-expert users, and on their impact on a successful healthcare process [START_REF] Mcgray | Promoting health literacy[END_REF][START_REF] Eysenbach | Poverty, human development, and the role of ehealth[END_REF]. Yet, it is not uncommon that patients and their relatives must face very technical health documents and information. Examples of this kind are frequent and usually the non-expert users are at loss in such situations:

 understanding of information on drug intake [START_REF] Vander Stichele | Promises for a measurement breakthrough[END_REF][START_REF] Patel | Errors in interpreting quantities as procedures: The case of pharmaceutical labels[END_REF], such as instructions related to the description and specification of steps necessary for the preparation and intake of drugs,  understanding of clinical documents [START_REF] Zeng-Treiler | Text characteristics of clinical reports and their implications for the readability of personal health records[END_REF], which contain important information on the healthcare process of patients,  understanding of clinical brochures or informed consents [START_REF] Williams | Inadequate functional health literacy among patients at two public hospitals[END_REF], which are specifically created for patients and which are typically read by patients during their clinical pathway,  more generally, understanding of information provided for patients by different websites [START_REF]Oregon Practice Center: Barriers and drivers of health information technology use for the elderly, chronically ill, and underserved[END_REF][START_REF] Brigo | Clearly written, easily comprehended? The readability of websites providing information on epilepsy[END_REF] in different languages (English, Spanish, French) and different medical specialties,  for the same reasons, communication between patients and medical staff [START_REF] Jucks | Choice of words in doctor-patient communication: an analysis of health-related internet sites[END_REF][START_REF] Tran | Internet et soins: un tiers invisible dans la relation medecin/patient[END_REF] remains complicated.

These various observations provide the main motivation to our work. We propose to address the needs of non-specialized users in the medical domain. As we noticed, the main need is related to the understanding of medical and health information. In what follows, we first present some related work (Section 2). We then introduce the material used (Section 3) and the proposed method (Section 4). Our results and their discussion are presented in Section 5. Finally, we conclude with some directions for future work in Section 6.

Related work

Related work is globally related to the detection of technical contents in documents and to their adaptation. Here, we are interested by the first aspect: detection and diagnosis of technical medical contents.

In the NLP (Natural Language Processing) area, work related to the diagnosis of technical medical documents is quite frequent. Traditionally, researchers exploit the readability measures. Among these measures, it is possible to distinguish classical readability measures and computational readability measures [START_REF] François | An AI readability formula for French as a foreign language //Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning[END_REF]. Classical measures usually rely on number of letters and/or of syllables a word contains and on linear regression models [START_REF] Gunning | The art of clear writing[END_REF], while computational readability measures may involve vector models and a great variability of features, among which the following have been used for processing the biomedical documents: combination of classical readability formulas with medical terminologies [START_REF] Kokkinakis | Comparing lay and professional language in cardiovascular disorders corpora[END_REF]; n-grams of characters [START_REF] Poprat | A language classier that automatically divides medical documents for experts and health care consumers[END_REF], manually [START_REF] Zheng | Filtering for medical news items using a machine learning approach[END_REF] or automatically [START_REF] Borst | Lexically based distinction of readability levels of health documents[END_REF] defined weights of terms, stylistic [START_REF] Grabar | Classication of health webpages as expertand non expert with a reduced set of cross-language features[END_REF] or discursive [START_REF] Goeuriot | Caractérisation des discours scientifique et vulgarisé en français, japonais et russe[END_REF] features, lexicon [START_REF] Miller | A Classifier to Evaluate Language Specificity of Medical Documents[END_REF], morphological features [START_REF] Chmielik | Détection de la spécialisation scientifique et technique des documents biomédicaux grâce aux informations morphologiques[END_REF], combinations of different features [START_REF] Zeng-Treiler | Text characteristics of clinical reports and their implications for the readability of personal health records[END_REF].

At a more fine-grained level, detection and diagnosis of technical medical words has been addressed much less frequently. In the general language, some research actions are often performed as part of the NLP challenges, such as the SemEval NLP challenge1 held in 2012. This challenge proposed the following task: for a short text and a target word, several possible substitutions satisfying the context have also been proposed. The objective was to rate and to order the substitutions according to their degree of simplicity [START_REF] Specia | Semeval-2012 task 1: English lexical simplification[END_REF]. The participants applied rule-based and/or machine learning systems. Combinations of various features, designed to detect the simplicity of words, have been used, such as: lexicon from spoken corpus and from Wikipedia, Google n-grams, WordNet [START_REF] Sinha | Untsimprank: Systems for lexical simplification ranking[END_REF]; word length, number of syllables, latent semantic analysis, mutual information and word frequency [START_REF] Jauhar | UOW-SHEF: SimpLex -lexical simplicity ranking based on contextual and psycholinguistic features[END_REF]; Wikipedia frequency, word length, n-grams of characters and of words, random indexing and syntactic complexity of documents [START_REF] Johannsen | Emnlp@cph: Is frequency all there is to simplicity?[END_REF]; n-grams and frequency from Wikipedia, Google n-grams [START_REF] Ligozat | Annlor: A naive notation-system for lexical outputs ranking[END_REF]; WordNet and word frequency [START_REF] Amoia | SB: mmSystem -using decompositional semantics for lexical simplification[END_REF]. The best systems reached up to 0.60 Top-rank and 0.575 Recall.

Another work has been done on scholar texts in French written for children with the purpose to differentiate between the texts from various scholar levels and to test various features suitable for that [START_REF] Gala | Towards a french lexicon with difficulty measures: NLP helping to bridge the gap between traditional dictionaries and specialized lexicons[END_REF]. This system reached up to 0.62 classification accuracy.

In the medical area, we can mention three experiments: manual rating of medical words [START_REF] Zheng | Filtering for medical news items using a machine learning approach[END_REF], automatic rating of medical words on the basis of their presence in different vocabularies [START_REF] Borst | Lexically based distinction of readability levels of health documents[END_REF], and exploitation of machine learning approach with various features [START_REF] Grabar | Automatic diagnosis of understanding of medical words[END_REF]. This last experiment achieved up to 0.85 F-measure on individual annotations.

The purpose of the current work is to propose novel machine learning approaches for a more efficient distinction of technical medical words which may present understanding difficulties to non-experts users. The medical data processed are in French.

3

Dataset description

Linguistic data description

For the text classification task aimed the data was collected and annotated as described in [START_REF] Grabar | Automatic diagnosis of understanding of medical words[END_REF]. The source terms are obtained from the medical terminology Snomed International [START_REF] Côté | Répertoire d'anatomopathologie de la SNOMED internationale, v3. 4[END_REF] in French, available from the ASIP SANTE website 2 . The purpose of this terminology is to provide an extensive description of the medical field. Snomed contains 151,104 medical terms organized into eleven semantic axes such as disorders and abnormalities, procedures, chemical products, living organisms, anatomy, social status, etc. For the purpose of the task, we chose five axes related to the main medical notions: disorders, abnormalities, procedures, functions, and anatomy. Our assumption is that terms in these categories are familiar to a layman, in contrast to contents of such specific groups as chemical products (hydrogen sulfide) and living organisms (Sapromyces, Acholeplasma laidlawii).

The 104,649 selected terms are lemmatized and tokenized into words (or tokens) resulting in 29,641 unique words such that 'trisulfure d'hydrogène' provides three words (trisulfure, de, hydrogène).

The dataset contains three morphological groups of words:

 compound words which contain several bases: abdominoplastie (abdominoplasty), dermabrasion (dermabrasion);  constructed words which contain one base and at least one affix: cardiaque (cardiac), acineux (acinic), lipoı̈de (lipoid);  simple words which contain one base, no affixes and possibly infections (when the lemmatization fails): acné (acne), fragment (fragment).

Annotation process

The set of 29,641 unique words was annotated by three French 25-40-yearold, without medical training, without specific medical problems, but with the linguistic background. The annotators are expected to represent the average knowledge of medical words among the population as a whole. The annotators are presented with a list of terms and asked to assign each word to one of the three categories:

 I can understand the word;  I am not sure about the meaning of the word;  I cannot understand the word.

The assumption is that the words, which are not understandable by the annotators, are also difficult to understand by patients. The annotators were asked not to use dictionaries during the annotation process. The annotation results are represented in Table 1.

Machine learning-based categorization

We propose to tackle the problem through the supervised categorization: the purpose is to categorize words, or terms, according to whether they can be understood or not by non-specialized people. The manual annotations of these words provide the reference data. The categorization pipeline is the following. First, for all words in the dataset Natural Language Processing (NLP) features were calculated. Then they were used for training the classifiers. Finally, the quality of the trained classifiers was evaluated using the cross-validation.

Standard NLP features for words

We will refer to previously used NLP features described in [START_REF] Grabar | Automatic diagnosis of understanding of medical words[END_REF] as "standard features" (opposed to "embeddings" described in the next subsection). They include 24 linguistic and extra-linguistic features related to general and specialized languages. The features are computed automatically and can be grouped into ten classes:  syntactic categories;  presence of words in reference lexica;  the frequency of words through a non-specialized search engine;  the frequency of words in the medical terminology;  number and types of semantic categories associated with words;  length of words as a number of their characters and syllables;  number of word's bases and affixes;  initial and final substrings of the words;  number and percentage of consonants, vowels and other characters;  classical readability scores.

Proposed usage of FastText word embeddings

Currently, word embedding vectors [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] (or word vector representations) are used in the most of state-of-the-art methods for various NLP tasks [START_REF]Repository to track the progress in Natural Language Processing (NLP)[END_REF]. Usually, word embeddings are pre-trained on the giant corpora of natural texts such as Google News, Wikipedia texts in an unsupervised manner to predict the context of the target words. They exploit the distributional hypothesis that semantically close words are next to each other in the sentence and that semantically close words share similar contexts.

FastText word embeddings [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF] is a good candidate as features for words difficulty detection task because they are able to use words' morphological information and generalize over it. The fact that word embeddings capture context and morphological information leads to the hypothesis that incorporating this information as features will improve classification accuracy for our specific problem. FastText embedding vectors are the sum of character n-gram representations, so that they could be generated even for unknown words. Nevertheless, being trained on Wikipedia texts the portion of known words from our dataset for current FastText embeddings is quite big. According to our analysis, 44.26% (13,118 out of 29,641) medical words in the dataset and 56.00% (16,598 out of 29,641) lowercased medical words in the dataset were used for training of the currently published French FastText3 model.

Experiments

In [START_REF] Grabar | Automatic diagnosis of understanding of medical words[END_REF] different algorithms of supervised classification methods were trained using standard NLP features (section 4.1) to detect the words' understandability. The success of the applied classification algorithms was evaluated within the three accuracy measures: accuracy (A), precision (P), recall (R) and F1-measure (F). These scores are weighted average for 1-vs-rest binary classifiers for each of three classes. They allow evaluation of the suitability of the methodology to the difference between understandable and non-understandable words and the relevance of the chosen features to the target problem.

Experiments on the reproduction of previous results

In order to ensure the consistency of the experiments, first we reproduced the WEKA results using pre-computed standard set of features from [START_REF] Grabar | Automatic diagnosis of understanding of medical words[END_REF]. Second, we developed a solution based on decision tree (DT) classifier from well-known scikit-learn library 4 . Here we got 0.85-1.41 lower F scores for scikit-learn compared to our own reproduction in WEKA (Table 2).

Table 2. Comparison of various implementations for decision tree classifier on three datasets (A1, A2, A3) in user-in vocabulary-out cross-validation

Results from [START_REF] Grabar | Automatic diagnosis of understanding of medical words[END_REF] Our Since the input features were identical for WEKA 5 and scikit-learn frameworks, we decided that this small degradation of quality is caused by the different implementations of decision tree classifiers in these frameworks. Nevertheless, in all subsequent experiments we will use the scikit-learn implementation because of it's ease of use for experiments.

Experiments on user-in vocabulary-out cross-validation

These experiments also follows the scenario from [START_REF] Grabar | Automatic diagnosis of understanding of medical words[END_REF]. The cross-validation is done on each dataset (i.e. each user's annotation) separately. The goal of these experiments is to measure the ability of the method to generalize class recognition on the known user and his known manner to annotate words (that is, his understanding of the meaning of medical words) for unknown words.

We carried out the experiments using (i) the standard features only, (ii) the FastText word embeddings only and (iii) their combination. Experiments with isolated FastText word embeddings as features and the data from three annotators resulted in poor F-scores (Table 3), that can be treated that contextual information which is dominant in the word embeddings is not enough to define the word understandability. Adding the FastText word embeddings to the standard feature set resulted in up to 1% higher F-score due to higher Precision (up to 1.8%), meaning that contextual information slightly impacts on the understandability of a word by a given person.

Experiments on user-out vocabulary-in cross-validation

In this experiment, we learn from all the annotations of one user and then test the model on annotations of another user. In this setting, we measure the ability of the classifier to generalize on all known words, but for unknown users (Table 4). This scenario is plausible to a real-world situation, where it is possible to obtain annotations from a couple of users but not from all the possible users, while it is necessary to predict the familiarity of medical words for all the potential users.

Table 4. Experiments on user-out vocabulary-in cross-validation

In these experiments we got a significant improvement of combined features in comparison to the standard features. When knowledge of words understandability of one user is used to predict it for another user, adding the FastText word embeddings provides up to 2.9 better F-score. Notice that used separately, standard features and embeddings shows similar performance as in user-in vocabulary-out cross-validation (Table 3). Our hypothesis is that there exists a robust nonlinear dependency between some subsets of standard features and subword-level components of FastText word embeddings. Testing this hypothesis is the topic of our further research.

Experiments on user-out vocabulary-out cross-validation

In this experiment, we take (k-1) folds of data from one user for training and use k-th fold for testing from the remaining user. In this case, we measure the ability of the method to generalize both on unknown users and unknown vocabulary.

The cross-validation setting is now the most strict and knowledge of words understandability of one user is used to predict whether another user will understand other medical words. In these experiments, embeddings provide approximately 0.5% higher F-score in case of learning on users A1 and A3 (Table 5). When learning on user A2, embeddings decrease F by 0.5, which means that annotations and health literacy of user A2 are different from users A1 and A3. It seems that adding embeddings makes overfitting of machine learning model to the dataset. As a result, tests on other "kind of word understandability" and on combined features are less successful compared to using standard features only for learning. This may be due to the lack of systematicity in annotations of A2.

Conclusions

We proposed to address the detection of medical words which understanding may be difficult for non-specialized users of the medical area. We exploit for this machine learning algorithms and several sets of NLP features: standard features (syntactic information, reference lexica, frequency, etc.), distributional features (such as provided by word embeddings), and their combination.

Our results indicate that adding FastText word embeddings provides a significant improvement of the performance for the generalization for unknown users (up to 2.9 Fscore) but provides a slight increase (0.5 -1 F-score) or even decrease (-0.5 F-score) of performance for unknown words. We consider this positive issue because it is important to be able to generalize annotations provided by a set of users on the whole population.

We have several directions for future work. For instance, we will try to understand the reasons of the decrease of performances with word embeddings, which may help to rectify the results. Besides, we currently use existing pre-trained word embeddings. Yet, we assume that their training on medical data may improve their impact on the categorization results. We also plan to implement and test other deep learning/neural networks/NLP methods which use the morphological information of words, such as character-level recurrent neural networks and character embeddings together with 1D convolutions. Indeed, when language data present stable patterns, which is the case in

Table 1 .

 1 Number (and percentage) of words assigned to reference categories by three annotators (A1, A2, and A3)

	Categories	A1 (%)	A2 (%)	A3 (%)
	1. I can understand	8,099 (28%)	8,625 (29%)	7,529 (25%)
	2. I am not sure	1,895 (6%)	1,062 (4%)	1,431 (5%)
	3. I cannot understand	19,647 (66%)	19,954 (67%)	20,681 (70%)
	Total annotations	29,641	29,641	29,641

Table 3 .

 3 Experiments on user-in vocabulary-out cross-validation

	Train user	Test user	Standard features only		Embeddings only		Standard features + embeddings
			A	P	R	F	A	P	R	F	A	P	R	F
	A1	A1 82.5 77.2 82.5 79.8 72.5	67 72.5 69.3 82.4	79 82.4 80.2
	A2	A2	82 78.9	82	80 73.5 69.9 73.5 71.3 81.9 79.5 81.9 80.3
	A3	A3 85.5 81.2 85.5 83.2 74.9 70.4 74.9 72.3 85.9	83 85.9 84.2

Table 5 .

 5 Experiments on user-out vocabulary-out cross-validation

http://www.cs.york.ac.uk/semeval-2012. Accessed 30 October

2 http://esante.gouv.fr/services/referentiels/referentiels-d-interoperabilite/snomed-35vf.Accessed 30 October 2018.

https://fasttext.cc. Accessed 30 Oct. 2018.

http://scikit-learn.org. Accessed 30 Oct. 2018.

https://www.cs.waikato.ac.nz/ml/weka. Accessed 30 Oct. 2018.

the medical field, processing of subword strings may help for the generalization over new and unseen words. As we presented above, this is one of the current limitations of our work.


Camille Chanial
email: camille.chanial@polytechnique.edu

Rédouane Dziri
email: redouane.dziri@polytechnique.edu

Helena Galhardas
email: helena.galhardas@tecnico.ulisboa.pt

Julien Leblay
email: julien.leblay@aist.go.jp

Minh-Huong Le Nguyen
email: minh-huong.le-nguyen@polytechnique.edu

Ioana Manolescu
email: ioana.manolescu@inria.fr

ConnectionLens: Finding Connections Across Heterogeneous Data Sources

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

MOTIVATION AND OUTLINE

The explosion of digital data sources on about every aspect of modern life has led journalists to collaborate with computer scientists, statisticians, and social scientists of various disciplines, aiming at novel digital tools to analyze and exploit this data. Since a seminal interdisciplinary study [START_REF] Cohen | Computational journalism[END_REF], the eld of data journalism [START_REF] Gray | The Data Journalism Handbook: How Journalists can Use Data to Improve the News[END_REF], that is, journalistic work signi cantly based on digital data, increasingly attracts the attention of journalists and computer scientists alike.

In this work, we focus on a core data journalism problem: identifying connections across a set of heterogeneous, independently produced data sources. We are inspired by investigative journalism work, which seeks out and explores connections between individuals, organizations, companies etc. Such work requires, rst, getting access to data sources, through any means (from verbal or phone communication, to paper mail, fax, and any electronic transmission method); second, analyzing this data, to nd out what valuable information it may contain. For instance, consider an example raised by journalists at Le Monde, a leading French national newspaper, with which we currently collaborate: France's national elections in 2017 brought about an unprecedented ratio of rst-time National Assembly elected members. Journalists scrambled to learn as much as possible about the nation's new representatives, in particular asking e.g., "what connections the new representatives have (or have had) with companies?" This question is interesting to identify areas of economic competence, but also possible con icts of interest. By law, French representatives must disclose direct nancial interests, but more indirect connections (e.g., being, for many years, a close collaborator of a company's current CEO) must be dug out by the press.

Many digital data sources which could be used to answer such questions are publicly available today. However, nding answers to journalists' queries is challenging for many reasons. (i) The data is large, precluding the use of the hand-crafted tools (often spreadsheets) that journalists are used to; (ii) there are many, structurally heterogeneous, independently-produced data sources; some sources, e.g. national company registry, are relational, some others, e.g. contracts, discourses are text, social media content comes as JSON documents, open data is often structured in RDF graphs etc.; (iii) an answer stating, e.g., that a certain representative has studied with the CEO of a given company, may require interconnecting information from several data sources, e.g., the history of company C, the Wikipedia page of A, and the public information available on D and its CEO, namely B; (i ) journalists do not know the shape, size, and structure of the connections they are seeking out, thus they need the ability to search through keywords; ( ) the set of data sources is highly dynamic, as journalists collect various data sources and try to see what insights they can get by combining them with existing ones. It is also worth stressing that newsrooms function under extreme time constraints, making it unfeasible to clean, consolidate and integrate all data sources in a uni ed warehouse. Finally, ( i) a staple of professional journalism is to be able to show evidence for a published claim. Thus, in an answer such as the one outlined in (iii) above, it is important to be able to show where each piece of information came from and how the connections were made.

C L is a prototype addressing the above challenges. At its core is a novel algorithm for keyword search across a set D of heterogeneous data sources, each of which can be: a relational table; a JSON document; a text le; or an RDF graph.

For instance, Figure 1 shows a JSON datasource DS 1 containing information about elected representatives, a text dataset DS 2 listing the alumni of Ecole polytechnique, where many French company executives have studied, and a relational source DS 3 providing information about companies and their CEOs. A query Q is a set of keywords {w 1 , . . . ,w n } for some integer n ≥ 1. In Figure 1, the query "En Marche company" seeks to nd out connections between elected representatives from the "En Marche" political party, and some company. An answer tree to Q over D is a tree, part of a virtual graph (which, as we explain later, is the way we view D). Each answer tree node n i comes from a dataset D i ∈ D, and each edge e j either comes from a dataset D j ∈ D, or is a link between two nodes (from the same or from two di erent datasets) whose data content we nd su ciently similar to consider them "the same" for the purpose of our querying. Further, each Q keyword matches a node or an edge of the answer tree. For example, in Figure 1, the red lines trace an answer tree to the sample query: one of the elected representatives of the party "En Marche" (node matching "En Marche" in DS 1 ) is Anne Martin, who studied at Ecole Polytechnique (alumni information from DS 2 ) just like Philippe Varin, the CEO of the Areva company (edge labeled "company" from DS 3 ). A query Q may have several answer trees on a given D, some more interesting than others; this can be captured by a answer tree score function s associating to every answer tree a number re ecting its interest (the higher, the better). The problem solved by C L then, is: given D, Q, the score function s and an integer k, nd the k answer trees across D with the highest s scores.

A novel contribution of our system is its keyword search approach capable of nding answers across heterogeneous data sources, approach based on a virtual graph we describe next. Our score function, tailored toward journalistic interests (Section 3) is also novel.

VIRTUAL GRAPH

To be able to exploit data connections occurring within and across all data sources D ∈ D, we view all data sources as a single, virtual graph G, and answer keyword search queries with subtrees of this graph. Each G node has a (globally unique) identi er, some of which are shown in Figure 1, in blue. A label function λ assigns to every node a (possibly empty) text label, re ecting its content in D. G edges are directed; each edge e carries a text label (we use λ also to denote the edge labeling function) and a con dence c e which is between 0 and 1, whose usage will be shortly explained below. Node and edge labels appear as quoted strings in Figure 1. Based on this gure, we explain below how G nodes and edges are (conceptually!) derived from each data source D.

Nodes and edges derived from a source

First, a dataset node n D is created in G to represent D itself, e.g., nDS1 to nDS3. Every G node corresponding to data from D (see below) is connected to n D , through an edge labeled origDS (standing for originating data source); we show such edges with dotted lines in the gure. They ensure that any two virtual graph nodes coming from the same data source D are connected at least through n D .

The other nodes and edges of G are de ned as follows: For instance, if S.spouse is a foreign key on T .id, then G comprises n s spouse ------→ n t . This graph view of a relational database is often used for keyword search, e.g. [START_REF] Hristidis | DISCOVER: keyword search in relational databases[END_REF][START_REF] Yu | E ective keyword-based selection of relational databases[END_REF].

( ) Any G node whose label λ(n) is longer than a threshold θ text is treated like a text data source, i.e., entity and relationship occurrences are extracted as in (iii); however, the G nodes created from these occurrences are all descendants of n, and their original data source is that of n. This both provides a uniform treatment of text regardless of where it appears, and records the origin of each text content.

All virtual G edges described above have a con dence of 1.0. Some extractors provide a con dence value for the extraction: this can be attached to the edges between the nodes, e.g. nDS1.V1, and their types, e.g. OC:Person.

SameAs edges

We add to G an edge labeled sameAs between nodes n 1 ,n 2 (from the same or from di erent data sources), as soon as they are similar beyond a certain threshold θ sim ; the con dence of such an edge is the similarity score, normalized to [0, 1]. Identifying when two data objects represent the same thing (aka approximate duplicate detection, etc.) is a thoroughly studied data integration problem [START_REF] Doan | Principles of Data Integration[END_REF], for which many solutions exist, especially among similarly-structured objects, e.g. [START_REF] Herschel | Scalable iterative graph duplicate detection[END_REF][START_REF] Zhu | Auto-join: Joining tables by leveraging transformations[END_REF]. C L interconnects disparate sources that may not even have the same data model, e.g., an extracted entity may match with a JSON value (e.g., "Anne Martin"), or with an attribute in a relational tuple, e.g. "Philippe Varin" and "P. Varin"; or, an interesting link may exist between, say, a person and a company, if they both mention a certain user in their tweets etc. We seek to nd all such value connections, and distinguish the trivial from the interesting ones using a score (Section 3). C L uses the labels λ(n 1 ),λ(n 2 ) of two G nodes to decide if they are the same. If the labels are shorter than a certain size limit L, the Jaro distance between them is compared with θ sim . If λ(n 1 ) or λ(n 2 ) are longer than L, we turn them both into bags of words and then compute their set-based Jaccard distance. If λ(n 1 ),λ(n 2 ) are identical URIs, we connect them through sameAs with a con dence of 1.0. Di erent distance functions or comparisons can be plugged in as users get familiar with data sources, in pay-as-you-go data integration fashion [START_REF] Sarma | Bootstrapping pay-as-you-go data integration systems[END_REF].

We do not build sameAs links based on nodes' adjacent edges and neighbor nodes. However, sameAs links based on labels alone, e.g., a common last name of two people, su ce to establish the data connections we are interested in.

Virtual graph indexing

We encode, index and store the G nodes and edges derived from a source D in a set of data structures, as follows. 1. We compute an ID id D for the dataset. 2. We derive nodes and edges as explained above from D. For each node n, we compute an ID id n pre xed with id D . This de facto encodes the edge n D or i D ------→ n into id n . 3. We compute λ(n) from the original text content of n, through stop word and punctuation removal, and stemming. 4. For each word w ∈ λ(n), we insert (w,id n ) in the index I(word, node). Similarly, λ(e) is computed for each edge e and its words indexed in I. 5. We seek to nd sameAs edges to which a node n derived from D may participate. We look up all the nodes n whose labels share at least a word with n, that is, (w,id n ) and (w,id n ) belong to I, and compute the similarity between λ(n) and λ(n ). All sameAs edges are stored in a bridge table B(id 1 ,id 2 ,c) which records that the nodes identi ed by n 1 ,n 2 are judged the same with con dence c. For instance, the B(nDS1.V1, nDS3.V2, 0.76) tuple encodes the sameAs link between the two nodes corresponding to Philippe Varin in Figure 1.

ANSWERING KEYWORD QUERIES

Given a query Q = {w 1 , . . . ,w n }, the problem of nding the k answer trees (recall their de nition in Section 1; we call them ATs, in short) with the best score is known to be NP-hard, by reduction to the Steiner tree problem. Thus, C L adopts a heuristic method to enumerate answer trees and returns the k highest-score ones.

The AT enumeration algorithm starts by looking up in the index I for the potentially interesting data sources for Q, denoted P (Q ), that is: the data sources from which G nodes matching some query keywords are derived. For each subquery Q of Q, and each source D ∈ P (Q ) containing exactly the keyword set Q , the procedure localSearch(D,Q ) returns the ATs (if any) whose nodes and edges derive only from D. The implementation of localSearch(D,Q) depends on the nature of D: it follows the lines of [START_REF] Hristidis | DISCOVER: keyword search in relational databases[END_REF] for relational sources, [START_REF] Chen | Supporting top-k keyword search in XML databases[END_REF] for JSON, and [START_REF] Le | Scalable keyword search on large RDF data[END_REF] for RDF data. The algorithm starts by optimistically asking each D ∈ P (Q ) for ATs for the largest subset of Q for which D has matches. If D has only one connected component, it is sure to contain at one such AT; also note that ATs are undirected, that is, G edges form an AT as soon as they share a node, regardless of the direction of the edges. For instance, an

AT may consist of three G edges n 1 a ← -n 2 b -→ n 3 c ← -n 4 .
This is because we are interested in data connections, and nd an edge such as n 1 a -→ n 2 interesting as a link, in both directions. Partial ATs (initially, those local to each data source) are inserted in a priority queue U, based on their score (discussed below). The algorithm greedily picks the top-score AT t from U, and adds it to the result set if it is an answer to Q and its score is among the k best so far. If t is just a partial answer, we try to nd another partial tree t to combine with t, through a sameAs edge between a node from t and one from t . The AT t resulting from t and t is again added to U and the process continues, until a time-out occurs or there are no ATs left to add in U. If answers to Q are not found after a certain time, localSearch calls are made with smaller Q subqueries, in the hope that the resulting ATs may participate to more sameAs edges allowing to combine them in answers to Q.

The score s (t ) of an AT t comprises for each w i ∈ Q, a matching score ms (t,w i ) re ecting the extent to which the labels of all t nodes and edges match w i , as well as a structure score ξ (t ) depending on the AT structure. We quickly found that unlike many previous keyword search works, small ATs are not always preferable because they may bring trivial (uninteresting) information. For instance, any French representative may be connected to any French company through a node labeled "France" (where they reside). Instead, for our journalistic applications, we favor a speci city metric, computed as follows. The edge n 1 a -→ n 2 is speci c in an AT, if n 1 has few outgoing a edges, and n 2 has few incoming a edges. Then, ξ (t ) is a weighted sum of: the average speci city of its edges; and the product of its edge con dences. We consider s (t 1 ) > s (t 2 ) if t 1 has non-zero ms scores for strictly more keywords than t 2 ; otherwise, a weighted combination of the ms and ξ values for both trees is used to decide who has the highest score.

IMPLEMENTATION AND SCENARIOS

C L is developed in Java; currently, we take advantage of Postgres' ability to store relational as well as JSON data, and also capitalize on prior work of ours [START_REF] Bursztyn | Optimizing Reformulation-based Query Answering in RDF[END_REF] storing and querying RDF within Postgres. We implemented localSearch for relational, JSON, (annotated) text and RDF data, with the rest of the algorithm from Section 3. We plan to show two scenarios.

Scenario 1 is based on (i) JSON data from the Regards Citoyens NGO that publishes all the French parliament activity, including all National Assembly representative (800KB); (ii) wikidata information about people, for example their past jobs (1.55GB JSON le); (iii) a text dump of recent articles published in main French media; and (i ) a text dump of Journal O ciel, France's o cial public law repository. Starting from political parties and companies, e.g. {"En Marche", "Areva"} from our motivating example, we will search for connections between them using the information available from the underlying data sources.

Scenario 2 Here, we aim at identifying political leaders who spread journalistic hoaxes. We will use: (i) a DBPedia RDF graph of French political leaders; (ii) a JSON document containing tweets from French political leaders; (iii) a known JSON database of hoaxes, provided by Le Monde. Given queries such as {"Front National", "Macron", "hoaxes"}, we will investigate connections between pairs of political leaders through hoaxes, e.g., who disseminates on the social media hoaxes about the other one.

Figure 2 shows a screenshot of the possible answers to the initial keyword-based query.

RELATED WORK

Our work is related to several areas of prior research. Large knowledge bases such as DBPedia or Yago are currently available, however, little-known people, events etc. are not covered there, yet they are described in details in other speci c and/or local data sources. C L is meant exactly to enable using together data sources of various nature, model, format etc. Exploiting together large data source collection is the goal of data integration systems, in particular dataspaces [START_REF] Franklin | From databases to dataspaces: A new abstraction for information management[END_REF] or more recently data lakes [START_REF] Deng | The data civilizer system[END_REF]; our work can be seen as keyword search for dataspaces, following [START_REF] Dong | Indexing dataspaces[END_REF]. However, we seek connections across distinct data sources, whereas they only nd local answers. Recent works focused on nding how to join large tables sets, e.g., [START_REF] Deng | Silkmoth: An e cient method for nding related sets with maximum matching constraints[END_REF]; we (also) work with semi-structured or text data, and identify connections among items from di erent sources. Keyword search is well-studied in text databases; more recently, algorithms have been proposed to answer keyword queries in structured databases including relational [START_REF] Oliveira | Ranking candidate networks of relations to improve keyword search over relational databases[END_REF][START_REF] Li | EASE: An e ective 3-in-1 keyword search method for unstructured, semi-structured and structured data[END_REF], XML [START_REF] Chen | Supporting top-k keyword search in XML databases[END_REF] and RDF [START_REF] Elbassuoni | Keyword search over RDF graphs[END_REF] ones; in all these works, each keyword query answer is local to one data source.

An interesting e ort with a goal quite similar to ours has been made by ICIJ developers working on the Panama Papers project (https://neo4j.com/blog/icij-neo4j-unravel-panama-papers/). This report describes the systems they used, specify that they have built a relational database and then exported it in Neo4J, but does not discuss if and whether links across data sources were established.

Figure 1 :

 1 Figure 1: Motivating example: data source collection D, corresponding virtual graph G, and an answer tree (red).

  (i) If D is an RDF graph, then G contains all its nodes and edges of D; λ attaches to each node its URI or literal (constant) label in D. The property labeling every edge becomes an edge label in G.(ii) If D is a JSON document such as DS 1 , G has a node for each constant, list and map occurring in D, and an edge labeled origDS connects the node representing D, to the one corresponding to the top list or map in D. For each (n 1 , 1 ) name-value pair in a map, n 1 becomes the label of the edge leading to the node corresponding to 1 .(iii) If D is a text such as DS 2 , we apply entity and relationship extraction to identify in D occurrences of entities (such as people, places, organizations etc.) and of relationships (such as bornIn, worksFor etc.) Any o -the-shelf extractor (or set of extractors) can be used; C L currently uses OpenCalais (http://www.opencalais.com) for entity extraction. A G node is created to each extracted entity (resp. relationship) occurrence; its λ label is the exact text snippet identi ed by the extractor; it has a type edge pointing to the entity type identi ed by the extractor, e.g., OC:Person stands for OpenCalais' Person type URI, and child nodes containing the o set and length of its appearance in the original text (omitted in Figure1to avoid clutter). Further, each node corresponding to an occurrence of a relationship between two entities, is connected to the nodes corresponding to the respective entity occurrences by edges identifying the entity roles in the relationship.(i ) If D is a relational database such as DS 3 , for each relation R(a 1 ,a 2 , . . .) ∈ D and each tuple r ∈ R, G contains a node n r , with outgoing edges labeled a 1 ,a 2 etc. toward G nodes, labeled with the values of the respective attributes of r . The label of n r is one of its primary keys (we add such a primary key attribute if R doesn't have one). Further, for any two relations S,T ∈ D such that S.a is a foreign key corresponding to the primary key T .b, and tuples s ∈ S,t ∈ T such that s.a = t .b, G comprises an edge n s a -→ n t .

Figure 2 :

 2 Figure 2: Sample view of the keyword query interface.

ACKNOWLEDGEMENTS

This project is supported by: (1) The ANR-15-CE23-0025 project ContentCheck; (2) the JSPS Bilateral Projects/Inria Associate Teams Program (WebClaimExplain team); J. Leblay is also supported by the New Energy and Industrial Technology Development Organization (NEDO) and a KAKENHI grant (17K12786); (3) Fundação para a Ciência e a Tecnologia, Portugal, with reference UID/CEC/50021/2013.


Jeanne Villaneau
email: jeanne.villaneau@univ-ubs.fr

Stefania Pecore
email: stefania.pecore@univ-ubs.fr

Farida Saïd
email: farida.said@univ-ubs.fr

Aspect Detection in Book Reviews: Experimentations

Keywords: Aspect Based Sentiment Analysis, aspect detection, opinion mining

Aspect Based Sentiment Analysis (ABSA) aims at identifying the aspects of entities and the sentiment expressed towards each aspect. Substantial work already exists in English language and in domains where aspects are easy to define such as restaurants, hotels, laptops, etc. This paper investigates detection of aspects in French language and in the books reviews domain where expression is more complex and aspects are less easy to characterize. On the basis of a corpus that we annotated, 21 aspects were defined and categorized into eight main classes including a catch-all class, General, which was found to be absorbent. Several methods were carried out to address this difficulty, with varying efficiency: Random Forest and SVM provided better results than kNN and Neural Net. Combining these methods with voting rules helped to improve noticeably the results. On another side, the difficulty of the task and the limits of a lexical approach were further explored with a qualitative analysis of errors and a topological mapping of the data using Self Organising Maps.

Introduction

Aspect Based Sentiment Analysis (ABSA) systems aim at detecting the main aspects (features) of an entity which are discussed in texts and at estimating the orientation of the sentiment expressed per aspect (how positive or negative the opinions are on each aspect) [START_REF] Liu | Sentiment Analysis and Opinion Mining[END_REF]. ABSA was first introduced as a shared task in SemEval-2014 [START_REF] Pontiki | Semeval-2014 task 4: Aspect based sentiment analysis[END_REF], with datasets in English in two domains: laptops and restaurants. The task was repeated in SemEval-2015 and SemEval-2016, and extended to new entities (hotel, restaurant, telecom, consumer electronics) and to other languages (French, Dutch, Russian, Spanish and Turkish) [START_REF] Pontiki | Semeval-2016 task 5: Aspect based sentiment analysis[END_REF].

ABSA is classically split into three subtasks: (i) extracting opinion expressions, (ii) determining the aspect of these expressions and (iii) determining their opinion value [START_REF] De Clercq | Towards an integrated pipeline for aspect-based sentiment analysis in various domains[END_REF]. In SemEval 2016, determining the aspects was the subtask of ABSA (task 5) which called the largest number of contributions (216 over 245 submissions in total). As an example, French data sets were proposed in restaurant domain with 6 types of entities and 6 types of attributes [START_REF] Apidianaki | Datasets for aspect-based sentiment analysis in french[END_REF]. On these data, the best system obtained a F 1 score of 0.612.

Despite challenges as SemEval, few studies were conducted in languages other than English and freely available data are scarse. We were interested in this work in investigating this task in French language and in a domain where aspects are more difficult to detect and where opinion is expressed in complex and varied forms. This paper presents a book reviews corpus which we collected and the work carried out to define aspects (Section 2) and to implement their automatic detection by using lexical statistical methods (Sections 3). It was found that these methods perform varyingly well and their performances can be improved when they are combined. Moreover, an analysis of the errors gives an idea of the difficulty of the task and the limits we have to go beyond to improve the results (Section 4).

2 Training and Test corpora -Task and Approach

Training Corpus and Annotation

We built a corpus of 900 reviews by concatenation of 450 book reviews from the French Sentiment Corpus (FSC), which was produced between 2009 and 2013 by [START_REF] Vincent | Construction et exploitation d'un corpus français pour l'analyse de sentiment[END_REF], and 450 more recent book reviews which we collected from the Amazon.fr website between 2016 and 2017 (NC).

The total number of words in the corpus is about 72,000 words.

We proposed an annotation schema suitable for all types of books, regardless of genre, which is based on 5 aspects and 20 attributes (see The complexity of the wording in book reviews makes difficult the task of allocating a unique aspect to an entity as usually done, for example in SemEval 2016 annotation task. The following examples, yet very simple, illustrate how entities, opinion phrases and context have to be taken into account to determine proper aspects.

-In the phrase "le livre est bien mal écrit" [the book is very badly written],

the part which expresses sentiment is "bien mal écrit" (very badly written) (value: -2) and the entity is le livre [the book]. The appropriate aspect is Text with Style for attribute, because of the verb écrire [to write]. -In the review, "la bobo au style frelaté" [the boho with degenerated style],

the word degenerated expresses a very negative opinion (-2). It relates to the entity Style and it is classified in Text#Style. Because of the reference to the style, one can say that bobo refers to the author; "la bobo" represents both the entity and the opinion of the reviewer.

Since it often happens that entity and aspect do not coincide, it is essential to include an aspect detection phase in the annotation process. For that, we proceed in three steps:

selection of a group of contiguous words which indicate an opinion (evaluated by an ordinal value), detection of the entity to which the opinion refers (when it is expressed), selection of an aspect and an attribute in the annotation schema.

The annotation task concerned about 4700 phrases related to 3300 opinion expressions. More information on the corpus (statistics, annotators, inter-annotators agreement) is given in [START_REF] Pecore | Complex and precise movie and book annotations in french language for aspect based sentiment analysis[END_REF].

Task, Test Corpus and Approach

Aspects were grouped into eight main classes because of the difficulty met by the annotators to separate certain aspects. More precisely, the following pairs of aspects were aggregated: General with Text#General, Text#Readability with Text#Style and Text#Interest with Text#Subject. The other considered aspects are Text#Pace-Narration, Text#Characters, Illustrations, Form and Authors regardless of attributes for the latter. Table 2 displays the relative importance of these classes in the training corpus. The large prevalence of the class General and the very limited size of the class Illustrations are to be mentioned.

The test corpus consists of 340 sentences or parts of text selected from the non-annotated part of the FSC corpus. The sentences were selected so as to present a unique aspect each and to cover all aspect classes, thereby reducing the prevalence of the class "General ". The resulting distribution of the aspect classes is given in the last column of Table 2.

As mentioned above, sentences presenting more than one aspect were removed during the selection process, as in:

"Tant dans le contenu que dans l'écriture je n'ai pu trouver aucun intérêt à cet ouvrage" [Both in the contents and in the writing I was not able to find any interest in this work.] Furthermore, it is whole sentences or their largest possible parts which were selected, as in: "Tout sonne faux, les relations entre les protagonistes, les dialogues qui semblent sortis de la bouche de mauvais acteurs, la psychologie des personnages." [Everything rings false, the relations between the protagonists, the dialogues which seem come out of the mouth of bad actors, the psychology of the characters.]

It should be noticed that some words which could seem to be key words in the determination of the target (Aspect#Attribute), can turn out to be false friends as in the previous sentence where the word personnage [character] can lead to misclassify the sentence in Characters while a human annotator would classify it in Interest.

Detecting opinion polarity meets several difficulties among which negation, use of humoristic or indirect expression, etc. On the other hand, the success of statistical methods based on simple bag of words (BoW) supports the hypothesis that determining aspects is essentially a lexical task. We investigated the efficiency of this approach on the corpus of book reviews.

Following lemmatization (with Treetagger), a list of lemmas (names, adjectives, verbs and adverbs excepting stop words) was selected according to their frequency in the corpus (i). Each annotaded expression in the training corpus is handled as a vector whose binary entries (0 or 1) code the co-occurrences of the expression with the lemmas (ii). A co-occurence matrix is built and then augmented with a column which specifies the aspect#attribute assigned to every annotated expression (iii).

Our attempts to enrich the model with linguistic parameters were not conclusive and the performances achieved were low below the results presented in the next section. Anyhow, the best results were obtained using lemmas rather than forms, possibly because of the modest size of our corpus.

Experiments and Results

Various experimentations were conducted using unsupervised and supervised classification approaches, namely SOM (Self-Organising Maps), kNN (k-Nearest Neighbours), NN (Neural Net), RF (Random Forest), SVM (Support Vector Machine). Linguistic contexts of words were taken into account through the use of Word2vec. The well known language and environment for statistical computing, R, was used all along this work.

The results of our experimentations are presented below and they reflect well the difficulty of the task. In all tables, aspect classes are identified by the abbreviations given in Table 2.

SOM

Self Organising Maps is a competitive learning network based on unsupervised learning. It provides a low dimension representation of the input data and it serves for representation as well as for clustering. We used in our experimentations the kohonen R-package.

The topological map in Figure 1 was obtained by combining the observationlemma matrix (weight of 5) with the vector of related aspect classes (weight of 1). Figure 1 shows the extension of the aspect General (black) on more than half of the topological map. Aspects Characters (green), Style (chocolate) and Form (grey) appear well grouped, but to a lesser degree for the last two classes. In contrast, aspects Illustrations (orange) and Authors (yellow) are more heterogeneous with a sub-class closely grouped and a subset of data highly dispersed and mixed with other classes.

These observations have to be put into perspective with the outcomes per class shown in Table 8. If SOM gives interesting representations of the data, attempts to use it as a predictive tool (with supersom from kohonen R-package) gave poor results in terms of precision and recall.

k-Nearest Neighbors (kNN) -Neural Networks -Fuzzy classification

The best results with kNN are displayed in Table 3; they were obtained for k = 2. These performances are disappointing and reflect the difficulties encountered, especially the absorbtion capacity of the class General, the only class showing a precision score lower than the recall. As predicted by SOM, Characters is the class that obtains the best results.

Ge Pa Ch St Au In Il Fo Ge 72 0 1 7 1 7 0 3 Pa 28 15 1 1 1 6 0 1 Ch 7 0 10 1 0 2 0 0 St 30 1 0 27 0 3 0 3 Au 14 1 0 3 2 0 0 0 In 32 3 1 3 0 14 0 0 Il 13 0 0 0 0 4 1 0 Fo 13 0 0 2 0 2 0 4 Fuzzy logic and Neural Networks already proved to be efficient in Sentiment analysis [START_REF] Afzaal | Fuzzy aspect based opinion classification system for mining tourist reviews[END_REF][START_REF] Khalil | Niletmrg at semeval-2016 task 5: Deep convolutional neural networks for aspect category and sentiment extraction[END_REF]. However, they provided very poor results when implemented on our data (R-package frbs and neuralnet), with almost all expressions classified in the class General.

Random Forest

The statistical approach using Random Forest (ntree = 500) gives encouraging results. The class General is still absorbent but all classes have their precision and recall scores greatly improved. In accordance with SOM (Figure 1), class Characters performs well. The results of class Author remain mediocre and those of class Form are poor, while the recall of class Illustrations is very low.

While names can be sufficient for the determination of aspects in certain domains, four parts of Speech are highlighted in our experimentation. Indeed, the top twenty words in Random Forest consist in 9 names, 5 adjectives, 5 verbs and 1 adverb and among them, the adjectives interesting, clear and likeable which are respectively associated with classes Interest, Style/Readibility, Characters and the adverb facilement [easily] which is associated with the class Style/Readibility.

SVM

In the field of ABSA, SVM classifiers made their proof for both aspect and polarity detection [START_REF] Kiritchenko | Nrc-canada-2014: Detecting aspects and sentiment in customer reviews[END_REF][START_REF] Wagner | Dcu: Aspect-based polarity classification for semeval task 4[END_REF]. The classic approach by SVM with linear kernel outclasses Ge Pa Ch St Au In Il Fo Ge 77 3 1 6 0 3 0 1 Pa 16 31 1 0 0 4 0 1 Ch 0 0 20 0 0 0 0 0 St 15 1 0 42 3 3 0 0 Au 9 0 0 3 8 0 0 0 In 20 1 1 3 0 27 0 1 Il 12 0 0 0 0 0 6 0 Fo 14 2 0 1 0 0 0 4 Random Forests, however the improvement is not general: classes Pace, Characters, Interest obtain poorer results. By contrast, the improvement of the results of the class Form is particularly remarkable. Besides, we still observe the trend to an overuse of the class General.

Ge Pa Ch St Au In Il Fo Ge 75 3 2 6 1 2 0 2 Pa 20 27 1 2 2 1 0 0 Ch 0 0 19 0 1 0 0 0 St 13 1 0 45 3 2 0 0 Au 9 0 0 2 7 2 0 0 In 25 1 0 1 2 24 0 1 Il 8 0 0 1 0 1 8 0 Fo 5 2 0 0 0 0 0 14

Class

SVM+Word2Vec (SVMW2V)

Many of the words used in the test corpus do not appear in the training corpus because of its small size. To deal with this difficulty, the last approach makes use of Word2Vec to enrich the space of words in the test corpus. Word2Vec was trained with the corpora FSC, NC and Wikipedia.

The training corpus remains unchanged and only the co-occurrence matrix is modified: the entry in the co-occurence matrix of every name, adjective, verb or adverb that does not appear in the training corpus, is replaced by its similarity score with its closest lemma in the training corpus.

The results were globally below expectations (except for class General) (cf. Table 6); a reason for that could be that the noise brought by W2V limited the global gain.

Combining approaches

The last three approaches (Random Forest, SVM, SVMW2V) obtain encouraging results and their performances are globally close. We combined them by adopting a majority vote with a special handling of the class General. The voting rules are presented in Table 7 The second rule states that if at least one system out of three chooses a class other than General, this class is favoured. The underlying purpose of this rule is to reduce the absorbing bias of class General which was observed in all single systems.

The third rule specifies that in case of total disagreement between the three systems, class General is chosen. This rule aims to avoid a random draw when there is no well defined class.

Results

Choice (1) Three equal results:

r1 = r2 = r3 = C C (2) Two equal results (example: r1 = r2 = C, r3 = C , C = C) if C =General C if C =General C (3) Three distinct results

General

Table 7. The rules of choice.

The outcomes of the combined system are given in Table 8. In global or on average across all classes, we notice that combining the three approaches leads to a slight reduction in the precision compared with SVM, which is widely compensated with an increase in the recall.

Table 9 gives the results of the combined system by class. Before combination, Random Forest outperformed the 2 other systems in 4 of the 8 classes, SVM in 3 classes and SVMW2V in the class General. Random Forest outperforms the combined system in 3 classes and SVM in the class Form. Seen from this perspective, no system outclasses totally the others.

Ge Pa Ch St Au In Il Fo Ge 71 3 2 6 1 5 0 3 Pa 13 31 1 2 2 3 0 1 Ch 0 0 19 0 1 0 0 0 St 7 2 0 49 3 3 0 0 Au 6 0 0 2 9 3 0 0 In 14 2 0 2 1 31 0 1 Il 6 0 0 1 0 1 10 0 Fo 5 2 0 0 0 0 0 14 4 BoW approach: efficiency and limits

In 78 out of 340 tests, none of the statistical systems selected the same aspect as the human annotators. A qualitative analysis of the disagreements allows to go deeper in the understanding of the limits of the lexical approach. Disagreements can be classified into three classes:

1. There are 19 "false errors" for which the human annotation may be questioned. For example : (a) "C'est drôle et enlevé, puissant et sensuel?: un chef-d'oeuvre de vie, dédié à la vie d'une ville incomparable." [It is funny and spirited, powerful and sensual?: a masterpiece of life, dedicated to the life of an incomparable city.] This test is classified as General by human annotators and as Style by the three statistical systems. Both choices are justified: the first choice is understandable if we consider the whole sentence and the second choice is essentially motivated by the first part of the sentence. This example shows the limits of a strict classification since classes are not necessarily mutually exclusive.

(b) "Attention: livre impossible à lâcher avant la dernière page?!" [Attention: book impossible to put down before the last page?!] BoW systems classified the sentence as Pace (SVMW2V) or as Interest (SVM and RF), while human annotators chose the class General, possibly because the choice is unclear. The significant number of false errors points out the difficulty of the task in the field of book reviews and the fuzzy outlines between the defined classes. One would hope that using Word2Vec would allow to go beyond the limits of training corpus' vocabulary by extending it. However, in our experiments, the noise introduced by the similarity scores negated the expected improvement. 3. Lastly, the vast majority of errors is related to the limits of BoW approach.

Firstly, representing a sentence as a bag of lemmas is very simplistic; on the other hand, the understanding of the reader uses contexts of various types: temporal, cultural, pragmatic, textual, of common sense, etc. [START_REF] Benamara | Evaluative language beyond bags of words: Linguistic insights and computational applications[END_REF]. (a) "il manque l'essentiel, les bonnes adresses, les accès, les plages, bref, aucun détail, c'est un TOP 10 sans le moindre intérêt." [The main part, the good addresses, the accesses, the beaches, in brief, no detail is missing, it is a PIP 10 without the slightest interest. ] Here, the aspect is expressed in the word "Interest" and yet, the test is classified by the systems as General, probably because the word is buried in many others, as "essential". It can be assumed that linguistic context, especially the adverb bref [in short] which introduces a conclusion, could make it possible to give more importance to this keyword. (b) "Je n'ai pas accroché à l'histoire, il convient sûrement à toutes les petites et jeunes filles ans des poupées, mais la trame est cousue de fils blancs Both tests express negative sentiments with a certain sense of humour (irony or sarcasm), which is a real challenge for automatic systems. For instance, a specific session of SemEval was devoted to sarcastic tweets [START_REF] Nakov | Developing a successful semeval task in sentiment analysis of twitter and other social media texts[END_REF] and numerous works addressed this topic (see for example [START_REF] Benamara | Evaluative language beyond bags of words: Linguistic insights and computational applications[END_REF]) .

Actual mistakes point clearly toward the need to take into account multiple contexts and knowledges to improve systems, as emphasized by Benamara and Co [2017]. Within our study, the most relevant aspects relate to the choice of wording and its structure, and to take into account the linguistic context of the words : expressions varyingly litteral ("cousu de fil blanc" [blindingly obvious]), linkage of adverbs and qualificatives... not to mention the detection of irony, a full study program in itself.

Conclusion

In a complex field where aspects are sometimes hard to sort out, even for a human annotator, a simple SVM approach with a linear kernel on words (lemmas in this instance) is, despite its lackings, relatively efficient. Regardless, the combination with other statistical approaches, especially with Random Forest, noticeably improves the attained results. Furthermore, an intake in lexical resources, like the list of authors, could help to better circumvent some classes.

Despite this, the analysis of errors brings to light the limits of the BoW approach. An improvement of the results inevitably requires a better analysis of contexts with the problems that come with the use of a language all-in-all lacking in normalization on one hand and, on the other hand in French language which proves much poorer in resources than English language.

At present, our research concerns polarity determination. Besides BoW approaches, we also take into account the linguistic context by implementing a surface analysis. First results seem to evidence that the use of linguistic parameters can allow to outclass widely a simple BoW approach in this task.

Fig. 1 .

 1 Fig. 1. SOM obtained by combining lexical data and classification into aspects in the proportions ( 5 6 , 1 6 ).

Table

Table 1 .

 1 ). The 21 resulting classes can be gathered into metaclasses to meet different needs. Aspects and Attributes for book reviews

	Aspects	Attributes
	General Feeling
	Text	General, Subject, Style, Characters, Pace/Narration, Readability,
		Translation/Adaptation,Interest/Accuracy
	Illustration	General, Interest/Accuracy, Graphic quality
	Author	General, Text Author, Translator, Illustration Author,
	Form	General, Bookbinding, Typography, Inner structure, Distribution

Table 2 .

 2 Aspect classes and their distribution in both training and test corpora.

	Class	Aspect/Attribute	% Training Test (Nb and %)
	General (Ge)	{General Feel. -Text#General}	44.9%	91 27%
	Pace (Pa)	Text#Pace-Narration	11.5%	64 19%
	Interest (In)	Text#{Interest-Accur., Subject}	21.0%	53 16%
	Characters (Ch) Text#Characters	8.5%	53 16%
	Style (St)	Text#{Style, Readibility}	3.2%	20 5.5%
	Authors (Au)	Author#{all attributes}	4.5%	20 5.5%
	Illustrations (Il) Illustration#{all attributes}	0.7%	18 5%
	Form (Fo)	Form #{all attributes}	5.7%	21 6%

Table 3 .

 3 Best result with kNN (for k = 2).

	Class	Precision Recall F-measure
	General	0.344	0.791 0.48
	Pace	0.41	0.75 0.411
	Characters 0.769	0.5	0.606
	Style	0.614	0.422 0.5
	Authors	0.5	0.1	0.167
	Interest	0.368	0.264 0.308
	Illustrations 1	0.056 0.105
	Form	0.363	0.190 0.25

Table 4 .

 4 Results with Random Forest (ntree = 500).

	Class	Precision Recall F-measure
	General	0.472	0.846 0.606
	Pace	0.816	0.585 0.681
	Characters 0.870	1	0.930
	Style	0.764	0.656 0.706
	Authors	0.727	0.4	0.516
	Interest	0.730	0.509 0.6
	Illustrations 1	0.333 0.5
	Form	0.571	0.190 0.286

Table 5 .

 5 SVM Results (linear kernel).

		Precision Recall F-measure
	General	0.484	0.824 0.610
	Pace	0.794	0.509 0.621
	Characters 0.864	0.95 0.905
	Style	0.789	0.703 0.744
	Authors	0.467	0.35 0.4
	Interest	0.75	0.453 0.565
	Illustrations 1	0.444 0.615
	Form	0.824	0.667 0.737

Table 6 .

 6 SVMW2V Results.

	Class	Precision Recall F-measure
	General	0.493	0.813 0.614
	Pace	0.703	0.491 0.578
	Characters 0.810	0.85 0.829
	Style	0.827	0.672 0.741
	Authors	0.389	0.35 0.368
	Interest	0.647	0.431 0.518
	Illustrations 1	0.444 0.615
	Form	0.722	0.619 0.667

Table 8 .

 8 Results of the combined system.

	Class	Precision Recall F-measure
	General	0.582	0.780 0.667
	Pace	0.775	0.585 0.667
	Characters 0.864	0.95 0.905
	Style	0.790	0.766 0.778
	Authors	0.529	0.45 0.486
	Interest	0.674	0.608 0.639
	Illustrations 1	0.556 0.714
	Form	0.737	0.667 0.7

Table 9 .

 9 F1-measure per aspect and system (left table) and Precision, Recall and F1-measure per system (right table). Final corresponds to the combined system.

  2. Another group of errors (about 12) can be related to training bias due to new words appearing in the tests. For example: (a) "Pouchkine est un écrivain au style sûr, simple et envoûtant" [Pushkin is a writer with a sure, simple and mesmerizing style] This sentence should be classified as Author since it expresses a general opinion on Puchkin's style. However, all systems classified it as Style because "Puchkin" did not appear in the training corpus. It is likely that a list of authors' names would improve the results of the class Author. (b) "Un très joli livre, avec de très belles peintures chinoises à l'intérieur." [A very attractive book, with very beautiful Chinese paintings inside.] This sentence related to Illustrations is miclassified as General by the systems. This error can be explained by the low occurrence of the keyword "peinture" [painting] in the training corpus.

  3 [I did not stick to the story, it is certainly advisable to all the girls and the girls the years of dolls, but the framework is a blatant lie...] The test is classified as General (instead of Pace) by all the systems despite the keyword "histoire" ["story"]. The word trame [framework], almost synonymic but much less common, was probably not taken into account by the systems, including W2V. (c) "le livre reste un catalogue d'interprétations déjà connues." [ the book remains a catalog of already known interpretations ] BoW systems classify this test in General instead of Interest. The adverb "déjà" plays a key role to show the lack of interest of the book. (d) "L'auteur abuse de mots aussi savants qu'inutiles qui détournent du sujet traité?; un défaut difficilement pardonnable." [The author makes excessive use of words as fancy as they are useless which divert from the handled subject?; a fault hardly overlooked ] The word "auteur" makes the test classified in class Authors, while the sentence relates to the style of the book and not to its author in general. (e) "Sauter de la page 288 à la 337 n'aide pas du tout à apprécier un roman, notamment si celui-ci doit tre le dernier d'une série." [Jumping from page 288 to 337 does not help at all to appreciate a novel, especially if it is the last of a series... ] "l'auteur oublie ici et là des mots qui AIDE à comprendre les phrase." [the author forgets here and there words which help to understand the sentences]

In French language, there is a play on words between trame, which means "weft" or "framework" depending on the context, and phrase cousue de fils blancs litteraly "sewn of white threads".