Sébastien Ferré
email: ferre@irisa.fr

What's New in SPARKLIS

Sparklis is a SPARQL query builder that can connect to any endpoint, and that interacts with users in natural language only. Users are guided in the building of their queries so that they do not have to know the schema, and so that empty results are almost completely avoided. This demo paper presents a number of recent extensions to Sparklis. Most notably, it now supports analytical queries, Wikidata statement qualifiers, and the display of results on a map or as a slideshow.

Introduction

Sparklis1 [START_REF] Ferré | Sparklis: An expressive query builder for SPARQL endpoints with guidance in natural language[END_REF] addresses the key problem of semantic search, striving to reconcile the expressivity of SPARQL, and the usability of point-and-click user interfaces. It succeeds in doing so by combining principles and techniques from query builders, faceted search, and natural language interfaces. It supports exploratory search [START_REF] Marchionini | Exploratory search: from finding to understanding[END_REF] by guiding users in the incremental construction of queries, avoiding empty results, and showing both queries and suggestions in natural language only. Complex queries can be built safely because intermediate results and relevant suggestions are shown at all steps.

Sparklis is based on theoretical work that started in 2010 [START_REF] Ferré | Conceptual navigation in RDF graphs with SPARQL-like queries[END_REF], and was first released online in 2014. Since then, it receives in the range of 200-2000 hits per month, and was officially adopted as the main search tool by two French institutions who had developed SPARQL endpoints for their data: Persée2 and INIST 3 . A startup, Askelys, has even been recently created to develop customized UI and other services on top of Sparklis.

The purpose of this demonstration paper is to shortly present the main features that have been added to Sparklis since the last publication in 2016 [START_REF] Ferré | Sparklis: An expressive query builder for SPARQL endpoints with guidance in natural language[END_REF]. They improve it by significantly increasing the coverage of SPARQL, by hiding some complex structures in RDF data, and by improving user experience.

Coverage of SPARQL

In 2016, Sparklis covered basic graph patterns (including cycles), UNION, OP-TIONAL, NOT EXISTS, basic filters, basic aggregations, ORDER BY, SELECT and ASK queries. The newly covered constructs are those related to expressions (FILTER, BIND, GROUP BY, aggregators) and named graphs (FROM, GRAPH). Apart from CONSTRUCT and DESCRIBE queries and updates, which are not yet covered, only a few features are still missing: transitive closures in property paths (partly covered by hierarchies of terms, see next section), SERVICE for federated queries, explicit LIMIT and OFFSET, and separator specification in GROUP CONCAT. The only serious limit in expressivity is that graph patterns must be connected at all steps, for the sake of tractability of query evaluation.

Expressions. SPARQL expressions, which are found in the FILTER and BIND constructs, and in the SELECT and GROUP BY clauses, bring a considerable increase in expressivity. They enable to derive values that are not explicitly present in data (e.g., people age from their birth date), and to answer analytical queries based on aggregations. Examples of analytical queries over geographical data are (permalinks to are provided as short URLs):

1. How many countries are there in Europe? (simple aggregation) http://bit. ly/2OMxbJw 2. Give me the average population and the average area of countries, for each continent. (grouping and multiple aggregations) http://bit.ly/2OMwKPm 3. What is the average GDP per capita, for each continent? (aggregation over a BIND) http://bit.ly/2OJsErj 4. Which continents have an average agricultural GDP smaller than 10% of their average service GDP? (FILTER over two aggregations) http://bit. ly/2OLAQaM 5. Give me the distribution of the number of islands in archipelagos. (aggregation over an aggregation) http://bit.ly/2NV71QF

For all those questions, an equivalent Sparklis query can be built over the MONDIAL dataset 4 . See the Learn/Examples page to try them live.

Named Graphs. Named graphs are found in the GRAPH construct, and in FROM clauses. They allow for controling the source of triples, and are required on some endpoints that put triples in different named graphs. Sparklis offers two ways to specify named graphs. The first way is to add FROM clauses in SPARQL queries by setting two configuration options: the default graph URI for retrieving factual triples, and the schema graph URI (if different from the default graph) for retrieving schema triples. The second way is to produce GRAPH-constructs in SPARQL queries for a finer control. Those constructs are introduced by the new query construct according to ..., which can be applied to any subquery to declare that this subquery should be matched on a named graph. The possible named graphs are shown in the results, and the ellipsis in the construct can be refined to select the desired named graphs. For example, the query: give me a city that according to a census whose year > 2000 has a population > 1000000 returns "the cities with over 1M inhabitants, according to a named graph that is a census posterior to 2000".

Complex Structures in RDF Graphs

Sometimes RDF datasets contain complex structures that, although covered by Sparklis 2016, generate ugly verbalizations, and are difficult to explore. We have added new constructs that abstract over those complex structures, so that they become more intuitive to query and explore.

Reified Relationships. Because RDF is limited to binary relationships (properties), a common solution to represent n-ary relationships or to add metadata to binary relations is to reify those relationships into statements or events. Unfortunately, various representations are used without consensus (e.g., standard reification, custom reification, singleton properties, named graphs, Wikidata predicates [START_REF] Hernández | Reifying RDF: What works well with wikidata[END_REF]), and it is difficult for Sparklis to automatically discover reified relationships through endpoints. Because of the importance and peculiarity of the Wikidata modeling, we have added a Wikidata mode as an option. In other cases, schema-level triples have to be added to declare which combinations of properties and classes represent reified relationships. For example, in the MONDIAL dataset, the relationships between countries and languages are reified into "spoken language" statements, and percentage values are attached to those statements to qualify the relationship. We have added the triple (mondial:ofCountry, nary:subjectObject, mondial:onLanguage) to declare that in the "spoken language" relationship, property mondial:ofCountry points to the subject, and property mondial:onLanguage points to the object. Every other property defined on the reified relationship (e.g., mondial:percentage) can be discovered by Sparklis, and suggested as a relation modifier that is verbalized similarly to prepositions. As a consequence, the verbose query give me every country that is the country of a spoken language whose language is English and whose percentage is > 25 can now be expressed more directly as give me every country that has as a language English at a percentage > 25 (http://bit.ly/2PHT0pD).

Hierarchies of Terms. Some properties code for the child-to-parent relationship of a hierarchy. Their range is equal to their domain, and in SPARQL queries it is usually their transitive closure that is used. General-purpose and well-known examples are rdfs:subClassOf or skos:broader. Domain-specific hierarchies exist for geographical locations, historical periods, anatomical parts, etc. We have added the construct (hierarchy) in ... that uses the last selected property as a hierarchical relationship. The effects are to apply transitive closure to the property in the generated SPARQL query, and to display the values of that property as a hierarchical tree. When a value in the tree is selected, it is equivalent to select all values under the selected value in the tree. It is possible to select several values and combine them by and/or. For example, the verbose and approximate query give me every place that is the part of the part of California or Arizona (assuming two levels between places and states) can now be expressed as give me every place (hierarchy) in California or Arizona.

User experience is important and we have also made improvements on this side.

-Responsive design. The UI has been given a responsive design with Bootstrap. It becomes therefore easier to use Sparklis on smaller screens. -Map view. When entities in the query have standard properties for latitude/longitude, they are detected and a special property has geolocation is suggested. If selected, this property triggers the display of the entities in the query results on a Google map (ex., http://bit.ly/2NT2Vs9). -Slideshow view. When the results contain images, those are collected and displayed as a slideshow. To provide context for the images inside the slideshow, the result row that contains the image is displayed under the image. For example, in DBpedia, a slideshow of scientist depictions can be obtained, along with biographic information http://bit.ly/2NRSwwT. -Full-text search. In order to provide a better experience when using the dynamic filtering features, support has been added for the full-text search capabilities of a few RDF stores (Jena/fuseki and Virtuoso so far). It assumes that full-text indices have been set up on the endpoint. For example, cities with name similar to "pétersbourg" are retrieved by http://bit.ly/2OH8sqg. -YASGUI editor. For advanced users who want to see and edit the generated SPARQL query, the well-known YASGUI editor [START_REF] Rietveld | YASGUI: Not just another SPARQL client[END_REF] has been integrated to Sparklis. It also brings useful tools for the download of query results in different formats, and for the visualization of results (pivot table and charts). -Dataset-specific UI customization. Properties of schema.org can be used to customize the appearance in Sparklis. Property schema:position can be defined for classes, properties and entities to control their ordering in lists of suggestions. Property schema:logo can be defined on entities with small images to be used as icons, in addition to the label (e.g., the logo of a company, the conservation status of a species). -Multilinguality. The UI is now available in four languages: English, French, Spanish, and Dutch. Similar languages would be easy to add.

Web application and screencasts at http://www.irisa.fr/LIS/ferre/sparklis/

http://data.persee.fr/explorer/sparklis/

https://www.loterre.fr/sparklis/

http://dbis.informatik.uni-goettingen.de/Mondial


Carlos Bobed
email: carlos.bobed-lisbona@irisa.fr

Laura Douze
email: laura.douze@univ-lille.fr

Sébastien Ferré
email: sebastien.ferre@irisa.fr

Romaric Marcilly
email: romaric.marcilly@univ-lille.fr

PEGASE: A Knowledge Graph for Search and Exploration in Pharmacovigilance Data

Pharmacovigilance is in charge of studying the adverse effects of pharmaceutical products. In this field, pharmacovigilance specialists experience several difficulties when searching and exploring their patient data despite the existence of standardized terminologies (MedDRA). In this paper, we present our approach to enhance the way pharmacovigilance specialists perform search and exploration on their data. First, we have developed a knowledge graph that relies on the OntoADR ontology to semantically enrich the MedDRA terminology with SNOMED CT concepts, and that includes anonymized patient data from FAERS. Second, we have chosen and applied a semantic search tool, Sparklis, according to the user requirements that we have identified in pharmacovigilance.

Introduction

The continuous research and advances in pharmacology improve significantly our life quality. However, despite being thoroughly tested before being released, all the possible side effects of the new drugs cannot be foreseen. Thus, along advances in pharmacology, we need methods to discover those adverse effects to improve the safety and efficacy of drugs. Pharmacovigilance is defined by the World Health Organization as "the science and activities relating to the detection, assessment, understanding and prevention of adverse effects or any other drug-related problem". In this work, we are concerned with supporting pharmacovigilance specialists in the search and exploration of their database of patient cases, which is generally the first step in the process of detecting new adverse effects of drugs.

In this context, the usefulness of standardized vocabularies to unify the codification of the reports is evident. MedDRA (Medical Dictionary for Drug Regulatory Activities) 3 is the vocabulary recommended by the ICH for the electronic transmission of individual case safety reports [START_REF]Electronic transmission of individual case safety reports[END_REF] to code adverse drug reactions (ADRs). However, as pointed out by Bousquet et al. [START_REF] Bousquet | Formalizing MedDRA to support semantic reasoning on adverse drug reaction terms[END_REF], "its main limitation comes from its standard terminological format, which restricts the possibility of accessing terms based on their semantics". To solve this problem, Bousquet et al. proposed OntoADR [START_REF] Bousquet | Formalizing MedDRA to support semantic reasoning on adverse drug reaction terms[END_REF], an ontology which makes it possible to work with MedDRA terms according to their actual semantics. In this paper, we present the solution we have developed in the PEGASE project to improve the way pharmacovigilance specialists search for cases. First, we have built a knowledge graph based on OntoADR integrating different knowledge sources, which makes it possible to have all the relevant data easily accessible, providing the flexibility required to be extended under demand. Then, we have chosen and applied Sparklis [START_REF] Ferré | Sparklis: An expressive query builder for SPARQL endpoints with guidance in natural language[END_REF], a query builder that eases the exploration and querying of any SPARQL endpoint, without requiring to master SPARQL itself. This choice was based on a requirement analysis conducted by ergonomists in the project. We are currently evaluating our proposal, along with other tools, in order to assess the benefits of our approach.

PEGASE Knowledge Graph

To build our knowledge graph, we have adapted OntoADR [START_REF] Bousquet | Formalizing MedDRA to support semantic reasoning on adverse drug reaction terms[END_REF], extending it with SMQs (Standardised MedDRA Queries) [START_REF]ICH: Introductory Guide for Standardised MedDRA Queries (SMQs) Version 21[END_REF] and anonymized patient data from FAERS dataset [START_REF]FDA's Adverse Event Reporting System (FAERS) Website[END_REF] to show how the integration capabilities of our knowledge graph can help pharmacovigilance specialists to ease their jobs. OntoADR The core structure of the PEGASE Knowledge Graph can be seen in Figure 1. It currently contains 3,257,389 triples without taking into account FAERS data (with the patient data of three months, it grows to 28,125,629 triples). To model MedDRA, we have introduced the concept MedDRATerm, which has five different subconcepts corresponding to the five levels of their hierarchy (see Figure 1). However, to model the hierarchy relationship between terms, instead of using the subclass relationship (i.e., formal subsumption), we have introduced the property medDRA parent. In this way, we can navigate the hierarchy without unexpected potential inferences.

To include SNOMED CT, we had to adapt its representation level. On the one hand, we had MedDRA terms, all of which were instances; on the other hand, we had SNOMED CT terms, all of which were concepts. To solve this mismatch, we materialized SNOMED CT concept hierarchy, and treated the concepts as instances 4 . This allowed us to introduce also different hierarchies to provide different navigation dimensions. In particular, we introduced a top-level hierarchy of SNOMED CT meta-concepts based on the semantic tags that SNOMED CT uses to further refine the concepts meaning. Note that this grouping cohabits with the subclass hierarchy of SNOMED CT concepts. This does not lead to inconsistencies as our knowledge graph is in RDFS, not in OWL. OntoADR relationships between MedDRA terms and SNOMED CT concepts (see [START_REF] Bousquet | Formalizing MedDRA to support semantic reasoning on adverse drug reaction terms[END_REF] for the complete list) were included as they are. SMQs SMQs are "groupings of MedDRA terms, ordinarily at the Preferred Term (PT) level that relate to a defined medical condition or area of interest" [START_REF]ICH: Introductory Guide for Standardised MedDRA Queries (SMQs) Version 21[END_REF]. In general, SMQs can be seen as disjunctions of terms which are used together in order to perform searches in a standardized way, although they can be grouped in more complex ways. We added each SMQ as a new node, related to the terms that it includes. The inclusion of SMQs is important because pharmacovigilants are used to work with them. FAERS Data The patient data provided by FAERS is split in seven different big tables, which we have integrated as shown in the resulting model in Figure 2. That model was obtained after an evaluation round with the ergonomists in the project's team, where we brought the FAERS model closer to the pharmacovigilants cognitive process.

Sparklis on the PEGASE Knowledge Graph

Sparklis5 is a query builder in natural language that allows people to explore and query SPARQL endpoints with all the power of SPARQL and without any knowledge of SPARQL [START_REF] Ferré | Sparklis: An expressive query builder for SPARQL endpoints with guidance in natural language[END_REF]. It reconciles the expressivity of SPARQL 1.1 and the usability of point-and-click user interfaces. Sparklis requires little configuration to be applied to the PEGASE Knowledge Graph. It is enough to provide the URL of the SPARQL endpoint6 , and to choose property rdfs:label for the labelling of entities, classes, and properties.

Figure 3 shows a screenshot of Sparklis on PEGASE data, taken during the process of building a query7 . The current query (at the top) select prefered terms (PT) in MedDRA whose finding site is (a subconcept of) "Skin and subcutaneous tissue structure", and whose associated morphology is (a subconcept of) various morphologic abnormalities. A first abnormality, "Blister" (dimmed font), has already been selected, and the user is in the process of selecting (at the center) a disjunction of three more abnormalities ("Vesicle", "Vesiculobullous rash", "Vesicular rash"). The keyword "vesic" was input at the top of the list of suggested terms in order to ease their retrieval among a long list of suggestions. The list of suggestions at the middle left contains classes and properties, i.e., types and relationships about the current focus (here, the focus is on the associated morphology of the selected preferred terms). The list of suggestions at the middle right contains query modifiers and operators (e.g., "and", "or", "number of"). The table of results of the current query is shown at each step (at the bottom). Here, it shows the selected preferred terms along with their finding sites and associated morphologies.

Fig. 1 .

 1 Fig. 1. Main modules of OntoADR within the PEGASE Knowledge Graph.

Fig. 2 .

 2 Fig. 2. FAERS integration in the knowledge graph. The companion numbers indicate the source of the data in the original FAERS tables.

Fig. 3 .

 3 Fig. 3. Sparklis' screenshot showing a query under construction (top) on the PEGASE Knowledge Graph, suggestions to refine the query (middle), and query results (bottom).

Abusing a little the language, we have flattened them in the RDF graph and allowed for meta-modeling, i.e., classes of SNOMED CT concepts.

http://www.irisa.fr/LIS/ferre/sparklis/

The URL is not provided here due to restrictive licences on MedDRA and SNOMED.

A screencast of the whole query building is available at http://www.irisa.fr/LIS/ common/documents/ekaw2018/#ExtraCase.

This research is supported by ANR project PEGASE (ANR-16-CE23-0011-08), project TIN2016-78011-C4-3-R (AEI/ FEDER, UE), and DGA/FEDER. 3 MedDRA R is a registered trademark of IFPMA (Int. Fed. Pharm. Manufact. and Assoc.)


Chrystopher L Nehaniv
email: c.l.nehaniv@herts.ac.uk

Kerstin Dautenhahn

Jens Kubacki

Martin Haegele

Christopher Parlitz

Rachid Alami

A Methodological Approach relating the Classification of Gesture to Identification of Human Intent in the Context of Human-Robot Interaction *

Keywords: interaction context, classification of gestures, human-robot activity and interaction

In order to infer intent from gesture, a broad classification of types of gestures into five main classes is introduced. The classification is intended as a generally applicable basis for incorporating the understanding of gesture into human-robot interaction (HRI). Examples from humanrobot interaction show the need to take into account not only the kinematics of gesture, but also the interactional context. Requirements for the operational classification of gesture by a robot interacting with humans are suggested and initial steps in its deployment are discussed.

I. INTRODUCTION: THE NEED FOR CLASSIFYING GESTURE

The word gesture is used for many different phenomena involving human movement, especially of the hands and arms. Only some of these are interactive or communicative. The pragmatics of gesture and meaningful interaction are quite complex (cf. [START_REF] Kendon | Movement coordination in social interaction: Some examples described[END_REF], [START_REF] Mey | Pragmatics: An Introduction[END_REF], [START_REF] Millikan | The Varieties of Meaning: The 2002 Jean Nicod Lectures[END_REF]), and an international journal [6] now exists entirely devoted to the study of gesture. Applications of service or 'companion' robots that interact with humans, including naive ones, will increasingly require human-robot interaction (HRI) in which the robot can recognize what humans are doing and to a limited extent why they are doing it, so that the robot may act appropriately, e.g. either by assisting, or staying out of the way. Due to the situated embodied nature of such interactions and the non-human nature of robots, it is not possible to directly carry over methods from humancomputer interaction (HCI) or rely entirely on insights from the psychology of human-human interaction. Insights from proxemics and kinesics, which study spatial and temporal aspects of human-human interaction [START_REF] Hall | The Dance of Life: The Other Dimension of Time[END_REF], [START_REF] Condon | A segmentation of behavior[END_REF], [START_REF] Kendon | Movement coordination in social interaction: Some examples described[END_REF] and some insights of HCI, e.g. recognizing the diversity of users and providing feedback acknowledgment with suitable response timing (e.g. [START_REF] Shneiderman | Designing the User Interface: Strategies of Effective Human-Computer Interaction[END_REF]), may also prove to be extremely valuable to HRI. Notwithstanding, the nascent field of HRI must develop its own methods particular to the challenges of embodied interaction between humans and robots. New design, validation, evaluation methods and principles particular to HRI must be developed to meet new challenges such as legibility, making the robot's actions and behaviour understandable and predictable to a human, and 'robotiquette', respecting human activities and situations (e.g. not interrupting a conversation between humans or disturbing a human who is concentrating or working intensely -without sufficient cause), as well as respecting social spaces, and maintaining appropriate proximity and levels of attention in interaction. Part of meeting these challenges necessarily involves some understanding of human activity at an appropriate level. This requires the capabilities of recognizing human gesture and movement, and inferring intent. The term "intent" is used in this paper in a limited way that refers to particular motivation(s) of a human being that result in a gestural motion directly or indirectly relevant for human-robot interaction.

In inferring the intent from a human's gesture it is helpful to have a classification of which type of gesture is being observed. Without a sufficiently broad classification, understanding of gesture will be too narrow to characterize what is happening and appropriate responses will not be possible in many cases.

Knowing how to recognize and classify gesture may also serve to inform the design of robot behaviour, including gestures made by the robot to achieve legibility and convey aspects of the robot's state and plans to humans. This in turn will contribute to robot interaction with humans that is legible, natural, safe, and comfortable for the humans interacting with the robot. To begin to approach the complexity of gesture in the context of situated human-robot interaction, the rough classes of gesture described below are developed in order to provide a broad level of description and the first steps toward a pragmatic, operational definition that could be used by an autonomous system such as a robot to help it (1) to infer the intent of human interaction partners, and (2), as an eventual goal, to help the robot use gestures itself (if possible) to increase the legibility of its behaviour.

II. SOME RELATED WORK ON RECOGNIZING GESTURE

AND INTENT The questions of how gestures are acquired and come to be recognized as meaningful by particular individuals in the course of their development (ontogeny of gesture and its recognition), and conventionalized, elaborated, or lost within particular cultures (evolution of gesture) are large and deep issues, but will not be addressed within the scope in this paper. Psychological/linguistic studies of human gesture use and understanding, related classifications relevant for interaction, language evolution, and language acquisition, e.g. by hearing or deaf children, have all been undertaken (cf. [START_REF] Volterra | Gesture and the emergence and development of language[END_REF]). Such understanding of the development of gesture and its functions may help shed light on gesture in human-robot interaction.

While this paper does not attempt a comprehensive survey of the role and recognition of gesture in humanrobot interaction, it does suggest inherent limitations of approaches working with a too narrow notion of gesture, excluding entire classes of human gesture that should eventually be accessible to interactive robots able to function well in a human social environment. Much work with data gloves, typically at a low level for hand gesture recognition for virtual reality or of manipulative grasping has been carried out since the 1990's (e.g. [START_REF] Dubois | Beyond user experimentation: notational-basd systematic evaluation of interaction technieuqs in virtual reality environments[END_REF], [START_REF] Bernadin | A sensor fusion approach for recognizing continuous human grasping sequences using hidden markov models[END_REF]). The important role of gesture for intent communication in human-robot interaction is increasingly being acknowledged, although some approaches still focus only on static hand poses rather than dynamic use of more general types of gesture in context; a survey of hand gesture understanding in robotics appears in [START_REF] Miners | Hand Gesture for Interactive Service Robots[END_REF].

Multimodal and voice analysis can also help to infer intent via prosodic patterns, even when ignoring the content of speech. Robotic recognition of a small number of distinct prosodic patterns used by adults that communicate praise, prohibition, attention, and comfort to preverbal infants has been employed as feedback to the robot's 'affective' state and behavioural expression, allowing for the emergence of interesting social interaction with humans [START_REF] Breazeal | Recognition of affective communicative intent in robot-directed speech[END_REF]. Hidden Markov Models (HMMs) have been used to classifying limited numbers of gestural patterns (such as grasps or letter shapes) and also to generate trajectories by a humanoid robot matching those demonstrated by a human [START_REF] Billard | Discovering optimal imitation strategies[END_REF]. Multimodal speech and gesture recognition using HMMs has been implemented for giving commands via pointing, one-, and two-handed gestural commands together with voice for intention extraction into a structured symbolic data stream for use in controlling and programming a vacuuming cleaning robot [START_REF] Iba | Interactive multimodal robot programming[END_REF]. Many more examples in robotics exist. Nevertheless, aproaches pursued so far in robotics thus tend to use very limited, constrained, and specific task-related gestural repertoires of primitives, and do not attempt to identify general gestural classes. They have tended to focus on a fixed symbolic set of gestures (possibly an extensible one, in which new gestures can be learned), or focus on only a few representatives from one or two of the gestural classes identified here (e.g. grasps (a subclass of manipulative gestures), or on symbolic and pointing gestures).

III. INSUFFICIENCY OF BODY MODEL FITTING ANALYSES: RELATING CONTEXT TO KINEMATICS.

It should be stressed that a single specific instance of a particular the kind of physical gestural motion could, depending on context and interaction history, reflect very different kinds of human intent. It will not always be possible to infer intent based solely on based the mechanical aspects of human movements (such as changes in joint angles) without taking context into account.

Gestures with identical or near identical human kinematics can be different classes. In general, the kinematic picture alone is not enough to determine the class of a gesture or the human's intent. Examples in shown in figure 1 require contextual information in order to be disambiguated. Relating the context and history of interaction to the kinematics is a key point for recognizing human gestures in HRI. Figure 1 using our classification (see below) illustrates this ambiguity.

IV. CLASSIFICATION OF GESTURES

To approach this problem, a classification of gesture for inferring intent and assisting in the understanding of human activity should closely relate gesture with limited categories of intent in situated human activity. The categories of the broad classification presented here thus correspond to and allow the attribution of limited kinds of intent to humans. This classification is developed as an aid for helping robots to achieve limited recognition of situated human gestural motion, so as to be able to respond appropriately if required, while these robots are working in an environment of ambient human activity (such as a home or office), in which, at times, the robots are also assisting or cooperating with the humans. Applications of this classification will require the mapping of physical aspects of gestural motion in interactional contexts to the five gestural classes (and their subtypes) suggested here.

The following is a rough, tentative classification. Gestures are classed into five major types with some subtypes. 1 Note that we are using the word "cooperative" in a sense that treats regulating communication or interaction as an instance of cooperation.

A. Five Classes (with

2 Some more subtle examples include putting one's hand on another person's arm to comfort them. Such actions, and others involving physical contact, may be quite complex to interpret as understanding them may require understanding and modeling the intent of one person to influence that state of mind of another. At this point, we class simply them with interactional gestures recognizing that future analysis may reveal deep issues of human-human interaction and levels of complexity beyond the rudimentary types of human intent considered here. A special case worthy of note is human contact with the robot, unless this is directly a manipulation of the robot's state via an interface -e.g. via button presses -which would fall into class 3 (symbolic gesture), non-accidental human contact with the robot is likely to be indicative of an intent to initiate or regulate interaction with the robot (class 4). Physical contact between humans might also involve expression of affection (kissing), or aggression (slapping, hitting) -which generally indicate types of humanhuman interaction it would be better for a robot to steer clear of! other directed motion, and/or eye gaze. Checking the eye gaze target of an interaction partner is commonly used to regulate reference and interaction. 3 Table I summarizes the five classes. Data on the interaction history and context may help in determining the class of a gesture. If the class is known, then the set of possible gestures can remain large, or be narrowed significantly. Symbolic gestures (class 3) correspond to discrete symbols in a finite set, of which there may be only be a small number according to context or size of the given repertoire of the given symbolic gestural code. Interactional gestures (class 4) are likely to comprise a small, constrained class. Class 1 gestures are either "irrelevant", or to be understood by seeking the intent of the associated motor action or object manipulation (e.g. grasping or throwing an object, arms moving as a side effect of walking). Class 5 (referential and pointing gestures) comprise a very limited class, although pointing can also at times carry affective force (e.g. hostility).

Knowledge of specific conventional codes and signs can help the identification of particular signs within class 3, and also in determining that the gesture in fact belongs to class 3, i.e. is a symbolic communicative signal. Machine learning methods such as Hidden Markov Models may be used successfully to learn and classify gestures for a limited finite set of fixed gestures (e.g. [START_REF] Westeyn | Georgia tech gesture toolkit: Supporting experiments in gesture recognition[END_REF]). It seems likely that HMM methods would be most successful with class 3 (symbolic gestures) or within narrow domains within other classes (manipulative grasps with class 1), but how successful they would be at differentiating between classes or for whole classes remains uninvestigated at present.

V. IMPORTANT ISSUES

A. Target and Recipient of a Gesture

If a gesture is used interactively or communicatively (classes 2-5), it is important to recognize whether the gesture is directed toward the current interaction partner (if any) -which may the robot, another person (or animal) present in the context, or possibly neither (target). If pointing, what is the person pointing to? Who is the pointing designed to be seen by? (recipient). If speaking, to whom is the person speaking? If the gesture is targeted at or involves a contact with an object, this suggests it may belong to class 1 (or possibly 5, even without contact). A gesture of bringing an object conspicuously and not overly quickly toward an interaction partner is manipulative (in the sense explained in the discussion of class 1, since an object is being manipulated), but it may well at the same time also be a solicitation for the partner to take the object (class 4). Similarly if the partner has an object, an open hand conspicuously directed toward the partner or object may be a solicitation for the partner to give the object (class 4). 3 Eye gaze following develops and supports joint attention already in preverbal infants. Language, including deictic vocabulary (e.g. demonstratives such as the words "these" and "that"), and other interactional skills, typically develop on this scaffolding (see [START_REF] Kita | Pointing: Where Language, Culture and Cognition Meet[END_REF]).

B. Multipurpose Gestures

It is possible for a single instance of a particular gesture to have aspects of more than one class or to lie intermediate between classes. As mentioned above, handing over an object is both class 1 and 4. And, for example, holding up a yellow card in football has aspects of classes 1 and 3, object manipulation and conventional symbolic signal. Many ritualized symbolic gestures (class 3) also can be used to initiate or regulate interaction (class 4), e.g. the 'come here' gesture: with palm away from the recipient, moving the fingers together part way toward the palm; waving forearm and open hand with palm facing recipient to get attention. More complex combinations are possible, e.g. a gesture of grasping designed by the human to be seen by a recipient interaction partner and directed toward a heavy or awkwardly-sharped target object as a solicitation of the partner to cooperatively carry the object with the gesturer (classes 1, 4, 5).

C. Ritualization: Movement into Classes 3 and 4

Gestures that originate in class 1 as manipulations of the non-animate environment and the person's relationship to it may become ritualized to invite interactions of certain types, e.g., cupping the hand next to the ear can indicate that person doing it cannot hear, so that the interaction partner should speak up. Originally cupping the hand near the ear served to improve a person's ability to hear sounds in the environment from a particular direction (class 1), but it may be intended to be seen by a conversational partner who then speaks up (class 4). The hand cupped at the ear can even be used as a conventionalized symbol meaning 'speak up' (classs 3). Other examples of ritualization toward regulation of interaction and also symbolic gesture include mimicking with two hands the motions of writing on a pad as a signal to a waiter to ask for the bill; miming a zipping action across the mouth to indicate that someone should be 'shut up'; or placing a raised index finger over lips which have been pre-formed as if to pronounce /sh/.

D. Cultural and Individual Differences

Different cultures may differ in their use of the various types of gesture. Some symbolic gestures such as finger signs (e.g. the "OK" gesture with thumb and index finger forming a circle) can have radically different interpretations in other cultures, or no set interpretation depending on the culture of the recipient (e.g. crossing fingers as a sign of wishing for luck, or the Chinese finger signs for some numbers such as 6, 7, 8). Tilting the head back (Greece) or nodding the head (Bulgarian) are used symbolically for 'no', but would certainly not be interpreted that way in many other cultures. Cultures also differ in their types and scope of movement in (class 2) expressive gestures: Consider, for example, the differences of rhythm, prosody, hand motions, eye contact, and facial expressions accompanying speech between British, Italian, Japanese, and French speakers.

Within cultures, differences between different individuals' uses of gestures can be regional, restricted to particular pointing of all kinds with all kinds of effectors (incl. eyes): referential, topicalizing, attention-directing social groups within the culture, and vary in particularities (such as speed, repertoire, intensity of movement, etc.) between individuals according to preference or ontogeny.

Elderly and young may employ gestures in different ways.

VI. INFERRING THE INTENT OF GESTURE

Being able to identify details of gestural kinematics and even to classify into one of the above classes gives us only starting points for inferring the intent of the person making the gesture due to frequent ambiguity. Resolving this points to the important roles of context and interactional history. Thus, it is necessary to develop operational methods for recognizing the class of gesture in a particular context. 4If the interactional context of recent activity in which a gesture occurs is known, this can suggest possibilities for which classes (and subtypes) of gesture might be involved. Information on the state of human (e.g. working, thirsty, talking, ...) often can limit the possibilities. Data on the following could help the robot classify the gesture and infer the intent of the human: (a) the activity of the gesturer is known, (b) previous and current interaction patterns are remem-bered to predict the likely current and next behaviour of the particular person, (c) objects, humans and other animated agents in the environment are identified and tracked. (d) the scenario and situational context are known (e.g. knowing whether a gesture occurs at a tea party or during a card game).

A programme to apply the above classification can be developed as follows: (1) Identify the many, particular gestural motions that fit within each of the five classes. Some gestural motions will appear in more than one class. For example, the same mechanical motion of putting a hand and arm forward with the forearm horizontal and the hand open could indicate preparation to manipulate an object in front of the human (class 1), to show which object is being referred to (class 5), or to greet someone who is approaching, or to ask for an object to be handed over (both class 4). ( 2) Gestural motions identified as belonging to several classes need to be studied to determine in which contexts they occur: determining in which class(es) particular a instance of the gesture is being used may require consideration of objects and persons in the vicinity, the situational context, and the history of interaction. (3) Systematic characterizations of a physical gestural motion together with interactional contexts in which they are occur could then be used to determine the likely class. (4) Deploy on-board characterizations of the relationships between classes and kinematic gestural motions for a range of typical interactional contexts to infer intent and guide robot behaviour. ( 5) Updating the Interaction History: Attribution of intent related to gesture can then feedback into understanding of the situational context, including motivational state of the human performing the gesture, and becomes part of the updated interaction history, which can then help in inferring intent from ensuing gestures and activity.

VII. HEURISTIC-BASED FAST RECOGNITION AND DISAMBIGUATION OF GESTURES WITH A TIME-OF-FLIGHT DEPTH SENSOR

There are mainly three methods to recognize and detect gestures: model-based approaches fit a kinematics model into the scene observed by sensors, recognition based on classifiers use learning algorithms to label gestures, and heuristic-based methods which directly search for hints related to a gesture. Depending on the context of the overall robotic control system, all of these may be of use. The model-based approach is followed by researchers in the Cogniron project, and, as shown above, this must be augmented by contextual and situational knowledge. The goal is to develop algorithms that geometrically fit a model maintained by the robot into the current scene observed by stereo vision systems and a time-of-flight depth sensor proposed in [START_REF] Oggier | User manual: Swissrangertm 2[END_REF]. Apart from this exhaustive approach there is also work related to a computationally much cheaper heuristic-based method only using data delivered by the depth sensor. The motivation for this is two-fold. Firstly, a 'quick' check of the existence of humans in the close vicinity of the robot and a first basic evaluation of possibly important gestures can be used directly for communication. Secondly, outputs of a fast algorithm related for instance to body, head or arm positions can serve to trigger more detailed investigation by e.g. model-based algorithms. Additionally, the data can be used to initialize model fitting.

The heuristic approach first divides the depth scene observed by the sensor into consecutive depth intervals each having a fixed distance and size. The two intervals containing the most measurements are used for binary segmentation of the humans profile. Within the profile the algorithm searches for a human's center point by summing all pixels belonging to the profile and averaging their coordinates. From this point the algorithm searches upwards and determines a bounding box for the head including the neck by incorporating estimates of the shoulder end points. They can be found as being the left and right extremes of the profile at a height roughly at the bottom of the head. Interestingly, the height of the bounding box around the head plus the width of the shoulder can give an estimate of the length of the upper arm as described in medical statistics. Based on this information four cases can be distinguished:

• Outstretched arm away from the body • Outstretched arm up or down • Bent arm next to body • Bent arm in front of body For each case further heuristic algorithms are used to determine the hand position and orientation. This can be used to recognize and discriminate between basic gestures. See Figure 2 for a visualization of how the program finds a WAVE gesture and SHAKE HANDS gesture; both are interactional gestures (class 4). By using additional information such as orientation and distance of the human towards the robot and internal state of the robot, tentative disambiguations between similar gestures have been made.

VIII. CONCLUSIONS, NEXT STEPS AND THE FUTURE

In order to infer the intent of a human interaction partner, it may be useful to employ a classification of gesture according to some major types -five in the tentative classification proposed here -whose intent may be (1) absent / directed to objects or environment, (2) incidentally expressive, (3) symbolic, (4) interactional, or (5) deictic. A summary of the classes is given by Table I.

In order to deploy the inference of intent on robots interacting with humans it is necessary to operationalize the distinctions between these (sometimes overlapping) classes. This may require the use of knowledge of human activity, recognition of objects and persons in the environment, and previous interactions with particular humans, as well as knowledge of conventional human gestural referencing and expression, in addition to specialized signaling codes or symbolic systems.

Work in Cognrion now focuses on the organization of the robot decisional abilities and more particularly on the management of human interaction. There is explicit management of the interactions between the robot companion and its human partners. This requires essentially task-oriented processes that each consist of establishing a common goal, achieving it and verifying commitment of all agents involved during the task performance. Indeed, perception of the human partner is one essential source of information all along the human-robot interaction process from the detection of human presence to the monitoring of human activity and the continuous estimation of its commitment level to a joint goal. This viewpoint is compatible with and served by the classification of gestures proposed here. It also helps us to operationalize use of the classification. Indeed, gestures of type 3 and many of type 1 may be considered as task-oriented and the inference of their intent can be done relative to the task at hand. Gestures of type 4 include generic interactional gestures that may serve to manage the session itself: inviting the robot to start an interaction, suspending or stopping an interaction session, etc. Many gestures of type 4 are consequently task independent.

The classification presented here suggests some requirements for the design and implementation of systems inferring intent from gesture based on this classification. These requirements might be realized in a variety of different ways using, e.g. continuous low-key tracking or more detailed analysis, event-based and/or scenario-based recognition, and prediction of human activity based on models of human activity flows (with or without recognition of particular humans and their previous interactions), depending the particular needs of the given human-robot interaction design and the constraints and specificity of its intended operational context. Design of a robot restricted to helping always the same user in the kitchen environment would be quite different from one that should be a more general purpose servant or companion in a home environment containing several adults, children and pets, but the classification presented here is applicable in informing the design of gesture recognition for inferring intent in either type of system, and for designing other HRI systems.

Finally, effective human-robot interaction will require generation of gestures and feedback signals by the robot. The classification given here can suggest categories of robotic gestures that could be implemented to improve the legibility to humans of the robot's behaviour, so that they will be better able to understand and predict the robot's activity when interacting with it.

Fig. 1 .

 1 Fig. 1. Gestures with similar kinematics but different functions. Top row: HELLO (left) an interactional gesture (class 4) is similar to STOP (right) a conventional symbol (class 3). Bottom row: PASS OBJECT (left) is similar and TAKE OBJECT are both multifunctional interactional (class 4) and manipulative (class 1) gestures. Activity and situational context -e.g. stage of interaction and current activity (top row), or location of manipulandum, here a bottle (bottom row) -are be used to disambiguate between such kinematically similar gestures.

Fig. 2 .

 2 Fig. 2. Heuristic-based fast recognition and disambiguation of interactional gestures. Detecting bounding box of head, hand and its orientation, and center of human (top row: color image of gesturing human in robot's vicinity. second row: depth image derived from camera images. Left column: SHAKE HANDS gesture, right column: WAVE gesture.

  Note that the degree of arbitrariness in such gestures may vary: The form of the gesture may be an arbitrary conventional sign (such as a holding up two fingers with palm forwards to mean 'peace', or the use of semaphores for alphabetic letters). On the other hand, a symbolic gesture may resemble to a lesser or greater extent iconically or, in ritualized form, a referent or activity.

	Further examples: holding up two fingers to indicate 'two'; opening both (empty) hands by turning palms down to indicate a lack of something. Nearly all symbolic gestures are used to convey content in communicative interactions. 4) Interactional Gestures. These are gesture used to regulate interaction with a partner, i.e. used to initi-ate, maintain, invite, synchronize, organize or termi-nate a particular interactive, cooperative behaviour: raising a empty hand toward the partner to invite the partner to give an object; raising the hand containing an object toward the partner inviting them to take it; nodding the head indicating that one is listening. The emphasis of this category is neither reference nor communication but on gestures as mediators for cooperative action.
	3) Symbolic Gestures. Gestural motion in symbol ges-ture is a conventionalized signal in a communicative interaction. It is generally a member of a limited, cir-cumscribed set of gestural motions that have specific, prescribed interpretations. A symbolic gesture is used to trigger certain actions by a targeted perceiver, or to refer to something or substitute as for another signal according to a code or convention. Single symbolic gestures are analogous to discrete actions on an interface, such as clicking a button.

Subtypes) 1) 'Irrelevant'/Manipulative Gestures. These include irrelevant gestures, body / manipulator motion, sideeffects of motor behaviour, and actions on objects. Broadly characterized, manipulation by a human is here understood as doing something to influence the non-animate environment or the human's relationship to it (such as position). Gestural motions in this class are manipulative actions (in this sense) and their side effects on body movement. These 'gestures' are neither communicative nor socially interactive, but instances and effects of human motion. They may be salient, but are not movements that are primarily employed to communicate or engage a partner in interaction. Cases include, e.g. motion of the arms and hands when walking; tapping of the fingers; playing with a paper clip; brushing hair away from the face with the hand; scratching; grasping a cup in order to drink its contents. (Note: it may be very important to distinguish among the subtypes listed above for robot understanding of human behaviour.) 2) Side Effect of Expressive Behaviour. In communicating with others, motion of hands, arms and face (changes in their states) occur as part of the overall communicative behaviour, but without any specific interactive, communicative, symbolic, or referential roles (cf. classes 3-5) Example: persons talk excitedly raising and moving their hands in correlation with changes in voice prosody, rhythm, or emphasis of speech. Examples: waving down a taxi for it to stop; use of a conventional hand signals (a command to halt indicated open flat hand; a military salute); nodding 'yes'; waving a greeting 'hello' or 'goodbye'. 1 Interactional gestures thus concern regulating the form of interactions, including the possible regulation of communicative interactions but do not generally convey any of the content in communication. Interactional gestures are similar to class 1 manipulative gestures in the sense that they influence the environment, but in contrast to class 1, they influence the "animated environment"doing something to influence human agents (or other agents) in the environment, but not by conveying symbolic or referential content. 2 5) Referential/Pointing Gestures. These are used to refer to or to indicate objects (or loci) of interesteither physically present objects, persons, directions or locations the environment -by pointing (deixis -showing), or indication of locations in space being used as proxies to represent absent referents in discourse. Deictic gesture can involve a hand, finger,

TABLE I Five Classes of Gesture. See

 I text for explanation, details and examples. Note that some occurrences of the same physical gesture can be used in different classes depending on context and interactional history; moreover, some gestures are used in a manner that in the same instance belongs to several classes (see text for examples).

Knowledge of the immediate context in some cases needs to be augmented by taking into account of the broader temporal horizon of interactional history (cf.[START_REF] Nehaniv | Meaningful information, sensor evolution, and the temporal horizon of embodied organisms[END_REF]).

under Contract FP6-002020. This paper extends and supercedes C.


Carlos Bobed
email: carlos.bobed-lisbona@irisa.fr

Laura Douze
email: laura.douze@univ-lille.fr

Sébastien Ferré
email: sebastien.ferre@irisa.fr

Romaric Marcilly
email: romaric.marcilly@univ-lille.fr

Sparklis over PEGASE Knowledge Graph: A New Tool for Pharmacovigilance

Keywords: Knowledge Graph, Semantic Search, Query Building, Pharmacovigilance, MedDRA, Sparklis, Usability

come   L'archive ouverte pluridisciplinaire

Introduction

Pharmacology has provided humanity with a huge improvement in life quality. However, while new drugs are thoroughly tested before being released for general consumption, all their possible side effects cannot be completely foreseen. Thus, we need methods to discover and track those adverse effects in order to improve the safety and efficacy of drugs. Pharmacovigilance is defined by the World Health Organization (WHO) as "the science and activities relating to the detection, assessment, understanding and prevention of adverse effects or any other drug-related problem".

Pharmacovigilance specialists manage and report adverse drug reactions (ADRs) noticed by healthcare professionals and patients to different healthcare authorities. For this, they must codify their reports using one or several terms that closely capture the original verbatim description of the ADRs. In this context, the usefulness of standardized vocabularies to unify the codification of the reports is evident. To this purpose, MedDRA (Medical Dictionary for Drug Regulatory Activities) 4 is recommended by the ICH 5for the electronic transmission of individual case safety reports [START_REF]Electronic transmission of individual case safety reports[END_REF], and is used in most countries. However, as pointed out by Bousquet et al. [START_REF] Bousquet | Formalizing MedDRA to support semantic reasoning on adverse drug reaction terms[END_REF], "its main limitation comes from its standard terminological format, which restricts the possibility of accessing terms based on their semantics". Therefore, depending on their experience, expertise, and interpretation of the meaning of the MedDRA terms, two pharmacovigilance specialists may use different MedDRA terms to code the same report. To solve this problem, Bousquet et al. proposed OntoADR [START_REF] Bousquet | Formalizing MedDRA to support semantic reasoning on adverse drug reaction terms[END_REF], an ontology that describes MedDRA terms by their actual semantics, expressed with a combination of SNOMED CT classes and properties. However, until now, pharmacovigilance specialists could not use those semantic descriptions due to the lack of appropriate tools to help them build the semantic queries required to retrieve sets of MedDRA terms.

In this paper, we present our solution to support pharmacovigilance specialists in the exploration and search of their database of patient cases, the first step in the process of detecting new adverse effects of drugs. First, we studied the existing tools, detecting a lack of proper tools to ease the documentation stage of pharmacovigilance specialists. Then, we built a knowledge graph integrating different knowledge sources, adopting the Semantic Web standards, which makes it possible to have all the relevant data easily accessible, providing the flexibility required to be extended under demand. Finally, in order to improve the way pharmacovigilance specialists search for cases, we adopted and extended Sparklis [START_REF] Ferré | Sparklis: An expressive query builder for SPARQL endpoints with guidance in natural language[END_REF], a query builder that eases the exploration and querying of any SPARQL endpoint, without requiring to master SPARQL itself. This work takes place in the PEGASE project, whose aim is to improve pharmacovigilance and signal detection.

The rest of the paper is as follows. Section 2 presents a literature analysis about usability evaluations of controlled vocabulary searching tools along with a benchmark of existing tools in pharmacovigilance. Section 3 describes the PEGASE Knowledge Graph. Section 4 describes Sparklis and the extensions we have implemented. Section 5 presents the knowledge graph validation we have performed, and the qualitative usability study we have carried out on Sparklis. Finally, Section 6 draws some conclusions, and presents the next stages of the project.

Analysis of Existing Tools

Our main objective was to identify the desirable usability qualities and the defects of the tools used by pharmacovigilance specialists. To do so, we performed: (i) a literature review of usability evaluations of tools supporting searches using controlled vocabularies (not limited to pharmacovigilance), and (ii) an evaluation of the tools currently used by French pharmacovigilance specialists. A total of 908 papers were identified in PubMed, Web of Science, and Scopus databases; six of which met all eligibility criteria6 and were analyzed in-depth: 1) Walji et al. [START_REF] Walji | Detection and characterization of usability problems in structured data entry interfaces in dentistry[END_REF] supports coding odontology diagnostics, 2) Bakshi-Raiez et al. [START_REF] Bakhshi-Raiez | A usability evaluation of a SNOMED CT based compositional interface terminology for intensive care[END_REF] supports coding reason for entrance in the intensive care unit (ICU), 3) Peute et al. [START_REF] Peute | The value of retrospective and concurrent think aloud in formative usability testing of a physician data query tool[END_REF] supports searching ICU patient data, 4) Shiri et al. [START_REF] Shiri | User evaluation of Searchling and T-saurus: Multilingual thesaurus-enhanced visual interfaces for digital libraries[END_REF] supports searching keywords over a multilingual thesaurus, 5) Vega-Gorgojo et al. [START_REF] Vega-Gorgojo | Semantic search of tools for collaborative learning with the ontoolsearch system[END_REF] evaluates the search of collaborative tools, and 6) Sutcliffe et al. [START_REF] Sutcliffe | Evaluating the effectiveness of visual user interfaces for information retrieval[END_REF] evaluates visual user interfaces for information search. Besides, we visited four French pharmacovigilance centers to identify the software used, where four main tools were identified: -BNPV (National Bank of PharmacoVigilance): the main tool used by the pharmacovigilance specialists to report and search cases of ADRs. -MedDRA browser: it allows searching MedDRA terms along with Standardized MedDRA Queries (SMQ). -PA.PV: a tool provided by the French Agency for the Safety of Health Products to help pharmacovigilance specialists choose MedDRA terms and SMQs. -Vigilyze: a world-wide pharmacovigilance database provided by the WHO. Two human factors specialists went through the different functionalities of each software to identify their significant usability qualities and defects. A total of 62 usability desirable features or defects were learned from the literature review, and 86 from the benchmark of the most used software. Overall, the main lessons learned were: 1. Searches should include synonyms and related terms. 2. One should be able to search terms either by entering keywords (without hierarchy level constraints) or by browsing the term hierarchy. 3. Lists and hierarchies of terms should be ordered in a meaningful way. 4. Display the query, the number, and list of results on the same page. 5. Results should automatically be updated with the query. 6. Provide users with hierarchy information to distinguish similar terms. 7. Inform the user about the number of ADR reports each term triggers. 8. Help users build their request by providing an intuitive interface. 9. Allow saving a request to be reused. Those recommendations led to the choice of Sparklis, and to its adaptation to the pharmacovigilance's context. Indeed, Sparklis [START_REF] Ferré | Sparklis: An expressive query builder for SPARQL endpoints with guidance in natural language[END_REF] (see Section 4 for a description) already meets to a large extent criteria 4, 5, 8, and 9 (as well as 7 in an approximate way) by guiding its users in the incremental building of their request while receiving results and lists of suggested terms at every step. The main missing feature was the handling of hierarchies, which concerns criteria 2, 3 and 6, which we have added as a result of the project.

PEGASE Knowledge Graph

In this section, we detail the knowledge sources we have used to build the PEGASE Knowledge Graph and its structure. First, we present MedDRA and OntoADR [START_REF] Bousquet | Formalizing MedDRA to support semantic reasoning on adverse drug reaction terms[END_REF] the core ontology we use, and then we move onto the additional sources we have included, Standardized MedDRA Queries (SMQs) and anonymized patient data (obtained from FAERS). The core structure of the PEGASE Knowledge Graph can be seen in Figure 1. It currently contains 3,257,389 triples without taking into account FAERS data (with the patient data of three months, it grows to 28,125,629 triples).

OntoADR This ontology [START_REF] Bousquet | Formalizing MedDRA to support semantic reasoning on adverse drug reaction terms[END_REF] is the result of progressive efforts to semantically structure the Medical Dictionary for Regulatory Activities (MedDRA) terminology. In the context of pharmacovigilance, MedDRA is a reference terminology to code and give a precise description of ADRs and related issues [START_REF] Harrison | MedDRA R : The tale of a terminology: Side effects of drugs essay[END_REF]. In particular, OntoADR is an ontology which connects MedDRA terms to SNOMED CT concepts, making it possible to search for MedDRA terms semantically, e.g. to search for all terms about ADRs taking place in the skin. To establish the mapping between MedDRA terms and SNOMED CT, OntoADR exploits the UMLS metathesaurus [START_REF]UMLS Website[END_REF], which provides an initial alignment. Apart from that, several curating steps and algorithms were applied to have a further refined set of relationships.

In order to include OntoADR in our Knowledge Graph, we had to do some adaptations. To model MedDRA, we introduced the concept MedDRATerm, which has five different subconcepts corresponding to the five levels of their hierarchy (see Figure 1). However, to model the hierarchy relationship between terms, instead of using the subclass relationship (i.e., formal subsumption), we introduced the property medDRA parent.

In this way, we can navigate the hierarchy without unexpected potential inferences. For example, "Aortic aneurysm" is a Preferred Term (PT), whose medDRA parent is "Aortic aneurysm and dissections", a High Level Term (HLT).

To include the SNOMED CT definitions, we had to adapt their representation level. On the one hand, we had MedDRA terms, all of which were instances; on the other hand, we had SNOMED CT terms, all of which were concepts. To solve this mismatch, we materialized SNOMED CT concept hierarchy, and treated the concepts as instances 7 . This allowed us to introduce also different hierarchies to provide different navigation dimensions. In particular, we introduced a top-level hierarchy of SNOMED CT metaconcepts based on the semantic tags that SNOMED CT uses to further refine the concepts' meaning (see Figure 2). Note that this grouping cohabits with the subclass hierarchy of SNOMED CT concepts. This does not lead to inconsistencies as our knowledge graph is in RDFS, not in OWL. OntoADR relationships between MedDRA terms and SNOMED CT concepts (see [START_REF] Bousquet | Formalizing MedDRA to support semantic reasoning on adverse drug reaction terms[END_REF] for the complete list) were included as they are. Following with the previous example, "Aortic aneurysm" (MedDRA term) is related to "Aneurysm" (SNOMED concept) via associatedMorphology (OntoADR property), and to "Abdominal Aorta Structure" (SNOMED concept) via findingSite (OntoADR property). Furthermore, "Abdominal Aorta Structure" is a BodyStructure (SNOMED meta concept) and a rdfs:subclassOf of "Descending aorta structure" (SNOMED concept). Note how we use the metaconcept hierarchy to group the SNOMED concepts according to their semantic tags.

SMQs SMQs are "groupings of MedDRA terms, ordinarily at the Preferred Term (PT) level that relate to a defined medical condition or area of interest" [START_REF]ICH: Introductory Guide for Standardised MedDRA Queries (SMQs) Version 21[END_REF]. In general, SMQs can be seen as disjunctions of terms which are used together in order to perform searches in a standardized way, although they can be grouped in more complex ways. We added each SMQ as a new node, related to the terms that it includes. The inclusion of SMQs in the graph has two main benefits: on the one hand, pharmacovigilants are used to working with SMQs and they had them directly accessible; and on the other hand, they extend the querying capabilities as now we have the full capabilities of SPARQL on top of them.

FAERS Data Finally, we integrated a source of patient data to show how the integration capabilities of our knowledge graph can help pharmacovigilance specialists to ease their jobs. FAERS (FDA Adverse Event Reporting System) is a dataset containing the anonymized reports of drug adverse events which is gathered and made public quarterly by the U.S. Food and Drug Administration (FDA). The patient data provided by FAERS is split in seven different big tables, which we integrated as shown in the resulting model in Figure 3. Such model was obtained after an evaluation round with the human factors specialists in the project's team, where we brought the FAERS model closer to the pharmacovigilants' cognitive process.

Sparklis on the PEGASE Knowledge Graph

Sparklis8 is a tool for retrieving information from RDF knowledge graphs, with the objective to reconcile the expressivity of SPARQL 1.1 and the usability of point-and-click user interfaces. We here present its principles as well as the main extensions that we have performed to address the needs of the PEGASE project.

Sparklis: a SPARQL Query Builder in Natural Language

Sparklis is a query builder in natural language that allows people to explore and query SPARQL endpoints without any knowledge of SPARQL [START_REF] Ferré | Sparklis: An expressive query builder for SPARQL endpoints with guidance in natural language[END_REF]. Sparklis is implemented as a Web client running entirely in the browser, which directly connects to SPARQL endpoints to retrieve query results and suggested query elements. It covers a large subset of SPARQL 1.1 select queries: basic graph patterns including cycles, union, optional, not exists, filter, bind, complex expressions, aggregations, group by, order by. All those features can be combined in a flexible way. Results are presented as tables, and also on maps and as slideshows. A configuration panel offers a few configuration options to adapt to different endpoints (e.g., GET/POST, labelling properties and language tags).

Applying Sparklis to the PEGASE Knowledge Graph

Sparklis requires very little configuration to be applied to the PEGASE Knowledge Graph. It is enough to provide the URL of the SPARQL endpoint 9 , and to choose property rdfs:label for the labelling of entities, classes, and properties. As the end users are French pharmacovigilance specialists, we also configure the user interface and the labels to the French language.

Figure 4 shows a screenshot of Sparklis on PEGASE data, taken during the process of building a query 10 . The current query (at the top) select prefered terms (PT) in MedDRA whose finding site is (a subconcept of) "Skin and subcutaneous tissue structure", and whose associated morphology is (a subconcept of) various morphologic abnormalities. A first abnormality, "Blister", has already been selected, and the user is in the process of selecting (at the center) a disjunction of three more abnormalities ("Vesicle", "Vesiculobullous rash", "Vesicular rash"). The keyword "vesic" was input at the top of the list of suggested terms in order to ease their retrieval among a long list of suggestions. Sparklis suggests modifications to be applied to the current focus (here, the focus is on the associated morphology of the selected preferred terms): at the middle left, Sparklis suggests classes and properties to refine the query; at the middle center, individuals denoting associated morphologies relevant to the query; and at the middle right, query modifiers and operators (e.g., "and", "or", "number of"). The table of results of the current query is shown at each step. Here, it shows the selected preferred terms along with their finding sites and associated morphologies.

Extending Sparklis for PEGASE

Handling hierarchies As the example in Figure 4 shows, it is important to take into account hierarchies of MedDRA terms or SNOMED concepts in the evaluation of queries. For instance, when asking for "preferred terms whose finding site is Skin...", what is really meant is: "preferred terms whose finding site is any subconcept of Skin...", i.e., any part of the skin (e.g., "epidermis", "subepidermal region"). The correct translation to SPARQL requires the use of property paths like ?x ontoadr:findingSite/rdfs:subClassOf* snomed:Skin .

A number of problems made it impossible to express in Sparklis, and thus required an extension to handle hierarchies in the building and evaluation of queries. First, it was possible to build a sequence of properties but not to apply the *-operator (transitive closure). Therefore, it was possible to reach the parent or grand-parent concepts but not all ancestors at once. Second, assuming the *-operator was available, it was tedious to explicitly cross the property defining the hierarchy (rdfs:subClassOf in the example), and to apply transitive closure. Third, once an element of the hierarchy was selected ("Skin..." in the example) and the query focus set on it, only that element was visible whereas the user would expect to see the subset of the hierarchy below and above that element.

We solved those problems by modifying Sparklis with several extensions and adaptations. What made it particularly tricky is that the new features had to combine smoothly with the numerous existing features: e.g., crossing properties forward and backward, Boolean coordinations of verb phrases and noun phrases. It was also important to make it in a generic way so as to handle various kinds of hierarchies. The PEGASE knowledge graph has two hierarchies based on two different properties (medDRA parent for Med-DRA terms, rdfs:subClassOf for SNOMED concepts), and other applications could have other hierarchies, e.g., hierarchies of geographical places or historical periods. First, we added a (hierarchy) in query construct that can turn any property into a hierarchy. It does not only apply transitive closure on the property but translates the current focus in a special way so that all terms below and above the selected terms are retrieved and can be displayed as a tree to render the hierarchical relationships between the terms. For example, the SPARQL translation of the above example, when the focus is on "Skin..." is ?x ontoadr:findingSite ?y . ?y rdfs:subClassOf* snomed:Skin ; rdfs:subClassOf* ?focus .

Second, we added schema-level declarations to automate the use of the hierarchy feature. For example, a declaration states that the range of property findingSite is hierarchically organized by property rdfs:subClassOf. Therefore, as soon as property findingSite is crossed, a hierarchy construct based on rdfs:subClassOf is inserted in the query so that the user immediately sees a hierarchy of concepts in which terms can be selected. We stated a similar declaration for each OntoADR property whose range is made of SNOMED concepts.

Other extensions Finally, a few other less original, yet important, extensions were implemented. We have added support for full-text search (compatible so far with Jena Fuseki and Virtuoso RDF stores), and for multi-selection. The former was required to speed up keyword searches and improve the robustness (e.g., handling accentuated letters). The latter, while it does not increase the expressivity of Sparklis, allows to build coordinations of values in much fewer interactions.

Evaluation

We have performed the expert evaluation in two subsequent steps: 1) the human factors specialists in the project team checked Sparklis against the list of recommendations derived from the literature analysis, comparing it to the main tools used by pharmacovigilance specialists in France (see Section 2); then, 2) they analyzed Sparklis over PEGASE Knowledge Graph, after the extensions were implemented (see Section 4.3) performing a cognitive walkthrough [START_REF] Polson | Cognitive walkthroughs: a method for theory-based evaluation of user interfaces[END_REF] in order to identify remaining usability issues.

Qualitative Evaluation

Two human factors specialists checked to which extent the extended version of Sparklis respects the requirements learned from the literature review and the benchmark of existing tools. Table 1 compares the BNPV tool and the extended Sparklis w.r.t. the requirements identified in Section 2. The evaluation showed that Sparklis has a better adherence to the requirements than the BNPV, the tool currently in use. Therefore, Sparklis may provide a better support to pharmacovigilance specialists' searches. Nonetheless, there is still room for some improvements regarding: criteria 1, synonyms and related terms are not considered; criteria 7, Sparklis provides the number of ADR reports linked to a given term based only on a sample of reports; and criteria 8, although it guides users during the request building, the current interface is not sufficiently intuitive for untrained pharmacovigilance specialists.

Cognitive Walkthrough

A cognitive walkthrough [START_REF] Polson | Cognitive walkthroughs: a method for theory-based evaluation of user interfaces[END_REF] was performed, following an ISO standard [START_REF]Ergonomics of human-system interaction -Part 210: Human-centred design for interactive systems[END_REF], to identify remaining usability issues in Sparklis. This inspection method focuses on how easy and intuitive for new users is their interaction with an interactive device. Representative pharmacovigilance specialists were asked for four prototypical use cases. Following those use cases, two human factors specialists went through the sequences of tasks supported by Sparklis stepping into pharmacovigilance specialists' shoes. For each subtask, they asked themselves four standardized questions:

-Will pharmacovigilance specialists try to achieve the effect the subtask has? -Will they notice that the correct action is available? -Will they understand that the wanted subtask can be achieved by the action? -Do they get appropriate feedback?

For each question, at each subtask, usability problems leading to answering "no" were noticed. While the evaluation was performed over Sparklis directly, we prepared some illustrative videos to show how the user can deal with the use cases using Sparklis, which are available at http://www.irisa.fr/LIS/common/documents/pegase2018/.

Despite the potential support provided by Sparklis to pharmacovigilance specialists, its current graphical user interface might not be intuitive enough from a pharmacovigilance specialists' perspective. While Sparklis has been proved to successfully reduce the gap between users and SPARQL queries, it still requires training in order to go through the first stage of the learning curve. This is specially relevant in this context, where the users (pharmacovigilance specialists) might have neither expertise nor training in computer and information sciences, and their related logic, concepts and wording. For instance, Sparklis provides all the flexibility of SPARQL in order to build the queries; however, it can be argued that this flexibility might confuse pharmacovigilance specialists who use daily a very limited range of requests.

As a conclusion of this evaluation, we can conclude that we might need two different versions of Sparklis interface depending on the user profile (i.e., advanced and beginner users): 1) for advanced users, an interface presenting all the relevant information to help pharmacovigilance specialists build a request, and 2) for beginners, a simplified graphical user interface presenting only the logical operators and concepts which are most likely to be used, and the others on-demand. In this way, we will allow the whole range of pharmacovigilance specialists to take advantage of its power from the start.

Conclusions and Future Work

To overcome the limitation of current pharmacovigilance tools, we have built a knowledge graph that integrates: 1) MedDRA, the main pharmacovigilance vocabulary, 2) On-toADR, its semantic enhancement using SNOMED CT, 3) SMQs, and 4) patient data. In order to search and explore that graph, we have adopted Sparklis, a SPARQL query builder in natural language. We conducted a literature review along with a benchmark of existing tools, which led to the detection of several requirements that pharmacovigilance tools have to meet, and which were taken into account in a Sparklis extension. Finally, we carried out an evaluation of the interface following an ISO ergonomics standard [START_REF]Ergonomics of human-system interaction -Part 210: Human-centred design for interactive systems[END_REF], showing the potential of our proposal. Usability evaluation with end-users is in progress.

Fig. 1 .

 1 Fig. 1. Main modules of OntoADR within the PEGASE Knowledge Graph.

Fig. 2 .

 2 Fig. 2. SNOMED meta-concept layer integrated to include the semantic tags hierarchy.

Fig. 3 .

 3 Fig. 3. FAERS integration in the knowledge graph. The companion numbers indicate the source of the data in the original FAERS tables.

Fig. 4 .

 4 Fig. 4. Sparklis' screenshot showing a query under construction (top) on the PEGASE Knowledge Graph, suggestions to refine the query (middle), and query results (bottom).

Table 1 .

 1 Adherence of BNPV and Sparklis to the identified requirements.

	Requirements	BNPV	Sparklis
	1. Include synonyms and related terms	No	No
	2. Search by entering terms or by navigating	Yes	Yes
	3. Meaningful order	No	Yes
	4. Display query, number and list of results together No	Yes
	5. Update automatically results	No	Yes
	6. Provide the terms' places within the hierarchy	Incomplete Yes
	7. Inform about the number of ADR reports	No	Incomplete
	8. Help users build their request	No	Incomplete
	9. Allow saving the request	Yes	Yes

MedDRA R is a trademark of the IFPMA.

Int. Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use

Omitted due to space restrictions. The most discriminative one was to include an objective evaluation of the interfaces. See http://bit.ly/2qA9Jkl for details on the selection process.

Abusing a little the language, we have flattened them in the RDF graph and allowed for meta-modeling, i.e., classes of SNOMED CT concepts.

http://www.irisa.fr/LIS/ferre/sparklis/ (includes the presented extensions)

The URL is not provided here because of restrictive licences on MedDRA and SNOMED.

A screencast of the whole query building is available at http://www.irisa.fr/LIS/ common/documents/pegase2018/#ExtraCase.

This research is supported by ANR project PEGASE (ANR-16-CE23-0011-08), project TIN2016-78011-C4-3-R (AEI/ FEDER, UE), and DGA/FEDER. 3 FDA's Adverse Event Reporting System