L E O N A R D O C A M P I L L O S -L L A N O S

T H E R I N E T H O M A S

É R I C B I L I N S K I

P I E R R E Z W E I G E N B A U M

S O P H I E R O S S E T

Designing a Virtual Patient Dialogue System Based on Terminology-rich Resources: Challenges and Evaluation

Virtual patient software allows health professionals to practice their skills by interacting with tools simulating clinical scenarios. A natural language dialogue system can provide natural interaction for medical history taking. However, the large number of concepts and terms in the medical domain makes the creation of such a system a demanding task.

We designed a dialogue system that stands out from current research by its ability to handle a wide variety of medical specialties and clinical cases. To address the task, we designed a patient record model, a knowledge model for the task, and a termino-ontological model that hosts structured thesauri with linguistic, terminological and ontological knowledge. We used a frame-and rule-based approach and terminology-rich resources to handle the medical dialogue. This work focuses on the termino-ontological model, the challenges involved and how the system manages resources for the French language.

We adopted a comprehensive approach to collect term variants and ontological knowledge, and dictionaries of affixes, synonyms and derivational variants. Resources include domain lists containing over 161,000 terms, and dictionaries with over 959,000 word/concept entries.

We assessed our approach by having 71 participants (39 medical doctors and 32 nonmedical evaluators) interact with the system and use 35 cases from 18 specialities. We conducted a quantitative evaluation of all components by analysing interaction logs (11,834 turns). Natural language understanding achieved an F-measure of 95.8 per cent. Dialogue management provided on average 74.3 (±9.5) per cent of correct answers. We performed a qualitative evaluation by collecting 171 five-point Likert scale questionnaires. All evaluated aspects obtained mean scores above the Likert mid-scale point. Finally, we analysed the vocabulary coverage with regard to unseen cases: the system covered 97.8 per cent of their terms.

Evaluations showed that the system achieved high vocabulary coverage on unseen cases and was assessed as relevant for the task.

Introduction

Medical education requires trainees and practising doctors to develop expertise in diagnosis or clinical reasoning. These skills are traditionally acquired through clinical practice, and they may be enhanced with the help of simulations with mannequins, role-playing games or virtual patients [START_REF] Rombauts | Patients virtuels: pédagogie, état de l'art et développement du simulateur Alphadiag[END_REF]. More broadly, the literature uses the term virtual patient (hereafter, VP) to refer to simulations such as case presentations, interactive patient scenarios, high-fidelity mannequins, virtual patient games, high-fidelity software simulations, human standardised patients-who are actors playing the role of interviewed patients paid for educational purposes-or virtual standardised patients (Talbot et al. 2012a). Virtual patients allow health professionals to practice their skills by interacting with a software 'that simulates real-life scenarios' [START_REF] Cook | Computerized virtual patients in health professions education: a systematic review and meta-analysis[END_REF]. In our work, virtual patient (VP) refers to virtual standardised patients. For the last few decades, VPs have allowed doctors to train clinical and history taking skills through simulated scenarios in digital environments [START_REF] Ellaway | An architectural model for MedBiquitous virtual patients[END_REF][START_REF] Danforth | Development of virtual patient simulations for medical education[END_REF].

Interactivity with a VP might be enhanced through a dialogue system, but such a component needs to address several phenomena to achieve a natural, user-friendly dialogue (Figure 1). As shown, medical doctors tend to begin by eliciting initial clues from the patient by using broad questions. Then, they use follow-up questions to focus on specific details. The system needs to deal with this behaviour by processing context information (ellipsis and anaphora) and updating its information state, so that it avoids providing redundant answers. In addition, term variants referring to the same concept need to be mapped accurately (e.g. hypertension ↔ high blood pressure) by means of linguistic and terminological knowledge.

This work describes our endeavour to create a dialogue system featuring unconstrained natural language interaction in a simulated consultation with a VP. We built this system to simulate history-taking in an educational software featuring an animated avatar with text-to-speech (Figure 2), and allowing students to simulate a physical exam. This project was developed in collaboration with a medical team specialized in simulation-based medical education at Angers University Hospital (CHU d'Angers) and several companies (Interaction Healthcare / Sim-forHealth, Voxygen and VIDAL R ). Within this partnership, 1 each group deployed their know-how: 1) to provide the system with advanced dialogue capabilities when users interview each VP: this was our contribution, and is the background for the present paper; 2) to endow each VP with text-to-speech and minor gestures; 3) to develop a dedicated interface for instructors to create unlimited avatars and patient records; 4) to manage the evaluation with potential end-users who tested the system; and 5) to provide domain texts and data for the task. In this project, which spanned over three years, four computational linguists and one software engineer (about 4.5 person-years) were involved to create the dialogue system. Fig. 1: Sample dialogue (D: doctor; P: patient, simulated by the system) and relation to the patient record. Phenomena to be addressed are shown in bold: discourse phenomena, linguistic variation (blue circles) and termino-ontological variation (green boxes). The patient record structure is simplified due to space constraints. We show real replies of the current version of the French system, but we translated them to English.

Fig. 2: Sample of dialogue with a virtual patient avatar

In this context, the medical team set specific design constraints:

• The system should manage a dialogue for unseen cases without being extended manually. Achieving high vocabulary coverage allows trainees to interact with VPs in a wide range of clinical conditions. The items to which a doctor must pay attention vary widely across cases and specialities. Practising in front of a variety of VPs should therefore be a good exercise to train students in medical history taking. Certainly, practising the history-taking skills with many cases needs to be accompanied by quality scenarios carefully designed by medical instructors to provide feedback and meaningful learning opportunities. • The VP should provide answers based on the actual content of the patient record, i.e. the information representing the patient's health or findings (e.g. disease history or medication, § 3.1). This brings up the need for providing correct replies, an aspect we refer to as correctness. By this, we mean that the simulated patient should not invent information that is not present in the patient record (faithfulness). Related to that aspect, the system should not omit data from the record (exhaustiveness of information). In our project, it was deemed that the system should provide all data available with regard to a given question. This design criterion seemed more adequate to avoid adding more difficulties to other possible sources of miscommunication arising with an artificial agent. Note that this was a pedagogical choice for a first version of the system: a virtual patient could indeed simulate situations in real life where the patient lies or makes mistakes in recalling the health record. Medical students should keep that in mind when conducting a medical history.

We can add the following observations related to the task and domain:

• The conversation does not take place in an open context but in a specific domain. The medical-history taking is a focused task, defined by the number and nature of the topics that are to be processed ( § 3.2).

• The medical domain involves a large concept space with a multitude of term variants. This requires entity linking (also called entity normalisation) in order to query and match input terms with those in the patient record. Our goal is to develop a system that, first, replies correctly (correctness), and second, can work with an unlimited number of cases and medical specialities (high vocabulary coverage). A method to evaluate these aspects at the natural language understanding step is to compute precision and recall : precision is related to correctness, whereas recall depends on vocabulary coverage. When evaluating dialogue management, correctness is a key aspect.

• A lexical simplification process is needed to simulate natural replies, i.e. using terms according to a patient viewpoint (e.g. saying appendix operation instead of the professional term appendectomy written in the patient record).

• Pre-existing dialogue data for the task and domain are not available, hence the system cannot be designed with a data-intensive approach.

Herein we explain how we address these requirements in our dialogue system and the challenges involved. To handle the needs of the domain and the diversity of its concepts, we applied a comprehensive approach to terminology collection. To the best of our knowledge, we integrated a larger volume of resources than in standard task-oriented conversational agents (see §3). We propose a framework for managing the linguistic and terminological needs in such a task, with different levels of knowledge representation, keeping coherence across components. The difficulty of term detection in our task motivates our approach based on rich terminological resources, which complement a frame-and rule-based dialogue management. The proposed methods aim to enhance the system's capability to adapt to new cases in a wide range of medical specialities and detect rare and unseen vocabulary items for a successful interaction. To show the extent to which it succeeds in doing so in a real-use scenario, we conducted evaluations which include user interactions (n=71) with 35 different VP cases from 18 different medical specialities, and an assessment of vocabulary coverage on 169 new cases.

We developed French, English and Spanish versions of the system. We report here work related to the French version because it is the only one evaluated to date. In the remainder of the paper, we first review the approaches to dialogue systems and to interactions with simulated VPs ( §2). We present our termino-ontological and linguistic models in §3 and summarise the architecture of our system in §4. We present the evaluation of resources and discuss its results in §5, then conclude in §6.

Approaches to dialogue systems

A textual or spoken dialogue system involves several modules, which typically include natural language understanding, speech recognition, generation, speech synthesis and a dialogue manager. More components may be added to correct input errors, query a database or search in a document collection, if the task requires it.

Different approaches exist for dialogue management [START_REF] Jokinen | Spoken dialogue systems[END_REF]): finite-state or graph-based approaches [START_REF] Cole | Tools for research and education in speech science[END_REF]; frame-based techniques [START_REF] Mctear | Handling errors and determining confirmation strategies-an object-based approach[END_REF][START_REF] Van Schooten | Handling speech input in the Ritel QA dialogue system[END_REF]; statistically-based approaches such as reinforcement learning [START_REF] Sutton | Reinforcement learning: An introduction[END_REF] or Partially Observable Markov Decision Processes (POMDP) [START_REF] Young | Using POMDPs for dialog management[END_REF]; and neural-network-based approaches, which have recently been reviewed [START_REF] Celikyilmaz | Deep Learning for Spoken and Text Dialog Systems[END_REF].

Due to the lack of existing dialogue corpora for this domain, statistically or neural approaches are not applicable to the design of our VP dialogue system. Our approach aims at endowing the system with capabilities to manage a comprehensive range of aspects of a dialogue task. This involves processing input so that it is understood in the context of previous questions and answers: interpreting discourse phenomena (such as co-reference or ellipsis) and maintaining the global dialogue state. This contrasts with question-answering approaches, designed to answer independent questions, in which a system analyses the user's natural language questions and produces a natural language answer, but commonly without processing of dialogue history [START_REF] Talbot | Natural language understanding performance and use considerations in virtual medical encounters[END_REF][START_REF] Maicher | Developing a Conversational Virtual Standardized Patient to Enable Students to Practice History-Taking Skills[END_REF][START_REF] Jin | Combining CNNs and Pattern Matching for Question Interpretation in a Virtual Patient Dialogue System[END_REF].

User interaction in healthcare applications

Interactive systems for healthcare applications address patient education and counselling [START_REF] Giorgino | Automated spoken dialogue system for hypertensive patient home management[END_REF][START_REF] Bickmore | Conversational agents for automated inpatient and outpatient health counseling[END_REF] or support to practitioners [START_REF] Beveridge | Automatic generation of spoken dialogue from medical plans and ontologies[END_REF]; a review is reported in [START_REF] Bickmore | Health dialog systems for patients and consumers[END_REF]. We focus on a dialogue with a VP in an educational context. Development challenges are similar across health dialogue systems [START_REF] Hoxha | Leveraging dialog systems research to assist biomedical researchers' interrogation of Big Clinical Data[END_REF]. Literature reviews on VPs are available [START_REF] Cook | Computerized virtual patients in health professions education: a systematic review and meta-analysis[END_REF][START_REF] Kenny | Embodied conversational virtual patients[END_REF][START_REF] Salazar | A case based reasoning model for multilingual language generation in dialogues[END_REF][START_REF] Rossen | A crowdsourcing method to develop virtual human conversational agents[END_REF][START_REF] Lelardeux | Serious game, simulateur, serious play : état de l'art pour la formation en santé[END_REF][START_REF] Rombauts | Patients virtuels: pédagogie, état de l'art et développement du simulateur Alphadiag[END_REF].

A key strategy for enhancing the simulation is to provide realistic user interaction. Integrating natural language interaction into a VP system requires managing domain terms-e.g. by formalizing ontological concepts (Nirenburg et al. 2008a)and Natural Language Understanding (NLU). The NLU unit may rely on text meaning representations for resolving paraphrases [START_REF] Nirenburg | A unified ontological-semantic substrate for physiological simulation and cognitive modeling[END_REF]), a corpus of questions and replies curated by experts [START_REF] Kenny | Evaluation of Justina: a virtual patient with PTSD[END_REF] or canned questions and answers [START_REF] Benedict | Virtual patients and problem-based learning in advanced therapeutics[END_REF][START_REF] Siregard | Apprendre le raisonnement clinique par jeu sérieux[END_REF]. The i-Human Patients R system2 allows users to choose the questions to ask to the VP, whose answers are parametrized by a patient record.

Few systems allow natural language input. As far as we can tell, current tools with natural language interaction are available for practicing patient assessment and diagnosis [START_REF] Hubal | The virtual standardized patient[END_REF])-e.g. in a pediatric scenario [START_REF] Hubal | Lessons learned in modeling virtual pediatric patients[END_REF]-and clinical history taking and communication skills-e.g. in a case of acute abdominal pain [START_REF] Stevens | The use of virtual patients to teach medical students history taking and communication skills[END_REF]; in a psychiatric consultation [START_REF] Kenny | Evaluation of Justina: a virtual patient with PTSD[END_REF]; or in a case of back pain [START_REF] Gokcen | A corpus of word-aligned asked and anticipated questions in a virtual patient dialogue system[END_REF][START_REF] Maicher | Developing a Conversational Virtual Standardized Patient to Enable Students to Practice History-Taking Skills[END_REF]. 3 Kenny and his team reported using 459 question variants mapped to 116 responses related to a posttraumatic stress disorder case [START_REF] Kenny | Evaluation of Justina: a virtual patient with PTSD[END_REF]. Gokcen and colleagues' system partially relies on manually annotated data-to date, 104 dialogues and 5,347 total turns [START_REF] Gokcen | A corpus of word-aligned asked and anticipated questions in a virtual patient dialogue system[END_REF]. The Maryland Virtual Patient, which simulates seven types of esophageal diseases, uses a lexicon covering over 30,000 word senses and an ontology of more than 9,000 concepts (Nirenburg et al. 2008b).

These interactive systems seem to be case-specific; i.e. they treat a limited number of cases. As far as we know, [START_REF] Talbot | Natural language understanding performance and use considerations in virtual medical encounters[END_REF] developed one of the few natural language interaction systems trained to cope with different clinical cases in the English language (e.g. ear pain, psychiatry and gastroenterology). 4 It relies on a medical taxonomy of 700 questions and statements and a supervised machinelearning model trained on over 10,000 training examples. For their part, the Virtual Patients Group (VPG, a consortium of North-American universities) also envisages a robust natural language interaction system. The VPG's platform Virtual Patient Factory5 allows users to create new cases and interact with virtual humans. Application scenarios range from psychiatry to pharmacy. To develop the NLU component, they used the Human-Centered Distributed Conversational Modeling (HDCM) technique [START_REF] Rossen | Human-centered distributed conversational modeling: Efficient modeling of robust virtual human conversations[END_REF], a crowd-sourcing methodology for collecting the corpus used to feed the system. Their method relies on a tight collaboration between VP developers and medical experts, a workflow that we specifically aim to bypass to make the system much more easily extensible to new cases.

Lastly, neural approaches are being explored for the NLU component in VP systems [START_REF] Datta | A Deep Learning Methodology for Semantic Utterance Classification in Virtual Human Dialogue Systems[END_REF][START_REF] Jin | Combining CNNs and Pattern Matching for Question Interpretation in a Virtual Patient Dialogue System[END_REF]. These are data-intensive methods and can be set up once enough data are collected from real interactions.

We applied a knowledge-based approach, mostly rule and frame-based, because of the lack of available dialogue and domain data to train a machine learning system. Due to the magnitude of the terminology in the medical domain, we also rely on rich terminological resources, which led us to give special care to the design of language resources management.

Models

Given a clinical case, the medical trainee will ask questions about various facets of the patient record, referring to entity types and concepts through domain terms.

In this section, we present the models designed to create our VP dialogue system.

Patient record model

We first need to specify what type of information is available to the dialogue system about the patient's state. This is the aim of the patient record model, from which the dialogue system will provide information about the specific patient it embodies. This model is similar to those that underlie electronic health records, and was refined iteratively when the first cases were created.

The VP is specified in a clinical record typically authored by a medical trainer, who aims at teaching students how to handle a given case. The clinical record describes common data found in patient records, structured into sections (e.g. Medical history or Current treatment) and lower-level subsections (e.g. Medical history has a subsection on Family history; Figure 3). Some basic elements can have associated attributes (e.g. the dose and frequency of a medical treatment). Most elements and attributes consist of free text. An example of a clinical record is shown in Table 1. food: items:

-the patient has normal eating habits physicalActivity:

items:

-the patient likes gardening addictions:

items:

-the patient smokes 20 cigarettes a day since 20 years old -the patient drinks a glass of wine per day medicalRecord:

allergies: nonmedicationAllergy:

-allergy: kiwi medicalHistories:

-disease: high blood pressure onsetTime: 10 years ago treatment: therapeuticClassValue: antihypertensive doseValue: 20 milligrams frequencyValue: 1 pill every day methodOfAdministrationValue: per os surgeries:

-operation: appendectomy lifeperiod: childhood -operation: inguinal herniorrhaphy anesthesia: GA age: at 51 other:

-observationsValue: the patient is up-to-date with his vaccines -observationsValue: the patient does not live in damp housing complaints:

-symptom: the patient has thoracic pain on the right side onsetTime: since yesterday night at 20 observationsValue: the patient had pain below the nipple, the patient was watching TV -symptom: the patient has a fever onsetTime: the fever started minutes after the pain feverValue: 38.9 degrees observationsValue: the patient did not take a medication -symptom: the patient is sweating observationsValue: the patient perspires because of the fever onsetTime: since yesterday night -symptom: the patient coughs observationsValue: the patient has a dry coughing since yesterday night at 23 -symptom: the patient has yellow sputum onsetTime: since today morning -symptom: the patient has shortness of breath onsetTime: since yesterday night currentTreatment:

-therapeuticClassValue: pain-killer 4). A preprocessing module extracts information from some text fields, e.g. time data from the symptoms (complaints) field. The system most often reuses text from the patient record to generate replies, which may make them less natural.

Knowledge model for the dialogue task

Our system operates in the context of the anamnesis stage of a consultation scenario-i.e., the medical-history taking step to collect diagnostic information from a patient. Therefore, the system must be capable of analyzing and replying to common questions (related to symptoms or treatments). The system also needs to process broader designated topics that are necessary to conduct history-taking; namely, demographic data (patient's name and surname, civil status, age, profession or family status) and questions on patient's lifestyle (recreational activities, eating habits, social life or family life). The knowledge model for the dialogue task defines:

• the range of topics that might be addressed during the clinical anamnesis, and the relation to each record section (as they were grouped in our project); • the question types to be processed in the dialogue manager, as well as the sections in the VP record associated to these question types; • the entity types to annotate in the user input by the natural language understanding (NLU) module ( §3.3.2); • dialogue acts, which define the function of the user input (e.g., greeting, acknowledging) analysed by the NLU module, and acts that define the reply type and content to be output by the natural language generation (NLG) module (e.g. inform symptom duration). A sample of the model with regard to symptoms is described in Figure 5.

We designed the model based upon the following sources. We collected questions used in a patient-doctor consultation scenario. We used 30 audio recordings with human standardised patients, who simulated consultations on anesthesiology, hypertension and pneumopathy. Several actors were recorded for each case, which allowed us to obtain varied versions of history taking for the same case. 6 We transcribed and analysed the recordings to detect interaction patterns. We also gathered questions from guides for clinical examination used by practitioners [START_REF] Bates | Guide de l'examen clinique -Nouvelle édition 2014[END_REF][START_REF] Epstein | Guide pratique de l'examen clinique[END_REF] and from resources for medical translation [START_REF] Coudé | Guide de conversation médicalefrançais-anglais-allemand[END_REF][START_REF] Pastore | How can I help you today? Guide de la consultation médicale et paramédicale en anglais[END_REF]. 7For entity types and dialogue acts, we defined 149 different labels: 62 entity types, 70 question types, and 17 dialogue acts (e.g. greetings). 52 labels are used for medical entity types and 10 for non-specifically medical entity types (e.g., frequency). 56 labels are used for medical question types (e.g. Qsymptom) and 14 for general question types (e.g. for dates, Qdate, when; or cause, Qwhy, why). Medical entity and question types are related to patient record sections: lifestyle (16.7 per cent of labels), medical history/symptoms (60.1 per cent), treatments (8.3 per cent), clinical examinations/analyses (4.6 per cent) and demographic data (10.2 per cent).

The knowledge model for the dialogue task was defined and refined in an iterative process during development: evaluators periodically interacted with the system, and a computational linguist analysed the logs to improve the questions types, the labels of entity types and the missing terms used in the interaction. The process extended over 7 iterations and about 24 months. We report the domain sources used for the lists of entity types in Table 3 ( §3.3.2).

Linguistic and termino-ontological model

Medical terminology brings up multiple processing difficulties. To illustrate their order of magnitude, let us first introduce the Unified Medical Language System R (hereafter, UMLS R ) [START_REF] Bodenreider | The Unified Medical Language System (UMLS): integrating biomedical terminology[END_REF]). The UMLS MetaThesaurus R is a large multilingual source of medical terminologies and ontological knowledge that come from close to 200 thesauri. In a similar way to how WordNet encodes synsets, the UMLS MetaThesaurus encodes concepts from different vocabulary sources with Concept Unique Identifiers (CUIs). CUIs map concepts and terms across multiple terminologies. Likewise, the UMLS Semantic Network R compiles semantic relations (e.g. is a or caused by) from the source ontologies. Terms in the UMLS are classified in 134 semantic types (STYs): e.g. diabetes is a Disease or Syndrome, and fever is a Sign or Symptom. UMLS semantic types are clustered into 15 semantic groups. For example, the group DISO contains various types of health conditions such as diseases, injuries, symptoms or findings. The UMLS MetaThesaurus contains 349,760 distinct French terms in the 2017AA version (4,011 of the type Sign or Symptom). Terms are often nested (e.g. heart failure) and variation includes derivation (heart vs. cardiac), compounding (cardiovascular ), abbreviation (MI, for myocardial infarction) and lay terms (heart attack ).

The above description shows that the number of different concepts and terms in this domain is larger than in usual dialogue systems. In such a context, we needed to provide the system with resources for concepts and words so that it knows about the domain and can handle term variation to interact adequately in it.

We developed for this purpose a linguistic and termino-ontological model, which hosts structured thesauri with linguistic, terminological and ontological knowledge. It is schematised in the central block of Figure 6 (left and right panels, respectively). Fig. 6: Linguistic and termino-ontological model at the core of system architecture With regard to the different language versions of the system, processing term variants is more challenging for French or Spanish, due to the higher number of verb forms or gender variants compared to English. For example, we needed resources for gender and number agreement to generate grammatically correct replies; accordingly, these resources are larger for the versions in French and Spanish.

Overall description

The variability of natural language expressions calls for linguistic knowledge. This includes word-level information: morphological information such as inflection (e.g. kidney ↔ kidneys), derivational variants (e.g. surgery ↔ surgical ), affixes and root elements (e.g. disease ↔ -pathy), and synonyms (e.g. operation ↔ surgery).

Termino-ontological knowledge defines the relations and concepts that are useful for the system to interact in the domain. The structure of thesauri is similar to that in the UMLS Metathesaurus, and is organised around entity types, terms linked to concepts, and relations between these concepts.

We distinguish entity types (semantic classes of the domain defined for the task, The linguistic knowledge is instantiated through language resources such as dictionaries; it has no direct link to concepts or entity types. The termino-ontological knowledge is typically instantiated by domain terminologies. Accordingly, we use different linguistic and terminological resources. There is a separate lexicon file for each component (e.g. a file for synonym variants and another for derivational variants in the linguistic model). Table 2 shows the types of variation phenomena and the resources needed for the generation, entity linking and normalisation steps. For instance, the generation step (first pane of the table) uses information stored in both the linguistic and termino-ontological models (this is detailed in §4.5). Here, linguistic knowledge consists of morphological information: gender, number, and part-of-speech, as well as correspondences between specific verb forms (e.g., has ↔ have). Termino-ontological knowledge maps scientific and lay term variants (e.g., per os ↔ oral ). Likewise, the entity linking and normalisation step (second pane of Table 2) uses linguistic knowledge to manage linguistic variation (see §3.3.3) and termino-ontological knowledge to manage terminological variation (see §3.3.4).

To build these lexicons, we extracted semi-automatically terms and ontology relations from the UMLS where possible. Due to the large size of the UMLS, we used the subset of its terminologies and semantic types that were relevant for our task; for example, we did not need entity types such as Regulation or Law (STY T089). We also used the National Agency for the Safety of Medicines list,8 and extended the lexicons as needed for our task. We built two types of lexicons: lexicons with terms recorded in the UMLS (with a CUI), and other lexicons that we created with terms not recorded in the UMLS. We explained the methods to collect them in a previous work [START_REF] Campillos-Llanos | Managing linguistic and terminological variation in a medical dialogue system[END_REF]), and we briefly describe them below.

Entity types

Vocabulary lists to label domain and miscellaneous entities (both mono-and multiword items) amount to 161,878 items. Table 3 shows the correspondence between patient record sections, entity types, NLU labels, the source of the data, and the sizes of the associated term lists. The counts presented do not aggregate typographical variants; e.g. anti-hypertensive and antihypertensive are counted separately. We provide the source used for extracting some domain terms from the UMLS (column Source): codes are UMLS semantic types (STYs; e.g. T184 for Sign or Symptom) or UMLS semantic groups. Our list of diseases has items of all DISO STYs except Sign or Symptom (T184). We specify in italics the Medical Subject Headings (MeSH) code where this terminology was used: e.g. D27.505 (Pharmacological action) for drug therapeutic classes. When there was not an UMLS semantic type for the entity type needed, we extracted terms by using regular expressions. For example, to collect the list of allergies, we applied the expression (allerg|intolér|réaction) on UMLS terms of type Disease or Syndrome (T047), and then manually revised results. Some lists come from the same STY, but semiautomatic and manual methods were used. For example, we extracted terms for gynecological and obstetric events from the type Disease or Syndrome. During development, after users interacted with the system, we observed that some question types were related to obstetrics entity types (e.g. miscarriage), and others, related to gynecologycal events (e.g. abundant menstruation). Hence we needed to distinguish between both of them and refined the scheme of entity types.

Managing linguistic variation

We address two types of linguistic variation: morphological variation (inflection and derivation) and synonymy. The former involves dealing with inflectional variants (e.g. to sweat and sweating) (Table 2, 2.1). We manage this through a generallanguage inflectional dictionary-we used DELA-type electronic dictionaries for French [START_REF] Courtois | Un système de dictionnaires électroniques pour les mots simples du français[END_REF]. Also, derivational variants may occur (e.g. impairment and impair ; Table 2, 2.2). We deal with them through deverbal nouns collected from a specialised general lexicon-for French, VerbAction [START_REF] Hathout | An experimental constructional database: the MorTAL project[END_REF]. Derivational variants of medical terms (e.g. fever ↔ feverish, thorax ↔ thoracic) come from the UMLF lexicon [START_REF] Zweigenbaum | A unified medical lexicon for French[END_REF]. Synonym relations (e.g. walk and stroll ) are processed by means of general synonym lexicons (Table 2, 2.4). For French, we reused the dictionary applied in a previous project [START_REF] Rosset | The LIMSI participation in the QAst track[END_REF]. This open-domain dictionary contains entries whose values are synonym words: e.g. opération|manipulation;action;intervention. Table 3: Correspondence between VP record sections, NLU labels and lists, and sources (cont.)

Managing terminological variation

Terms referring to the same concept For terms recorded in the UMLS (Table 2, 2.5), we can map term variants associated to the same concept by using their CUI: e.g. high blood pressure and hypertension (CUI C0020538). Term variants come from the following UMLS types and groups [START_REF] Mccray | Aggregating UMLS semantic types for reducing conceptual complexity[END_REF]): Terms not recorded in the UMLS need special processing (Table 2, 2.6). For example, the UMLS lacks French verbs for symptoms (e.g. to perspire), which can not be mapped to noun forms (e.g. sweating, C0038990). We created ad-hoc lists to link them, gathering single-and multi-word verbs/idioms and lemmatised forms. We collected them manually and iteratively: after users interacted with the system, we analysed interaction logs and added items to lexicons. With a similar procedure, we collected a secondary list for remaining lay variants (e.g. per os ↔ oral ).

• Anatomic entities (
When other methods fail, medical affixes and roots/stems (Table 2, 2.3) help match terms with no UMLS relation. To build these resources, we adapted the lexicon of affixes and roots of DériF, a morphosemantic linguistic-based parser for processing medical terminology in French [START_REF] Namer | Acquiring meaning for French medical terminology: contribution of morphosemantics[END_REF]). This analyser decomposes terms into morphological constituents and classifies them into domain semantic types. We also translated to French some neoclassical compounds (i.e. Latin or Greek prefixes, roots or suffixes, e.g. amygd-, 'tonsil') from the UMLS Specialist lexicon R (McCray, Srinivasan and Browne 1994). This lexicon is an English dictionary (over 200,000 entries) gathering biomedical terms and frequent words. Each entry records syntactic, morphological and orthographic information.

Using hierarchical relationships Some dialogue contexts involve concepts with a different degree of specialisation to that in the record. In these cases, we use UMLS relationships to map general to specific concepts (or vice versa), especially relations from SNOMED CT (Table 2,3). The Systematized Nomenclature of Medicine -Clinical Terms (SNOMED CT) (Donnelly 2005) is a clinically-oriented multilingual terminology distributed by the International Health Terminology Standards Development Organisation (IHTSDO). SNOMED CT gathers codes, hierarchical concepts, relations between them, and descriptions, and it is included in the UMLS.

Namely, UMLS child of (chd) relations are used to map a type of disorder (e.g. cardiovascular disease) to a specific disease in the record (e.g. high blood pressure). Relationships are also used when term variants fail. Terms referring to classes of disorders with the pattern disease gen + anatomy are related to their anatomical site by SNOMED CT relation has finding site: e.g. kidney disease and glomerulosclerosis. Other SNOMED CT relations are used: has procedure site, to map anatomic terms and surgical procedures with pattern surgery gen + anatomy (e.g. appendectomy ↔ appendix ); and has procedure morphology or To relate symptoms or disorders to physiological functions (e.g. dyspnea and to breathe), we extracted lists of correspondences between these types of entities from UMLS terminologies-namely, ICD10, MeSH and SNOMED-together with hierarchical relationships between concepts of these types.

Table 4 reports the size of the resources for managing terminological and lexical variation: number of variants, minimum, maximum and mean values per CUI or lexical entry (mono-and multi-word items), and number of lexical entries or CUIs in each resource (for relations between CUIs, we give the number of related pairs).

Implementation

In this section, we first describe the general architecture of the system ( § 4.1). We then detail the resources and processes used in the NLU stage ( §4.2 and §4.3), the dialogue manager and patient record querying ( §4.4) and the generation step ( §4.5).

Architecture of the dialogue system

The architecture of the system (Campillos-Llanos et al. 2015) is based on the modular schema of the RITEL interactive question-answering dialogue system for opendomain information retrieval [START_REF] Rosset | Integrating Spoken Dialog and Question Answering: the Ritel Project Proc. of InterSpeech[END_REF]. The RITEL platform is an in-frastructure for developing dialogue systems. We adapted the NLU engine and the processing functions for dialogue management and generation. Our contributions are the lexicon and models for the VP dialogue task. Our system has these modules:

• The terminology and linguistic management module provides the terminoontological and linguistic knowledge needed by the various components. • The natural language understanding (NLU) module analyses the user input. It recognises medical entities and question types, and includes spelling correction to deal with errors in user input. The user input is a turn and consists of one or more utterances. For instance, Hello. How are you doing? is a turn and consists of two utterances. • The dialogue manager interprets the results of the NLU. Based on the entity and question labels produced by the NLU, rules are applied to determine the semantic frame that best represents the type of user question and drives the construction of a query to the record. The required information is looked for in specific sections of the record. The dialogue manager then passes the query results and a suitable reply type to the generation module.

The dialogue manager takes into account and updates a dialogue history, using a shallow implementation of the information state approach [START_REF] Traum | The information state approach to dialogue management[END_REF]. This history keeps track of the interaction at each move, and is used to process ellipsis and anaphora (see examples in Figure 1). This is an important device to manage longer, more natural conversations than what a series of independent question-answer turns would provide. • The natural language generation module creates the output utterances. Instead of using predefined replies, the system relies on templates to generate new answers according to the contents of the patient record.

Figure 8 illustrates how an input question is analysed and processed to output a reply according to the VP record. After the spelling correction, the NLU module detects the terms in the user input and annotates the question and entity types. Then, the dialogue manager processes the semantic frame filled with the entity types in the input. The terminology management module looks for the corresponding data in the VP record, performing entity linking or normalisation if needed. If these data are found, the lexical module looks for a lay variant of the term (e.g. appendicitis operation for appendectomy). Lastly, the generation module applies the type of reply that corresponds, and the record items are instantiated in the generation template.

NLU: Spelling correction

For each out-of-vocabulary word, we attempt spelling correction. To handle common types of misspellings, we implemented a correction algorithm based upon [START_REF] Norvig | How to write a spelling corrector[END_REF]. We use morphological information, word length and corpus frequency of each word in the system together with edit-distance metrics to choose the most likely correction. The spelling corrector relies on a dictionary gathering all mono-word terms in the system. Each word form corresponding to the same We used a domain corpus to develop the dictionary of the spelling corrector. The VIDAL R company provided for the project its dictionary of medical drugs, which is a reference book used by health professionals and is copyrighted (therefore, not freely available).9 Texts include data on common disorders and medications in French, and the corpus size amounts to 7,678,363 tokens.

Spelling correction is a useful component but is not the focus of the present paper, hence we do not expand further on its description.

NLU: Entity recognition and semantic annotation

The NLU module performs the following tasks:

• Entity recognition and semantic annotation: we use rules and domain lists and gazetteers semi-automatically curated for each entity type. More detail on these lists were provided in §3.3 and Table 3. Table 5: NLU rules (actual code) for dialogue acts (greetings and acknowledging), questions on hours, and entity types (diseases and surgeries). Labels start with the character; & indicates a sub-expression that can be reused in different locations; | indicates alternation; ∧, start of string; <!, left negative lookahead; ?!, right negative lookahead; ?:, non-substituting grouping; .{0,n}: 0-n matching; ∼, word lemma tional dialogue acts (e.g. greetings) and classify question types (e.g. yes-no questions bear label Qyesno).

In this step, we use Wmatch rules [START_REF] Van Schooten | Handling speech input in the Ritel QA dialogue system[END_REF][START_REF] Galibert | Approaches and methodologies for automatic Question-Answering in an open-domain, interactive setup[END_REF]. Wmatch is a regular expression engine of words for natural language processing. It Table 6: Types of input questions uses domain lists to detect words in user input and allows defining local contexts for matching and semantic categorisation. Each of the 149 entity types, dialogue acts and question types is defined by a grammar that is expressed by a combination of abstract rules and gazetteers. Each grammar generates a complex graph that is used ultimately at run time. Table 5 shows sample rules. Two computational linguists, with the expertise of a senior researcher, developed the rules in an iterative process of analysing interaction logs and refining matching contexts.

With regard to medical entity types, we distinguish two levels of specialisation: general (the top-level entity types of the domain, such as surgery gen) and specific (the descendants of these top-level types, such as surgery spec). Patrick and Li's taxonomy also differentiates general and specific clinical questions [START_REF] Patrick | An ontology for clinical questions about the contents of patient notes[END_REF]), but we did not adapt it to our knowledge model for the dialogue task due to the different application setting.

We consider two variants of question types: Wh-questions (open questions) and yes-no questions (polarity questions) [START_REF] Quirk | A comprehensive grammar of the English language[END_REF]. This distinction is needed to determine the type of reply (i.e., yes-no questions are replied with yes or no).

The resources used in the NLU step vary according to the type of entities in input questions (see Table 6):

• Questions on general entities: Lists of top-level entities (i.e. towards the top of the hierarchy or ontology) referring to a entity type in the domain: e.g. operation belongs to the entity type surgery gen, and disorder, to the entity type disease gen. • Questions on specific entities: specific entities referring to a more detailed entity type in the domain. Two types of queried entities may appear depending on the type of question topic:

-Classes of entities: e.g. cardiovascular disease (disease spec).

-Subclasses of entities: e.g. appendectomy (surgery spec) or hypertension (disease spec).

In the NLU step, lists and rules aim at balancing both precision and recall to consider different term and question structures or spelling variants. To increase precision, we needed comprehensive lists of terms and rules defining precise matching contexts. To improve recall, we expanded term lists (e.g. by including frequently misspelled words) and relaxed the context of some rules-the less specific the con-text, the higher the recall. During system development, we removed noisy terms in lists and fine-tuned greedy matching rules.

Dialogue manager and patient record querying

At each dialogue move, the dialogue manager interacts with the lexical modules. First, the dialogue manager processes user input according to the semantic frame from the NLU step. In addition, the information state module updates the input content representation dynamically according to the current dialogue state. The reference of an anaphoric pronoun or an elliptic element is interpreted according to the previous dialogue state. For example, in the sample dialogue of Figure 1, the system interprets the ellipsis of the medical term in since when? as the symptom expressed in the previous reply (shortness of breath). This allows the system to manage the semantic interpretation of user input in context.

To query the record, inflected forms of entities are transformed to a base or canonical form (e.g. the singular noun or the infinitive verb form) by using the lexicons in the linguistic model. Medical entity verbs (e.g. Have you bled? ) undergo some steps of lemmatisation (bled → bleed ) before the base form is mapped to any variant (e.g., bleed → hemorrhage). Multiword entities also need another step to remove some pronouns and obtain a canonical form: e.g. Respirez-vous avec difficulté ? ('Are you breathing with difficulty?') is reduced to the base form respirer avec difficulté ('breathe with difficulty'); then, this form can be mapped to a mono-or multi-word variant term in the patient record (e.g. shortness of breath or dyspnoea).

In the patient record query step, postprocessed entities are dynamically looked for in the record. The dialogue manager uses the entity type to restrict the search for data in the corresponding record section. For example, a question on a disease is looked up in the section concerning disease history.

There is a continuum between types of entities and questions types (as exposed in §4.3). Their nature (general or specific) affects the size of processes and resources for querying the record. At one extreme, questions on general entities only require an accurate identification of the entity type. For example, a question such as What diseases do you have? requires identifying diseases as a generic term (label disease gen). At the other extreme, questions on specific entities also demand entity linking (also called entity normalisation) to check whether the input entity and that in the record refer to the same concept. For example, a question such as Do you have tension problems? requires labelling tension as symptom, and managing term variants when checking these data in the record (e.g. tension problems ↔ high blood pressure). Questions on a class of entities require matching this class with any of its subclasses in the record. If a user asks a question such as Do you have cardiovascular diseases?, which contains a term referring to a broad class, we need to map it to a specific disease in the record (e.g. high blood pressure, a subclass of cardiovascular disease). To do so, we use ontological relations.

Algorithm 1 Pseudocode of function to match terms through UMLS CUIs. The function returns True when an input term and a term in the patient record refer to the same UMLS concept. Dictionaries in the termino-ontological model are selected according to a semantic code (ANAT, DISO, PROC or T121) corresponding to the input entity type. The linguistic model model is used to get the lemma of the input word form. [START_REF] Levenshtein | Binary codes capable of correcting deletions, insertions, and reversals[END_REF] or any of the resources defined in the termino-ontological model. The specific lexicons and/or ontology knowledge to be used rely on the entity type of each term. Terms whose entity types are related to pathologies (e.g. label disease spec or symptom) are looked up in lexicons of term variants extracted from the UMLS DISO group. That way, hypertension can be mapped to high blood pressure or hypertensive disorder. Likewise, terms belonging to procedure entity types (e.g. appendectomy, label surgery spec) are looked up in lexicons with variants extracted from the UMLS PROC group. The input terms to be matched with terms in our lexicons belong to the same entity type; variants are not expected to be found among terms of other entity types. This restriction of the search space speeds up the dictionary look-up process. By focusing on the relevant parts of the record, the cor-Algorithm 2 Pseudocode of function to query surgery terms in the patient record. The function returns True when an input term is found in the patient record. Dictionaries and ontology relations are used from the termino-ontological model. Function Match Term through CUI (see algorithm 1) is used to match terms through UMLS CUIs. The linguistic model is used to match terms through affixes and roots. return false 24: end function rection of answers is also expected to increase. Algorithms 1 and 2 are pseudocode examples of how these queries are implemented.

In this step, the correction of answers depends on the ability of the system to map input terms to items in the patient record. This in turn depends on the coverage and quality of the linguistic and termino-ontological resources of the system.

Generation

Resources for generating replies cover three types of information:

• Linguistic data for gender/number agreement: e.g. fever is feminine in French.

We use DELAS-type [START_REF] Courtois | Un système de dictionnaires électroniques pour les mots simples du français[END_REF]) dictionaries with inflectional information (Table 2, 1.1). • Correspondences between 3rd and 1st person verb forms, to output the content expressed in the record (in 3rd person) with the patient's viewpoint (1st person): e.g. The patient has a fever → I have a fever. We clustered pairs of verb forms from the mentioned dictionaries (Table 2,1.2). • Lay variants of terms: e.g. appendectomy → appendicitis operation. These were selected by processing domain corpora of different degrees of technicality [START_REF] Bouamor | Transfer-based learning-to-rank assessment of medical term technicality[END_REF] and manual revision (Table 2, 1.3 and1.4).

Evaluation methods and results

We present our evaluation goals and criteria ( §5.1) and explain how we gathered evaluation data ( §5.2). Next, we detail our evaluation methods and results for different aspects, and we end with a discussion of results ( §5.8).

Overview of evaluation principles

One of the difficulties in evaluating dialogue systems lies in the lack of benchmarks and comparable or agreed standards [START_REF] Paek | Empirical methods for evaluating dialog systems[END_REF]. Frameworks such as PARADISE [START_REF] Walker | PARADISE: A framework for evaluating spoken dialogue agents[END_REF]) established a foundational methodology, especially with regard to distinguishing objective and subjective metrics-or performance and usability [START_REF] Roy | Methods for evaluating software architecture: A survey[END_REF]. Human judgements on dialogue performance are thus relevant and necessary to complement other measures. We designed and ran both quantitative and qualitative evaluations of system performance with a focus on its vocabulary coverage. Evaluating at these two levels provides us with an overall picture of how objective metrics reflect subjective assessments [START_REF] Paek | Empirical methods for evaluating dialog systems[END_REF]. More specifically, we performed the following evaluations:

• A quantitative evaluation of the natural language understanding unit ( §5.3).

• A quantitative evaluation of dialogue management, i.e. dialogue control and context inference ( §5.4). • A qualitative evaluation of the overall functioning of the system and of its usability (end-user satisfaction) ( §5.5). • A quantitative evaluation of the system's vocabulary coverage with regard to processing new cases ( §5.6). • A qualitative evaluation of vocabulary usage in the task ( §5.7).

Collection of interaction data

During system development, we collected interaction data by having computer science students and researchers (n=32) interact with the system (3 VP cases) and evaluate it through an online interface and questionnaire. 10 For the evaluation presented here, in the following rounds of tests, medical students and doctors (n=39) interacted freely with the system and then evaluated it. We used 35 different VP cases; each case was tested by an average of 3.74 users (±2.8; minimum number of different users per case=1; maximum=13). We gave instructions concerning the types of dialogue acts the system can process (e.g. avoid instructions or out-oftask requests such as Give me your telephone number ). Table 7 includes a sample of the instructions provided. Note that we did not provide examples of question formulations, removing the risk of priming effects.

You are a medical doctor and a 41 year old woman, married, arrives for a preoperative assessment. You need to get information about the following aspects:

• Patient's demographic data: weight, height, profession, family... • Lifestyle: eating habits and diets, addictions, physical activities... • Medical history: allergies, diseases, surgery history, obstetric history, treatments... • Patient's symptoms, treatments and observations

The system cannot reply (or replies wrongly) to:

• Instructions or prescriptions: e.g. I will examine you, I will prescribe you a medicine • Complex or coordinated questions: e.g. Do you smoke or drink?, How much and since when?

Try asking your questions in a varied way and avoid abbreviations if possible. You can fill in an evaluation questionnaire whenever you want by clicking on the button Evaluate the system.

Table 7: Instructions available on the evaluation interface

The data reported in this evaluation were collected between March 2016 and February 2018. During the development tests with a small set of cases, we collected from computer science students and researchers around 1,987 pairs of turns, i.e. user input and system reply (a total of 3,756 turns, 11,960 tokens in user input). Users with this profile have more knowledge of the human-computer interaction limits and helped us improving system's response behaviour. Dialogues they conducted provided us with variants of question types before real end-users evaluated the system. For the evaluation presented here, we gathered from 39 medical doctors a total of 8,078 turns in 131 interaction dialogues (21,986 tokens of user input and 21,921 tokens in replies). After manually inspecting our data, we removed 149 pairs of turns (3.7 per cent) corresponding to out-of-task questions (e.g. What is your favourite colour? ) or declarative statements the system is not expected to answer (e.g. Please give me reasonable replies). We are aware that some types of declarative statements are important in doctor-patient interactions, e.g. to show empathy or counsel the patient (e.g. You should stop smoking). The current version of the system includes rules to process some declarative statements related to the patient's additive behaviour. Table 8 breaks down our evaluation data (all collected turns) corresponding to medical users. Note that in some cases the system did not reply due to processing errors of the dialogue manager.

Users #D Turn pairs Words #T #T/D stdev #W #W/D

Methods

The natural language understanding module analyses the user input and as a result provides labels for its semantic analysis. To evaluate it, we used the standard metrics of Precision, Recall, F-measure, and Slot Error Rate (SER) [START_REF] Makhoul | Performance measures for information extraction[END_REF]). These were computed by counting the number of correct and incorrect entity types labelled; for wrong labels, we counted insertions (I), deletions (D) and substitutions (S) (Table 9). We also computed the Sentence Error Rate (SeER), which is the proportion of sentences with at least one error. The definitions of all these metrics are given in Table 10. A better performance is normally reflected in higher P, R and F-measures and lower SER and SeER values. We also evaluated the spelling correction module by counting the number of errors in dialogues. However, we found that most spelling errors affected grammatical words or items not requiring semantic annotation. We thus manually corrected the orthographic errors in the input and evaluated the NLU again. Our results showed that the F-measure did not largely improve (+0.4 per cent). This confirmed that spelling errors had a minor impact on the semantic annotation. Given those results and the space limits of this article, we do not report the evaluation of this module.

Results

We reported a preliminary evaluation based on 242 turn pairs in a previous article [START_REF] Campillos-Llanos | Managing linguistic and terminological variation in a medical dialogue system[END_REF]. Herein, we present the results based upon the data obtained to date from 39 medical users (around 4,044 turn pairs). Table 11 shows that the NLU achieves a high F-measure (95.8 per cent), balancing precision (96.8) and recall (94.9). Recall is lower than precision due to missing entity types and unannotated terms. The 6.1 per cent of Slot Error Rate implies that, on average, one entity type was incorrectly labelled every 16.4 labels. The 10.7 per cent of Sentence Error Rate means that, overall, one turn with incorrect labels occurred every 9.3 turns. Both these error rates are low. In terms of impact on dialogue flow, errors at this level made users reformulate their questions or change the topic of the dialogue.

Precision (P) is the ratio between the correct labels for entity types and all labels in the hypothesis:

P = C Hyp
Recall (R) is the ratio between the correct labels in the hypothesis and the reference labels:

R = C Ref F-measure (F)
is the harmonic mean of P and R; it is usually weighted with β = 1 and provides a global summary of the system performance:

F = (1 + β 2 )P R β 2 P + R = 2P R P + R
The Slot Error Rate (SER) is similar to the Word Error Rate and is used in information extraction tasks [START_REF] Makhoul | Performance measures for information extraction[END_REF]). The SER is the total number of annotation errors (I, S and D) divided by the total count of labels in the reference (insertions are not considered in the reference, i.e. the denominator):

SER = S + D + I Ref
The Sentence Error Rate (SeER) is the ratio between the sentences with at least one error and all of the correct sentences:

SeER = #W rong sentences #Correct sentences
The proportion (in per cent) of correctly-corrected spelling errors (CC) is calculated by dividing the number of CC errors and the sum of errors that were not corrected (NC ), wrongly well-corrected errors (WC ) and correctly-corrected errors:

P roportion of spelling errors (per cent) = CC (N C + W C + CC)
Table 10: Evaluation metrics for the natural language understanding evaluation 5.4 Quantitative evaluation of dialogue management

Methods

To evaluate the dialogue manager, we adopted the framework explained by Dickerson and colleagues, who evaluated a system developed for the same task and domain [START_REF] Dickerson | Evaluating a script-based approach for simulating patient-doctor interaction[END_REF]. We analysed the logs of user-system interactions and manually classified each turn pair as correct, incorrect, not understood or clarifica- Table 11: Evaluation of the entity detection (NLU). We report the average number of entities per turn annotated in system hypothesis, standard deviation (stdev ), minimum and maximum number of entities. Values for precision, recall, F-measure, SER and SeER are given in percentages, as well as figures between brackets tion requests [START_REF] Purver | On the means for clarification in dialogue[END_REF]. We define a correct reply as that providing both: 1) a coherent answer with regard to the user question; and 2) correct information from the VP record. Conversely, incorrect replies are those not succeeding at providing a coherent answer, or those giving erroneous information. Three of these criteria also map to those applied by Traum and his team [START_REF] Traum | Evaluation of a multi-party virtual reality dialogue interaction[END_REF]: (1) Correct utterances correspond to Traum and colleagues' Get response and Appropriate continuations;

Entities in reference

Entities in hypothesis

Average

(2) Clarification utterances correspond to Request for repair ; (3) Incorrect utterances correspond to Inappropriate response or continuation. A computational linguist analysed interaction logs and classified them; then, a subset of 350 (8.6 per cent) turn pairs, which were hard to interpret and classify, was double-checked by a senior computational linguist; finally, a consensus was reached. We computed the percentage of turn-reply pairs for which the annotations made by one researcher were confirmed by the senior researcher at the consensus stage. The agreement between both linguists was of 93.6 per cent.

Results

An analysis of the user-system interaction logs was performed on the data collected from non-medical users in the development stage, and showed that 72.9 per cent of the replies were correct. The analysis of the interaction logs collected from medical users is shown in Figure 9 and Table 12. Segment labeled 1 in Figure 9 stands for correct replies, which represents 74.3 per cent; segment 2, the ratio of incorrect replies (14.9 per cent); segment 3, the proportion of not-understood replies (7.8 per cent); and segment 4, the ratio of clarification requests (2.9 per cent). Performance across VP cases varied (standard deviation, stdev, of 9.5) due to the different number of dialogues conducted with each case, and also in relation to the medical specialities of the cases. We obtained the best results (93.8 per cent) with a VP case suffering from diarrhea, and poor results (53.6 per cent of correct replies) with a postpartum case from the obstetrics speciality; however, both of these were tested by only one evaluator. In our error analysis of the logs of the postpartum case, we noticed that some of the evaluator's questions referred to the patient's newborn. The dialogue manager provided wrong replies because these question types did not refer to the patient's medical condition, but to that of her newborn, and the system could not distinguish them. Among incorrect replies, about 37.8 per cent were due to errors in the dialogue manager and 26.2 per cent were caused by unforeseen question types (e.g., we did not prepare rules for questions on the patient's blood group). Among the not-understood replies, 48.2 per cent were caused by unforeseen question types and about 10.2 per cent were caused by missing variants of questions.

5.5 Qualitative evaluation of system performance and usability

Methods

Right after users interacted with the system, they filled in a questionnaire with questions using a 5-point Likert scale. The survey addressed the following aspects:

• Global functioning: an overall assessment of system performance.

• Coherence: adequateness of system answers in relation to user input.

• Informativeness: satisfaction with the information provided by the system.

• User-understanding: degree of comprehension of system replies by the user.

• System-understanding: system's degree of comprehension of user input.

• Speed: system quickness in replying.

• Tediousness: verbosity of information answered by the system.

• Answer concision: quality of replies in terms of length.

• Naturalness of replies: realism of the utterances produced by the system.

Results

The 131 questionnaires collected from medical users scored highly the degree to which users understood system replies (64.1 per cent of evaluators assessed it very good ) and the speed in providing an answer (very good, 69.5 per cent). The following aspects were in general considered good : overall performance (63.4 per cent of users), informativeness (62.6 per cent), coherence of replies (61.1 per cent), system understanding of input (56.5 per cent), concision of replies (45.8 per cent) and their (absence of) verbosity or tediousness (51.9 per cent). The naturalness of replies was scored as good by 45.0 per cent of participants; 29.8 per cent gave a neutral score, and 9.9 per cent assessed it as poor. There is still room for improvement for this and other aspects; lower scores, however, represented only a small proportion of users.

Figure 10 depicts the results of the evaluation through the 40 questionnaires collected from participants with computer science backgrounds (left) and the 131 questionnaires filled by medical doctors (right). Assessments were rather similar excepting slight variations regarding informativeness and system understanding (slightly higher for computer science users) or naturalness (slightly higher for medical users). These differences might be due to the more strict criteria applied by medical users and to the improvements made to the system between evaluation rounds.

Users provided free comments concerning aspects to be improved. Table 13 shows some of them (translated from French). Several users commented upon difficulties in getting more details after a general question. Sometimes the record lacks detailed information: this raises the question of what the system should answer if the user asks for such missing information. For example, some users asked for the patient's disease or symptoms, and after the system replied, they wanted to know specific observations, which were not present in the record. In that situation, currently, the dialogue manager gives an explicit answer (I cannot answer that question. This piece of information is not present in the record ). This does not always satisfy users due to the missing data or to the lack of naturalness of the reply (see Table 13).

Because medical users need accurate information from the patient, we chose to give a neutral reply when no data are available. Likewise, some context processing errors have hindered the correct interpretation of questions. The first case requires to improve the ergonomy of the system, the latter case requires to improve its robustness.

Negative comments Positive comments

Replies are very stereotyped (...) As soon as one goes out of the strict context of expected questions, the system is lost I have noticed its limitations, but it is often possible to reformulate to get a coherent answer since the system replies that it did not understand

The patient always said "I cannot answer to that question [There is no information in the record]", which makes the dialogue less natural For some questions, the patient only replied almost always the same thing, but apart from that, the dialogue is natural and fluid, the patient understood many things.

Sometimes not too much memory of previous question

System replies are fine and make it possible a fluent interaction I didn't have the impression that it was possible to link several questions, that is, to clarify certain answers

Very very coherent replies, some sentences where syntax was not completely correct (sometimes a verb is missing). The patient gives a lot of information anyway and the dialogue is fluid.

Table 13: A selection of positive and negative user comments in the qualitative evaluation (translated from French).

Quantitative evaluation of vocabulary coverage

Methods

We assessed how robust our lexicons are by comparing them to domain data not used for developing the system. Because no preexisting library of VP cases existed in French, we used the most similar source we could find. We collected 169 cases from Epreuves Classantes Nationales ('National Classifying Tests', hereafter ECN), which are used to prepare exams in medical universities. 11 We used the description of the case, not the feedback for students. Table 14 shows a sample. The procedure was as follows. We lowercased and tokenised the ECN texts; we removed numbers, dates, punctuation and stop words; and we expanded common abbreviations (e.g. mg → milligrams). Then, we compared the word types (i.e. different word forms, not tokens, which represent the occurrence of each type) in these texts against all the terms in the lexical resources used by the NLU, entity Monsieur B., 71 ans, consulte pour des douleurs abdominales et des épisodes de constipation. Il présente également des épisodes de selles molles. Son médecin traitant lui a parlé d'intestin irritable, ce qui ne l'a pas vraiment rassuré. Il est comptable retraité. Il a des antécédents d'hypertension artérielle et d'artérite pour lesquelles il prend un traitement par hydrochlorothiazide ( Ésidrex R ), amlodipione (Amlor R ), et acétyl salicylate de lysine (Kardégic R ). Il est très anxieux et prend régulièrement du bromazépam (Tranxène R ).

'Mr. B, 71 year old, consults for abdominal pain and episodes of constipation. He also presents episodes of soft stools. His family doctor has talked to him about irritable bowel disease, which has not reassured him at all. He is a retired accountant. He has a history of arterial hypertension and arteritis for which he takes a treatment with hydrochlorothiazide ( Ésidrex R ), amlodipione (Amlor R ), and lysine acetylsalicylate (Kardégic R ). He is very anxious and takes bromazepam (Tranxène R ) regularly.' Table 14: Sample case of the Epreuves Classantes Nationales (ECN, 'National Classifying Examination'), and its translation; ECN were used for evaluating the vocabulary coverage of the system with regard to new clinical cases normalisation and generation steps. We computed for each text the proportion of in-vocabulary and out-of-vocabulary words, and their average over all texts.

We also evaluated to what extent the ECN texts are different to the first cases used in system development. For this purpose, we compared the word types in the ECN cases to the word types in the initial cases and computed the percentage of ECN words that were not present in the development cases.

Results

Cases in our development set included 1,504 tokens (428 types), and two cases had several consultations. We evaluated the system's vocabulary coverage of the 169 new cases found in ECN texts (24,521 tokens, 4,112 types). These counts do not include dates and numbers, which we removed from these texts because they are not managed through dictionaries and hence raise no coverage problem.

We measured that 3,805 (92.5 per cent) of the word types occurring in the ECN texts did not occur in the development cases. This shows that the ECN texts used to test vocabulary coverage are really different from the cases used in system development.

The system's resources recognised an average of 97.85 per cent of ECN word types (stdev=1.91). A very low percentage of ECN word types were out of vocabulary (OOV) items (2.15 per cent), with an average of 2.34 per case (stdev=2.72).

Figure 11 shows that missing terms were mostly acronyms and abbreviations (e.g. . Other OOV words were names of people or locations (e.g. France), other words (e.g. rendezvous), and spelling errors (*oédème instead of oedème, 'edema'). Note that names of people and locations are not useful in the system, and are thus not included in our lexicons. Accordingly, the actual rate of OOV words is even lower than measured: resources covered an average of 98.09 per cent of ECN word types (stdev=1.86), and the average of OOVs per case in the ECNs was of 2.14 (stdev=2.67).

Qualitative analysis of vocabulary usage

Methods

To illustrate the difficulties in managing the variety of terms in our task, we provide a qualitative analysis of domain term usage in the interaction data. First, we analysed how medical evaluators used domain terms in dialogue logs. We only analysed dialogues in cases tested by more than one medical doctor (28 different cases). As an illustration, we focused on terms related to entity types of specific references to diseases (disease spec) and symptoms (symptom vb, which labels verb forms, and symptom). We examine here the actual occurrence of the terms observed in the interaction logs. Second, we analysed the term distribution in the corresponding VP records (only for those 28 cases). We did not include stop words nor the record section containing demographic data (e.g. proper names or civil status). For both aspects of the analysis, we obtained frequencies of usage of each item, and plotted them to examine their distribution across user interactions or records.

Results

In the interaction logs and patient records of the 28 analysed cases, we found 1,408 different tokens. Less than 60 (∼4 per cent) have a frequency over 10 and around 30 (∼2 per cent) occur in at least 10 cases. Figure 12 summarises our results. The frequency distribution of the terms for symptoms and diseases are represented in two plots above (A & B). Terms were analysed in the dialogues of all VP records (plot A) and we also counted the number of different cases where each term occurred in the interaction dialogues (plot B). Secondly, we plot the analysis of tokens in all A quick look showed that very low-frequency items are domain terms (e.g. cianotic or acetylleucine). This illustrates the difficulty of term detection in our task and justifies our approach based on rich terminological resources. The methods we propose enhance the system's ability to adapt to different cases and detect rare and unseen vocabulary items for a successful interaction. Table 15 breaks down the most frequent terms for specific references to symptoms (labels symptom and symptom vb) and diseases (disease spec) observed in the dialogues; and Table 16 reports the frequency analysis of tokens in the records.

Discussion

Our system is aimed at dealing with new cases and therapeutic areas. To achieve that, we rely on robust and comprehensive terminology components, which, as far as we know, are unparalleled in current VP systems [START_REF] Maicher | Developing a Conversational Virtual Standardized Patient to Enable Students to Practice History-Taking Skills[END_REF]. The method which consists in crowd-sourcing the evaluation of the answers of the dialogue system and the extending its question-answer database with the correct answer [START_REF] Rossen | Human-centered distributed conversational modeling: Efficient modeling of robust virtual human conversations[END_REF] does not seem scalable for extending the components of our system and its terminology. Because these variation phenomena are easier to process through thesauri, we rely on domain lexicons and ontological

Most frequent

Least frequent (selection) knowledge. Our approach is closer to that based on a taxonomy of questions [START_REF] Talbot | Natural language understanding performance and use considerations in virtual medical encounters[END_REF]; however, we use terminologies available in the UMLS Metathesaurus. Evaluation outcomes showed that the core lexical and terminological components seem stable and able to process new clinical cases. From a quantitative point of view, the NLU module achieved an F-measure of 95.8 per cent, balancing precision (96.8 per cent) and recall (94.9 per cent) when annotating entities in user input.

The test of vocabulary coverage brought out the few types of terms that occurred in unseen patient cases and were still missing in our lexicon: these only represented 2.16 per cent of terms in a collection of 169 descriptions of clinical cases, and most missing terms were acronyms and abbreviations. From a qualitative point of view, users who tested the system did not mention any error related to terminology needs.

According to the evaluations, the most important causes of failures in the dialogue manager are beyond terminology needs and might define the limits of a purely rule-and frame-based system. The difficulty of the task also accounts for processing failures of follow-up queries after general questions, especially in cases of missing information in the record. Medical doctors tend to ask general questions to begin to circumscribe a diagnosis; then, if the patient replies with the searched bit of information, they ask for more details. This requires both processing correctly the implicit information in the dialogue context and foreseeing all details to be queried: e.g. observations, descriptions (e.g. intensity) or temporal data related to a condition. The lack of pre-existing task-specific dialogue data hinders achieving a comprehensive coverage of question types, query variants and interaction contexts.

We would like to improve the naturalness of replies, especially those with long sentences and negative symptoms. The realism of responses depends on technical aspects as well as on how medical instructors input data.

The methods we propose should be valid for other dialogue tasks in other domains where rich lexical and/or ontological resources exist and a semi-structured database is available. That makes it possible to develop a system to collect interaction data, which can then be used in statistical or machine-learning approaches. Nonetheless, a key takeaway is the fact that, even when rich resources exist, these need an extensive effort of iterative filtering and task adaptation before production mode. As we explained, we created ad-hoc lexicons with lay variants or equivalences between noun terms and verbs (which are missing in the UMLS). Similar needs were reported when adapting the UMLS for concept indexing [START_REF] Nadkarni | UMLS concept indexing for production databases: a feasibility study[END_REF].

In terms of dialogue management, we contribute with an approach for querying the database at each dialogue move by considering the semantic content of each dialogue state.

Conclusion

In this paper we highlighted the difficulties raised by the terminological needs of a dialogue system that aims at providing natural language interaction in a simulated medical consultation context, robust enough for multiple clinical cases. We designed three models involved in such a dialogue task: a patient record model, the knowledge model for the task, and a termino-ontological model.

This work focused on the termino-ontological model, which manages terminological and linguistic resources. To populate the model, we adopted a comprehensive approach to lexicon and terminology collection. We collected term variants for domain concepts based on existing medical terminologies, which helped us structuring terms according to the concepts they describe. We compiled large dictionaries of inflectional and derivational word variants. These resources enabled the system to: 1) recognise a large number of entities in the NLU step; 2) handle general and specific entities in the NLU or dialogue manager modules; 3) perform entity linking, entity normalisation and hierarchical reasoning; and 4) give priority to lay variants in the generation step. The quality of the collected resources allowed the system to obtain a high vocabulary coverage when tested on a large number of unseen cases: the system proved stable for the task and robust enough to cope with the vocabulary of new cases. Our system stands out from current research on VPs by its ability to handle a large variety of clinical specialities and cases. We developed the system with 35 different records from 18 medical specialities.

A total of 32 non-medical users and 39 medical students and doctors evaluated the system. Overall, the majority of users evaluated it as good or very good in most dimensions. The evaluation also highlighted aspects that deserve further work. User comments in the evaluation reflected these shortcomings, especially regarding follow-up utterances after general questions, improving the naturalness of some replies and handling missing information in the patient record.

We make the evaluation corpus used in this work available for the community.12 In the context of a lack of dialogue resources, especially in the medical domain, we believe these data will be useful for moving ahead in the field. Now that we have collected interaction corpora, we are focusing our research on machine-learning based methods. Specifically, we have begun exploring the classification of question types according to the system needs, i.e. the system's current rules or an alternative processing strategy (Campillos-Llanos, Rosset and Zweigenbaum 2017). Designing such a fallback strategy is our research interest, with a view to providing a satisfactory answer when a question cannot be handled with the current approach. We estimate that a subset of system's wrong and not-understood replies (overall, less than 20 per cent of replies) would need a fallback strategy.

The system has been adapted to English and Spanish, following the same design procedures and models (e.g. entity types scheme) as explained. Domain lists contain over 116,000 terms in English and 103,000 in Spanish; and dictionaries gather over 1,886,000 word/concept entries in English and 1,428,000 in Spanish. A thorough data collection and evaluation are needed to improve these versions of the system. 13

Fig. 3 :

 3 Fig. 3: Patient record model

Fig. 5 :

 5 Fig. 5: Knowledge model for the dialogue task (sample). The upper part shows the relation between questions on the patient record, questions types and topics, and entity types. The lower part shows how system components instantiate this model. NLU: natural language understanding; NLG: natural language generation.

Fig. 7 :

 7 Fig. 7: Relation between entity types, concepts and terms. Linguistic knowledge processes affixes or morphological variants; ontological knowledge classifies terms into entity types; UMLS terms are indexed by Concept Unique Identifiers (CUIs)

  photoallergy

Fig. 8 :

 8 Fig. 8: Example of question-reply processing

•

  Dialogue act and question type annotation: we use rules to label conversa-# Grammar and rules for acknowledging acknowledging: ( (?: ˆok | ˆokay | all right | yes ) (?! "?" ) ); # Grammar and rules for greeting greeting: ( how are you doing | how are things going | what's up ) # Contexts and words for time expressions (hours) &hour: half-hour | half-hours | ∼hour | ∼minute | noon | midnight | o' clock | pm | am | p.m. | a.m. ; # Contexts and words for time expressions (parts of the day) &part of day: (<! good) ( morning | night | evening ) | diurnal ; # Grammar and rules for questions on hours Qhour: ( (?: what | which ) .{0,2} time | (?: at | around ) about? (?! between | from ) %cardinal(0,24) &hour | &part of day ); # Specific terms of surgeries (appendectomy, tonsillectomy...) &surg spec trms: include L I S T O F S U R G E R I E S ; # General, unspecified terms referring to surgical procedures &surg gen trms: intervention | operation | procedure | surgery ; # Terms of anatomy (appendix, chest, knee, ligaments, tonsils...) &anatomy: include: L I S T O F A N A T O M I C E N T I T I E S; # Grammar and rules for specific types of surgery procedures surgery spec: ( &surg spec trms | &surg gen trms of &surg spec trms | (?: &surg gen trms | ∼operate | ∼remove ) .{0,4} &anatomy ) ) ; # Specific terms of disorders (cancer, diabetes, hypertension...) &dis spec terms: include L I S T O F D I S E A S E S; # General, unspecified terms of disorders &dis gen terms: disease | disorder | illness | pathology ; # Grammar and rules for specific types of disorders disease spec: ( (?: &disease gen terms .{0,5} &anatomy ) | &dis spec terms );

  ) Total of correct and wrong labels in the system annotation Reference (Ref ) Total of correct and wrong labels in reference annotation Correct (C ) Number of correct labels in the hypothesis (true positives)

Fig. 9 :

 9 Fig. 9: Dialogue manager evaluation

Fig. 10 :

 10 Fig. 10: Qualitative evaluation by non-medical users (left) and medical users (right)

Fig. 11 :

 11 Fig. 11: Breakdown of types of out-of-vocabulary words in ECN texts

Fig. 12 :

 12 Fig. 12: Qualitative analysis of term distribution in dialogues and VP records

Table 1 :

 1 Sample clinical record: the format used is YAML Fig. 4: Dedicated interface to input data on the virtual patient record Medical instructors prepared patient records through a dedicated interface (Figure

Table 2 :

 2 Types of resources used in each processing step for managing linguistic and terminological variation (and examples of each type), listed according to linguistic aspects (morphology and synonymy) and termino-ontological aspects (concepts and ontology relations)

	Morphology (LINGUISTIC KNOWLEDGE)	1.1. Gender/number/PoS data antihypertensive, .N:ms (noun, masculine, singular)	1.2. Verb correspondences [the patient] has ↔ [I] have	Concepts (ONTOLOGICAL KNOWLEDGE)	1.3. Scientific/lay correspondences (with CUIs) high blood pressure ↔ hypertension (C0020538)	1.4. Scientific/lay correspondences (no CUIs) per os ↔ oral
		Generation

  Table 2, 2.5.1): ANAT semantic group • Health states (i.e. diseases/symptoms, Table 2, 2.5.2): DISO semantic group • Pharmacologic substances (Table 2, 2.5.3): T121 semantic type • Therapeutic/diagnostic procedures (Table 2, 2.5.4): PROC semantic group

Table 4 :

 4

Resources for managing linguistic and terminological variation has direct morphology, to map entities referring to surgeries with the structure disease spec + surgery spec (hernia ↔ inguinal herniorrhaphy).

Table 8 :

 8

	stdev

Data collected during the evaluation by medical doctors; #D: count of dialogues; #T : count of turns; #T/D: average turns per dialogue; #W : count of words; #W/D: average words per dialogue; stdev : standard deviation 5.3 Quantitative evaluation of NLU

Table 9 :

 9 Evaluation schema for the NLU module (with examples)

Table 12 :

 12 Dialogue manager evaluation (medical doctors)

		Count Average per cent (stdev) Min Max
	1. Correct	2,835	74.3 (±9.5) 53.6	93.8
	2. Incorrect	628	14.9 (±6.3)	0.0	31.6
	3. Not understood	315	7.8 (±5.3)	0.0	25.0
	4. Clarification request	117	2.9 (±2.7)	0.0	11.5
	Total evaluated	3,895	100

Table 15 :

 15 Most and least frequent terms (translated from French) observed in dialogues with 28 different VPs for entity types of symptoms and diseases. Note that some terms cannot always be assigned to symptom or disease, and the dialogue context or VP record are needed to make a distinction. For example, vertigo is most commonly a symptom, but it might be a chronic condition in a specific case.

		Term	Freq # cases		Term	Freq # cases
	Symptoms	ache pain pains fever lost weight	112 96 65 45 29	25 14 19 21 12		cramps cyanotic itching distress difficulty urinating	1 1 1 1 1	1 1 1 1 1
	Diseases	injury hypertension 22 24 diabetes 16 psoriasis 12	1 13 11 2	abuse accident cerebrovascular accident 1 1 1 accidents 1	1 1 1 1
		pancreatitis	7	1		alzheimer	1	1
		Most frequent tokens		Least frequent tokens (selection)
		Token	Freq # cases	Token	Freq	# cases
		patient (masc) 483	21		abdomen	1	1
		patient (fem) 199	10		accelerates	1	1
		years	120	26		acetate	1	1
		months	60	19 acetylleucine 1	1
		day	57	23		acyclovir	1	1

Table 16 :

 16 Most and least frequent tokens (translated from French) observed in the 28 analysed VP records (only a selection of the least frequent tokens is shown)

More information about the project is available at: https://pvdial.limsi.fr/

http://www.i-human.com/

The OSU VP Project: http://128.146.170.201/WEBGL/JackWilson/

The system can be tested at: https://prod.standardpatient.org/

http://www.virtualpeoplefactory.com/Classic/Home

We thank the doctors from the Angers Medical School who recorded the sessions. Since the partner team collected the data, we do not know the number of recorded actors.

http://anglaismedical.u-bourgogne.fr/

https://ansm.sante.fr/

If free texts are needed for a similar domain, a corpus of the European Medicines Agency is available at: http://opus.nlpl.eu/EMEA.php

Available at: https://pvdial.limsi.fr/data/PG-logs-eval.zip

Acknowledgements

This work was funded by BPI through the FUI Project PatientGenesys (F1310002-P) and by SATT Paris Saclay through project PVDial. We greatly thank all the evaluators who tested and assessed the system, and we thank VIDAL R , who provided a domain corpus of medications and disorders. We also thank Brigitte Grau and the anonymous reviewers for their valuable remarks on the manuscript.