Jim Hunter

Albert Gatt

François Portet

Ehud Reiter

Somayajulu Sripada

Using Natural Language Generation Technology to Improve Information Flows in Intensive Care Units

In the drive to improve patient safety, patients in modern intensive care units are closely monitored with the generation of very large volumes of data. Unless the data are further processed, it is difficult for medical and nursing staff to assimilate what is important. It has been demonstrated that data summarization in natural language has the potential to improve clinical decision making; we have implemented and evaluated a prototype system which generates such textual summaries automatically. Our evaluation of the computer generated summaries showed that the decisions made by medical and nursing staff after reading the summaries were as good as those made after viewing the currently available graphical presentations with the same information content. Since our automatically generated textual summaries can be improved by including additional content and expert knowledge, they promise to enhance information exchange between the medical and nursing staff, particularly when integrated with the currently available graphical presentations. The main feature of this technology is that it brings together a diverse set of techniques such as medical signal analysis, knowledge based reasoning, medical ontology and natural language generation. In this paper we discuss the main components of our approach with a critical analysis of their strengths and limitations and present options for improvement to address these limitations.

INTRODUCTION

The modern intensive care unit (ICU) is data rich. Most bedside computers are equipped with software that helps medical staff to record, retrieve and visualize patient-related information. Several megabytes of physiological data (e.g. heart rate), as well as laboratory results and the results of other investigations (such as Xray results), are recorded per patient per day. Medical staff need to assimilate these data rapidly to provide real-time patient care. In the healthcare sector there is increased interest in deploying technology to deal with this information overload. Over several years, we have been working closely with medical staff in a neonatal ICU to understand how to achieve more effective use of patient data in order to improve the quality of patient care. Our studies led us to the following hypotheses: 1. Timely and accurate information flow between medical staff is the key to better patient care. Nurses and doctors require tools that accurately summarise the key facts about patients rather than expert systems that generate clinical recommendations. 2. The informational requirements of these summaries vary between the different groups of people involved in health care such as nurses, doctors and even family and friends. While 1 Department of Computing Science, University of Aberdeen, UK. email: j.hunter@abdn.ac.uk nurses need information that helps them to make detailed patient care plans, doctors require information to support clinical decision making. Family and friends have more variable requirements depending on how close they are to the patient. 3. Staff roles in ICUs are well defined and any new technology has to suit the work flows associated with existing staff roles [START_REF] Strople | Can technology improve intershift report? What the research reveals[END_REF].

This paper presents an extensive programme of work whose ultimate goal is the automatic generation of summaries in natural language (text) of the history of a baby in a neonatal ICU over a period of several hours. In the rest of this introduction, we set out the background to this endeavour. In Section 2 we describe our initial prototype; the outputs from this system have been evaluated experimentally. This evaluation and its results are presented in Section 3; full details can be found in [START_REF] Van Der Meulen | When a Graph is Poorer than 100 Words: A Comparison of Computerised Natural Language Generation, Human Generated Descriptions and Graphical Displays in Neonatal Intensive Care[END_REF]. We conclude with a discussion of what has been achieved so far and the challenges for the future.

Data-intensive clinical environments

Apart from clinical environments in which imaging plays a central role, the largest volumes of clinical data are generated where several physiological variables are measured at high frequencies (say once per second) over considerable periods of time. The data are collected for good reason, in that when correctly interpreted, they enable the tracking of the clinical state of the patient over time so that action can be taken to rectify any undesirable deviations.

For example, in an ICU, up to ten physiological variables (including heart-rate, blood pressure, O 2 and CO 2 saturations, and temperatures) may be measured continuously every second (i.e. almost a million measurements per day). In addition there will normally be a number of additional discrete data items which are entered sporadically: laboratory results, blood gases, daily organ failure scores, medication, equipment settings, patient observations, etc. The example from a neonatal ICU shown in Figure 1 illustrates many of the characteristics of data from such environments -they are multi-channel, voluminous, noisy and full of artifact.

Even in intensive care, where the ratio of patients to staff is low, medical errors are not uncommon. Errors often result from missed symptoms and signs, or a lack of appreciation of their importance which may be due to staff ignorance, to attentional or informational overload. Presenting these data in an effective way is central to informed medical decision making.

Normal practice is to present the physiological variables graphically as shown in Figure 1. However previous research [START_REF] Cunningham | A randomised control trial of computerised physiological trend monitoring in an intensive care unit[END_REF] showed that displaying data in this way does not automatically lead to improved patient care. One of the reasons for this was established by the Cognate project [START_REF] Alberdi | Expertise and the Interpretation of Computerised Physiological Data: Implications for the Design of Computerised Physiological Monitoring in Neonatal Intensive Care[END_REF], which showed that junior staff (who are responsible for most of the immediate care of the baby) spend a small fraction (about 5%) of their time looking at the displays. Cognate concluded that if the information present in these complex multi-channel time-series was to be fully utilised, some form of additional processing was essential.

In the Neonate project, a follow-up to Cognate, it was hypothesised that reporting information textually would be more effective than the graphical displays because textual descriptions, by virtue of their structure, can present a coherent picture of all the relevant information, while graphical displays require their user to filter out a lot of irrelevant detail in order to establish the relatively few important facts about the patient state. To test this hypothesis, an experiment was carried out to compare the effect on decisionmaking of two different modes of presentation [START_REF] Law | A Comparison of Graphical and Textual Presentations of Time Series Data to Support Medical Decision Making in the Neonatal Intensive Care Unit[END_REF]. One mode was graphical (as in Figure 1); the other was a summary of the data in natural language written by expert clinicians. The data periods presented had durations of about 45 minutes. This experiment demonstrated clearly that participants took better decisions when informed by the textual presentation. However it took considerable effort and involved a considerable amount of time for the medical experts to generate the text. If the benefits of textual summary are to be exploited, the texts need to be generated automatically. This can be achieved by mean of recent advances in data-to-text technology.

Data-to-text NLG

Natural Language Generation (NLG) is an area of research focused on developing techniques for automatically generating natural language (such as English) descriptions of non-linguistic information [START_REF] Reiter | Building Natural Language Generation Systems[END_REF]. An implicit assumption of most NLG techniques is that the non-linguistic input information comes from knowledge bases with well-defined semantics. In practice, however, in most application domains where automatic textual descriptions are desperately required, such knowledge bases do not exist. The dataintensive clinical environments described in the previous section generate large amounts of clinical data that are not structured into logical forms in a knowledge base. Data-to-text NLG is a recent extension to traditional NLG to allow such naturally occurring data to be described linguistically.

Since the pioneering work of Kukich [START_REF] Kukich | Design and Implementation of a Knowledge-Based Report Generator[END_REF] in summarising stock market data, various applications of data-to-text NLG have been reported in the literature. Yu et al. [START_REF] Yu | Choosing the Content of Textual Summaries of Large Time-Series Data Sets[END_REF] reported an NLG system that produces textual descriptions of time series data from an operational gas turbine. Sripada et al [START_REF] Sripada | Lessons from Deploying NLG Technology for Marine Weather Forecast Text Generation[END_REF] reported an NLG system that produces textual weather forecasts for the offshore oil industry. This system has actually been deployed in the industry to produce drafts of marine weather forecasts for supporting oilrig operations in the North Sea. User evaluation studies found computer generated forecasts to be more consistent in their language and therefore of better value than the expert written forecasts. Sripada and Gao [START_REF] Sripada | Linguistic Interpretations of Scuba Dive Computer Data[END_REF] reported a system that described data recorded by a scuba (Self Contained Underwater Breathing Apparatus) dive computer used by recreational scuba divers.

In the Medical domain, the recent CLEF project [START_REF] Hallett | Structural variation in generated health reports[END_REF] aims at generating summaries of multiple text-based health reports. In [START_REF] Cawsey | The evaluation of a personalised information system for patients with cancer[END_REF], the authors describe a system that dynamically generates hypertext pages that explain treatments, diseases, etc related to the patient's condition, using information in the patient's medical record as the basis for the tailoring. Perhaps the most successful medical data-totext applications have been tools that (partially) automate the process of writing routine documents, such as the Suregen-2 system [START_REF] Hüske-Kraus | Suregen-2: A Shell System for the Generation of Clinical Documents[END_REF], which is regularly used by physicians to create surgical reports; see [START_REF] Hüske-Kraus | Text Generation in Clinical Medicine -a Review[END_REF] for a review of text generation in medicine. However, the complete summarisation of ICU data is more complex, involving the processing of time series, discrete events, and short free texts, which seems not to have been done before.

BABYTALK PROJECT

The studies discussed above led us to conclude (i) that textual summaries of ICU data might have a significant role in decision support, and (ii) that the automatic generation of textual summaries of complex time series data was feasible. A preliminary study [START_REF] Sripada | Summarising Neonatal Time Series Data[END_REF] demonstrated the problems specific to the ICU domain. The main challenge in developing a computer application for the ICU context that delivers the required functionality is bringing together a diverse set of techniques that were originally developed quite independently -medical signal analysis for processing the raw physiological data, knowledge based reasoning to integrate and interpret the results of signal analysis along with other discrete data and finally the application of NLG to generate the required descriptions. These issues are being tackled in the BabyTalk project.

Example data

Input to our system consists of several channels of physiological data such as heart rate, blood pressure, and temperature. In addition to these continuous data, discrete data about equipment settings, laboratory results and actions taken by medical staff were also available to our system. Figure 1 shows a window of continuous channel data with annotations of discrete data. Independently, our human experts produced their own summaries of the data; Figure 2 shows the text they generated for the data shown in Figure 1. An example of the automatically generated output of our system for the same data period is presented in Figure 3.

Architecture

The main architecture of the prototype is presented in Figure 4. All of the modules in the architecture are driven by information stored in the knowledge base of the NICU Ontology (1). While developing ontology-driven systems is nothing new in both medical and NLG applications, developing an ontology that captures the knowledge required by both of these components is one of the novel features of this architecture. Moreover, as shown in the figure, the ontology is programmatically integrated into the system.

The system creates a summary of the clinical data period in four main stages. All the terms used to describe the discrete events are related to our ontology which is still being extended to include about 1800 concepts. To enable future sharing of this valuable knowledge source we synchronised our ontology with UMLS, a meta-thesaurus that brings together several popular medical taxonomies (such as SNOMED-CT, WHO classification). The first stage of the processing is Signal Analysis (2) which extracts the main features of the physiological time series (artifacts, patterns, and trends) using modelling based on a baby's physiological values, auto-regressive filtering, and adaptive bottom-up segmentation techniques. Data Interpretation (3) performs some temporal and logical reasoning to infer higher medical abstractions and relations ('re-intubation', "A causes B", etc.) from the signal features and the clinical observations using an expert system linked to the ontology. From the large number of events generated, Document Planning (4) selects the most important based on an importance factor either determined by experts (e.g. a surgical operation is must always be reported) or computed from the signals (e.g. the importance of some patterns depends on their duration). The most important events are then structured into a tree. Finally, Microplanning and Realisation [START_REF] Hunter | Workshop on Intelligent Data Analysis in Medicine and Pharmacology and Knowledge-Based Information Management in Anaesthesia and Intensive Care[END_REF] translates this tree into coherent text. Details of all these stages are available in [START_REF] Portet | Automatic Generation of Textual Summaries from Neonatal Intensive Care Data[END_REF]. Over the next 24 minutes there were a number of successive desaturations down to 0. Fraction of Inspired Oxygen (FIO2) was raised to 100%. There were 3 successive bradycardias down to 69. Neopuff ventilation was given to the baby a number of times. The baby was re-intubated successfully. The baby was resuscitated. The baby had bruised skin.

Blood gas results received at 16:45 showed that PH = 7.3, PO2 = 5, PCO2 = 6.9 and BE = -0.7.

At 17:15 FIO2 was lowered to 33%. TcPO2 had rapidly decreased to 8.8. Previously T1 had rapidly increased to 35.0." There are several ways of evaluating the quality of computer generated texts, including post-editing [START_REF] Sripada | Evaluation of an NLG System using Post-Edit Data: Lessons Learnt[END_REF] (where one measures the amount of editing required to render the text acceptable to a user) and detailed comparison with human generated texts based on the same input data. However, we are strong believers in taskbased evaluation, where end-users are asked to perform a meaningful task under different conditions; a judgment is then made on the extent to which the task was carried out satisfactorily -any differences are attributed to the conditions under which the task was performed.

Experiment

The participants in our experiment were nurses and doctors from the neonatal ICU in Edinburgh with different levels of experience: 7 junior nurses and 8 senior nurses, 8 junior doctors and 8 senior doctors. They attended three separate sessions during each of which they had to look at the presentation of data from each of 8 scenarios and indicate which action or actions they would take at the end of the period; the actions were selected from a set of 18 possible actions. The experiments took place away from the ward and suitable practice was provided each time before the test began.

For each scenario a participant was given some information about the baby at the start of the period and the data for the period was presented graphically or as text (human-authored or computergenerated). Participants were under some time pressure (three minutes per scenario), both to simulate the pressures that arise on the ward and to ensure that the time that the participant was away from her duties was predictable.

Considerable care was taken to ensure that the human-authored text was purely descriptive i.e. that it did not interpret the data in a way which would have given a definite hint to the participant as to the correct action.

Further information about all aspects of the experiment (including more detailed results) is given elsewhere [START_REF] Van Der Meulen | When a Graph is Poorer than 100 Words: A Comparison of Computerised Natural Language Generation, Human Generated Descriptions and Graphical Displays in Neonatal Intensive Care[END_REF].

Results

For each scenario our experts judged some actions to be appropriate, some to be inappropriate and the remainder to be neutral. Each participant was scored on each scenario, the score consisting of the percentage of the appropriate actions that they selected minus the percentage of inappropriate actions that they selected. For each condition an average score over the eight scenarios was calculated.

Subjects from all the groups performed better with humanwritten textual summaries (mean score = 0.39), compared to their performance with graphical presentations (mean score = 0.33); note that graphical presentation is the currently available form of data presentation to medical staff on the ward. This result is in agreement with the earlier experiment reported in [START_REF] Law | A Comparison of Graphical and Textual Presentations of Time Series Data to Support Medical Decision Making in the Neonatal Intensive Care Unit[END_REF] and provides additional support for our hypothesis that textual summaries are valuable in presenting patient data to medical staff.

On the other hand, subjects did not perform as well with the computer-generated summaries (mean score = 0.34) as they did with human-written texts. Although this is disappointing, the observation that they found the computer-generated texts as useful as the existing graphical presentations is positive and encouraging.

As with the previous experiment, the textual descriptions (both human-authored and computer generated) deliberately avoided interpreting the data in ways that gave clues about the correct decisions to the subjects. However this is an artificial restriction imposed to allow meaningful comparison with the existing presentation and need not limit the future development of our approach.

Because human (expert) written summaries seem to be superior, we have started to analyze the differences between these texts and those generated automatically. Among other things, our initial analysis shows that the human written texts take a narrative approach to the presentation of the content; this is currently missing from the computer generated texts.

Moreover, in the experiment, textual and graphical presentations were treated as alternatives. But in the future there is no reason why they should not be used to complement each other with the textual summaries reinforcing important messages. One of the most important factors in favour of computer generated texts is the amount of time they save for already stressed medical staff, who may otherwise be required to write these textual summaries (e.g. nursing shift summaries).

CHALLENGES AND CONCLUSIONS

Besides the need to improve automatic generation to produce human-quality texts, there are several issues that need to be addressed before routine operational deployment.

The existing technology for producing textual descriptions of ICU data only processes data captured on the ward over 45 minutes whereas in reality we need to be able to handle several hours of data. Also the current textual descriptions are not targeted at any specific group of people in the ICU. In our experiment, the performance of different groups such as junior nurses and senior doctors was not the same, almost certainly because the informational requirements of these groups are different. So one objective of the ongoing project is to develop significant extensions to the current working prototype for three distinct purposes: 1. To assist nurses in writing summaries at the end of their 12 hour shifts; these shift summaries help the incoming nurse to plan her care of the baby; the benefits to the nurses of automatic generation would be time saving and consistency. 2. To generate summaries on demand for junior doctors;

summaries will cover approximately 12 hours and be designed to support decision making. 3. To provide parents and relatives with a basic summary of the progress of the baby over the previous 24 hours; the Edinburgh neonatal ICU has recently made such reports (written by clinicians) available to parents and close friends; we note that immediate family members often require more detailed information than others.

These extensions will attempt much more complex tasks than the simple summarising of a 45 minute period. The times to be covered are an order of magnitude larger and the level of abstraction must be correspondingly greater. The systems are targeted at very different classes of user; the content, structure and language used must be appropriate. In addition to the above, the following issues require attention in future extensions:

1. Uncertainty in information: Many discrete events recorded by medical staff on the ward do not bear accurate time stamps. Some events may be recorded before they actually happen; most are recorded after their occurrence. This uncertainty in temporal information should either be resolved by the system or communicated to the user with appropriate linguistic cues. There are several domain independent techniques for handling uncertainty such as the possibility theory. We are currently developing techniques for resolving uncertainty in temporal information which exploits domain knowledge. For example, if it is known that an event such as intubation (the task of inserting a tube for artificial breathing) has been performed, based on the domain knowledge about the expected duration of the event, it may be possible to estimate the start and end times of that event. 2. Missing information: Several events happen regularly on the ICU and it is humanly impossible to record all of them. When inspecting recorded data, medical staff can infer, from experience, the occurrence of some of the unrecorded events. Certain events also manifest their occurrence as observable patterns in the physiological data which can be computationally recovered. We are currently investigating techniques for automatically detecting 'missing' information from the existing recorded information. Certain types of information such as visual observations may always elude automatic detection. This may place an upper bound on the amount of information that can be communicated by the textual descriptions. 3. Unstructured (free) text: Medical staff often record details about specific patients in the form of unstructured (free) textual comments in the patient record. Summarising patient status without this information may be incomplete but it is not possible simply to insert the textual comments into the computer-generated descriptions. We are currently exploring Information Extraction (IE) techniques for automatically extracting important data from the free text comments and integrating them into our textual descriptions. 4. Integrating text and graphics: In our interactions with the medical staff, we found that the eventual acceptance of our new technology will only be possible if it integrates seamlessly into the existing informational infrastructure on the ward. Although our system presents patient information in textual form, there is much to be gained if the texts can be integrated into the graphical presentations. Textual presentations seem to be appropriate for certain contexts -graphical presentations for others. 5. Generating narratives: As observed earlier, human written texts follow a narrative style which helps readers' comprehension. We are currently working on developing techniques to produce texts that mimic expert narrative style.

Figure 1 .

 1 Figure 1. Graphical Display of NICU data

Figure 4 .

 4 Figure 4. Architecture of BabyTalk

Figure 3 .

 3 Figure 3. Computer generated summary of the data shown in Figure 1.

Figure 2 .

 2 Figure 2. Human generated summary of the data shown in Figure 1.

"

  You saw the baby between 16:40 and 17:25. The pCO 2 continues to rise to 10.1. The baby is pale and unresponsive. ET suction is given, baby is turned and at 17:02 the ETT is removed; the baby is again given Neopuff ventilation. Baby is re-positioned and the NGT aspirated. By 17:08 the baby is reintubated; the oxygen saturation has increased to the 80s and the HR has risen to 176 the pO 2 = 0.1 and pCO 2 = 0.2, T1 is 32.7ºC and T2 34.7ºC.

	Initially the HR baseline is 140-160; pO 2 is 8-10; oxygen
	saturation = 92%, T1 and T2 are 36.9º and 36.6ºC.
	At around 16.45 ET suction is performed; there is a drop in
	oxygen saturation to 50% and pO 2 to 3.3 and a rise in pCO 2 to
	around 9. The FiO 2 is increased from 61 to 100%. By 16.51
	the HR is at 155, the pO 2 is 6.7 and the oxygen saturation is
	88% and the pCO 2 is 9.2. There is an upward spike in the pO 2
	to16.9 and a corresponding downward one in pCO 2 to 3.1; the
	oxygen saturation has fallen to 78%. T1 is now reading 36.9ºC
	and T2 35.7ºC. At 16.57 the ventilator rate is increased to 30.
	Baby is given Neopuff ventilation. The oxygenation continues
	to decrease: pO 2 = 0.2 and oxygen saturation = 20% at 17:00
	and the HR falls to 60. At 17:15 the FiO 2 is reduced to 33% and the rate put back to
	15.
	At 17:24 the oxygen saturation falls to 65 and the FiO 2 is
	increased to 56%.
	At 17:25 the HR is 165, the oxygen saturation is 100%, T1 is
	35.7º and T2 is 34.5ºC."

ACKNOWLEDGEMENTS

The authors are grateful to the UK Engineering and Physical Sciences Research Council for funding of the BabyTalk projectproject under award EP/D049520/1.

We would also like to thank the other members of the BabyTalk team: Yvonne Freer, Felix Gao, Robert Logie, Saad Mahamood, Neil McIntosh, Wendy Moncur, Marian van der Meulen and Cindy Sykes.


Belén A Baez

Sybille Caffiau

Catherine Garbay

François Portet

Task based model for récit generation from sensor data: an early experiment

Keywords: 1998 ACM Subject Classification I.2.7 Natural Language Processing: Language generation, I.2 Artificial Intelligence:Knowledge Representation Formalisms and Methods narrative generation, task model, real world story analysis, ambient intelligence Digital Object Identifier 10.4230/OASIcs.xxx.yyy.p

Automatic story generation is the subject of a growing research effort. However, in this domain, stories are generally produced from fictional data. In this paper, we present a task model used for automatic story generation from real data focusing on the narrative planning. The aim is to generate récits (stories) from sensors data acquired during a ski sortie. The model and some preliminary analysis are presented which suggest the interest of the approach.

Introduction

Stories are pervasive in everyday life and humans have developed cognitive abilities which are tuned to the production and comprehension of stories. Philosophers, linguists and communication scientists have recognised since the Antiquity the fundamental role of story as primary human communication means [START_REF] Barthes | Introduction à l'analyse structurale des récits[END_REF][START_REF] Propp | Morphology of the Folktale[END_REF][START_REF] Brémond | Logique du récit[END_REF][START_REF] Labov | La transformation du vécu à travers la syntaxe narrative[END_REF][START_REF] Genette | Narrative Discourse: An Essay in Method[END_REF][START_REF] Fisher | The narrative paradigm: An elaboration[END_REF][START_REF] Herman | Genette meets vygotsky: narrative embedding and distributed intelligence[END_REF]. The advantage of story over other communicative modes lies in the availability of various means at the structural and linguistic level to convey temporal and causal information, to allow perspective-taking and framing of events, and to emphasise the most relevant information. It is therefore unsurprising that written and oral stories are the main form in which collections of events are conveyed by humans, whether the intention is to report, explain, illustrate opinions or transmit knowledge. According to the Narrative Paradigm [START_REF] Fisher | The narrative paradigm: An elaboration[END_REF], stories are widely used even in technical communication (e.g., medical case histories, reports of engineering faults, forensics) and are central in social interaction to establish and strengthen social groups.

Stories are also the discursive basis on which our identity is fleshed out. Computerised story generation, modelling and analysis has already been the subject of research in Computing Science and has recently emerged as being part of what it is called Computational Narratology [START_REF] Mani | Computational Modeling of Narrative[END_REF]. Contrasting with this field of research, our aim is to make story generation from real non-linguistic data possible. Our goal is more precisely to record daily life events from sensors capturing people's location, physiological state or activity and to process this data in order to generate stories of daily activities. The widespread development of lifelogging systems, the dissemination of mobile devices such as smartphones able to sense their environment open the door to these new applications. However, to the best of our knowledge, this has been an under-developed area.

The amount of applications that story-generation from real data could support is very large : social interaction for people with disabilities; education; cultural heritage; publishing news or stories for local communities; or helping patients understand complex medical data about themselves. For instance, robots with enhanced communication capabilities for people lacking social interactions will also benefit from a technology that seeks to simulate basic human communicative abilities [START_REF] Sasa | Socio-affective interactions between a companion robot and elderly in a Smart Home context: prosody as the main vector of the "socio-affective glue[END_REF].

In this paper, we focus on the narrative planning which, in narrative generation system, is in charge of defining the overall structure of the story. The hypothesis is that a narrative plan of a story generated from real world can be abstracted by a task model that can be used to ensure consistency in the planning process. In the creativity domain, narrative planning has often be approached by automated planners which find the possibles sets of actions that change an initial state (beginning of the story) to a final desired state (end of the story) [START_REF] Meehan | Tale-spin, an interactive program that writes stories[END_REF][START_REF] Turner | Minstrel: A computer Model of Creativity and Storytelling[END_REF] with more of less constraints on characters, objects, etc. [START_REF] Solis | Planning author and character goals for story generation[END_REF][START_REF] Riedl | Narrative planning: Balancing plot and character[END_REF], or interaction with the users [START_REF] Cavazza | Character-based interactive storytelling[END_REF][START_REF] Pérez Y Pérez | Three computer-based models of storytelling: BRUTUS, MINSTREL and MEXICA[END_REF]. Another emerging field of research is concerned with automatic learning of narrative structure from annotated corpus such as StoryBank [START_REF] Finlayson | Inferring Propp's Functions from Semantically-Annotated Text[END_REF] or NarrativeML [START_REF] Mani | Computational Modeling of Narrative[END_REF]. Nevertheless, they are based on models build from the fictional literature which are not adapted to the narrative generation from real world data where, for instance, characters or goal does not fit into predefined or stereotyped categories. Some systems, such as Tag2Blog [START_REF] Ponnamperuma | Tag2blog: Narrative generation from satellite tag data[END_REF] or BabyTalk [START_REF] Portet | Automatic generation of textual summaries from neonatal intensive care data[END_REF], aim at generating reports of events from real data. However, the produced texts are still far from a story [START_REF] Reiter | The importance of narrative and other lessons from an evaluation of an NLG system that summarises clinical data[END_REF]. Our approach based on task model would help to deal with real data that are uncertain and incomplete in order to obtain story-like texts or at least close to a story.

The main application domain of the project is pervasive computing domain, exemplified by smart phones equipped with several different types of sensors, to generate everyday life stories that we call récits. More precisely we focus on the ski touring domain which is introduced in Section 2. Section 3 provides some formal definitions regarding the notion of récit and how it is formalized in the proposed approach. Section 4 provides a focus on task-based narrative planning and gives a presentation of the methods and tools involved in this specific step. Early experiments with annotations and an evaluation about them regarding the task-based narrative planning are reported Section 5. The paper finishes with a short discussion on related work and issues to address in further work.

Generating stories of ski sorties: main issues

In this paper, we focus on the ski touring. In this sport, skiers go either alone or in team, following a predefined route, to climb in the snow, often in difficult weather conditions. After their sortie, skiers regularly use online portals to share their experiences with their peers in the form of stories (e.g., www.skitour.fr). These stories are intended to help other skiers to organise their sorties and to alert on potential security issues. They combine environmental data (weather, snow conditions, slope. . . ), temporal data and social or evaluative elements. Events, such as route changes, avalanches or accidents are often reported. These stories are framed to give more emphasis on recreational, environmental or emotional aspects, depending on the profile of the writer and also on the experience that has been lived. Hence, this domains is particularly well-suited to récit generation in that it provides a large collection of human stories about their experience, while making raw data collection possible using smart phones which are equipped for the purpose. Figure 1a provides a narrative from the ski touring domain. Physiological and actimetric data were specifically collected for this sortie using two devices (smart phone and physiological sensors). These data involve time, location, altitude, heart and breath rate, etc. An exemple of the collected data is provided in Figure 1b. For details about the data processing the reader is refereed to [START_REF] Portet | Automatic generation of textual summaries from neonatal intensive care data[END_REF][START_REF] Vettier | Starting artificial intelligence research symposium[END_REF].

Various processing steps are needed to cross the abstraction levels from raw data to full récit generation, as may be seen in Figure 2. We distinguish between (i) the physical world, which is captured through sensors; (ii) the abstracted world, which is an interpretation of the observations captured; (iii) the story world, which is 'one' way of making sense of a subset 10.00, rotten weather, we went to Chamechaude, a usual destination in case of bad weather. In order to add some more climbing, we start 100 m below the Col de Porte, down the lift. The weather is not beautiful, objectively not very cold but we slippers under a fine rain that freezes a bit. We climb quickly and we warm up quickly. Above the rain stopped and I even have the feeling it was too hot in the humid atmosphere! We took only a few breaks, and I do not remember having eat or drink. The snow is pleasant and the track easy to follow up to half of the meadow, then it gets too stiff to our taste and we retrace the end in a thicker snow before attacking the final meters. of the interpretations by choosing, ordering and relating them to each other; and (iv) the natural language world, which is 'one' way of expressing the story world discursively (i.e., the narrative). Each of these steps raises difficult issues, involving signal processing, data interpretation, event graph construction, story planing and Natural Language Generation (NLG). Viewed from a bottom-up perspective, the processing of raw data involves recognition, inference and/or abstraction. Viewed from the opposite perspective, top-level representations provides framing constraints that gives context to drive the interpretation process, e.g. directing attention to some relevant aspects. This process must be seen as dynamic: the framing evolves as different phases of the story unfold. For instance, elements that are necessary for the introduction can be rendered in a factual manner while more emphasis can be put on external conditions or emotional/personal information, depending on the events occurring in the course of the sortie. For instance, in the example in Figure 1, the fact that few short resting periods are observed during a sortie, might be used as a frame to drive story generation.

In the rest of the paper we focus on core issues of the story planning: what is the structure of a ski touring récit? What are the elements that are relevant to a récit? How to ensure coherence in a récit? This calls for proper models that will provide the skeleton on which the overall process will draw. These elements are described in more details in the following sections.

Defining récit

Narrative defines a large amount of different literary forms and artefacts as well as events collections (e.g., narrative in the Event Calculus definition). In this approach, we consider the Mieke Bal's [START_REF] Bal | Narratology: Introduction to the Theory of Narrative[END_REF] definitions:

"A narrative text is a text in which an agent relates ('tells') a story in a particular medium, [..]. A story is a fabula that is presented in a certain manner. A fabula is a series of logically and chronologically related events that are caused or experienced by actors. Actors are agents that perform actions. [..] To act is defined here as to cause or to experience an event."

In our approach, the fabula is the set of connected events that are either directly observed by sensors (i.e., the abstracted world). We call this set the graph of events where nodes are particular instance of events and edges are relationships between events. From this graph of events, scenes of interest are identified as the atomic components of a récit plan.

For Brémond, a récit is a discourse about a succession of events in the same unit of action relevant for humans [START_REF] Brémond | Logique du récit[END_REF]. Genette suggested that 'récit' indicates the content of the narrative and that 'narrative' is the statements used for communicating the story [START_REF] Gunning | Griffith and the Origins of American Narrative Film: the Early Years at Biograph[END_REF]. The Barthes' definition of récit implies a sequence of events and actions linked by causality [START_REF] Barthes | Introduction à l'analyse structurale des récits[END_REF]. In [START_REF] Adam | Narrativité et généricité des textes[END_REF], Adam provides the characteristics of a récit, whether fictional or real with respect to the other kinds of stories. The following definitions are in line with the Adam's perspective.

◮ Definition 1 (Récit). A real-life récit is seen as a sequence of activities with unity of theme and action focused on communicating the actors' experience. It is a succession of events related to facts that have been effectively experienced, observed or captured. In the ski tour context, example events are the tracks actors followed, which dangers were encountered, who was met along their way, etc.

◮ Definition 2 (Récit Plan). A récit plan is a set of selected scenes from the fabula that are ordered and logically related to each other so as to ensure temporal and causal coherence. In ski tour récits, the beginning and end are always identified as the starting point and the ending point of the sortie. Scenes are selected to produce a particular inflection of the fabula.

In this paper, coherence and ordering is ensured by using an a priori model based on a task model (cf. Section 4.2). All input data should percolate up to the story through different layers of abstraction, filtering and aggregation constrained by the task model. The planning process should thus be driven by data and knowledge at the same time. One important feature in the récit is the temporal aspect. There is a distinction between the story time and discourse time [START_REF] Chatman | Story and Discourse: Narrative Structure in Fiction and Film[END_REF]. Here, the récit presents details and context of scenes in the story time.

◮ Definition 3 (Scene).

A scene is the basic temporal unit of récit plan. It belongs to a predefined set of classes of a ski sortie (climbing, resting, observing the environment, etc.) and present a unity of trajectory or location, of activity and of actors.

Task-based Récit Planning

In the creativity domain, narrative planning problem is often defined as finding the sequence of character's actions that transforms an initial world state into a world state in which goal propositions hold [START_REF] Riedl | Narrative planning: Balancing plot and character[END_REF]. However, in the real world, the problem is, given raw data, to extract the scenes that might be connected that can explain the transformation from an initial state to a final state. Important challenges are to be addressed to be able to perform such as planning, such as dealing with the uncertainty and incompleteness in the data. Indeed, while in fictional story, the world is completely defined (e.g., in the phrase "the king's death causes grief to the queen", the consequence of death to cause grief is explicit and thus can be used as the reason for the queen's sorrow) it is incompletely known in real data (e.g., the reason why 'the group had a break' may not be extractable from the data). This incompleteness in the data impacts many other functions of the story generation, such as perspective taking as what a character knows might be unknown.

To account for this uncertainty, the récit planning does not uses crisp logical planning but a task modelling that mostly bases its reasoning on time constraints and preconditions rather than a chain of state transforms in a closed world. The task model does not only ensure consistency of the sequence of scenes but also tolerates a certain amount of incompleteness in the data. This model is particularly well suited to the fields in which the events are relatively constrained (e.g., ski tour, tourism visit, intensive care operations). Other approaches have recognized the power of task modelling for interactive storytelling [START_REF] Cavazza | Character-based interactive storytelling[END_REF]. The proposed approach for the generation of récit plan relies on two models: Domain model: based on an ontology of the domain knowledge, this defines the atomic components of the fabula emerging from the data. Récit model: based on a task model which describes actions, schedule, links between tasks and actors

The instances of the ontology and récit are extracted from the actual data according to the model. Thus, a récit plan generated from these data which populate the ontology will be conform to the the task model schedule (here a ski sortie). The two following subsections give more details about these models.

Ontology

In order to support definition, reasoning and to ensure uniqueness of semantic meaning of the information handled by the different processing step, the domain knowledge was represented by an OWL (Web Ontology Language) ontology. This ontology is expressed in Descriptive Logic (DL) a formal knowledge representation language. In DL, the knowledge is distinguished between the TBox (Terminological Box) and the ABox (Assertional Box). The TBox contains the definition of the concept hierarchies while the ABox contains definitions of the individuals (relations between individuals and concepts). Figure 3a shows an excerpt of the ontology TBox and ABox. The taxonomy is divided into abstract and physical entity as well as event. All events involve at least one entity. For instance physiological events relate one actor with a physiological state at a certain time.

Task Modelling

Task models have been used for decades in the Human Computer Interaction field (HCI) to express user's activities. It describes how a user acts with interactive systems by hierarchically composing the activities by goals and sub-goals. Thus, a task model describes human activity by composition of actions (i.e., tasks) that are linked by abstraction level and temporal relations. Several notations have been developed to express task models [START_REF] Balbo | Choosing the right task modelling notation: A taxonomy[END_REF] which highlight different activity points according to the designer's preoccupations [START_REF] Caffiau | Increasing the expressive power of task analysis: Systematic comparison and empirical assessment of tool-supported task models[END_REF]. One of them is K-MADe (Kernel of Model for Activity Description environment) 1 . Figure 3 shows the task model in K-MADe currently employed to represent a ski tour récit. In this model, the activities of actors are represented in the form of task trees, from the most general (e.g., 'Doing a ski touring sortie') to the most detailed (e.g., 'Moving forward'). Each task is characterized by an actor (individual, group, system. . . ) and associated with objects describing the actor's environment. These objects influence the course of the activity (e.g., snow storm . . . ) and can handled by the actors. Constraints between the tasks include 'Enabling' (the task on the left side must precede the task on its right), 'Concurrent' (tasks can be performed in parallel), etc. Thus, this model makes it possible to represent temporal constraints between tasks, dependencies as well as causality. In this model, each récit starts with an introduction ('Arriving at start point'). Then, each scenes consists in going from the current point to another ('Going to the target') till the starting point in a sequential order (cannot be concurrent). Each target task can then be refined into analysing the situation (e.g., checking the weather or the route, keep going or meeting other skiers).

A task is instanced only if the conditions for execution are valid (e.g., preconditions). Moreover a task is associated with side effects (e.g., postconditions such as creation or removal of objects, events being generated during the execution of the task, etc.). A succession of tasks respecting the expressed constraints and objects values, namely a scenario, expresses a specific execution flow of the activity. In this work, a récit plan is represented by a scenario.

The planning problem becomes to build a scenario that respects the task model or more concretely to find a path in the graph of events that respect the task model constraints. Given the incompleteness of the data, in the course of the plan building, missing scenes are identified by the task model. Once identified, the process can either generate an empty event that will be rendered as an ellipsis or discard the path if the scene is mandatory to ensure coherence.

Early Experiment

To validate the task model approach, data were extracted from real textual stories written by skiers. These textual stories have been annotated using a schema annotation that has been defined from studies applying the methods used to construct a task model from interviews [START_REF] Sebillotte | Methodologie pratique d'analyse de la tache en vue de l'extraction de caracteristiques pertinentes pour la conception d'interfaces[END_REF] and from a study identifying the terminology in the field. The annotations are then transformed into scenarios (i.e., an instance of the task model) and checked with the reasoner of K-MADe. Inconsistencies with the model can thus be emphasized.

Figure 4 Example of an annotated story using Callisto.

Corpus Collection

For the experiment, we collected a few texts from the skitour website (c.f. www.skitour.fr).

The website offers the opportunity for who practice the ski touring to share their experiences about their journey. Some skiers publish texts containing advices, recommended landscapes, difficulties or, a really important feature, the itinerary of the sortie. Many stories were written as if the author was directly interacting with the reader (recreational conversation, private jokes etc.) and are thus not conform with our definition of a ski tour récit. Thus, "conversational stories" were discarded and the selection was restricted to a few summits. Up to now, 17 texts were selected. We also collected a small parallel corpus consisting in people going for a sortie equipped with sensors (smart-phones and physiological sensors). These people then wrote the récit of their sortie so that most of their experience can be found in the raw data. This corpus will grow to become the main development corpus.

Annotation

The texts were annotated with the Callisto annotation tool. The software allows the creation of personalized tag-set which are used to annotate the selected phrases with a highlight colour. Callisto employs a Document Type Definition (DTD) to characterise each the tag-set as well as its attributes. The output format file is XML. Figure 4 shows an annotated text (in French, translation can be found in Figure 1a). Each of coloured text segments corresponds to an annotated scene. The elements attributes are put automatically in the lower half of the window in the form of a HTML form (most of them, text fields). It is possible to specify all the attributes of the scene (in the task sense) and to connect each scene to one or several actors. Actors are also annotated in the text. Up to now, 11 texts were annotated.

First analysis of the annotated texts

During the annotation, some phenomena linked to the nature of human-written text were observed. For instance, some text segments indicate several pieces of information at the same time. This was addressed by labelling several times the same segment (e.g., implicit information about the trajectory in a observation). Moreover, in some cases the narrator decided to omit some details that, however, did not impede the reading. These ellipses were mainly related to trajectory, information that will hopefully be captured by sensors. Others discarded information were references to the past or to the future. Regarding the actors, it was decided to identify the main actor (can be a group) for each récit and to indicate the actor in the scene only when the event was not experienced by the main actor ("we climbed quickly" vs. "I do not remember").

Even if emotional expressions were found in the texts (e.g. surprise, frustration . . . ), the récit structure is mainly a sequential narration of the sortie phases which describe the itinerary followed as well as the environmental conditions (weather and ground characteristics). Recall that the purpose of the récit generation is to generate from sensor data, thus information hard to measure, such as skiers' opinion and emotion are not the focus of the present study. 14 tag types were used for the annotation and from the 11 récits (125 segments), 44% corresponds to a moving forward, 16 % to a terrain description, 10% breaks, 8.8% were meetings, 6.4% mention the starting point. Making a decision and the weather conditions got 5.6% each while checking the physical condition and having an accident got a 0.8% each the 1.6%. Finally, even if all the human aspects exposed were not included in the annotation scheme, we found that the annotation system and their later validation in the K-MADe simulation were capable of capture the itinerary of all the ski touring récits.

Validation plan of the Task Model

After being annotated, the annotated text was translated into récit plan, that is scenario in the K-MADe format. In this transformation, we assumed that the text order was the chronological order (actually the annotation was performed to make sure this was the case). The purpose was to know if the task model is able to cover all the executed activities in the ski touring. K-MADe has a task editor in which we can set as parameters the activities linked to a specific task and how many times it could be executed. The simulation starts with a point of departure and after that we must choose between all the activities proposed the next one. Each time an activity of a task is chosen, we fill in the attributes with the values that belongs to the activity. Once the simulation done, an XML document containing the succession of tasks chosen with their respective values is obtained. It is the récit plan. This plan is going to be evaluated in a human based experiment to assess the coherence of the final generated texts based on the annotations.

6

Discussion and Future work

The early experiments undertaken in this study suggest that task modelling imposes a flexible domain dependent récit structure and ensures coherence in the way the scenes are organised. However, there are many issues to be addressed to definitely conclude about its relevance. The first issue is related to missing information in the text. For instance some of the scenes are implicit such as in "The snow is pleasant and the track easy to follow up to half of the meadow" where the description of the snow suggests that the group had moved forward. When some scenes are missing (e.g., a decision without explanation) we intend to use ellipses (i.e., empty task in the task model). How many and what kind of ellipses can be permitted for a human reader is still an open question. Another issue is the lack of information about the temporal aspect in the annotation. Annotation scheme such as TimeML [START_REF] Pustejovsky | Iso-timeml: An international standard for semantic annotation[END_REF] could help capturing important temporal elements in the text. Our work follows a line of research similar to the one that was early developed by Mark Finlayson's StoryBank [START_REF] Finlayson | Inferring Propp's Functions from Semantically-Annotated Text[END_REF]. As we are not dealing with fictional stories, but rather with real world stories, generated from real world data to represent daily life activities, we were mainly concerned by a dedicated dimension of annotation that we might called "task-driven", a part of the "Plans and Goals". As discussed by him, other dimensions for annotation should be considered in further research. The specificity of Daily life récits in this respect should be studied in depth. An important step will be the text generation. The approach will be a simple translation of the scenes and their links in order to help system evaluation by humans. However, the most interesting issues will be the study of the dependences between the récit generation layers for dealing with uncertainty in the data. For instance if a break is identified with a low confidence, the récit planning might not take it into account or might include it as justification for an other scene then the NLG part would express this uncertain using modals or other linguistic means (e.g., "the group discussed a lot may be during a break") [START_REF] Gatt | If it may have happened before, it happened, but not necessarily before[END_REF]. The evaluation of the récit planning will be a big issue in the project. Though recent attempt to approach this challenge methodologically (see for instance [START_REF] Rowe | Storyeval: An empirical evaluation framework for narrative generation[END_REF]) there is no consensus. In this study, we will introduce simulation techniques involving stories of varying complexity built at different levels of detail and abstraction. The validation will be conducted in a qualitative manner, by human "experts" and "naive" participants. These assessments will also validate narrative and linguistic choices according to the user model. Indeed a naive participant will not look at the same events and will not use the same language as an expert. Probably one of the most decisive parts is the long term is the data processing that will be put in place. We plan to perform data abstraction based on an hybrid top-down bottom-up approach based on statistical models and logical model to recognized a set of high-level scenarios in physiological data [START_REF] Vettier | Starting artificial intelligence research symposium[END_REF] as well as on statistical-relational models which were used to recognize human activities from pervasive sensors [START_REF] Chahuara | Making Context Aware Decision from Uncertain Information in a Smart Home: A Markov Logic Network Approach[END_REF]. With this approach, raw data would be abstracted in a graph of events logicically and probabilistically linked.

Our work is comparable to others that are focused on computational narrative structure representations, such as the Hierarchical Task Networks planning (HTN). In this work, the HTN has been applied to storytelling [START_REF] Cavazza | Character-based interactive storytelling[END_REF] to structure a story through the decomposition of a main task in subtasks. However, while the authors in [START_REF] Cavazza | Character-based interactive storytelling[END_REF] are interested by the emergence of story variants from the interaction of autonomous actors, we are rather interested by the emergence of story variants from the interaction between a planned activity, with some a priori known starting and ending point (the car park and the summit) and a set of unpredictable contextual conditions (weather or quality of the snow for example). How these contextual elements will cause variants to story generation is an open line of research to be followed.

In any of the issues mentioned above there is a need for a larger corpus. We are pursuing the data acquisition in order to record this 'parallel' corpus with to the best of our knowledge is currently non-existent in the community. Despite the limited outcome of this study, the findings could be beneficial for others fields. For instance, the ski touring domain is close to domain in which a predefined route and wearable devices can be used such as in rescue operation, city travel or tourism.

Figure 1 Figure 2

 12 Figure 1 Example narrative from the ski touring domain and the corresponding raw data captured along a ski touring activity (two persons involved P1 and P2)

Figure 3

 3 Figure 3 (a) Domain Ontology in OWL (b) the task model employed to support récit planning.

www.lias-lab.fr/forge/projects/kmade

Acknowledgements

The authors would like to thank the support of the Pôle Grenoble Cognition, France. We are also grateful to Albert Gatt, Pablo Gervás and Ehud Reiter for thorough discussions on this project. Thanks are extended to Ky Nguyen and Issam Zrelli.