Markku Turunen
email: mturunen@cs.uta.fi

Jaakko Hakulinen

Olov Ståhl
email: olovs@sics.se

Björn Gambäck
email: gamback@sics.se

Preben Hansen
email: preben@sics.se

Mari C Rodríguez Gancedo

Raúl Santos De La Cámara

Cameron Smith
email: c.g.smith@tees.ac.uk

Daniel Charlton
email: d.charlton@tees.ac.uk

Marc Cavazza
email: m.o.cavazza@tees.ac.uk

Multimodal and Mobile Conversational Health and Fitness Companions

Keywords: Companions, Embodied Conversational Agents, Conversational Spoken Dialogue systems, Mobile interfaces, Cognitive Modelling, Dialogue management

Multimodal conversational spoken dialogues using physical and virtual agents provide a potential interface to motivate and support users in the domain of health and fitness. In this paper we present how such multimodal conversational Companions can be implemented to support their owners in various pervasive and mobile settings. We present concrete system architectures, virtual, physical and mobile multimodal interfaces, and interaction management techniques for such companions. In particular, we present how knowledge representation and separation of low-level interaction modelling from high-level reasoning at the domain level makes it possible to implement distributed, but still coherent, interaction with Companions. The distribution is enabled by using a dialogue plan to communicate information from domain level planner to dialogue management and from there to a separate mobile interface. The model enables each part of the system to handle the same information from its own perspective without containing overlapping

Introduction

Most existing spoken dialogue systems provide a single interface to solve a well-defined task, such as booking tickets or providing timetable information. However, there are emerging areas that differ dramatically from task-oriented systems. In domain-oriented dialogues [START_REF] Dybkjaer | Evaluation and usability of multimodal spoken language dialogue systems[END_REF] the interaction with the system, typically modelled as a conversation with a virtual humanlike character, can be the main motivation for the interaction. These systems are often multimodal, and may be implemented in pervasive computing environments where various mobile, robotic, and other novel interfaces are used to communicate with the system.

In the EC-funded COMPANIONS project [START_REF] Wilks | Is there progress on talking sensibly to machines?[END_REF] we have developed conversational Companions that build long-lasting relationships with their users via mobile and physical agent interfaces. Such systems have different motivations for use compared to traditional task-based spoken dialogue systems. Instead of helping with a single, well-defined task, a companion provides long lasting support and companionship on a daily basis.

One of the developed companions, and the topic of this paper, is the conversational Health and Fitness Companion, which helps users to maintain a healthy lifestyle. There are several examples of commercial systems in the domain of health and fitness, in particular in exercise domain. Some of the most well known examples are miCoach from Adidas and NIKE+. 1 More in line with the present work, MOPET [START_REF] Buttussi | MOPET: A context-aware and user-adaptive wearable system for fitness training[END_REF]) is a PDAbased personal trainer system supporting outdoor fitness activities. MOPET is similar to a Companion in that it tries to build a relationship with the user, but there is no real dialogue between the user and the system and it does not support speech input or output. Neither does MPTrain/TripleBeat [START_REF] Oliver | MPTrain: A mobile, music and physiology-based personal trainer[END_REF][START_REF] De Oliveira | TripleBeat: Enhancing exercise performance with persuasion[END_REF], a system that runs Page 3 of 42 A c c e p t e d M a n u s c r i p t on a mobile phone and aims to help users to more easily achieve their exercise goals. The system selects music indicating the desired pace and different ways to enhance user motivation, but without an agent user interface model. InCA [START_REF] Kadous | InCa: A mobile conversational agent[END_REF]) is a spoken language-based distributed personal assistant conversational character with a 3D avatar and facial animation. Similar to the Mobile part of Companion, the architecture is made up of a GUI client running on a PDA and a speech server, but the InCA server runs as a back-end system, while the Companion utilizes a stand-alone speech server.

The difference between the Health and Fitness Companion and most of the existing health related systems is that our Companion prototype aims to be a peer rather than an expert system in health-related issues. The ultimate goal of the Health and Fitness Companion is to support overall lifestyle changes in the user's daily habits rather than giving detailed advice on any specific health related issues. A lifestyle change often requires great motivation and the Health and Fitness Companion aims to support building and maintaining this motivation. The method the Health and Fitness Companion uses to support the user motivation is to set up a long lasting social and emotional relationship with the user. Since people build relationships mostly in face-to-face conversations, a conversational embodied agent is a potential platform to build such a relationship [START_REF] Dybkjaer | Evaluation and usability of multimodal spoken language dialogue systems[END_REF].

Both mobile usage and physical and virtual agent interfaces can support the goal of making a spoken dialogue system part of its user's everyday life, and building a meaningful relationship between the system and the user. There are numerous examples of virtual embodied conversational agents, but only a few examples of physical conversational agents. Physical agents, however, can be particularly efficient in applications that try to evolve dialogue systems into being part of people's lives.

While naturalistic human-like physical robots are under development, there is room for a variety of different physical interface agents ranging from completely abstract (e.g., simple devices with lights and sound) to highly sophisticated anthropomorphic apparata. For example, [START_REF] Marti | Physical embodiments for mobile communication agents[END_REF] used several toy animals, such as bunnies and squirrels, as physical embodied agents for a conversational system. Another example is an indoor guidance and receptionist application involving a physical human-like interface agent that combines pointing gestures with conversational speech technology [START_REF] Kainulainen | A speech-based and auditory ubiquitous office environment[END_REF]. Some physical agent technology has also been commercialized. Examples include the Paro Therapeutic Robot A c c e p t e d M a n u s c r i p t2 , and the wireless Nabaztag/tag rabbit3 , which has resulted active user community.

In this work, we have used the Nabaztag/tag device as a multimodal physical interface to create a conversational Health and Fitness Companion. In addition to the physical interface we have used a mobile device interface for outdoor usage, and a graphical, lifelike virtual agent interface in certain dialogues to provide rich multimodal interface when those features are needed.

In these kinds of applications where multiple user interfaces can be used to access parts of the same functionality, e.g., the same physical exercise information, and the system interacts with a user many times over a long time period, modelling the interaction and domain quickly becomes complex. To address this, the system must include a model capable of reasoning about the domain, and learn from the user and his/her actions to provide meaningful interaction, such as to provide reasonable guidance on the user's health and progress as the user's condition alters over time. Such reasoning should be concentrated on one component, instead of duplicating the logic to keep the system maintainable. Still, the information must be communicated over different interfaces and to the components inside them. Therefore, modularization of the system and appropriate knowledge representation become vital.

On the dialogue management level, a common way to take some complexity away from the dialogue manager and limit its tasks more specifically to dialogue management is to separate domain specific processing, such as database queries, into a back-end component. Many researchers have worked with separating generic dialogue management processes from the domain specific processes. Example solutions include shells [START_REF] Jönsson | A natural language shell and tools for customizing the dialogue in natural language interfaces[END_REF] and object oriented programming methods [START_REF] Salonen | Flexible dialogue management using distributed and dynamic dialogue control[END_REF][START_REF] O'neill | The Queen's Communicator: An object-oriented dialogue manager[END_REF]. On the other hand, a simple back-end interface, e.g., SQL queries, can be included as configuration parameters [START_REF] Pellom | The CU Communicator: An architecture for dialogue systems[END_REF]. Since dialogue management is usually based on some clearly defined model, such as state transition networks or form filling, keeping domain specific processing in the back-end makes it possible to implement dialogue management purely with the selected model without the unnecessary complexity of domain specific The Health and Fitness Companion is based on a model where the domain specific module is more than just a simple backend database and includes active processing of domain information, reasoning, learning, and other complex processes. We call such a component cognitive model. While the task of a dialogue manager is to maintain and update the dialogue state, the cognitive model reasons using domain level knowledge. The separation of the tasks between the different parts is not trivial. For example, managing dialogue level phenomena, such as error handling and basic input processing, are tasks clearly in the area of a dialogue manager. However, cognitive modelling can help in error handling by spotting input that seems suspicious based on domain level information, and in input parsing by providing information on potential discussion topics. The solution we have devised is to have the cognitive model to produce a dialogue plan for the dialogue management in the Home Companion system. The dialogue manager in the Home Companion provides parsed user inputs to the cognitive model and to the Mobile Companion. The Mobile Companion provides similar input back to the Home Companion, which communicates it back to the Cognitive Model.

In the rest of this paper, Section 2 introduces the concept of Health and Fitness Companion including the different types of mobile, physical and virtual agent interfaces that realize the concept. For the Cooking Companion we present the initial prototype interface developed to demonstrate the concept. For the Home and Mobile Companions, which have been fully implemented, we present the technical solutions to implement such Companions. Section 3 details the software architectures used to implement physical and mobile Companions interfaces. Section 4 introduces a novel interaction management solution which separates cognitive modeling from dialogue management and enables their flexible interoperability and distributed but coherent dialogues between different agent interfaces. Together, the solutions presented in these sections result in concrete software components to implement further Companion applications. Some of the solutions, such as the PART system used in the Mobile Companion and the Jaspis architecture and the jNabServer software used for the Home Companion have been released as open source software. Finally, Section 5 presents the key results from the early evaluations of the first Health and Fitness Companion prototypes. Section 6 recapitulates the key parts of the discussion.

A c c e p t e d M a n u s c r i p t

Companion Interfaces

As discussed previously, the Health and Fitness Companion interacts with its user in three main forms: as a physical Home Companion to provide general support for a healthy lifestyle, as a Mobile Companion to support the user during physical outdoor activities, and as a virtual Cooking Companion to support eating and cooking related lifestyle activities. Together, these interfaces aim to build a long lasting relationship with the user.

Example Dialogue

The example below show a complete dialogue which a user can have with the three different Health and Fitness Companion interface prototypes. Dialogues 1 and 2 are carried out with the Home Companion interface, Dialogues 3 and 4 are done with the Mobile Companion interface, and Dialogue 5 demonstrates a conversation with the Cooking Companion interface. (4) After 12 minutes C10: You have been running for 12 minutes. The distance is 2.5 kilometres; the current pace is 5 minutes per kilometre.

U9: Okay, please play some relaxing music

(5) Cooking with the Cooking Companion C11: Good, let's start preparing the ingredients. You should remove the stalk and seeds from the red pepper, then chop it up.

U10: And I should chop up the onion now too?

C12: Yes, that's right, peel and chop the onion. You could peel and crush the garlic too afterwards.

Home Companion Interface

The Home Companion interface is a physical agent that resides in the home of its user, for example, in the kitchen table, where it is natural to have a conversation before going to work or start a physical activity. This setup is illustrated in Figure 1. The physical agent is implemented with the Nabaztag/tag WLAN rabbit that provides audio output and push-totalk audio input, RFID-based interaction, moves its ears, and operates four coloured lights to signal its status.

In order to provide meaningful advice to the user on his or her daily activities the Home Companion communicates with the user in two main dialogue phases; the planning phase where the system talks about the coming day with the user (as demonstrated in Dialogue 1), and the reporting phase where the users actual activities are assessed with reference to what was agreed on earlier (Dialogue 2). The covered topics include a range of common situations and activities relevant for health and fitness conversations. In overall, the Home Companion is able to discuss the following topics: travelling to work, getting lunch, activities to be performed before dinner, getting dinner, and activities to be performed after dinner. It knows activities such as playing The Home Companion interface in a typical usage situation (see http://www.youtube.com/watch?v=KQSiigSEYhU for a video presentation) football, squash, or badminton; going to the gym or shopping; and watching television or reading a book.

Mobile Companion Interface

The Mobile Companion runs on Windows Mobile devices and can be used during outdoor exercise activities such as walking, jogging or cycling. The Mobile Companion can download the plan of the day which the user has agreed on with the Home Companion. The Mobile Companion will then suggest an exercise, based on the user's current location, time of day and the plans made earlier, or, if there are no suitable exercises, ask the user to define one, as in Dialogue 3. Once an exercise has been agreed upon, the Companion will track the progress (distances travelled, time, pace, and Page 9 of 42 calories burned) using a GPS receiver. While exercising, the user can ask the Companion to play music or to give reports on how the user is doing. When the exercise is over, the Mobile Companion can upload the result to the home system. The data gathered during an exercise is also stored locally in the device's record store, which allows for access to data from previous exercises (for comparison purposes) even if the mobile internet connection is temporarily unavailable.

The Mobile Companion's graphical user interface, as seen in Figure 2, consists of a single screen showing a static image of the Nabaztag rabbit, along with a speech bubble. The rabbit image is intended to give users a sense of communicating with the same Companion, no matter the interface (Home or Mobile). To further the feeling of persistence and help users associate the two interfaces, the Home and the Mobile Companions use the same All spoken messages are also shown as text in the speech bubble. The user can provide input via voice, by pressing hardware buttons on the mobile device, and in some situations, by tapping on a list of selections on the touch screen. Screen-based input is used, for example, when identifying the current exercise route (from a list of route names, defined by the user in previous sessions), but also when ASR errors occur, to perform error correction. In the latter case, the user is presented with a list of input strings that the Companion is able to understand in the current context, and must select one to continue the dialogue.

Cooking Companion Interface

One of the main aims of the Health and Fitness Companion is to provide tips for a healthier lifestyle, and one of the key aspects is a correct nutrition. The Cooking Companion is a virtual embodiment of a dietary advisor, which:

Page 11 of 42 A c c e p t e d M a n u s c r i p t 1. Presents a set of possible recipes to the user, based on availability of ingredients or home delivery food. 2. Informs the user of the appropriateness of each choice, based on available information such as the user's likes and dislikes, authorised medical and nutritional databases, hers/his current physical condition, special dietary requirements (e.g., allergies) and the physical exercise scheduled by the rest of the Health and Fittness Companion. 3. Helps in the preparation of the selected recipe using multimedia and/or dialogue.

As an example, the user wants to eat seafood paella. The system then inspects all of the aforementioned parameters and conclude advising the user to prepare vegetable paella, based on his current exercise schedule and high blood fat levels (see Dialogue 5 for an example). Then the system instructs the user on how to prepare the dish using videos and speech.

In Figure 3, the interface for the Cooking Companion prototype is shown. The system augments plain spoken dialogue with the help of an ECA, powered by a third party engine4 and a finger-operated touchscreen interface. The interaction is multimodal, intertwining speech utterances and finger pointing. The ECA, modelled as a photorealistic human, includes some advanced gesturing capabilities that have been demonstrated to make the avatar both more engaging and understandable [START_REF] López Mencía | A good gesture: Exploring nonverbal communication for robust SLDSs[END_REF][START_REF] López | Design and validation of ECA gestures to improve dialogue system robustness[END_REF][START_REF] López | ECA gesture strategies for robust SLDS[END_REF].

The Cooking Companion interface is presented here as an example of how the Companions paradigm can use different agent interfaces to achieve certain aspects of companionship. Currently, a prototype to demonstrate its functionality has been implemented. In the rest of the paper, we focus on the Home and the Mobile Companions, which have been fully implemented as described in the forthcoming sections. Further information on the Cooking Companion and its relation to the overall Health and Fitness Companion is presented in [START_REF] Hernández | Modular definition of multimodal ECA communication acts to improve dialogue robustness and depth of intention[END_REF] and [START_REF] Turunen | Multimodal agent interfaces and system architectures for health and fitness Companions[END_REF].

Companions Architectures

In order to construct Health and Fitness Companions, we need concrete system architectures and software components that are flexible enough to A c c e p t e d M a n u s c r i p t implement different interfaces running on different hardware and software platforms. At the same time, we need to maintain shared knowledge for the Healt and Fitness Companion. In the following, the Home Companion and the Mobile Companion architectures and their main components are presented. These provide foundations for the interaction management solutions presented in the following sections.

Home Companion

The Home Companion is implemented on top of Jaspis5 [START_REF] Turunen | An architecture and applications for speech-based accessibility systems[END_REF], a generic open source software architecture designed for adaptive spoken dialogue systems. In order to implement the physical Health and Fitness Companion, the Jaspis architecture was extended to support interaction with physical agents, and the Nabaztag/tag device in particular. For this purpose, the jNabServer software6 was created to handle local communication with Nabaztag/tag. This solution offers full control over Nabaztags, including RFID-reading, and makes it possible to use custom programs and technologies to process inputs and outputs, such as the speech recognition and TTS software used in the Health and Fitness Companion.

Figure 4 illustrates the Home Companion architecture. The top-level structure of the system is based on managers, seven of them in total, which are connected to the central Interaction Manager (a HUB/Facilitator style component) using a star topology structure. In addition, the application has an Information Manager (a database/blackboard style component) that is used by all the other components to store and share information. The Information Manager provides a high-level interface to the Information Storage, which contains all the dialogue management and the cognitive model structures (e.g., confirmation pools and activity models). The communication between the these two models is based on a dialogue plan, which is also in the Information Storage. These are presented in detail in Section 4. The Information Manager and the Information Storage do not modify this content, they just handle the technical management of data in XML format. The details of this layered information management approach are beyond the scope of this paper, and are discussed in detail in [START_REF] Turunen | Jaspis -A Spoken Dialogue Architecture and its Applications[END_REF].

All components in the system are stateless, meaning they save all their internal data to the central database when they end processing (or go to Page 13 of 42 sleep), and update their internal models when they start processing (wake up) again. This way, the overall system is modular and can be extended easily. We believe this is a requirement needed to construct complex conversational system such as the Health and Fitness Companions, since it separates data from its processing, allowing efficient iterative development, for example, to visualize, debug, and teach complex systems easily [START_REF] Hakulinen | Visualization of Spoken Dialogue Systems for Demonstration, Debugging and Tutoring[END_REF]. Most importantly, it makes possible the architecture level modularization and adaption scheme described next.

Modularity is further supported in this architecture by the agents-managersevaluators paradigm that is used across all system modules. In this approach, all tasks are handled by compact and specialized agents located in modules and coordinated by managers. When one of the agents inside a module, such as the Dialogue Manager, is going to be selected, each evaluator in the module gives a score for every agent in the module. These scores are then multiplied by the local manager, which gives the final score, a suitability factor, for every agent. As an example, the multi-agent architecture of Jaspis [START_REF] Turunen | An architecture and applications for speech-based accessibility systems[END_REF] is used heavily in dialogue management; in the current A c c e p t e d M a n u s c r i p t prototype, there are 30 different dialogue agents, some corresponding to the topics found in the dialogue plan, others related to error handling and other generic interaction tasks. These agents are dynamically selected based on the current user inputs and overall dialogue context with using three rule-based reasoning evaluators.

For speech input and output, Loquendo ASR and TTS components (Loquendo, 2008) have been integrated into the Communication Manager. ASR uses grammars in "Speech Recognition Grammar Specification" (W3C) format. In these grammars, semantic tags in "Semantic Interpretation for Speech Recognition (SISR) Version 1.0" (W3C) format are used to provide information for the NLU component (described in detail in the following sections). Domain specific grammars were derived from early informal testing and Wizard-of-Oz sessions conducted during the iterative development of the system. The data collected from these initial studies has been used to build the HFC grammars. There is a set of about 50 grammars and the system dynamically selects these according to the current dialogue state. The grammar size for most grammars is about 1400 words, with a total of about 900 grammar rules. The vocabulary coverage is balanced across the four relevant domain parts of the activity model: transportation, physical activity, leisure and food. Natural language generation is implemented using a combination of canned utterances and Tree Adjoining Grammar-based generation. The grammar-based generation is used mostly in confirmations, while canned utterances with multiple options for each situation are used for most questions about users' day.

Mobile Companion

The Mobile Companion is realised by two components running on a Windows Mobile device, a Java midlet that handles the graphical user interface and the dialogue with the user, and a speech server that performs ASR and TTS operations on request by the midlet (see Figure 5). The Java midlet is built using the PART library, 7 and uses the Hecl scripting language 8 for GUI and dialogue management. The speech server uses Loquendo ASR (speakerindependent) and TTS libraries for embedded devices [START_REF] Loquendo | Loquendo embedded technologies: Text to speech and automatic speech recognition[END_REF], and SRGS 1.0 grammars. Pre-compiled grammars are loaded dynamically. The While the Mobile Companion itself is running as a stand-alone system, the communication with the Home Companion requires the Mobile Companion to have access to the Internet, for instance via WLAN or 3G/GPRS. The Mobile Companion maintains a persistent data store in device to store user model (name, age, gender, weight, etc.), and the exercise results. Saving the exercise results allows the Companion to compare the progress of an exercise with previous exercises of the same kind. For instance, if the Companion knows that the user is currently cycling from home to work, it can provide feedback on how the user is doing compared to previous sessions. This allows for status messages like "You are currently 1 minute and 23 seconds behind yesterday's time." The Mobile Companion is described in detail by [START_REF] Ståhl | A mobile fitness Companion[END_REF][START_REF] Ståhl | A mobile health and fitness Companion demonstrator[END_REF].

A c c e p t e d M a n u s c r i p t

Dialogue Management and Cognitive Modelling

As mentioned in Section 1 interaction management easily becomes too complex without clear separation of conversational, task, and domain specific information and components. To address these challenges, we introduce a novel solution for dialogue management and cognitive modelling in this domain.

Traditional architectures for dialogue includes three parts: an input module, which receives user input and parses it into a logical form, dialogue management, which maintains and updates dialogue state based on user input and generates output requests, and an output module, which generates natural language output to user based on the requests. In the case of the In the rest of this section we present how the DM of the Home Companion, the CM component, and the Mobile Companion interact to crete coherent long-lasting dialogues with the Companion.

Dialogue Plan

The communication between the Home and the Mobile Companion dialogue managers and the CM is based on a dialogue plan. Various kinds of dialogue plans [START_REF] Larsson | Information state and dialogue management in the TRINDI dialogue move engine toolkit[END_REF][START_REF] Jullien | Plan revision in person-machine dialogue[END_REF] have been used inside dialogue managers in the past. A plan usually models what the system sees as the optimal route to task completion.

In Health and Fitness Companion, the CM provides a plan on how the current task (planning a day, reporting on a day) could proceed. The plan consists of a list of domain specific predicates. Figure 6 During the interaction, new relevant information may become available from the user. The information includes statements on the user's condition (tired ), user commitments to the system (will walk to work ), user preferences (does not like cafeterias) and user reports on past activity (took a taxi to work ), which can be accomplishments or failures of earlier commitments.

The DM provides this information to the CM, piece by piece as it becomes available. As the CM receives the information, it updates the dialogue plan as necessary and builds up the day plan or report. Query type items, whose information has been gathered, disappear from the dialogue plan, while new items may appear when the dialogue progress.

As discussed above, the messages sent by the DM to CM can add new information (predicates) to the CM state. The DM can also remove information from the CM if previously entered information is found to be untrue. At the same time, the information is uploaded to a web server, where the Mobile Companion interface can access it any time. <plan> <plan-name>Generate-Task-Model-Questions</plan-name> <plan-item> <action>QUERY-PLANNED-ACTIVITY</action> </plan-item> <plan-item> <action>SUGGEST-TRAVEL-METHOD</action> <param>CYCLING-TRAVEL</param> <param>HOME</param> <param>WORK</param> </plan-item> The DM in the Home Companion can follow the dialogue plan produced by the CM step by step. Each step usually maps to a single question, but can naturally result in a longer dialogue if the user's answer is ambiguous, or error management is necessary, or if the DM decides to split a single item into multiple questions. For example, the two dialogue turn pairs seen in Figure 7 are the result of a single plan item (QUERY-PLANNED-ACTIVITY). Since the first user utterance does not result in a complete, unambiguous A c c e p t e d M a n u s c r i p t predicate, the DM asks a clarification question. A single user utterance can also result in multiple predicates (e.g., will not take the bus, has preference for walking).

When the Mobile Companion interface is activated, it downloads the current day plan from the web server and uses it as a basis for the dialogue it has with the user. The exercise which will then take place can be linked to an item in the day plan, or it can be something new. As the exercise is completed (or aborted), information about this is uploaded to the web server. The DM of the Home Companion can download this information from the server. This information is relevant to the DM when the user is reporting on a day. The DM reports the downloaded information back to the CM when the dialogue plan includes related items. The DM may also provide some feedback to the user based on the information, for example, how the exercise relates to the current plan and the overall situation. It is noteworthy that the CM does not need to differentiate in any way, whether the information about the exercise is originating from the Mobile Companion, or was gathered in a dialogue with the Home Companion.

Similarly, clarifications and confirmations are not directly visible to the CM. The DM can confirm items immediately (for example, when low confidence scores are reported by the ASR component) or it can delay confirmations to generate a single confirmation for multiple items at an appropriate moment.

In this model, the DM has freedom to choose how to operate on the base of the CM information. Most importantly, when presenting questions and suggestions to the user, the DM is free to choose any item in the plan, or even to do something not included in the plan at all. When information from the Mobile Companion is available, it can direct where the dialogue starts from. The DM could also decide to do some small-talk to introduce sensitive topics, which can be useful in managing the user-system relationship [START_REF] Bickmore | Establishing and maintaining long-term human-computer relationships[END_REF]. The communication that is not related to the domain does not reach the CM at any point. In another words, DM takes care of the conversational dialogue topics and meta-communication, while CM is involved in task-related topics.

In addition to the essential information, such as those presented in Figure 7, the CM can include additional annotation in the plan. First, DM can mark certain information requests as being important. If information is marked important, it is likely, but not certain, that the DM will explicitly confirm it. Second, CM can explicitly request a confirmation by generating (<pred> <action>PLANNED-ACTIVITY</action> <param>ACTIVITY-JOGGING</param> <param>unknownTime</param> </pred> ) C2: Is that jogging exercise before dinner? U2: No, its after.

( <pred> <action>PLANNED-ACTIVITY</action> <param>ACTIVITY-JOGGING</param> <param>AFTER-DINNER</param> </pred> ) Figure 7: A dialogue fragment and corresponding plan items and predicates a separate plan item. For example, if the users report on having run much more than they are likely to be capable of in their condition, the CM can generate a confirmation plan item. It is worth noting, that the DM cannot do reasoning on such a high level (as it would be not meaningful to duplicate such information in DM), and therefore the CM must participate in error handling in such cases.

Cognitive Modelling

In order to make the interaction with the Health and Fitness Companion coherent, we need to provide a shared cognitive model for the Companion.

The system architecture provides support for this in the form of the separate The Health and Fitness dialogues have currently two phases. The planning phase involves the system operating through a global interaction cycle, integrating dialogue with planning to construct the user's activity model. This process is illustrated in Figure 9 with a detail of the graph in Figure 10. The first step consists of generating an initial activity model for the user, based on default knowledge, along with any previously captured information on user preferences. Throughout this interaction cycle, each time the Planner generates a candidate activity model, it generates a corresponding dialogue plan, which is then used as a basis for the dialogue to enquire about user preferences and make suggestions in relation to the planned activities.

User responses are used to update the activity model, with the user utterance mapped to the predicates used within the planning domain and semantic categories associated with the domain methods. If a user response validates part of the existing activity model, that part of the model is marked as planned and will not appear in the dialogue plan. If the user response is incompatible with the current activity model, either through explicit rejection or stating of a conflicting preference, the preference is stored so that the relevant items remains unplanned and inactive.

The cycle continues with the activity model being re-generated. Those tasks that have been accepted, that is, marked as planned, are preserved while the unplanned parts are re-planned making use of the latest preferences provided by the user. An updated dialogue plan is then generated and the dialogue with the user continues until the user has agreed on a fully planned The reporting phase is accomplished in a similar manner, with a dialogue plan asking the user questions about what he/she did and the user utterances being used to produce an activity model with completed activities.

Language Understanding and Dialogue Management in the Home Companion

Natural language understanding relies heavily on SISR information (see previous section). The context specific ASR grammars contain tags, which provide logical items to NLU component as part of ASR results. These logical items are either complete predicates, or in most cases, individual parameter values and lists of parameter values. This provides a basis for further input processing, where input is parsed against the current dialogue state to compile full, logical representations compatible with the planning implemented in the Cognitive Model. Input is parsed first against the previous system output. In practise, NLU component tries to unify the logical items into the space of predicates, which match the last system utterance. For example, if the utterance was "Why don't you cycle to work", the possible predicate space for anwers includes acceptance and rejection of the suggestion, specification of an alternative travel method, and an expression of preference for or against the suggested travel method. If user input was "yes, that would be fine", the SISR based item is "ACCEPT-TASK", and the unification results in the complete predicate "ACCEPT-TASK TRAVEL-METHOD CYCLING-TRAVEL HOME WORK".

If the first attempt does not result a full parse, then a new predicate space is built from predicates related to the current topic. In the case of the above example, this would be everything related to the different possibilities when traveling to work. The final parsing option is the parsing against the entire space of known predicates. This set includes generic things like help requests A c c e p t e d M a n u s c r i p t and enabled topic shifts in this domain. If needed, deeper analysis can be made by parsing against the entire dialogue history and the current plan. This way, the information produced by the CM is used in input parsing. The dialogue plan could also be used in dynamic construction of recognition grammars to support this on ASR grammar level.

In addition to unification of predicates, a reduced set of DAMSL dialogue acts [START_REF] Core | Coding dialogs with the DAMSL annotation scheme[END_REF]) is used to mark functional dialogue acts using rule-based reasoning.

As said, the task of the DM is to maintain and update a dialogue state. In the Home Companion, the dialogue state includes a dialogue history, a stack of active dialogue topics, and current parsed user input, including ASR confidence scores and N-best lists. In addition, two pools of items that need to be confirmed are stored; one for items to be confirmed individually and another for those that can be confirmed together with a single question.

The DM receives user inputs as predicates parsed by the NLU component. The DM is aware of the relations of the predicates on the topics level, i.e., it knows which predicates belong to each topic. This information is used primarily for parsing input in relation to the dialogue context. The DM also has understanding of the semantics of the predicates which relates to interaction. Namely, relations such as question answer pairs (suggestion agreement, confirmation acceptance/rejection, etc.) are modelled by the DM.

If an utterance is successfully parsed and matches the current dialogue plan, the DM does not need to know the the actual meaning of the input. Instead, it takes care of confirmations, when necessary, and provides the information to the CM. Similarly, when generating output requests based on the plan, the DM can be unaware of the specific meaning of the plan items. Overall, the DM does not need to have the deep domain understanding the CM specializes in.

Dialogue management is implemented as a collection of separate compact dialogue agents, each taking care of a specific task. In each dialogue turn one or more agents are selected by the DM, as presented in Section 3.1. In the current Home Companion, there are over 30 dialogue agents. There is a separate agent for each topic that can occur in the plan. In practice, one topic maps to a single planitem. These agents are all instances of a single class with specific configurations. Each agent handles all situations related to its topic; when the topic is the first item of an active plan, the agent produces related output, and when the user provides input matching to the topic it forwards that information back to the Cognitive Model. In addition, A c c e p t e d M a n u s c r i p t topic specific agents handle explicit topic switch requests from the user (e.g., "let's talk about lunch") and also take turns if the topic is found on top of the dialogue topic stack. A topic ends up on the stack if it has not been finished when a new topic is activated. The Dialogue manager processes stack items before returning to dialogue plan items. Currently, we have not encountered any situations where the dialogue stack needs to be flushed. This might be necessary in more complex dialogues.

The generic, non-topic specific agents include one that generates a confirmation if the ASR confidence score is too low, one that repeats the last system utterance when the user requests it ("eh, come again? "), and an agent to handle ASR rejection errors. There is no need to model such information in the CM.

Dialogue Management in the Mobile Companion

In the Mobile Companion, the dialogue with the user is managed by the script code running inside the Java midlet, using a finite state machine model. At any point in time there is one active dialogue state. Each state is represented by four script procedures, enter, leave, input, and error.

A typical use of these procedures in the Mobile Companion is to output a question in the enter procedure, and then analyse the user's reply and move on to another state or ask another question in the input procedure. The leave procedure is used to do state clean-up tasks (if necessary), and the error procedure will output an error message and repeat the question.

SISR expression are used in the Mobile Companion as well. The SISR expressions consist of predicates of the form #TASK_SELECTED (walking, home, work) and are used directly by the script code to analyse the input. The user can provide input using voice, by pressing buttons, or tapping on a list on touch screen. However, all user input has the same form, so it does not matter how the input was produced.

Natural language generation is handled by the state procedures directly. There are some canned utterances and some dynamically constructed output, for instance, based on the content of the activity plan downloaded from the Home Companion.

Page 25 of 42 A c c e p t e d M a n u s c r i p t

Benefits of the Model for Conversational Distributed Dialogues

The presented model for interoperability between the Mobile Companion, the DM of the Home Companion, and the CM has provided great flexibility to each component. While the dialogue plan generated by the CM provides a base for dialogue management, which, in most cases, is followed, the DM can deviate from it when feasible. The DM can handle confirmations as it pleases, add small talk, and process the plan items in any order. The model also supports mixed-initiative dialogues; while the DM may follow the plan, the user may discuss any topic.

Most importantly, all this is possible without including domain specific knowledge into DM. All such information is kept exclusive in the CM. Similarly, the CM does not need to know the interaction level properties of the topics, such as recognition grammars and natural language generation details. These are internal to their specific components. The Mobile Companion uses the same knowledge representation as the CM, but the CM does not need to be aware of its existence at all. Similarly, the Mobile Companion can use any part of the information it receives, but is not forced to do anything specific. The DM just feeds all the information to the mobile Companion and lets it decide what to do with the information. When the Mobile Companion provides information back to the Home Companion, the DM handles the access to the information and the CM can ignore completely the fact that different parts of the information it receives were generated using different systems. Similarly, the Mobile Companion does not see any of the internals of the Home Companion.

At the implementation level, the model is independent of the mechanics of either the DM or the CM. The DM can be implemented using state transition networks (a network per plan item), forms (form per item), an agent-based model, like in the case of the Mobile Companion, or any other suitable method. Similarly, the plan does not tie the CM to any specific implementation.

Evaluation of Companions

Since Companionship in the sense described in the present paper is a fairly new concept, no agreed-on evaluation strategies of it exist, and one of the goals of the Companions project is to develop such evaluation strategies. In order to gain deep understanding on the Companions paradigm, user studies must be conducted both in laboratory and field settings. Thus an initial set 25 A c c e p t e d M a n u s c r i p t of user studies of the Health and Fitness Companion prototypes has focused on out-of-the-box functionality to set a baseline for further studies. Two mechanisms have been utilized for the evaluation: qualitative and quantitative. Qualitative surveys are used to acquire subjective opinions from the users; including Likert-based surveys, focus groups and interviews, while the quantitative measures are based on the following set of metrics:

Speech Metrics, such as WER (word error rate) and CER (concept error rate);

Dialogue Metrics, e.g., dialogue duration, number of turns, word per turn dialogue structure;

Task Metrics, e.g., task completion; and

User Metrics, e.g., user satisfaction, requirement elicitation.

The metrics mentioned are quite traditional, and they are included in evaluation frameworks such as PARADISE [START_REF] Walker | PARADISE: a framework for evaluating spoken dialogue agents[END_REF]. One option would have been to use such a complete evaluation paradigm. However, our aim was a lightweight evaluation at this phase of the project instead of running a comprehensive and laborous evaluation. Furthemore, the existing methods are mainly focusing on task-driven dialogues, and while we use here metrics such as Task Completion rate (see definition below), our tasks are rather loose, so we believe it would not be feasible to try to find similar correlations between objective and subjective metrics as in more more straightforward task-oriented systems.

Two evaluation sessions have been carried out so far. The first evaluation session focused on WER and CER metrics for the Home Companion. The CER is based on correctly identifying the semantic result of a user utterance. For example, correctly identifying whether the user accepted or rejected a proposal from the HFC. If a user utterance contained distinct semantic results, these were then counted as separate concepts when counting the score. For example, if in reply to a proposal to have a rest after dinner the user replies "No, I'll play a game of football." the system would have to identify both a rejection of the HFC's proposal and a counter-proposal of playing football. Since we wanted to find out a baseline for further work, we did not include any clarification dialogues to this evaluation. Furthermore, there Page 27 of 42 A c c e p t e d M a n u s c r i p t were no issues such as grounding taking place in the dialogues. For these reasons, there was no need to model these when calculating CER.

Another metric calculated is Task Completion rate, which in the case of Health and Fitness companion is specified as the percentage of the activity model correctly instantiated. That is, it provides the percentage similarity between the activity model as instantiated and the instantiation of an ideal activity model with all utterances optimally understood.

The study involved 20 subjects who interacted with the Home Companion in both planning and reporting phases. They were briefly introduced to the concept of the Companion and the scenario, and provided with a set of slides illustrating (via images of activities and food types) what was known to the system. To avoid bias, the subjects were not shown examples of possible utterances nor allowed to witness experiments with other subjects.

For the planning dialogues, the WER was 42% and CER 24%. For the reporting dialogues, the WER was 44% and CER 24%. The fact that no user training or speaker adaptation was carried out, along with the realistic experimental conditions, explains the high Word Error Rate level. The Concept Error Rate is lower, indicating some resilience to misrecognition of portions of the user utterance. Still, Task Completion was high, with, on average, 80% of the Activity Model being correctly instantiated in planning dialogues and 95% being correctly instantiated in reporting dialogues. This is better than the corresponding per utterance CER as the system was able to eliminate some errors over the course of the entire dialogue.

The significant improvement of CER over WER is due to the reasonable robustness in identifying the semantic meaning of the user's utterances. This is partially due to the semantic results being constrained to a relatively low number of possibilities compared to the reasonably large vocabulary user input to the ASR. For example, in response to the HFC asking "How about bringing a packed lunch to work?" the user responds "Yes, I will make a sandwich". When this is recognised as "yes I love that sandwiches" this results in a WER of 50% but the system is still able to identify the concepts (agreement to the proposal) correctly. There is a similar degree of robustness when instantiating the activity model. For example, the HFC can infer from a counter-proposal that the original proposal was rejected despite potentially missing out on an explicit rejection.

Also worth noting is that reporting dialogues tended to involve simpler user utterances, such as basic confirmations, than those in the planning phase, which was reflected by smaller average utterance length and higher A c c e p t e d M a n u s c r i p t task model completion rate. The initial results show that even with relatively high WER acceptable task completion rates can be obtained in this domain.

In the second evaluation session, the Home Companion and the Mobile Companion were evaluated in more a complex setting. Eight participants completed a study protocol of four distinct tasks: introductory tutorials, using prototypes, on-line surveys, and interviews. Each session began with an introductory tutorial. These presentations (10-16 slides) introduced the prototype, established its intentions, its limitations, what the prototype would say and do, how to use the prototype, and gave the user suggestions in how to respond. Participants then used the Home and Mobile Companion for about 10-15 minutes each, completing an on-line questionnaire after each session. Researchers were sitting in the background while the participants interacted with the prototypes, and the participants were video-taped during their interaction with the systems. At the end of the session, each participant was interviewed. In this study, the Home Companion language model was more complex than in the previous session evaluation. The Word Error Rate ranged between 51% and 79% (however, it must be noted that in one case the error rate was over 100% because of massive amounts of rejection errors). While the Word Error Rate was extremely high, the Concept Error Rate (calculated by ignoring the order of recognized concepts) was somewhat lower, ranging between 33% and 65%. These numbers are still high, but most of the time the Concept Error Rate was reasonably good, with the majority of the errors being insertions of several concepts in some specific cases. The average length of user utterances varied between participants from 3.0 to 8.3 words for the Home Companion, while the average system utterance length was 12 words. Even though the data set is too small for statistical testing, we could see differences in how verbose different people were. The user vocabulary size was surprisingly small, and only varied between 18 and 116 words, with an average of 55 words.

Future plans for the evaluation efforts include to target areas of performance required in long-term collaborative conversational agents. These are discussed in more detail in [START_REF] Benyon | Evaluating human-computer conversation in Companions[END_REF].

Summary

In this paper, we have introduced the concept of multimodal Companions that build long-lasting relationships with their owners to support their everyday health and fitness related activities. While traditional spoken dialogue Page 29 of 42 A c c e p t e d M a n u s c r i p t systems have been task-based, the Health and Fitness Companion is designed to be part of the user's life for a long time, months, or even years. This requires that they are part of the life physically, i.e., that interactions can take place in mobile settings and in home environments outside of traditional, task-based computing devices. With the physical presence of the interface agents, and spoken, conversational dialogue we aim at building social, emotional relationships between the user and the Companion. Such relationships should help us in motivating the users towards a healthier lifestyle. The mobility of the interface integrates the system into the physical activities it aims at supporting users in.

We have shown what kind of architectures such distributed agent interfaces need, and how they can be realized with proper dialogue and cognitive management techniques. When dialogue systems move beyond limited taskbased domains and implement multimodal interfaces in pervasive computing environments, complexity increases rapidly. Dialogue management, which in most cases is handled with well-understood methods such as form filling or state transition networks, tends to grow more complex. Therefore, a model to modularize dialogue management and domain reasoning is needed. At the same time, distributed systems require various kinds of information to be communicated between components and systems.

We have presented a novel interaction management model, which separates cognitive modeling from dialogue management and enables flexible interoperability between these two. This model also enables sharing the gathered knowledge to the mobile part of the system and back. This division, while similar to separation of a back-end from dialogue management, draws the line deeper into the area of interaction management. The presented model has been implemented in the Health and Fitness Companion prototype, and it has enabled the Cognitive Model, the Dialogue Manager, and the Mobile Companion interface to be developed in parallel by different groups using various programming languages.

On software architecture level, the solution has enabled the Dialogue Manager to focus only on interaction level phenomena, such as initiative and error management, and other meta-communication while the congitive model takes care of the domain level processing. The Dialogue manager can also include input from a mobile interface of the system without making this explicit to the cognitive model. One example of flexibility is error management; while the actual error correction is the task of the Dialogue Manager, domain level knowledge can reveal errors. Using the dialogue plan, the cog-A c c e p t e d M a n u s c r i p t nitive model can provide such information to the Dialogue Manager without knowledge of details of error management. The model also enables user initiative topic shifts, management of user-system relationship, and other such issues relevant in domain-oriented dialogue systems.

In overall, the model presented has enabled a clear division and interoperability of the different components handling separate parts of the interaction. We can recommend the model for other applications, where complex domain modeling is used in a situation where error correction and other dialogue managements task must be handled as well. Using the model, the different component can each focus on their own tasks.

In addition to the interaction model presented, we have produced concrete software components to implement further Companion applications. The PART system used to implement the Mobile Companion and the Jaspis architecture (including the jNabServer software) used for the Homa Companion have been released as open source software. Together, they can be used to construct similar distributed multimodal applications with virtual, physical, and mobile conversational agents.

Finally, we have introduced the first evaluation results of the Companions paradigm. These initial results show that even with relatively low speech recognition accuracy meaningful conversations can be carried out in this domain.

Page 31 of 42

( 1 )

 1 During breakfast C1: Do you have anything arranged for today? U1: I'm playing Squash C2: Is that before dinner? U2: Yes C3: Okay. Are you going to take the train to work? U3: Yes C4: Right. You could take in a packed lunch to work? U4: Yes, that would be fine (2) Later on in the evening C5: Welcome back! How did your game of Squash go? U5: The game was cancelled, so I went to the gym C6: Okay. Did you take the train to work? U6: Yes, I did C7: Right. Did you take in a packed lunch? U7: No. I was in a rush so I went to the cafeteria A physical exercise with the Mobile Companion C8: Hello, what kind of exercise would you like to do? U8: I would like to do some running C9: Ok, running it is. Select start when you are ready.

  Figure 1:The Home Companion interface in a typical usage situation (see http://www.youtube.com/watch?v=KQSiigSEYhU for a video presentation)

Figure 2 :

 2 Figure 2: Mobile Companion interface

  9

Figure 3 :

 3 Figure 3: Cooking Companion interface

  A c c e p t e d M a n u s c r i p t

Figure 4 :

 4 Figure 4: The Home Companion architecture

7

  Pervasive Applications RunTime: part.sourceforge.net 8 www.hecl.org Page 15 of 42 A c c e p t e d M a n u s c r i p t

Figure 5 :

 5 Figure 5: Mobile Companion Architecture

  Health and Fitness Companion, a special component, Cognitive Model (CM) is introduced as separate component from the Dialogue Manager (DM), as seen in Figure 8. The Cognitive Model contains what can be considered the higher-level cognitive processes of the system. The Dialogue Manager takes care of conversational strategies. It presents questions to a user based on the dialogue plan, maintains a dialogue history tree and a dialogue stack, and communicates facts and user preferences to the Cognitive Model. The Dialogue Manager also handles error management, supports user initiative topic shifts, and takes care of top level interaction management, such as starting and finishing of dialogues. The Dialogue Manager and the Cognitive Model communicate using a dialogue plan.

  contains two items from the start of a plan for planning the day with the user in the morning. The first plan item (QUERY-PLANNED-ACTIVITY) can be realized as the Page 17 of 42 A c c e p t e d M a n u s c r i p t question "Anything interesting planned for today? " by the NLG component.

Figure 6 :

 6 Figure 6: Start of a plan

  Good morning. Anything interesting organized for today? U1: I'm going jogging.

  19

Figure 8 :

 8 Figure 8: Dialogue management and cognitive modelling components and their information passing

Figure 9 :

 9 Figure 9: Activity model cycle

Figure 10 :

 10 Figure 10: Activity Model Items

  Figure

Figure

  Figure

Figure

  Figure

Figure

  Figure

http://www.parorobots.com/

www.nabaztag.com

Haptek Player, Haptek Inc.: www.haptek.com

www.cs.uta.fi/hci/spi/Jaspis

Java-based Nabaztag/tag Server: www.cs.uta.fi/hci/spi/jnabserver

www.companions-project.org

Acknowledgements

This work was funded by the COMPANIONS project 9 sponsored by the European Commission as part of the Information Society Technologies (IST) programme under EC grant number IST-FP6-034434.