Rainer Holve

Silke Goronzy

A Model-based Approach towards Human-Machine-Interfaces

Keywords:

Specification and development of Human-Machine-Interfaces (HMI) for infotainment systems pose a difficult challenge for those automotive OEMs who claim this topic as one of their core competences. Experience shows that a major problem arises from the non optimal communication of all involved parties during the HMI development. This article describes a model-based approach towards HMI-development which combines conceptual design, specification and implementation based on a single complete HMI-model which serves as an improved means of communication between designers, ergonomists and software developers.

Introduction

The complexity of display-based driver information systems (DIS) in passenger cars steadily increases. While current vehicles face the driver with usually a single monitor, future systems will include fully graphics based cluster instruments, head-up displays and monitors for rear seat entertainmentall of this probably combined with speech dialogue systems. Thus, the DIS gain such a prominent role inside the vehicle that they have a considerable impact on the entire vehicle's usability and the Human Machine Interfaces (HMI) of the information systems become key factors for developing and preserving customer retention. It is no surprise that the leading automotive manufacturers (OEM)especially those who offer cars in the premium segment -now regard HMI related issues as their core competence and no longer rely on first tier suppliers' "off the shelf" solutions.

HMI as OEM Core Competence

Currently, the approaches followed by the OEMs to express their HMI-related core competence can be characterized into three levels, each coming with certain tasks and challenges for both, the OEM and the first tier supplier.

Select and Modify

In this weakest form of competence, the major burden of HMI design and development remains at the first tier supplier. However, the OEM expects to be presented with various different designs from which he can choose. Also, the OEM might demand minor changes to the design such as varying the colour schemes or selecting certain fonts and bitmaps. Therefore, the OEM needs personnel with a background in design and human factors who can make qualified decisions while evaluating the design proposals of the first tier. The supplier on the other hand must be able to produce and modify several different HMI designs and present them in a form suitable for a qualified evaluation by the OEM (rapid prototyping, simulation).

Design and Specify

The currently most common approach followed by the leading OEMs focuses on a complete and detailed specification of the HMI related issues. Here, the OEM determines the look and feel for a new DIS by a user centred design process [START_REF] Hamberger | Toolsuite for multi-modal operating concepts: Usabilityengineering from concept to code[END_REF], i.e. a large number of different aspects need to be addressed like fonts, bitmaps, layouts, application flow, internationalization, speech dialogue etc. For this, the OEM needs to employ a group of highly skilled people who know how to use various design tools, perform usability studies and communicate their design to the first tier supplier. The supplier needs to establish a software architecture which allows to easily exchange one OEM-specific HMI with another and to incorporate the specification provided by the OEM as efficiently as possible.

Design and Develop

In this strongest form of expressing HMI competence, the OEM not only precisely specifies the HMI for his new system (see 2.2) but also performs the implementation of parts of the HMI software himself [START_REF] Bruckmeier | Human-Machine-Interface: Software becomes OEM core business[END_REF]. So, additionally to the detailed specification, the OEM has to deal with all responsibilities that come with SW-development including the maintenance phase which easily stretches over several years after the initial launch of the new DIS. The advantage of this approach lies clearly in a maximized reuse of the HMI software and thus drastically reduced development costs. The first tier supplier now faces the task to integrate software components developed by a new party and he has to provide a software platform that can easily be adapted to different APIs as they are required / expected by the OEM.

Problematic Aspects of HMI Development

Regardless of the level of competence followed by the OEM, a development project for a new DIS HMI has to deal with a number of issues which are responsible for occurring problems.

Time Pressure

Currently -and very probably in the future as well -OEMs are integrating more and more features to their automobiles.

Most of these features are software driven and either provide more information to the driver or assist him/her in handling the vehicle. It is reasonable to assume that the bulk of these new functions will be integrated in or accessible through the DIS, causing an increased complexity of the HMI. While this alone is not a major problem per se, OEMs are reacting to the diversification of the automotive market by offering multiple variants of vehicles in different geographical market segments. These two aspects have to be managed within steadily shortened development cycles.

Change Pressure

As mentioned above, an increasing number of features are to be integrated in a decreasing amount of time. Driven by the very short development cycles of e.g. the consumer electronics market, features are bound to be modified, added or removed from the current planning on a very short notice. Also, many changes become necessary when technical restraints of the target hardware and/or software platform emerge when the development is already in an advanced stadium. Lastly, it is not unheard of that certain changes are caused by "management override", simply because a certain aspect of a HMI's look & feel does not appeal to executives who review the finished system only at milestones with a relatively high grade of completion.

Communication Barrier

Through the mentioned aspects of time and change pressure, a third facet gains increasing relevancethe communication between all parties in the development process of the HMI. Currently, in most projects the communication flow is similar to the one described in Fig. 1. On the OEM side, driven by the desire to describe the newly developed HMI as precisely as possible, a very heterogeneous team of graphics designers, ergonomists, speech dialogue designers, translators etc. create a set of specification documents. This set of documents (mostly simply paper based and not machine readable) is then handed over to the developer team of the first tier supplier. While this approach is -in theory -perfectly valid, real projects suffer from a couple of drawbacks.

Figure 1: Classical communication flow

Firstly, the people involved all come from different areas of expertise and -necessarily -approach the project from very different viewpoints. Naturally, everyone expresses him/herself in a way that is most convenient and most suitable to the task at hand. As a result, the developers of the first tier suppliermostly engineers -are faced with a multitude of specification documents, created with different tools, containing information in different "languages". In the end, only the final implementation of the target system will add a real semantic to the specification of the OEM since there is no way to get a complete impression of all facets covered in the specification documents. Secondly, driven by time and change pressure, the specifications are not simply created once and then left to the OEM for implementation. Rather, the OEM's designers will expand and modify the specifications several times, even when the implementation process of the first tier is right under way. Thus, the processes on the OEM side and the tier one side are parallel and iterative. Now things can become really complicated: Is the modified specification still consistent (i.e. are all cross references between the different documents up to date)? Are all parties involved informed in time about the modifications? How do changes fit into the feature release schedule? Which sample of specifications is currently in effect as agreed upon by the OEM and the tier one? 3.4 Room for Improvement? Given these aspects -time pressure, change pressure and communication difficulties -which when combined pose a serious threat to a successful project completion -one might ask where the room for improvement lies. Realistically, time pressure and need for changes are very much market driven and can hardly be averted from within the HMI development. Rather, we will probably have to cope with the fact that specification and realization of HMI related issues are iterative processes and will take place in parallel. That leaves us with the communication process used to transport ideas between the involved parties and the way they interact.

Model-based HMI Development

4.1 Principal idea Instead of the "classical" and mostly unidirectional communication flow from the OEM to the tier one, based on paper bound specification documents, we propose a model-based approach (see Fig. 2). The idea behind this is to establish a cooperative process around a central HMI model that contains all relevant information concerning the HMI in a uniform way.

Benefits of the model-based approach

The main goal one hopes to achieve with this approach is to keep the HMI model (i.e. the description of how the final HMI should look like) consistent at all times and to get all involved parties informed immediately when changes to the model are made. Additionally, a model that contains all relevant data in a machine readable fashion lends itself to an early simulation and rapid prototyping of the HMI. Thus, the HMI can be evaluated, "debugged" and presented for acceptance quite early in the specification process, even before the tier one starts the implementation. The tier one on the other hand can draw a massive profit from the uniform HMI model by incorporating it in his software development process by the means of automatic code generation. This can drastically decrease the development time and costs, improve the reactivity to change requests of the OEM while increasing the quality of the HMI software at the same time.

Requirements for a model-based approach

In order to perform properly and to deliver the mentioned advantages, there are a set of certain requirements for the HMI model, the tools used to modify it and the way to cooperate with it.

Abstraction versus detailedness

As mentioned above, the people involved in the HMI development, coming from the OEM and the supplier, form a very interdisciplinary group. Thus, the semantic features of the HMI model must be flexible enough to express the properties of the modeled HMI in different grades of abstraction according to the needs of the various involved designers/ engineers. To give an example: Modeling, modifying, simulation and evaluation of a HMI could (and quite often actually is -at least in the early phase) quite easily be carried out by means of macromedia flash on the side of the OEM. For the tier one on the other hand such a model is hardly more useful than a paper based specification since the suitability for code generation of a flash animation is virtually nil. The other extreme would be to have the ergonomists and dialogue designers specify the HMI in form of configuration scripts suitable for a code generator which outputs target code for the embedded hardware platform of the tier one. This is also hardly a passable way. Thus, the means of expression provided by the HMI model have to perform a "balancing act" between the abstract view HMI designers favor and the grade of detailedness needed by the software engineers to incorporate the model into their code generation process.

Completeness

In order to not leave out certain facets of the developed HMI, the model has to cover all relevant aspects in a suitable form. To mention the most important: Graphics/Layout This concerns the arrangement of single screen menus (other common terms are view, panel or mask) as well as the look and behavior of the basic graphical elements (widgets) like e.g. buttons, sliders, dropdown-boxes or spellers. The choice of a suitable font and font sizes as well as different colorschemes for e.g. daylight and night-design are also important.

Application flow A modern driver information system covers literally hundreds of different functions and is therefore composed of a huge number of distinct menus. These functional menus must be arranged in such a fashion that the user can easily learn and memorize how to use the system. While the application flow is mainly driven by the user's inputs via keys, rotary knobs or voice commands, internal system events must also be considered Internationalisation Current DIS offer the possibility to change the system's language at runtime. So, all static HMI texts have to be defined in several languages and one has to consider different character sets as well as language dependent icons and units (liter-gallon, km/h -mph, bar -psi etc.) Variants Although The OEM wish to establish a brand specific HMI that is as uniform as possible, it is necessary to consider variants of a HMI's look &feel. This stems from different equipments throughout an OEM's models (e.g. color/monochrom) as well as from different customer expectations in the different markets, which require different application flows.

Speech dialogue

The increasing functionality of DIS leads to an increasing utilization of speech dialogue systems (SDS) in vehicles. A complete HMI concept must therefore consider a suitable combination of graphics, haptics and speech to feature a multimodal dialogue (see also [START_REF] Goronzy | Fully exploiting the potential of speech dialog in automotive applications[END_REF]).

Ability to simulate

In order to make maximum use of a model-driven development, the HMI model must lend itself to a simulation that comes as closely as possible to reality and recognizes all aspects covered by the model. Ideally, the simulation is permanently accessible during specification without the necessity of model transformation, compilation or invocation of additional tools.

Machine readability

Regarding the requirements mentioned as well as the described problems with the paper based specification, this demand certainly goes without saying. It is also a necessity when one thinks about how a model driven approach can be integrated in the already existing processes and tools used by the designers and developers. Generally, the format of the HMI model is arbitrary, but the general acceptance of XML and the wide availability of parsers and tools strongly recommend a XML-based storage format.

Tool based modeling process

The complexity of a HMI model that serves the desired purposes is necessarily very high. Thus, in order to work with such a model, we need a modelling tool (or a set of tools) that supports the modelling process in all stages of development and provides access to all aspects of the model's semantics. Specifically, the toolbase needs to fulfil the following requirements: Specialized editors The many different aspects of a complete HMI model as well as the different people who will create the model make the necessity for specialized editors apparent. The editors should provide access to the semantic model facets in a fashion that suits the professional domain of the most likely user as well as possible. Simulation The simulation of the HMI model should be integrated in the tool suite as seamlessly as possible, so that the user can evaluate his modifications to the model "on the fly" without the need to transform the model code or to launch external tools. Preservation of model consistance A HMI model of any kind that tries to fulfil the mentioned requirements will consist of literally thousands of different model elements that are highly connected and dependent on each other. Unfortunately, the more interdependencies a model has, the easier it is to mess it up with a thoughtless modification, e.g. the deletion of an element that other elements still depend on. Therefore, a tool base for HMI modelling should offer permanent consistency and syntax checks as well as the ability to gather information about the model itself, e.g. which and how many other model elements depend on a particular element. Multi-user support Since the whole approach described in this article circulates around a centralized HMI model which is accessed by multiple users, the support of multiple parallel accesses to the model as well as e versioning system is indispensable. Embedding of code generators In order to enable the tier one supplier to integrate the HMI model into his software development process, it is preferable if the modelling tool allows embedding code generators which transform certain model facets to source code for the target system. While this feature is not absolutely necessary, since the code generation can as well be based on the XML format of the HMI model as input, the direct embedding of code generators can help to speed up the development process.

Openness for extensions This requirement stems from two sources. Firstly, the integration of a model based approach into an already existing development process calls for the ability to adapt to the existing process by integrating available or newly developed tools into the modelling tool base. Secondly, technological progress in HMI related issues will confront us with new technologies, paradigms and requirements. In order to be able to adapt the HMI modelling process to these new challenges, the ability to add new features to the tool base is essential.

A proposal for a HMI model

In this chapter we briefly describe the structure of a HMI model that has been applied successfully to the development of several automotive and nonautomotive HMI development projects. 3, the model consists of five submodels which are interconnected by references, actions and condition. Combined, this features a HMI model that is suitable for simulation as well as for code generation for embedded target systems. The different sub-models are depicted in the subsequent sections.

Data-Model

The requirement of an immediate "on-the-fly" simulation of the HMI model makes it necessary to sufficiently describe all data represented by the HMI during the specification. The proposed HMI model utilizes the principle of a globally visible "data-pool" which contains all data relevant to the HMI. The other sub-models can arbitrarily refer to this data and request a notification if certain data is modified (observer pattern). Assumptions about the origin of the data are not necessary; the data-model therefore follows the motto:

"I know that the data exists and I notice when it changes, but I don't care where it's coming from."

Once defined, all elements of the data-model are available to all other sub-models and a notification mechanism informs about changes to the datamodel anytime. A variant mechanism within the proposed HMI model allows for holding certain subsets of the data-model in arbitrarily many variants which enables the modeler to handle aspects like skinning (daylight, nighttime design) and internationalization (different text translations and language dependent layout parameters). 6.2 Event-model The event-model describes all events, which affect the specified HMI (e.g. user input via keyboard or speech, external events like an incoming phone call or system events like error messages or warnings). As the content of the data-model, elements of the event-model are globally visible and can be referenced, fired ore processed by other submodels.

Layout-model

The layout-model contains the description of all views (also called masks, panels, screens, contexts) within the HMI-model. Views are constructed from widgets (graphical base elements with a predefined behaviour like buttons, sliders, checkboxes etc.). The widgets are linked to the data-model to provide them with the necessary data as well as to the event-model to trigger actions after occurrence of certain events, e.g. user input.

Flow-model

The application-flow-model combines all defined views to a complete menu structure. The model uses UML-like statecharts which are illustrated with small depictions of the referenced views (see Fig. 4). The transitions between the certain states are labeled with events from the event-model and can also be extended with conditions to model even complex application flows.

Model-driven HMI development process

The described HMI model has already proven suitable in various development projects for automotive as well as for non-automotive HMIs. The underlying development process, however, is very closely coupled. This means, that all involved parties are required to share a central model. While this process is quite mature, effective and very efficient, there is still room for improvement. Depending on the level of trust, the geographical proximity or the underlying business model between OEM and tier one, it might be desirable to feature a more loosely coupled development process (see Fig. 5). In such a process, the OEM periodically provides an increasingly complete simulation model to the tier one who extends the model to get an implementation model suitable for code generation. However, since the two parties no longer share a single HMI model and work quite asynchronously we now face the need for a merge process that integrates changes in the OEM's simulation model into the (older) implementation model of the tier one. The implementation of such a model-merge is quite a challenge and the efforts undertaken by the authors are not yet completed.

Conclusion

This paper has motivated the need for a model driven approach towards HMI development to

Figure 2 :

 2 Figure 2: Model-based HMI development

Figure 3 :

 3 Figure 3: Structure of a HMI model As shown in Fig. 3, the model consists of five submodels which are interconnected by references, actions and condition. Combined, this features a HMI model that is suitable for simulation as well as for code generation for embedded target systems. The different sub-models are depicted in the subsequent sections.

Figure 4 :

 4 Figure 4: Modeling the application flow

Figure 5 :

 5 Figure 5: Loosely coupled development process

overcome communication difficulties between the involved parties. The requirements for such a HMI model as well as for the underlying tool base were outlined. The described model driven approach developed by the authors has proven successful in several embedded HMI development projects. While the closely coupled model drive development process based on this approach is quite mature, further work is necessary to allow for a loosely coupled process which requires the ability to merge two HMI-models.


M Sabeur

D Schulz

M Schabacker

S Vajna

Vajna Insides -A New

INSIDES -A new Virtual Prototyping Platform of Human Machine Interactions Systems for Automotive and Aerospace Applications

whether they are published or not. The documents may come

INSIDES -A new Virtual Prototyping Platform of Human Machine Interactions Systems for Automotive and Aerospace Applications

Abstract: Human Machine Interactions Systems are decisive for the acceptance and the safety of new cockpits in the automotive as well as in the aerospace industries. A new design and simulation platform called INSIDES will be presented where virtual cockpit prototypes are being built based on 3D CAD geometry e.g. from CATIA and integrated with logical interaction data derived from UML specifications. This new development platform enables the continuous validation and check of new interaction concepts by involving usability engineers in the very early stage of the development cycle. Since the simulation work is being done in the context of the entire aircraft cockpit/car interior with all instruments, control commands as well displays devices a better validation of the HMI systems can be achieved.

Motivation

Human and machine interact (HMI) via a so-called human machine interface, which is influenced by several factors like the operating environment, both natural and systems-produced; the characteristics of the machine; and the performance of the human operator in the operational situation.

The Frontloading Process for the HMI Design

The early phases in the development of airplanes and automobiles are becoming more and more critical in achieving time and cost savings. The highest priority is however product quality and safety and the related processes. Cost savings and a concentration on "time-to-market" principles must still take a second rank versus to these main concerns. For this reason new development strategies are being used with more emphasis on concepts such as "frontloading". In a very early phase of development, various virtualization technologies linked with full functionalities are used to test the physical properties in simulations. This process can partially replace the involvement of physical prototypes. Specially the implementation of nearly any kind of product functionalities scenarios will increase the use of the frontloading approach and amplify the degree of details and complexity for virtual prototypes [START_REF] Sabeur | 3D basierendes Virtual Prototyping von Mensch-Maschine-Schnittstellen ein neuer Ansatz zur Optimierung des Entwicklungsprozesses von Cockpitsystemen[END_REF], [START_REF] Sabeur | 3D basierende Entwicklung von Mensch-Maschine-Schnittstellen[END_REF].

Figure 2: The HMI frontloading process in the automotive industry

The presented design and simulation platform for HMI is ideal for implementing the frontloading process. Since it has an open architecture and it can be linked with any high end technologies for visualization and simulation like VR/AR of driving and flight simulators. The new platform takes also advantage of the unified modelling language (UML) to describe logical behaviour. With its integrated code engine code is being generated for the targeted embedded systems. For further and in dept system validation the integration with the Rhapsody UML software has been developed.

The CAD modelling process

In the aerospace industry the mechanical design and CAD departments build the first model of new airplanes concepts in a digital manner by using a CAD system. Due the fact that no CAD data of the Airbus was made available to us we had no choice rather then to reconstruct the cockpit model entirely.

A380 cockpit geometry data collection

The goal of the study was to build 3D virtual prototypes based on CAD data in order to simulate the real workflow for creating HMI cockpit concepts in 3D. Because within the automotive and the aero-space industry the CAD software CATIA from Dassault is widely used, it was mandatory to have CATIA version 5 geometry. In the following it will be described how the CAD model of the Airbus A380 cockpit was generated.

The assembly of the A380 cockpit is based on several components (the armatures and the exterior surface cockpit, different switches and levers, seats etc.).These components contain a series of elements (features), which consist of different geometrical objects themselves, too. Since it was not possible to get original CAD data of the A380 from Dassault as mentioned before, the CAD laboratory of the Chair of Information Technologies in Mechanical Engineering from the Otto-von-Guericke University Magdeburg (Germany) had the challenging task to reconstruct this CAD geometry by using CATIA. The CADlaboratory is well equipped with all the state-of-the art software and hardware technologies which are used in the industry today. To rebuild the A380 cockpit from scratch, one started by collecting information in paper and digital form which helped to produce the necessary basic drawings which represent the shape of the cockpit. The input data like sketches and photos were found in various publications of several flight magazines (figure 3).

The modelling process

The so collected information was the starting point of the modelling process. For example pictures were scanned. Their contours were used to build the first forms of the cockpits components with the help of the tool "Sketch Tracer". By using the scale function in this tool, the correct component proportions were defined.

Figure 3: Illustrations of the A380 cockpit

The contour outlines from the pictures were generated and transformed into splines, which formed the base for the surface model (figure 4). For building the cockpit interior photos were very helpful to understand the dash board and the instrument clustering. By using the extrusion tool the 3th dimension of the cockpit interior was generated. Several iteration loops were necessary to reach the level of realism by letting new and recent pictures information converging with the final model. So at the end we were able to get the look and feel of the real A380 cockpit (figure 6).

The modelling task was fulfilled by using the CAD system CATIA version 5 to simulate the industrial design work, we wanted to present. The highlight of CATIA version 5 is its parametric capabilities.

With "parametric modelling" a design is developed from the beginning in such a way that isometric changes are accomplished later, simply by input of new parameter values [START_REF] Vajna | General Procedure for Paramertrisation with 3D-CAD Systems[END_REF]. The CAD solid model is for itself particularly suitably and thus supports the variants and change design. For example the different screen surfaces of the armatures were parameterised. If the position of a screen surface was changed, the other screen surfaces were adapted automatically. Moreover, it was meaningful to parameterise various switches and their positions. Thus fast different changes could be made.

During the parameterisation it should be paid attention that certain changes of the basic solid cannot be fulfilled, because different relations could be lost among themselves. Furthermore, CATIA version 5 is based on the "master model concept". That means that the base is always the CAD model. If design changes are made, then these are changed also in the assembly. Thereby, this saves the change effort in the assembly. This is very important within CATIA and helped to make changes and to optimise the final model which was the base in the following step for creating the HMI prototype. Weekly meetings took place with the CAD designers to discuss the cockpit model. This shows that a tight co-operation between these two groups is fundamental to build the appropriate model for the prototyping simulation. As a result the complete CAD model was generated as an assembly with all components and sub-components (figure 7).

Therefore the HMI design and simulation is a team work and cannot be seen as insulated process.

For this purpose the 3D functionality and the CAD compatibility of the new HMI platform INSIDES were absolutely necessary.

Figure 7: The complete CAD Model

The Audi A6 cockpit geometry

The geometry of the A6 cockpit was made available to Princess Interactive for demonstration purposes by the courtesy of Audi. This geometry data was delivered in CATIA 4 and 5 Format. This helped to validate the principles of the workflow, we are presenting in this paper as well

The HMI design process

HMI are essential for the safe operating of technical systems. Therefore the development of user friendly HMI is of great importance. The cockpit in modern transportation vehicles consists of several in-and output-components such as displays instruments, information systems, multifunction command devices, Head Up displays which are being integrated within a complex cluster. The design and simulation of this kind of system has to be applied on the entire aircraft or car interior to see the interaction and influence of all the components between each-other. Therefore a mixed 3D and 2D representation is necessary. Furthermore on the fully operational virtual prototype usability engineering will be concurrently conducted in the very early phase of the development process and not at its end. This is multidisciplinary task where executive managers, marketing and sales specialists, designers, electronic engineers like hard-and software programmers as well as psychologists and other partners or customers are in-volved. In this section the design process used in the study will be outlined.

CAD data preparation for virtual prototyping

For this purpose another tool, which is in the same time a CAD data acquisition tool as well as an optimisation tool, was used. With this software CATIA native geometry was imported and tessellated as well as level of details were generated. Also a very important feature of the tool called "Geometry filtering" was very helpful. This enables the elimination of all the internal parts, which are not relevant for the HMI simulation. The entire assembly of the geometry was structured in a logical scene graph. Material and textures were applied.

The so optimised geometrical model was then converted into the .PID format, which is the internal representation of the HMI design and simulation platform.

The HMI prototype

The optimised geometry is the first step for building a dynamic and fully operational HMI virtual prototype. It has to be well structured and must represent a systematic hierarchy, so single referenced objects as well as groups of objects can be handled separately. By using special tools within this new HMI simulation platform every single object or group of referenced objects will receive properties like movements (e.g. rotation or translation or a combination of them), appearance or textual information parameters. The assignment of this data will then be completed by setting boundary conditions to all parameters, which show, how they can be operated dynamically and specially interactively. When all the reference objects are defined with their respective properties parameters their dynamic behaviour within the entire HMI cluster will be programmed. For this task, an hierarchical finite state machine will help by the definition and the deployment of all interaction scenarios which come from a detailed specification compiled by the system engineering people. The hierarchical finite state machine engine of the HMI design and simulation platform is completely graphically driven and is based on the principle of events, actions and transitions. The design of the HMI prototype is very modular. So each HMI display or HMI command device are represented by a single module, which carry on graphical and logical information and can be replaced, changed and enhanced or removed from the entire HMI cluster at any time. Also the separate modules can be designed concurrently at the same time by different people, which can work separately in different locations on the projects. The final HMI cluster can be assembled later on.

Simulation and Validation

After the first version of the entire prototype was finished, the simulation of the cockpit could be started.

In this phase consistency checks of the HMI cluster as well usability studies were fulfilled. Due the lack of time and financing only the tests were used based on the A6 cockpit. For this purpose several test benches were built. The simulation in both benches runs under the Windows and Linux operating systems.

a) Desktop Test Bench By using a graphical workstation and interfacing the the HMI Audi virtual prototype with a real steering wheel and the physical MMI haptic device one was able to conduct the first interaction checks. This was a very cost effective method to validate the HMI during the development cycle. It helps also to detect inconsistencies within HMI cluster.

b) Test Bench in conjunction with a virtual human model The open architecture of this new design and simulation platform allowed the integration of the virtual human model "RAMSIS" which is widely used in the automotive industry. In this case two separate simulation environment were build and connected in real time together.

• The real environment

This environment represents the test person, which is wearing the head mounted display and is being tracked by an optical tracking system. The test person is able to see and manipulate interactively with entire virtual cockpit.

• The virtual environment

This environment describes virtually the entire 3D car interior of the A6 with all operational HMI devices and displays. Within this virtual environment the human model "RAMSIS" was put in place. It was completely synchronised with the real tester e.g. for hands actions and manipulations, which was possible by using data gloves. This two environment are synchronised together in real time. Any action operated by the test person is being fulfilled by the human model RAMIS within the virtual environment. A series of interaction scenarios can be tested to validate new human machine interfaces in the car interior. This test bench set up is very complex but allows to conduct usability studies when nothing real is avail-able and can be used in the very beginning to back up the front loading process.

Code Generation

The new HMI design and simulation platform has a code generation engine, which generate separately graphical data as OpenGL or OpenGLES and logical information as C-or Java-code for embedded devices. For the study pure OpenGLES code combined with logical information has been generated and tested under a "Microsoft Windows" environment.

Future Development

The project has shown that 3D HMI prototypes are necessary to investigate complete HMI cluster within aircrafts or car cockpits. 2D based design and simulation are not enough. However the actual and new 3D based HMI software has to be extended with new features, which are already in development.

The following work is already in progress: a) Integration of embedded systems from the beginning to test the performance of the HMI displays. For this case a flexible embedded environment will be worked out and proposed for users of the design and simulation platform INSIDES. b) Integration of a specification methodology to allow the automatic generation of specification documents [START_REF] Hamberger | Toolsuite für multimodale Bedienkonzepte, Usability-Engineering von der Konzeptskizze bis zur Codegenerierung VDI-Congress 2005[END_REF], [START_REF] Müller-Bagehl | Teledrive Vision -a Consistent HMI Development Process 4[END_REF]. c) Integration with well known UML Tools like Rhapsody [START_REF] Bortolazzi | Integrierte Methoden und Werkzeuge zur Optimierung der Prozeßkette Elektrik/Elektronik -VDI[END_REF].

Conclusion

The frontloading within the HMI design process is essential to reduce costs and time in respect to quality and safety. The actual 2D based HMI tools cannot fulfil the above requirements. Therefore a 2D and 3D combined approach is necessary. The new presented HMI design and simulation platform INSIDES could contribute to help the acceptance of this process. Also the study has shown that one can conduct interaction checks and usability engineering with very limited costs in the very beginning of the development cycle. This can have a very positive impact on the entire product development cycle in term of time and costs saving.

Acknowledgement

All the people involved in the success of this study have to be acknowledged for their contribution. It is important to mention the co-operation of Princess Interactive with the Chair of Information Technologies in Mechanical Engineering from the Otto-von-Guericke University Magdeburg (Germany), which helped to conduct this project at a very low cost budget.

M. Sabeur 1 ,

 1 D. Schulz 2 , M. Schabacker 3 ,S. Vajna 4 , 1: Princess Interactive, Rogätzer Str. 8, D-39106 Magdeburg, Germany 2: Princess Interactive, Rogätzer Str. 8, D-39106 Magdeburg, Germany 3: University of Magdeburg, Universitätsplatz 2, D-39106 Magdeburg, Germany 4: University of Magdeburg, Universitätsplatz 2, D-39106 Magdeburg, Germany

Figure 1 :

 1 Figure 1: The tension field between human, machine and its environment

Figure 4 :

 4 Figure 4: Generation of the surface modelFunctions like e.g. filling surfaces, surfaces with multiple cuts, free forming junctions or free forming extrapolating were used for the creation of the surfaces. Finally, a solid body was created by setting an offset on top of the surfaces. The final result is the outer skin of the cockpit (figure5).

Figure 5 :

 5 Figure 5: Solid body of the A380 cockpit

Figure 6 :

 6 Figure 6: CAD model with the instruments locations During the different iteration loops the HMI engineers from Princess Interactive were always involved.Weekly meetings took place with the CAD designers to discuss the cockpit model. This shows that a tight co-operation between these two groups is fundamental to build the appropriate model for the prototyping simulation. As a result the complete CAD model was generated as an assembly with all components and sub-components (figure7).

Figure 8 :

 8 Figure 8: The A380 cockpit built with INSIDES

Glossary


H Ailisto

L Pohjanheimo

H Keränen

P Alahuhta

Physical Browsing -a novel HCI paradigm for people on the move

Keywords: Human Computer Interface, Mobile phone, RFID, Physical Browsing

RFID tag readers have recently appeared into mobile phones and other types of tags and readers are also coming -most notably visual tags which can be read by a software in a camera phone. This will open up new possibilities for using RFID and visual tags broadly in consumer and industrial applications. In this paper we outline a concept called Physical Browsing, present some implementations of it and discuss the implications this concept could bring to car and transportation applications.

Introduction

Conventional Human Computer Interaction (HCI) methods, such as keyboards and displays work well in office environments and desktop computing. However, in situations where people are on the move, e.g. walking or driving a car, and they have to divide their attention, the conventional methods are not the most suitable ones since they 1) generate too heavy cognitive load (need total concentration), 2) rely on equipment which are cumbersome and heavy and can not be used with one hand, e.g. when walking, 3) do not use the context information available in the usage situation.

Physical browsing paradigm can be seen as an answer to these problems. According to the physical browsing paradigm the way user interacts with the computer should be linked to the task at hand. For example, if the user wants information about an object he should get it by touching the object with his handheld computer instead of keying in a sequence of characters, the web address; or if the user wants to pair his mobile phone with the audio system of a car or train, he should be able to do this by touching a RFID empowered symbol of the pairing instead of performing a tour in the labyrinth of pull down menus in his device. These examples show how the cognitive load can be minimized, mobile devices can be used and the context (location, situation) can be utilized in the human computer interaction with physical browsing paradigm.

The realisation of Physical Browsing paradigm requires 1) hand held devices with computing power, communication capability and modest graphical user interface 2) identifier tags such as RFIDs and 3) back-end services supporting the applications. The tags can be embedded in the ambient and the user has a reading device, e.g. in his mobile phone, or the tags can be carried by the users and the tag readers are in the ambient infrastructure. The choice between these alternatives depends on the number of tags and readers needed and on their cost. The tag and reader technology can be, for example, RFID, barcode, matrix code, human readable characters or infrared beacon based [START_REF] Välkkynen | A user interaction paradigm for physical browsing and near-object control based on tags[END_REF].

Traditionally car and transportation applications have utilised tags carried by users and readers in vehicles. Examples of these are electronic immobilizer systems (RFID tag in the car key and reader in the lock), identifying and authenticating driver based on car keys in the pocket and adjusting car preferences to drivers' settings. Public transport has used a lot of RFID bus cards or visual tags printed on cardboard.

Recently RFID and visual tag readers have become available for mobile phones. This establishes new kind of possibilities and more cost effective applications, since the expensive reader hardware is already available and the tags do not need much infrastructure e.g. power and they can easily made resistant to hostile conditions and vandalism.

In our approach we have chosen RFID tags embedded in the ambient environment while the user's mobile device is equipped with a tag reader. We have implemented a number of prototype applications in order to study the potential of the physical browsing concept. These prototype applications include basic phone usage like calling, sending SMS, coupling devices and opening communications (Bluetooth, GPRS), ordering screen savers, getting information (mobile weather report), and making small payments e.g. at a vending machine.

This paper is organized as follows. We will elaborate the concept of physical browsing in Section 2, then a mobile phone and RFID based implementation of the paradigm is shown in Section 3, we describe some demonstration cases in Section 4 and discuss the implications of the paradigm in Section 5. Conclusions are made is Section 6.

Physical browsing paradigm and related work

Välkkynen and co-workers define Physical browsing "as getting hyperlink information from physical objects" [START_REF] Välkkynen | A user interaction paradigm for physical browsing and near-object control based on tags[END_REF]. This can happen if the object has a way to communicate a URL to a user, which requests it. This URL can be transmitted for example with an information tag and it can be read with a mobile device like a cell phone. We define an information tag (hereafter: a tag) as a small and inexpensive unique identifier, which 1) is attached to a physical object but has limited or no interaction with the object itself, 2) contains some information, which is typically related to the object, and 3) can be read from near vicinity.

The idea of physical browsing can be broadened from getting hyperlink information to include

• any data acquisition initiated by physical browsing, reading the data in the tag or reading a sensor linked to a tag • initiation of action, for example making a phone call, sending a certain message, opening a door • performing a (financial) transaction, for example paying a purchase at a wending machine or ticket in a buss • retrieving an user interface for a device having no user interface or using the phone as a remote user interface.

Ideas close to physical browsing have been suggested [START_REF] Want | Bridging Physical and Virtual Worlds with Electronic Tags[END_REF][START_REF] Ullmer | mediaBlocks: Physical Containers, Transports, and Controls for Online Media[END_REF][START_REF] Schmidt | C Enabling implicit human computer interaction a wearable RFID-tag reader[END_REF][START_REF] Kindberg | Places, Things: Web Presence for Real World[END_REF][START_REF] Ljungstrand | WebStickers: Using Physical Objects as WWW Bookmarks[END_REF]. For example, Want et al. [START_REF] Want | Bridging Physical and Virtual Worlds with Electronic Tags[END_REF] summarise the goal of augmented reality and physically-based user interfaces: "The goal of these projects is to seamlessly blend the affordances and strengths of physically manipulable objects with virtual environments or artefacts, thereby leveraging the particular strengths of each." Ulmer and Ishii [START_REF] Ullmer | mediaBlocks: Physical Containers, Transports, and Controls for Online Media[END_REF] developed the idea of Phicons, which serve as physical icons for the containment, transport and manipulation of online media in an office environment. Their paper does not discuss the role of mobile personal terminals, such as smart phones or PDAs, but instead relies on fixed devices, such as digital whiteboards, projectors, and printers. Kindberg and co-workers study infrastructure to support "web presence" for the real world [START_REF] Kindberg | Places, Things: Web Presence for Real World[END_REF], their main idea being connecting physical objects with corresponding web sites. Infrared (IR) beacons, electronic tags or barcodes are suggested for creating the connection.

Implementation of physical browsing paradigm

In this chapter we briefly describe our physical browsing platform consisting of hardware and software. The platform is described more comprehensively in [START_REF] Keränen | Tag Manager: a Mobile Phone Platform for Physical Selection Services in International conference on Pervasive Services (ICPS 2005)[END_REF][START_REF] Pohjanheimo | Implementing TouchMe Paradigm with a Mobile Phone[END_REF].

We have used two similar RFID technologies. Our research started before any RFID reader was commercially available for mobile phones or the interfaces for them were public. Therefore we have used a prototype RFID reader developed for Nokia 6600 phone (Figure 1). We have also used a commercial barcode reader pen, Baracoda, which connects to the mobile phone by Bluetooth, but we feel that reader to be convenient to use it has to be built into the phone.

Our software platform consists of a middleware component called TagManager located in the mobile phone. It provides an interface for different tag readers and makes possible for third party vendors to write tag aware applications. TagManager performs service resolution in a way which makes it possible for different applications to utilise the same tags if needed. The applications can either function locally in the phone or utilise network resources to perform the desired functionality.

Demonstration cases

This chapter cations pilot system implemented in presents demonstration appli realized with the physical browsing paradigm.

4.1 Drivers log system Drivers log system is a collaboration with the City of Oulu. In the scenario employees are using their cars for job related tasks. Driven kilometres are refunded to the employee. Previously users had to write the kilometres in a logbook and then enter them to the employer's backend system by hand. In the pilot case RFID tag reader and tags are used to ease the process. User can input the kilometres to the back-end system in real time. When starting the trip, user touches the tag attached to the car with his mobile phone and enters the kilometres and other related data to his phone as seen in the figure 2. After the trip user touches the tag again and the mobile phone asks him to enter the ending kilometres. Phone communicates with the back-end systems database in the background and uploads the entered information. This method saves the user the trouble of entering the kilometres several times to different media, thus minimizing the possibility of errors in the process. This Pilot is currently having a trial period with employees of City of Oulu ario was demonstrated bought from vending yable for these kinds of er .2 Vending machine case Financial transaction scen with paying a soft drink machine by touching a tag attached to the machine (Figure 3). The demonstration was built on the already available service where user could buy the drink by calling a certain number. This demonstration replaced the need to key in the phone number and to listen the automated announcement by action of touching the tag. The much simpler way of doing the purchase would lower the decision making limit while considering the purchase. Same kind of method is viable for example when paying the public transport tickets, parking tickets and possibly in car rentals where the car could be rented and access to the car could be gained by using a mobile phone. NFC Standard and NFC compatible FeliCa technologies are emplo scenarios and few such pilots have been already made.

Figure 3: Soft drink purchase made by touching the tag .3 Fan remote control emonstrates the ability to using tags (Figure 4). When e the space for UI, the cost 4 Fan remote control d retrieve a user interface the fan control tag is touched the mobile phone loads a user interface for the fan. The fan speed can then be controlled using the mobile phone. In the demonstration mobile phone uses Bluetooth to connect to the X10 home automation system which in turn controls the fan. This kind of user interface (UI) has its place in devices which do not hav of making an UI for each device is too high, or an UI is omitted for aesthetical reasons. These kinds of user interfaces could be used when, for example the electronic service system of a car must be accessed on the road or when there is need to have remote control for one of the electronic systems of a car, e.g. radio. 5). With a tag containing the connection information user can skip the tedious Bluetooth connection establishment procedure and use the service just by touching. Simple explicit connection establishment is essential for examp system of the car, in a case where there are several users for the car with similar phones. The demonstration also explores the possibility of having several services offered by TagManager middleware presents a menu when there is several services available and the user can choose one of them (Figure 6). This actually is in contradiction with simple one tag one service principle of physical bro some cases it is more convenient to have one tag to offer several services than having many tags stacked in a limited space. In this case evoking a phone call was made using a RFID tag embedded to a business card, see Figure 7. This was perceived as a very convenient way to reduce cognitive load of the user when placing the call. Call was also easy to make when the user cannot use the phones keyboard properly, e.g. when using thick gloves.

Figure 7: Making a call from business card

Discussion

In the previous section we have described some cases where physical browsing could be used to make Human Computer Interaction more natural, pleasant or effective. In this section we discuss some potential cases which are related to car and transportation applications. The benefits of physical browsing paradigm should be especially important in this domain, since people on the move do not have the access to the normal office type input-output facilities. Furthermore, they should not be burdened with extra cognitive load and, and if possible, information about their current context should be used to help the task at hand. The concept of physical browsing can find various applications in the domain of car travel. Examples showing this include:

• Driving log in company or rental cars. The driver reads the car's identifier tag, records the odometer reading and starting position and sends them to the back-end system in real time via the mobile phone. The trial by VTT and City of Oulu can be regarded as an early version of this. • Switching the mobile phone to "driver mode" when entering a car just by touching a tag instead of a more complicated Bluetooth setup procedure. The driver mode would include using the audio system of the car for phone calls instead of the phone itself and setting the presence status into "driving". The mode switching is initiated by touching a tag in the car when entering it. • Personalizing the car used by a family, car pool or a company. In this case the user would carry the identifier (in his mobile phone) and the car the reader. Personalization might include seat settings, automatic transmission settings and radio / CD settings.

• By using local RF communication provided by NFC standard a cell phone could work as a key for a car. In fact the car keys could be delivered to a cell phone on demand over cellular network. Keys could also expire after intended time of use. Also in the case of misuse the keys could be deleted from the phone remotely. This could be used for car or vehicle renting or delivering keys for workers of a company. • The phone could work as a remote user interface for the car for example to check the status of burglar alarm or turn on the preheat system in the winter. The tag to launch these services could be located in the wall in the home or in the car keys. In the future all vehicles could provide a standardised software and hardware interface for obtaining key information from the vehicle, like odometer reading, usage hours, register number and status of different components. This would enable the vehicle to be used more seamlessly as a building block of company infrastructure without any modifications needed to the vehicle itself. In the domain of public bus transport, we have identified two main usages for physical browsing: information retrieval and ticketing & payment.

• Some bus stops, especially in cities and big towns, have information displays showing when the next bus is coming and which route number it has. These displays work well, but they are costly to install and vulnerable to vandalism. Same functionality could be achieved, particularly in small town or rural areas, by installing identifier tags in the bus stops where people could then interact with the bus travel information system via physical browsing paradigm, i.e. reading the tags with their mobile devices.

The system could offer all the current services (the time to next buses, their route numbers) and additional personalized services such as route planner: "take bus number N to home, switch at Market place to number M, time to home 30 min." • Physical browsing concept can be applied also to payment and ticketing in buses and other public transportation. In fact, the widely used remotely readable bus cards are an instance of physical browsing. If they are integrated inside personal mobile devices, additional services, such as payment history and balance, can be added to the available services.

Using the mobile phone, equipped with RFID reader capability, as a user interaction device with cars, public transport and many other functions would involve certain benefits. The mobile phone has become an integral part of the life of modern people, and thus it is always available. It has a familiar user interface, making its use easy, increasing amount of processing power and memory capacity as well as local (NFC and Bluetooth) and global communication capabilities.

Many challenges exist on the way towards realizing the vision of using RFID enabled mobile devices with cars and public transportations. For example, the standardisation issues and legacy systems (the cars!) pose a big challenge for this vision.

Conclusion

This paper has presented the Physical Browsing concept and some examples of implementing and using it. Mobile phones with RFID tag readers will provide a cost effective way to realize new kind of applications for transportation or replace existing solutions with more user friendly and effective ones. Some potential cases were introduced and discussed.

Acknowledgement

The support of Dr. Heikki Huomo from Nokia Corporation in realizing the prototypes is acknowledged. The contribution of our colleagues including Ilkka Korhonen, Pasi Välkkynen, Tua Huomo, Tapio Matinmikko and Jussi Liikka in the formulation and demonstration of the physical browsing concept is acknowledged.

Glossary

Figure 1 :

 1 Figure 1: RFID reader replacing the back cover of Nokia 6600 phoneThe RFID reader has a reading distance of 1 to 2 centimetres and it reads 1.5 x 1.5 cm non powered RFID tags.The second technology we have used is the Near Field Communication (NFC) standard proposed recently [http://www.nfc-forum.org]. The NFC has many features, such as support for peer to peer communication and compatibility with contactless smart card standard. Mobile phone manufacturers have launched models, such as Nokia 3220, for which the NFC reader is available. The NFC standard eventually makes our prototype reader obsolete when it will be available for Symbian-based smart phones which offer more flexible programming possibilities.

Figure 2 :

 2 Figure 2: Mobile phone used in the driv s' log pilot

Figure 4 :

 4 Figure 4: Equipment used in the fan remote control demonstration

Figure 5 :Figure 6 :

 56 Figure 5: Phone sends SMS messages to the computer for archiving


S Goronzy
email: silke.goronzy@3soft.de

R Kompe

R Holve

Fully exploiting the potential of speech dialog in automotive applications

Keywords: speech dialog, multimodal interfaces, integrated interface design

Today users are faced with infotainment devices and applications of increasing complexity. The design of easy-to-use and intuitive interfaces becomes a more and more challenging task. Users are usually not aware of the underlying applications and their restrictions when they want to use certain functionalities.

Therefore, hierarchical menu structures are difficult to handle especially in situations where eyes and hands are occupied with other tasks, such as driving. For quite a while speech-enabled interfaces have been used to solve this problem since they allow users to control various applications without occupying hands and eyes. However, state-of-the-art multimodal applications often do not exploit the full potential that speech dialog offers simply because this modality is not well integrated with the "traditional" modalities such as graphics and haptics. The resulting speech interfaces do not run smoothly, exhibit plenty of inconsistencies concerning the GUI and are thus more or less tedious to use. Such kind of interfaces result in low acceptance because users do not see the immediate benefit. In this paper we present an approach that develops multimodal interfaces in an integrated way, thus ensuring highly consistent interfaces that closely couple the involved modalities and are thus easier to use.

Introduction

It is commonly accepted that the use of multiple modalities for human machine interfaces (HMIs) potentially facilitates the handling of complex systems for users. Especially drivers of cars, who want to operate e.g. the infotainment system while driving need particularly designed interfaces that do not distract them from their main task -from driving. Therefore, speech dialog systems have been frequently added to the graphical/haptical interfaces because speech control potentially allows drivers to keep their hands on the wheel and their eyes on the street. Speech recognition technology itself was optimised over the past years for the usage in cars.

High recognition rates above 90% can be achieved even with considerable background noises. Still, many of the speech-enabled interfaces are not perceived as easy-to-use by most of the users. The reasons are manifold, ranging from design issues (such as 'boring' prompts being played over and over again) to severe inconsistencies between the GUI and the speech dialog (e.g. a hardware having 6 buttons for directly selecting a CD from the CD changer, only 3 of which being speech enabled).

Speech dialog interfaces are often designed separately from the GUI. This is also true for research prototypes for multimodal dialog systems such as SmartKom [START_REF]Project: Dialog-based Human-Technology Interaction by Coordinated Analysis and Generation of Multiple Modalities[END_REF][START_REF] Wahlster | SmartKom: Symmetric multimodality in an adaptive and reusable dialog shell[END_REF] or Embassi [START_REF] Elting | Modeling Output in the EMBASSI Multimodal Dialog System[END_REF]. In the former the integration of the different modalities was achieved by developing specially tailored modules for modality fusion and by using a special XMLbased language for information exchange between the different modules. Such an architecture, however, is prohibitive for most embedded applications so that in the embedded area no such integration exists. In this case separate development processes pose a severe problem. During speech dialog design (at least part of) the GUI's functionality and application flow is rebuilt using formalisms of different development environments, resulting in a considerable amount of different documents and/or files whose consistency has to be ensured at great costs. Change management is particularly difficult in this case. The danger of diverging application flows for GUI and speech dialog is thus very high.

The problems range from potentially 'simple mistakes' such as speech commands differing from what is displayed on the GUI to more severe inconsistencies such as different system behaviour depending on whether the user used a speech command or navigated via haptics. Another problem is that the multimodality is often rather restricted in a way that a dialog once started by speech cannot be continued by haptics. However, there are many situations where it would be beneficial if users could freely choose between the different modalities depending on personal preferences and/or the current situation. Such behaviour might confuse naïve users more than helping them.

In order to avoid inconsistencies and provide truly multimodal HMIs we developed a tool that allows a model-based specification and uses one data model for all modalities involved to describe the complete HMI. Such a common data model allows automatic consistency checks and helps to develop sound interfaces with the modalities really supplementing each other. The tool was originally developed as a tool for graphical/haptical user interface specification and development and was recently extended for speech dialog. Its architecture allows further extension if more modalities, like gesture recognition, are to be added. It can interface to various speech recognition and synthesis engines and can thus be flexibly employed for various applications.

HMI Specification and Development

The specification for HMIs of infotainment systems in the automotive domain typically involves the graphical layout design, the determination of menu logic, the speech dialog, and the administration and maintenance of different languages. In a first step it is usually defined how the different views or screens of the GUI look like, which kind of widgets are used and how these behave (e.g. how a button looks like if it gets pressed). In order to describe the system behaviour the menu logic or application flow must be determined in a way that the user can later easily navigate in the menus to find the desired functions of the infotainment system. In addition to potential user actions, like selecting radio stations or programming the navigation system, also internal system events, like incoming telephone calls, have to be considered. Finally, since HMIs need to be available in multiple languages, all language specific GUI texts and particularly speech dialog prompts, vocabulary, grammars etc. need to be administered in such a way that the change of the HMI language becomes possible at the runtime of the system. All these different tasks are usually carried out using different tools in different formats resulting in a HMI description that is spread among many different documents. As a result, it becomes particularly difficult to keep the HMI specification consistent, even if only the graphical/haptical part is involved.

A Tool for Generating GUIs

In order to avoid this problem a tool that allows a model-based specification and design of HMIs was developed. This model contains all HMI-related information and thus is the complete specification in one model. The core of the system is a data pool, in which all information that is relevant for the HMI is stored. Whenever HMI related information changes, e.g. because the user selected a title in the mp3 application or because the navigation application returned a list of matching destination cities, the corresponding information is written to the data pool and the affected HMI components are notified about the changes, so that they can display the updated information on the screen. The main advantage of the data pool architecture is that the HMI can be developed completely independent of the applications. It can be fully tested by manually writing HMI relevant data to the data pool, thus simulating the complete HMI without the need to integrate any application at that stage. However, integrating applications as well as e.g. hardware control panels is easily possible at any time.

Specialised Editors

The tool uses several editors for the different tasks at hand:

• View Editor: graphical layout design for each view • Event Editor: Definition of all events that influence the HMI • State-chart Editor: Definition of the application flow and specification of possible paths between states.

A state can be linked with a view to be displayed and the transitions between the states/views are triggered by events. The state-charts used are UML (Unified Modeling Language) compliant. This means in particular that functions like e.g. 'history' or 'deep history' nodes or the inheritance mechanism for events is available for specification. Another advantage of the model-based specification is that it allows to automatically generate code for the target platform to extents of 60-80%. The tool furthermore allows multiple users to work on the same HMI, which is particularly relevant in the case of speech dialog development.

In [START_REF] Holve | A model-based approach towards human-machine-interfaces[END_REF] a detailed description of the system architecture and features can be found.

Integrated Design of GUI and Speech Dialog

Speech dialog is nowadays very often added as a further modality to the HMIs, yielding multimodal interfaces. The look & feel and the behaviour of the speech dialog need to be appropriately specified.

We need to describe what kind of vocal interaction can be understood and how the system reaction should look like. For the graphics part, consistency is ensured by using one globally visible data model, which can be referenced by different parts of the GUI using specialised editors. Correspondingly, the speech dialog is integrated in exactly the same way, using yet another editor for specifying the speech dialog and its properties, which are stored in the global data model. As the basis for the dialog flow the previously defined application flow can be used.

If desired, also a dialog flow that deviates from the graphical dialog flow can be specified. In case the goal is to design very simple, command & controllike speech dialogs where users can "speak what they see" the desired prompts and speech commands can directly be connected to the GUI states. In this case, the speech commands often correspond directly to a haptic interaction such as pushing a button on the GUI. Therefore, it might make sense to also re-use the previously generated 'haptic events'.

If the speech dialog is somewhat more complex and also (at least) partly deviating from the graphical/haptical application flow (e.g. if special help dialogs or confirmations of user input needs to be designed) it makes sense to design the speech dialog flow separately. In this case a second state machine can be constructed that purely shows the speech dialog flow. Then special 'speech events' are used instead of re-using the graphical/haptical events. Still it can be ensured that both modalities are perfectly synchronous because

• both modalities have access to the data pool and can thus react on user actions no matter which modality was used for the previous input

• special events can be used in one state machine to elicit transitions to certain states in the other state machine This means that the speech dialog can trigger a certain view to be displayed as well as the haptic part can cause a certain speech dialog to start.

Figure 2

GUIDE+SPEECH: The Speech Dialog Editor

Just like a view of the GUI is associated with a state in the state-chart, speech dialog and its properties are attached to states. With a special dialog editor, the allowed user utterances can be defined.

To define a valid user utterance the words that can be understood in this state need to be defined as well as the allowed sentence structure, the so-called grammars. Here special care needs to be taken to select words (at least in the list of synonyms) in such a way that all commands that are currently displayed on the GUI are included in the speech recogniser's vocabulary. Users usually tend to use the commands they see on the screen and are confused if those words are not recognised in the speech dialog.

Base Vocabulary

A special feature that helps to ensure the consistency with the GUI is the generation of the socalled base vocabulary. It can be automatically generated from the corresponding view, suggesting all haptical commands for the recognition vocabulary. This base vocabulary can be extended by synonyms which can be manually added to the vocabulary to allow more freedom for the user. Also more complex grammars can be specified, defining optional words and phrases etc.

Prompting

For outputting information in speech dialog either pre-recorded prompts or text-to-speech (TTS) engines are used. During specification one can use both for prompt definition. Also multiple prompts can be defined per speech dialog state.

Dialog Building Blocks

When specifying speech dialogs, certain sub-dialogs might occur more than once in more or less the same form, e.g. in help dialogs. Here the structure of the message can be defined to be the same systemwide, e.g. it can be organised in three steps, in which each time an error occurs a little more help is provided. Other examples for such recurring dialog turns are e.g. entering sequences of numbers. For these sub-dialogs templates can be specified and reused throughout the whole dialog.

Concepts vs. Wording

The dialog flow is generally held independently of the actual wording, so that both can be administered separately. This is important for maintaining several languages for both GUI and speech dialog. This means that during dialog specification concepts like 'DIAL_TEL_NUM' are used. Which wording can actually be used is defined separately.

Dynamic Vocabularies

Very often the speech recognition vocabulary cannot be fully defined during system design because it contains dynamic content such as radio station names that are currently received, media content such as mp3 titles, or street names that are loaded in a navigation task once the destination city was chosen. The tool allows specifying such dynamic vocabularies. This is necessary to allow users to vocally reference information that is dynamically loaded by the different applications. Of course it depends on the recognition engine that is connected, whether it supports dynamic vocabularies or not. If the underlying recognition engine does not support such dynamic vocabularyespecially selecting mp3 titles by speech is a big problem because a title name might contain words from different languages -a special numbering scheme can be used to make the selection based on the position in the list ("first title").

Inheritance of Speech Commands

Commands/phrases/sentences that are defined at parent states in the state-chart can be inherited to the children states. Commands defined at children states are only valid at this particular state. Commands inherited from parent states can be deactivated explicitly if e.g. the vocabulary should be restricted to 'yes/no' answers in clarification dialogs. Also the inheritance of single commands instead of the whole set of commands is possible.

Multiple Dialog Strategies

In some applications it might be necessary to define different dialog strategies that e.g. distinguish whether the user is an expert in using the system or a novice user. On the one hand, novice users might need very detailed help messages telling them what actions are possible in the current state. On the other hand such lengthy explanation might be bothering for the expert user who does not need any or very little help. Thus, different dialog strategies that e.g. use different prompts and potentially also command sets can be defined.

Multimodality

Applications that integrate multiple modalities are not necessarily multimodal. Very often users cannot freely decide when to use which modality or they are restricted to use only one at a time. For example speech interfaces are often designed in such a way that once a dialog turn that requires several subsequent user utterances was started by speech, it also needs to be completed by speech, i.e., no haptic interaction is possible. If the user pushes a button nevertheless, the speech dialog is aborted. In our tool, in order to achieve genuine multimodality, all modalities access the data pool. Here all HMI relevant data is stored so that the speech dialog can access any HMI information and data the user previously entered by haptics and the other way round. As a consequence, speech dialog can be started in the middle of an interaction and also a dialog that was started by speech can be continued by haptics.

On-the-fly Simulation

Due to the data pool architecture and event mechanism chosen for this tool, an on-the-fly simulation of the HMI becomes immediately possible while designing the interface. This means that the look & feel of the overall HMI, i.e. all modalities at once, as well as the usability can be evaluated at very early design stages. Figure 3 shows an example simulation screen for an infotainment system. In this simulation both the haptic interaction and the speech dialog and how they interact with each other can be tested. Usually, if the speech dialog is designed separately from the GUI, also the simulations are separated. In one tool the GUI simulation is tested, in another the speech dialog is simulated. Both modalities are then integrated very late on the target. Inconsistencies can only be discovered then. This means that even if the GUI simulation and the speech dialog simulation work perfectly on their own, problems might arise when they are integrated.

In our simulation the overall HMI can be experienced immediately. This helps to optimise the HMI early enough if it comes to mass production. The resulting multimodal HMIs are thoroughly consistent and truly multimodal in a sense that the user can freely choose between the different modalities at any time.

Conclusion

Interfaces using different modalities often lack proper integration and consistency. This is a result of the design process that usually consists of separate development tracks of graphical/haptical interface and speech dialog. Due to these separate development processes many inconsistencies can arise.

We follow an integrated design of multimodal interfaces, where the complete HMI is kept in one model. This allows a close coupling of the different modalities. We use specialised editors for the GUI design as well as for speech dialog design, still keeping all data in one overall model. A key feature is the capability to do an on-the-fly simulation of the HMI model including all modalities. Thus, the behaviour of the different modalities in combination can be immediately simulated.

In such a way thoroughly consistent, multimodal interfaces can be developed that allow users at any time to switch between modalities. In this way speech dialog is realised as a fully-fledged input modality that can help users to intuitively control the different applications easily in situations where the use of other modalities is problematic.

Figure 1 :

 1 Figure 1: The HMI model used for graphical/haptical input and speech input

Figure 2 :

 2 Figure 2: Specification of speech dialog using a separate state-chart

Figure 3 :

 3 Figure 3: Simulation of an infotainment HMI using graphical/haptical input and speech input