Marcos Serrano
email: marcos.serrano@imag.fr

Laurence Nigay
email: laurence.nigay@imag.fr

Temporal Aspects of CARE-based Multimodal Fusion: From a Fusion Mechanism to Composition Components and WoZ Components

Keywords: H.5.2 [Information Interfaces and Presentation]: User Interfaces-Input devices and strategies, Interaction styles, Prototyping, User interface management systems (UIMS); D.2.2 [Software Engineering]: Design Tools and Techniques -User interfaces Algorithms, Human Factors, Theory Multimodal Interaction, Fusion, Component-based Approach, Design and Implementation Tool, Wizard of Oz

  L'archive ouverte pluridisciplinaire

INTRODUCTION

Several research projects have demonstrated very innovative modalities as part of prototypes and we are facing a huge range of possibilities in terms of interaction modalities. In this context, there is a specific need for generic models and tools allowing us to do fast prototyping of multimodal applications. These generic tools should enable the rapid development of multimodal prototypes to be experimentally tested as part of an iterative usercentered process.

One of the key points concerning those generic tools is the fusion mechanism. Through the fusion of input modalities, new combined modalities are defined (i.e., pure and combined interaction modalities are introduced in [START_REF] Bernsen | Modality Theory in support of multimodal interface design[END_REF]), enriching the design space for interaction. As explained in [START_REF] Pfleger | Context Based Multimodal Fusion[END_REF], fusion is classically classified as early fusion or late fusion. Early fusion represents fusion at the signal level, and late fusion represents fusion at the semantic level. Furthermore, semantic fusion can be divided into subprocesses. In [START_REF] Landragin | Physical, Semantic and Pragmatic Levels for Multimodal Fusion and Fission[END_REF], three subprocesses for multimodal fusion are identified: coordination, content fusion and event fusion. The coordination subprocess is in charge of creating pairs of coordinated events coming from two input modalities. Content fusion obtains a coherent sense from incomplete information. Event fusion produces a resulting complex act from pragmatic monomodal acts. Content and event fusion subprocesses are closely related to the application, while the coordination one is related to temporal fusion. In this paper we focus on those temporal aspects of generic fusion mechanisms.

For studying the fusion, we based our works on the CARE properties. The CARE properties (Complementarity, Assignment, Redundancy and Equivalence) define various forms that multimodal interaction can take. While Equivalence and Assignment express the availability and respective absence of choice between multiple modalities for performing a given task, Complementarity and Redundancy describe relationships between modalities and require fusion mechanisms. Since our first definition [START_REF] Coutaz | Four easy pieces for assessing the usability of multimodal interaction: the CARE properties[END_REF], the CARE properties have been used to implement several multimodal applications.

In this paper we present a summary of the different technical solutions for multimodal interaction that have been defined based on the CARE properties and we illustrate them by considering several examples of multimodal applications. Those solutions have evolved from a monolithic algorithm integrated in the PAC-Amodeus software architecture to the implementation of several software components dedicated to each CARE property. We also introduce a new contribution based on the use of the CARE properties and Wizard of Oz (WoZ) components.

For presenting our work on the CARE-based fusion mechanisms, this paper is organized as follows. We first explain in detail the combination space defined by the CARE properties and the temporal relationships between input modalities. We then describe the different approaches that have been taken for implementing multimodal interaction based on the CARE properties, from a single fusion mechanism to several composition components. Among the composition components, we distinguish operational components and WoZ components.

COMBINATION SPACE

The combination space of interaction modalities is organized along two dimensions. The first dimension defines the type of relationships between modalities based on the CARE properties. The second dimension is related to the temporal relationships between input modalities.

CARE properties

The CARE properties [START_REF] Coutaz | Four easy pieces for assessing the usability of multimodal interaction: the CARE properties[END_REF] were proposed as a simple way of characterizing and assessing aspects of multimodal interaction: the Complementarity, Assignment, Redundancy, and Equivalence that may occur between the interaction modalities available in a multimodal user interface. They have been shown to be useful concepts for the design and evaluation of multimodal interfaces in [START_REF] Coutaz | Four easy pieces for assessing the usability of multimodal interaction: the CARE properties[END_REF]. In this paper, we focus on the technical solutions for implementing the CARE properties.

The formal expressions of the CARE properties rely on the notions of state, goal, modality and temporal relationships. A state (s) is a set of properties that can be measured at a particular time. A goal (g) is a state that an agent intends to reach. An agent is an entity (user, system or component) capable of performing actions. A modality (m) is an interaction method that an agent can use to reach a goal. A sequence of successive steps is called an interaction trajectory. Two examples of a modality can be the general terms 'using speech' or 'using microphone'. A temporal relationship (TR) characterizes the use over time of a set of modalities. The use of these modalities may occur simultaneously or in sequence within a temporal window (TW), that is, a time interval. Figure 1 illustrates these notions, used for expressing the CARE properties.

Figure 1. Notation for expressing the CARE properties [8].

From those concepts, the CARE properties are defined as a set of properties that characterize four types of relationships between modalities for reaching state s' from state s. The letter T in Figure 1 represents the CARE properties in the diagram.

Equivalence expresses the availability of choice between multiple modalities but does not impose any form of temporal constraint on them. More formally, modalities of set M are equivalent for reaching s' from s, if it is necessary and sufficient to use any one of the modalities.

In contrast to equivalence, assignment expresses the absence of choice. More formally, modality m is assigned in state s to reach s', if no other modality can be used to reach s' from s.

Two modalities are used redundantly to reach state s' from state s, if they have the same expressive power (they are equivalent) and if they are used within the same temporal window. In other words, the two modalities are required to reach state s' if they are used redundantly, and they convey the same meaning.

Modalities of a set M are used in a complementary way to reach state s' from state s within a temporal window, if all of them must be used to reach s' from s, i.e., none of them taken individually can cover the target state.

As opposed to Equivalence and Assignment, Redundancy and Complementarity imply fusion of input modalities. The formal expressions of the CARE properties include the notion of temporal relationship (TR) that we further refine by the second dimension of our combination space.

Temporal relationships

We define the temporal relationships between two modalities by using Allen's properties [START_REF] Allen | Maintaining Knowledge about Temporal Intervals[END_REF]. We used Allen properties to define the different combination schemas between two modalities related to the combination aspects (temporal, spatial, syntaxic, semantic) in [START_REF] Vernier | A Framework for the Combination and Characterization of Output Modalities[END_REF]. In this paper we focus on the temporal aspect of the fusion mechanisms. Figure 2 presents the different combination schemas for the temporal aspect. Each bar represents the temporal length of a modality (considering a left-to-right time axis). Two modalities are combined anachronously if there is a temporal gap between their usage. Anachronism and sequence are distinguished by the size of the temporal window between the usage of the two modalities.

Two modalities are concomitant when one modality overlaps another one with a time interval during which the two modalities coexist.

Finally, the coincidence of two modalities is when one modality is used in the temporal context of another one. Such a combination is necessary to implement a modality that can only be used with another one.

A concomitant, coincident as well as a parallel combination implies that the two devices corresponding to the two modalities can function in parallel. These three temporal relationships refine the notion of simultaneous usage of modalities introduced in the previous section (TR) while anachronism and sequence further refine the sequential usage of modalities (TR).

Multimodal fusion examples

The CARE properties along with the temporal relationships have been shown to be useful for multimodal input interaction as well as for multimodal output interaction.

A first example of multimodal input interaction using the previous combination space can be found in [START_REF] Coutaz | Four easy pieces for assessing the usability of multimodal interaction: the CARE properties[END_REF], the MATIS system. MATIS stands for Multimodal Air Travel Information System. MATIS allows a user to retrieve information about flight schedules using speech, direct manipulation, keyboard and mouse, or a combination of these methods, supporting individual and combined use of multiple input modalities. For example, using a single modality, the user can say "show me the flights from Boston to Pittsburgh", can type sentences in pseudo-natural language in a dedicated text window, or can fill in a form using the keyboard and the mouse. When exploiting combined usage of modalities, the user may also combine speech and gesture as in "show me the flights from Boston to this city" along with the selection of "Pittsburgh" with the mouse on the screen ("put that there" paradigm [START_REF] Bolt | Put-that-there: voice and gesture at the graphics interface[END_REF]).

Although we focus on input multimodality in this paper, we present an example of output multimodality highlighting the fact that our combination space can also be applied to output multimodality. We consider a multimodal application that we developed with only one output device (i.e. screen). This example consists of a visualization system for large tables of numerical data, the MulTab system [START_REF] Vernier | A Framework for the Combination and Characterization of Output Modalities[END_REF]. The system uses several output modalities based on the same output device, the screen. There are three different modalities: M1 displays the entire table. M2 is used to color each cell of the table according to the numerical data. M3 displays a part of the table deformed. Those three modalities can be combined in different ways. In Figure 3, M1 and M3 are combined in a Complementary way (CARE dimension, Section 2.1), and the temporal aspect of the combination is parallel (Temporal dimension, Section 2.2).

Figure 3. Combination of two output graphical modalities in the MulTab system (from [23]): a case of complementarityparallelism.

After defining our CARE combination space and illustrating it, we now present the different implementations of it, from a monolithic fusion algorithm to composition components.

CARE: FUSION ALGORITHM IN PAC-AMODEUS

The CARE properties were first implemented as a fusion engine in the PAC-Amodeus software architecture [START_REF] Nigay | A Generic Platform for Adressing the Multimodal Challenge[END_REF]. The PAC-Amodeus model along with the fusion engine formed a reusable global platform applicable to the software design and implementation of multimodal applications. PAC-Amodeus is a conceptual model that blends together the principles of both Arch [START_REF]A Metamodel for the Runtime Architecture of an Interactive System[END_REF] and PAC [START_REF] Coutaz | an Object Oriented Model for Dialog Design[END_REF]. PAC-Amodeus has been used to implement the MATIS system, presented in the previous section.

The fusion mechanism relies on a uniform representation: a melting-pot which is a 2D structure. On the vertical axis, the "structural parts" model the composition of the task objects. For example, destination and time departure are the structural parts of the task objects handled for MATIS. The horizontal axis represents the time. Fusion is performed on those melting pots, as shown in Figure 4. There are three types of implemented fusion in PAC-Amodeus: microtemporal fusion, macrotemporal fusion, and contextual fusion. Microtemporal fusion is used to combine input events produced in parallel or in a pseudo-parallel manner (i.e., Parallelism, Coincidence or Concomitance along the temporal dimension of our combination space of Figure 2). Macrotemporal fusion is used to combine input events produced sequentially (i.e., Anachronism or Sequence in Figure 2). Contextual fusion is used to combine related input events produced without attention for temporal constraints.

Figure 4. Macrotemporal fusion of two melting pots in PAC-Amodeus (from [7]).

While this fusion mechanism defines a reusable software solution implementing the redundancy and complementarity of input modalities, we also study the development of a fusion mechanism by adopting a component-based approach. Indeed such an approach offers the established advantages of verifying the software engineering properties of reusability maintainability and evolution [START_REF] Szyperski | Component Software: Beyond Object-Oriented Programming, Second edition[END_REF]. The next section presents our general componentbased model for multimodal interaction as well as the composition components based on CARE.

CARE-BASED COMPOSITION COMPONENTS

Before focusing on the composition components for performing fusion, we present our underlying component-based approach for multimodal interaction.

Component-based approach for multimodal interaction

Our approach [START_REF] Serrano | A Three-dimensional Characterization Space of Software Components for Rapidly Developing Multimodal Interfaces[END_REF] defines a set of characteristics of components, understanding the term "component" as any type of software unit (e.g., software component, service). It does not define the execution behavior of the components as do traditional component-based models, such as the Corba Component Model (CCM) [START_REF]OMG -CORBA[END_REF] or JavaBeans [START_REF] Ejb | [END_REF]. The aim of this approach is to be used to define and implement high-level characteristics of components that can be implemented in any of those technologies.

Our characterization space of software components for multimodal interaction is constructed along three dimensions [START_REF] Serrano | A Three-dimensional Characterization Space of Software Components for Rapidly Developing Multimodal Interfaces[END_REF]. In the present work on fusion, we use two of those three dimensions (Figure 5). The first dimension is related to the data-flow from input devices to an interactive application. Along this axis, there are four types of components: Device, Transformation, Composition and Task. Fusion is performed at the Composition level.

The second dimension describes the genericity of the components. The space includes both generic and tailored components. Generic components represent high-level reusable abstractions. Tailored components implement operations for specific devices/modalities or for application-dependent tasks. It is possible to integrate existing tailored composition components in our approach, such as the fusion mechanism presented in [START_REF] Miki | An integrative recognition method for speech and gestures[END_REF], which combines speech and pointing. This fusion mechanism is tailored because it is an integrative recognition method based on a set of possible associations between an utterance and a gesture in the application context of explanations of geometry problems (i.e., modality and task dependent). However, in this paper we only present generic composition components, defining elementary temporal fusion operations, independent from modalities and application tasks. The Composition components are related to Redundancy and Complementarity of CARE.

Generic composition components

We have developed two implementations of the Composition components, first in ICARE [START_REF] Bouchet | ICARE Software Components for Rapidly Developing Multimodal Interfaces[END_REF] and more recently in OpenInterface [START_REF]IST Framework 6 STREP funded by the European Commission[END_REF]. Those two software tools are based on a similar component-based approach that consists of visually assembling components in a graphical editor in order to define the pipeline corresponding to the abstracting function from actions using devices to tasks. The JavaBeans technology [START_REF] Ejb | [END_REF] was chosen to develop ICARE software components. As opposed to ICARE, the OpenInterface (OI) framework [START_REF]IST Framework 6 STREP funded by the European Commission[END_REF] is independent of the component technology and is able to handle distributed heterogeneous components based on different technologies (Java, C++, Matlab, Python, .NET). Figure 6 illustrates the graphical editor of ICARE and Figure 7 the graphical editor of OpenInterface, namely the OIDE (OpenInterface Interaction Development Environment). The OIDE allows the designer to define a multimodal interaction by assembling components. This assembly of components can be directly executed (no code is generated): each component will run separately, sending and receiving events from other components. We have implemented generic composition components within those two frameworks corresponding to Redundancy and Complementarity of CARE. Composition components implemented in ICARE are described in [START_REF] Bouchet | ICARE Software Components for Rapidly Developing Multimodal Interfaces[END_REF]. They were used for developing several multimodal systems, such as the FACET system (plane cockpit simulator [START_REF] Bouchet | ICARE Software Components for Rapidly Developing Multimodal Interfaces[END_REF]), MEMO, a GeoNote system [START_REF] Bouchet | ICARE: A Component-Based Approach for the Design and Development of Multimodal Interfaces[END_REF] or MID, a Multimodal Identification system [START_REF] Bouchet | ICARE: A Component-Based Approach for the Design and Development of Multimodal Interfaces[END_REF] that enables the user to identify herself/himself through three modalities.

In this paper we focus on the two composition components implemented in OpenInterface. Those composition components have two inputs, one for each input modality, and one output, for sending the resulting combined event. In the case of Redundancy, when the component receives two events within the same temporal window, it sends one of them (first port event by default). As the two events contain redundant information, there is no need to send both events. The Complementarity component waits for receiving two events within the same temporal window and sends a concatenation of both events. The Redundancy and the Complementarity components have a configuration port that is used to define the size of the temporal window. This temporal window is defined by the designer/developer using the OIDE and before executing the component assembly.

Based on these two composition components, we have implemented several multimodal applications using the OpenInterface framework, such as a multimodal map navigator [START_REF] Serrano | A Three-dimensional Characterization Space of Software Components for Rapidly Developing Multimodal Interfaces[END_REF] and a multimodal arkanoid game. We use a multimodal slide viewer application to illustrate the composition components. This application allows the user to navigate within a list of slides using a combination of gestures. Two devices are used for the interaction: the wiimote and the "wiisotf" (Figure 8). The 'wiisoft' is a torsion sensor inside a wiimote plastic cover (InterfaceZ sensor [START_REF]Interface-Z[END_REF]). The pressure on the "wiisoft" is used to change the slide or to zoom. The wiimote is used to change the slide by rotating the device or to point on the current slide. In an equivalent combination of the modalities, which does not imply fusion and therefore composition components, the user can move to the next slide by rotating the wiimote to the right or by pushing the "wiisoft".

In a redundant combination of modalities, the user can move to the next slide by rotating the wiimote and by pushing the "wiisoft" at the same time. The user must perform the two gestures at the same time (simultaneously within a temporal window TW) in order to accomplish the task. Figure 9 illustrates this redundant combination using the notation presented in Section 2. Figure 10 shows the corresponding component assembly of the redundant combination of the two modalities. Such assembly of components is graphically specified in the OIDE (the component assembly of Figure 7). In a complementary combination of modalities, the user can zoom on a point of the current slide by pointing with the wiimote and pressing the 'wiisoft'. The wiimote will define the point to be zoomed while the 'wiisoft' will define the zoom value. Figure 11 illustrates the complementarity combination using the notation presented in Section 2. Figure 12 shows the component assembly of the complementarity combination of these two modalities. We can note two differences from the previous assembly (Figure 10): the 2D Position component and the composition component (reusability of the component assembly). Such modifications can therefore be graphically done quickly using the OIDE (Figure 7) in order to explore different multimodal design solutions.

Figure 12. Complementary component for combining two input modalities.

Within an assembly of components, several Composition components can also be used to define further complex combinations, involving several types of fusion. For example, let us consider a new modality based on a pedal. In order to reduce recognition errors from the 'wiisoft' sensor, which is too sensible and launches erroneous zoom actions, we add a redundant composition of the 'wiisoft' and the pedal. The user has to push the pedal and the 'wiisoft' at the same time in order to be able to specify the zoom action (i.e, push-to-gesture similar to push-totalk in the case of speech recognition). Figure 13 illustrates this example using two Composition components, the new added Redundancy component and the Complementarity component as in Figure 12.

We have presented Composition components for performing fusion as part of a model-based approach for prototyping multimodal interaction. Based on the CARE properties, the two composition components are generic and reusable for different input modalities. For the fine tuning of the temporal window of these composition components, we have also developed a novel approach: Wizard of Oz (WoZ) composition components.

WOZ composition components

As explained in [START_REF] Davis | SketchWizard: Wizard of Oz Prototyping of Pen-Based User Interfaces[END_REF], "a Wizard of Oz prototype is an incomplete system that a designer can simulate 'behind curtain' while observing the reactions of real end users". As pointed out in [START_REF] Cheng | A Wizard of Oz Framework for Collecting Spoken Human-Computer Dialogs[END_REF],

WoZ studies have been shown to allow fast prototyping of interactive applications by allowing a human operator ( i.e., a wizard) to simulate missing functions.

For the case of the Composition components, the Redundancy and Complementarity components presented before, it can be relevant to allow a human operator (i.e., a wizard) to decide in real time which type of temporal relation is used. Indeed, as pointed out in [START_REF] Oviatt | Individual Differences in Multimodal Integration Patterns: What Are They and Why Do They Exist?[END_REF], individual differences appear between users in their multimodal integration pattern. Analyses revealed that everyone has a dominant integration pattern, either simultaneous or sequential, which is consistent and remains stable over time. In order to quickly identify a person's dominant integration pattern in real time and to adapt the system, we use a Wizard of Oz (WoZ) approach. A WoZ approach for adapting the system to user's dominant pattern was implemented in [START_REF] Oviatt | Toward a Theory of Organized Multimodal Integration patterns during Human-Computer Interaction[END_REF]. Our approach generalizes that work by proposing a component-based WoZ approach, which offers the benefit of being reusable and generic. We first introduce our WoZ component-based approach for multimodal interaction before presenting the WoZ Composition components.

WoZ components for multimodal interaction

We define WoZ components as parts of our component-based approach described in Section 4.1. These WoZ (non-functional) components are characterized according to the roles that a WoZ component can play in the data-flow of input multimodal interaction, from devices to tasks (vertical axis in Figure 5).

This approach allows building different types of prototypes with different degrees of functionality (according to the number of functional/non-functional components). This enables testing and improving functional components while evaluating the overall multimodal interaction. During the evaluation, the wizard can identify errors in those functional components that can be fixed later. Moreover, at each iteration of the design cycle, the designer/developer can replace some of the non-functional components by functional components, towards a final functional prototype that better fits the users' needs and preferences.

All generic WoZ components have the same interface configuration. Such a graphical user interface (GUI) is manipulated by the wizard. The WoZ component GUI is composed of several panels. Figure 14 shows the general GUI configuration of a WoZ component corresponding to a onedimension device. This WoZ component allows the wizard to generate events corresponding to a one-dimension device (e.g., rotatory pot, button). Our experiences show that simulating a device is possible when the data generated by the device is simple.

When dealing with complex input (images, sound), the wizard will use WoZ command components, which allow generating data at a higher level of abstraction. At the bottom of the window, two console panels display input and output events. Input events correspond to the events being received by the component. Some WoZ components may not have any input data. Output events correspond to the events being generated by the wizard. On the top left, a panel is dedicated to generating WoZ events, in this case with a slide bar. On the top right, a panel contains the configuration elements. Those two top panels are different for each type of WoZ components. The GUI of the WoZ components does not contain the video frame where the wizard will observe the end-user in action. This video frame is displayed on a dedicated terminal next to the computer, as it is usually done in evaluation studios. Two main solutions have been taken in order to reduce the work load of the wizard in case there are several WoZ components in the assembly of components. Firstly, we consider multi-wizard configurations. In this case, the GUI of the WoZ components are displayed and controlled by different wizards on different computers (Figure 15).

Secondly, in the case of a single wizard, all the GUI of the WoZ components being displayed on the same computer are integrated into a single GUI. In the following section, we illustrate this point in the case of WoZ Composition components.

WoZ Composition components

In order to adapt the temporal fusion to the user's dominant integration pattern, the designer or developer can use a WoZ Composition component. WoZ Composition components also allow replacing of a missing input modality, while still using the events coming from the other one. Based on the received input events as well as the actions of the wizard, the Composition components generate the outputs.

For illustrating WoZ Composition components, we use the same zoom task of the slide viewer example (Figure 12), but this time for the case of a multimodal map navigator. It consists of a map that can be controlled using several interaction modalities in order to perform a set of interactive tasks such as panning or zooming.

For example, the zoom on a specific point of the map can be specified by combining speech and pointing gesture (Figure 16). Capture of the finger position action can be done using several innovative sensors. Figure 16 shows the use of the DiamondTouch tactile surface to capture the finger position. In this example, the wizard observes the user and generates commands such as <zoom in> or <zoom out> according to the user's spoken utterances. The WoZ Composition component generates a combined output event by coordinating the command event (launched by the wizard using one of the command buttons) and the input event coming from the DiamondTouch. In order to define the temporal combination of the two input events and to modify it in real time according to user's dominant integration pattern, we use the temporal relationships presented in Figure 2.

The temporal relationships are displayed in the GUI (Figure 17). The wizard can select a temporal combination in the list and define its time value by double-clicking on it (Figure 17). This will define which event of the DiamondTouch is used to generate the combined event, when the wizard clicks on a command button. In case the designer or the developer wants to fully simulate a multimodal interaction, they can create a non-functional prototype using WoZ components. For instance, in our running example, we want to simulate the tactile surface in order to allow a vertical setting (map projection on a wall) since we currently have no component for capturing finger pointing on a wall. We connect two WoZ components, one for simulating the speech commands and another one for simulating the 2D finger pointing, to a WoZ Composition component. As seen previously, this Composition component will integrate different elements from the GUI of the other two components in order to define a unified wizard interface (Figure 18). This graphical interface will allow the wizard to simulate a multimodal interaction based on complementarity of two simulated modalities. In order to generate combined events, the wizard has two choices. Firstly, s/he can push a button to put the WoZ component in a certain mode. For example, if s/he pushes the "zoom in" button, s/he will put the component in "zoom in" mode, and each time s/he will click on the map, s/he will generate a zoom command on that point. Secondly, s/he can use key shortcuts to simulate a multimodal interaction using the keyboard and the mouse. S/he clicks on the map, and generates a command using a key. On Figure 18, we can see that the wizard assigned the 'H' key to the "zoom here" command.

CONCLUSION

In this paper, we presented a summary of the different solutions for multimodal interaction that have been defined based on the CARE properties. The CARE properties (Complementarity, Assignment, Redundancy and Equivalence) define various forms that multimodal input interaction can take. Redundancy and Complementarity describe relationships between modalities and require fusion mechanisms. The CARE properties have been shown to be useful concepts for the design and evaluation of multimodal interaction in [START_REF] Coutaz | Four easy pieces for assessing the usability of multimodal interaction: the CARE properties[END_REF]. In this paper we showed how we make such concepts explicit while designing/implementing multimodal interaction and in particular for the cases of fusion (Redundancy and Complementarity). Two main approaches have been presented: a single fusion mechanism anchored in the PAC-Amodeus software architecture and a component-based approach.

As part of our component-based approach for multimodal interaction, we introduced a new contribution based on generic composition components and WoZ composition components.

Future work will focus on other temporal aspects of the WoZ composition components, such as the integration of highfrequency data. We are currently studying the synchronous aspects of WoZ components for the case of simulating modalities that provide continuous events. Our goal is to avoid asking the wizard to perform repetitive tasks such as generating the same event at a fixed frequency. A further complementary challenge we plan to address is to define tools for analyzing the test data in the context of our component-based approach for rapidly prototyping and testing multimodal interaction.

Figure 2 .

 2 Figure 2. Allen's temporal properties.

Figure 5 .

 5 Figure 5. Two dimensions of our component-based approach [20].

Figure 6 .

 6 Figure 6. ICARE: Graphical assembling of components.

Figure 7 .

 7 Figure 7. OpenInterface: Redundant composition of modalities graphically defined using the OIDE.

Figure 8 .

 8 Figure 8. Multimodal slide viewer using the 'wiisoft' and the wiimote.

Figure 9 .

 9 Figure 9. Redundancy of Wiimote and Wiisoft expressed with the CARE notation.

Figure 10 .

 10 Figure 10. Redundancy component for combining two input modalities.

Figure 11 .

 11 Figure 11. Complementarity of Wiimote and Wiisoft expressed with the CARE notation.

Figure 13 .

 13 Figure 13. Complementarity and Redundancy for combining three input modalities.

Figure 14 .

 14 Figure 14. General User Interface of WoZ components.

Figure 15 .

 15 Figure 15. Multi-wizard configuration.

Figure 16 .

 16 Figure 16. Illustrative example of multimodal interaction on a map and the corresponding component assembly.

Figure 17 illustrates

 17 Figure 17 illustrates the use of a WoZ Composition component along with a WoZ command component, which replaces the speech command recognition. The WoZ Command component is for example useful to perform experimental tests without having biased results due to a poor recognition accuracy of speech commands. For the case of several WoZ components and one wizard, the Composition component imports GUI elements from

Figure 17 .

 17 Figure 17. Graphical interface of the WoZ Composition component in the case of a partially simulated multimodal prototype.

Figure 18 .

 18 Figure 18. Graphical interface of the WoZ Composition component in the case of a fully simulated prototype.


Renaud Blanch
email: blanch@imag.fr

Michaël Ortega
email: ortega@imag.fr

Rake Cursor: Improving Pointing Performance with Concurrent Input Channels

Keywords: Fitts' law, multi-channel pointing, rake cursor. ACM Classification H.5.2 [Information Interfaces and Presentation (e.g., HCI)]: User Interfaces -Graphical user interfaces, Input devices and strategies

We investigate the use of two concurrent input channels to perform a pointing task. The first channel is the traditional mouse input device whereas the second one is the gaze position. The rake cursor interaction technique combines a grid of cursors controlled by the mouse and the selection of the active cursor by the gaze. A controlled experiment shows that rake cursor pointing drastically outperforms mouse-only pointing and also significantly outperforms the state of the art of pointing techniques mixing gaze and mouse input. A theory explaining the improvement is proposed: the global difficulty of a task is split between those two channels, and the sub-tasks could partly be performed concurrently.

INTRODUCTION

Pointing is a fundamental task in graphical user interfaces (GUIs). Many interaction techniques have been proposed to reduce pointing time. This paper explores a new approach: using two concurrent input channels to perform a pointing task. The first channel is the regular mouse. The mouse movements move the cursor in a standard manner. The only modification is that the standard cursor is replaced by a grid of cursors, all moving together. The second input channel is the gaze. The eye movements do not move the cursors, they have no motor effect. The gaze position is only used to select which cursor is active, i.e. where the traditional mouse events are send to the system. Figure 1 illustrates this principle with a hexagonal (hex) grid of cursors, the active one being the closest to the gaze position (figured by a red disc), the others being semi-transparent. We call this technique rake cursor. Rake cursor can be seen as a merge of several previously proposed interaction techniques. Gaze position to select object in GUIs has been proposed and evaluated for a while [START_REF] Jacob | What you look at is what you get: eye movement-based interaction techniques[END_REF][START_REF] Sibert | Evaluation of eye gaze interaction[END_REF]. However, using the gaze to interact is still under investigation (e.g., [START_REF] Kumar | EyePoint: practical pointing and selection using gaze and keyboard[END_REF]) and not very much widespread, perhaps because most interaction techniques use the gaze as a motor input which is unnatural as noted by Zhai et al. [START_REF] Zhai | Manual and gaze input cascaded (MAGIC) pointing[END_REF]. The MAGIC interaction technique they proposed to circumvent this problem has a limitation that could explain that rake cursor performs better: the mouse and gaze input are sequential and the cursor jumps near the target but at a position difficult to predict for the user.

Using a grid of cursors has been recently investigated with the Ninja Cursors technique [START_REF] Kobayashi | Ninja cursors: using multiple cursors to assist target acquisition on large screens[END_REF]. The problem that impairs this technique is the possible ambiguity occurring when multiple cursors hover over different potential targets at the same time. Instead of requiring a suplemental interaction to resolve this ambiguity, as Ninja Cursors does, rake cursor prevents it by using a suplemental input channel which states explicitly at any time which cursor is the active one.

After describing the rake cursor technique and its implementation, we describe a controlled experiment that compares our technique to normal pointing and to the state of the art of gaze-enhanced pointing. We then propose a theory to explain the observed benefit of the rake cursor technique. Finally, we discuss potential extensions of this technique.

RAKE CURSOR

Figure 2 illustrates rake cursor in action during a drag-anddrop interaction: the user wants to put a folder in the trash. The normal interaction implies to traverse the whole screen while holding the mouse button depressed, which is a tedious task. With the rake cursor, the cursor grabbing the folder is active when the user starts the drag (left). Since the goal is the trash, it becomes naturally the gaze focus, which makes the closest cursor active (right). Since the cursor was already at that location the change does not introduce visual discontinuity and thus does not perturb the eye. The movement remaining to complete the task is easy: its amplitude is bounded by the distance between the cursors (D R ).

Implementation

Given the system cursor position, we compute the possible positions for the cursor on a hex grid. We choose this grid because of its regularity and because it is known to be the densest plane lattice packing1 thus giving the best tradeoff between cursor density and D R .

Those rake positions are used each time a mouse movement is detected as illustrated by Algorithm 1:

• the position of the gaze is monitored and recorded in order to be used by the mouse movement handler; • the motion of the mouse is also monitored, it triggers the recording of the current cursor position, and then schedules an immediate redisplay; • when redisplaying, a semi-transparent cursor is drawn at each rake position, and the position closest to the last known gaze position is recorded (best position); • finally, if the current system cursor position is not the best, it is wrapped to this position (triggering a mouse motion event, which reenters the process described here).

Two implementation details should be noticed. First, after each redisplay the gaze position is updated to the computed best position. We do this to reduce the impact of a lack of gaze tracking: the cursor is not trapped in the neighborhood of the last known gaze position if it is not updated. This enables a graceful degradation of the technique. Second, the gaze does not produce any modification of the state of the

for all p ∈ COMPUTE_RAKE_POSITIONS(p c ) do d ← p -p g if d < d b then p b ← p d b ← d if p = p c then draw supplemental cursors DRAW_CURSOR(p) if p b = p c then wrap system cursor MOVE_SYSTEM_CURSOR(p b ) p g ← p b
GUI by itself. The active cursor can change only when a mouse motion event occurs. Thus, when the user does not move the mouse, the display is totally stable. A reference implementation for Mac OS X is made freely available 2 .

Applications

The rake cursor can be used directly on any GUI. It only requires to have a gaze tracker. This requirement may seem high given the cost of current eye tracking solutions, but the tracking our technique needs does not have to be very precise. Since the gaze is not used to point but to select the active cursor, the requirement on the precision of the tracking is low: it only needs to disambiguate between cursors that are D R (typically ≈ 400) pixels distant from each others. This precision is much less than the one expected for standard eye trackers. A pure software tracking using a webcam could thus be sufficient for the rake cursor technique. Rake cursor would be of particular interest for disabled people with limited movement capacity. It reduces the amplitude of movements, thus limiting the effort needed. In contrast to other eye tracking based input techniques, it does not use the gaze as a motor channel, and thus it does not stress the user.

Another good property of the rake cursor is that the cursor is literally anywhere at anytime. The little (but annoying) trouble of loosing the cursor is totally suppressed by the rake cursor technique: the cursor can not be loosed because it is where you look. More seriously, extending the rake to multiple displays would generalize the Multi-Monitor Mouse technique [START_REF] Benko | Multi-monitor mouse[END_REF]. This requires an eye trackers spanning the displays or multiple eye trackers but also solves other issues such as drag-and-drop spanning displays [START_REF] Baudisch | Mouse ether: accelerating the acquisition of targets across multi-monitor displays[END_REF].

EXPERIMENTAL VALIDATION

To test the rake cursor idea, we ran a controlled experiment. Since we just wanted to show that the idea is worth exploring, we choose to use the well established, minimalist, à la Fitts, 1D protocol. In such a setup where there is only one target, there is no possible ambiguity for the Ninja Cursors technique, so it would at least be as good as the rake cursor technique. Comparing the two techniques would require a more elaborated setup (e.g., a 2D setup inspired by the ISO 9241-9) stressing the techniques with distractor targets.

Task

Participants had to perform successive 1D discrete pointing tasks. They had to move the cursor, represented by a one pixel thick vertical black line, to the start position marked by a gray rectangle on the left of the screen, rest there for 0.5 s, start moving to the target -a blue rectangle-as soon as it appeared on the right, and click it (Figure 3). After each block, their error rates were displayed and they were encouraged to conform to a nominal 4% error rate by speeding up or slowing down.

Conditions and Procedure

The control (C) condition used a mouse without acceleration. The magic (M) condition used a 1D implementation of the MAGIC technique: on a mouse motion, if the cursor is farther than 120 pixels from the gaze, it is wrapped to the gaze with a 120 pixels undershoot (distance suggested by [START_REF] Zhai | Manual and gaze input cascaded (MAGIC) pointing[END_REF]). The rake (R) condition used a 1D rake i.e. an array of 6 one pixel thick vertical black lines 200 pixels distant. Four IDs [START_REF] Fitts | The information capacity of the human motor system in controlling the amplitude of movement[END_REF][START_REF] Hick | On the rate of gain of information[END_REF][START_REF] Jacob | What you look at is what you get: eye movement-based interaction techniques[END_REF][START_REF] Kobayashi | Ninja cursors: using multiple cursors to assist target acquisition on large screens[END_REF] and two sizes (D = 511 or 1023 pixels) were used, giving eight possible tasks. W s are then given by Equation (1) that links ID, D and W . A pseudo-random series of 80 trials (10 times each possible task), balanced to minimize order effects, was build. This series was split into 2 blocks of 40 trials to allow a pause in the middle of the series. Those two blocks were repeated for each technique condition, making each participant perform 240 trials.

An order for the three conditions was chosen for each participant: the first pair of blocks were performed using the C condition, the second and third pairs were performed using the R then M conditions for half of the participants, and start area rake cursor target

D W

Figure 3. Experimental setup with a rake cursor (the active cursor on the target is the one which is looked at by the user).

the M then R condition for the other half. Each series was preceded by 10 randomly-chosen tasks using the same condition to train the participants.

Subjects and Apparatus

Sixteen unpaid adult volunteers, 5 female and 11 male, served in the experiment. We had to discard one of the them because the gaze tracker could not produce accurate gaze position for him (presumably due to reflects on his glasses).

The experiment was conducted using a custom software and a Tobii ET-17 eye tracker (17-inch 1280 × 800 monitor).

Results

The effects of the technique were explored by analyzing four dependent variables: error rate (ER), reaction time (RT ), movement time (MT ), and total time (T T = RT + MT ). Repeated measures analyses of variance were performed on these four variables. We analyzed the effects of the three factors (3 conditions, 4 indices of difficulty, and 2 sizes) in a within-participant full-factorial design.

Error rate ER is 3.86% on average (slightly better than the 4% consign). The strongest effect on ER is the technique (F 2 = 5.96, p = 0.00263 ), the second one is ID (F 3 = 4.55, p = .0035). The rake condition gives the best error rate (2.25 ± 2.2%4 vs. 2.75 ± 2.0% for M; 6.58 ± 4.1% for C).

Movement time

The strongest effect on MT is the technique condition (F 2 = 120.66 *5 ). The rake condition gives the lowest movement times (0.64 ± .14s vs. 0.88 ± .10s for M; 1.39 ± .22s for C).

As expected, ID has also a very strong effect (F 3 = 45.57 * ).

We can also notice a significant effect of the interaction between D and the condition on MT which is probably due to the fact that for R, the difficulty of the pointing task is not ID but ID S which depends on D and D R .

Reaction time

The strongest effect (F 2 = 131.82 * ) on RT is the condition. This is not usual for pointing experiments. Further investigation shows that the RT are pair-wise significantly different (Student's t test), although less significantly for C and M. The rake condition gives the slowest reaction time (0.40±.06s vs. 0.30±.04s for M; 0.28±.03s for C). A plausible explanation is that the multiple cursors of the rake adds a cognitive load: the selection of the active cursor is not fully parallelized with the pointing.

Total time

Since RT is longer for the R condition, and MT shorter, the best way to compare the techniques is to consider the total time T T . The main effects on T T are the condition (F 2 = 88.18 * ) and ID (F 3 = 47.46 * ). As for MT , the interaction between D and the condition is significant. A Student's t test shows that T T is significantly different for each pair of conditions with R being the fastest technique, M the second fastest, both outperforming C (1.04 ± .15s vs. 1.18 ± .09s for M; 1.68 ± .22s for C, Figure 4). On average, the rake cursor interaction technique performs better (less errors, reduced pointing time), providing a 38.1% gain on the total pointing time, while the magic interaction technique provides a 29.8% benefit.

PROPOSED INTERPRETATION

In this section we propose a theory that could explain the gain observed in the experiment. For a normal pointing interaction, Fitts' law [START_REF] Fitts | The information capacity of the human motor system in controlling the amplitude of movement[END_REF] is accurate at modeling the movement time (MT ) as a linear function of the index of difficulty (ID), itself a function of the target distance (D) and width (W ):

MT = a + b × ID, with ID = log 2 D W + 1 . (1
) With the rake cursor technique, this expression of the difficulty is not relevant: the distance to the target depends upon the active cursor. If we consider that the user gaze is on the target, the active cursor is the one closest to the target. The distance to the target is then bounded by the distance between the cursors of the rake (D R ). The difficulty of the mouse pointing task (ID M ) is thus bounded:

ID M ≤ log 2 D R W + 1 . (2)
The other sub-task is a selection performed the gaze. Such a task is known to follow Hick's law [START_REF] Hick | On the rate of gain of information[END_REF]: the time to choose between n equally probable options is proportional to log 2 (n + 1). In our case, the number of choices is given by the number of cursors in the rake, which is inversely proportional to the distance between them (D R ) so the selection time should be proportional to the selection difficulty (ID S ):

ID S = log 2 k D R + 1 . (3)
ID can be interpreted as bits of information to transmit to the system. The "information theory" interpretation of Fitts' law [START_REF] Mackenzie | A note on the information-theoretic basis for Fitts' law[END_REF] states that those bits have to be transmitted through a channel -the movement-with bounded throughput (b) thus leading to MT . The same bits have to be transmitted by our rake cursor technique so we have ID M + ID S = ID. The movement time and the selection time (T M = a + b × ID M , and T S ∝ ID S ) could be overlapped if the movement and selection sub-tasks could be performed concurrently. In the experiment, the longer reaction time for rake cursor means that the overlap is not complete: the selection task retards the pointing. But the total time shows that the bandwidth between the user and the system is overall a bit better than with MAGIC.

CONCLUSION

We presented the rake cursor input technique, aimed at facilitating pointing in GUIs by using two concurrent input channels: the mouse motion to move a grid of cursors; and the gaze position to select the active cursor. We explained the details of the technique and provide a working implementation for the Mac OS X system. We have shown that the rake cursor technique outperforms the MAGIC technique while also not overloading the visual channel with a motor control. We have proposed an explanation for this improvement: the rake cursor technique allows to use the motor and visual channels concurrently. We expect that rake cursor will be valuable for any user but especially people having limited movement capacity. Rake cursor can also solve problem arising in multi-display setups.

In the future, we would like to compare the rake cursor technique to other techniques such as Ninja Cursors in a more realistic setup that could test the validity of our proposed interpretation. We also would like to study the impact of the form of the grid. We know that D R impacts the performance: when it becomes larger than the screen, the rake cursor degenerates to the normal cursor. On the other hand, D R = 1 pixel would mean that the pointing is done only by sole the gaze channel. Since both limit case are not efficient, an optimum D R value must exists somewhere in between. The determination of this optimum, and more generally the impact of the form of the rake on its efficiency will be some of the next question we will investigate.

Figure 1 .

 1 Figure 1. Rake of cursors. The gaze (figured by the red disc) selects the active cursor, other cursors are semi-transparent.

Figure 2 .

 2 Figure 2. Multiple cursors drag-(left) and-drop (right). The gaze selects the active cursor (red discs figure the gaze position added, cursors magnified).

Figure 4 .

 4 Figure 4. Total time vs. index of difficulty. From slowest (top) to fastest (bottom): control (C); magic (M); and rake (R) condition.

  Algorithm 1 Rake cursor -handling events Variables starting with p are 2D positions on the screen.

	global p g , p c	gaze and cursor positions
	procedure ON_GAZE_MOVE(p)
	p g ← p	record gaze position
	procedure ON_MOUSE_MOVE(p)
	p c ← p	record cursor position
	REQUEST_REDISPLAY( )
	procedure ON_REDISPLAY( )
	p b ← p c	compute best cursor position
	d b ← p c -p g	(i.e. minimum distance to gaze)

E. W. Weisstein. Circle Packing.

http://mathworld.wolfram.com/CirclePacking.html

The source code is available at: http://iihm.imag.fr/blanch/projects/rake-cursor/.

The total number of degree of freedom (371) is omitted for Fs.

µ ±σ gives the mean (µ) and standard deviation (σ ) across users.

* denote p < .0001.

ACKNOWLEDGEMENTS

The authors would like to thank N. Mandran (MARVELIG) and B. Meillon (LIG-MULTICOM) for helping with the statistics and providing the gaze tracker. Part of this work is performed within the OpenInterface consortium (EU STREP FP6-35182).