Joëlle Coutaz
email: joelle.coutaz@imag.fr

User Interface Plasticity: Model Driven Engineering to the Limit!

Keywords: User interface plasticity, user interface adaptation, user interface generation, run time adaptation, user interface composition, dynamic service composition, model driven engineering (MDE), service-oriented architecture (SOA). ACM Classification D.2.2 [Software Engineering]: Design Tools and Techniques -User interfaces

Ten years ago, I introduced the notion of user interface plasticity to denote the capacity of user interfaces to adapt, or to be adapted, to the context of use while preserving usability. The Model Driven Engineering (MDE) approach, which was used for user interface generation since the early eighties in HCI, has recently been revived to address this complex problem. Although MDE has resulted in interesting and convincing results for conventional WIMP user interfaces, it has not fully demonstrated its theoretical promises yet. In this paper, we discuss how to push MDE to the limit in order to reconcile high-level modeling techniques with low-level programming in order to go beyond WIMP user interfaces.

INTRODUCTION

Ten years ago, I introduced the notion of "user interface plasticity" to denote the capacity of user interfaces to adapt, or to be adapted, to the context of use while preserving usability [START_REF] Taleb | Interactive Systems Engineering: A Pattern-Oriented and Model-Driven Architecture[END_REF]. My proposal was motivated by the emergence of ubiquitous computing with the need for accommodating a large degree of variability in terms of heterogeneity, dynamicity, and scalability. Although these challenges have permeated the whole ICT community, they have not been addressed in a holistic, systemic manner.

Typically, virtualization, as developed for cloud computing, does not cover that of interaction resources. SLA (Service Level Agreement) developed for dynamic service composition, does not cover any of the HCI-centered concerns. Research in autonomic systems conveniently keeps humans out of the loop. Service composition as supported by mash-up tools, boils down to the assembly of ready-for-use services whose UI'Ss, which are tightly coupled with business code, cannot be "plastified". Unfortunately, during this period, the HCI community has not been any better, developing a variety of disjoint tools and concepts. HCI researchers have addressed user interface plasticity from different starting points, depending on their "credo": at the toolkit level by those who advocate "hard core development" and fine grained control of user interfaces [START_REF] Crease | Caring, Sharing Widgets: a toolkit of sensitive widgets[END_REF][START_REF] Demeure | COMET(s), a Software Architecture Style and an Interactors Toolkit for Plastic user Interfaces[END_REF], at the infrastructure level with the development of dedicated middleware by those who strive for generic computational substrates [START_REF] Balme | Interfaces homme-machine plastiques : une approche par composants dynamiques[END_REF][START_REF] Newman | Designing for Serendipity: Supporting End-User Configuration of Ubiquitous Computing Environments[END_REF][START_REF] Ponnekanti | Icrafter: a Service Framework for Ubiquitous Computing Environments[END_REF][START_REF] Sousa | The Aura Software Architecture: an Infrastructure for Ubiquitous Computing[END_REF][START_REF] Tandler | Software Infrastructure for Ubiquitous Computing Environments: Supporting Synchronous Collaboration with Heterogeneous Devices[END_REF], to task level modeling by those who believe in the top-down development of user interfaces [START_REF] Berti | Migratory multimodal interfaces in multidevice environments[END_REF][START_REF] Nichols | Huddle: Automatically Generating Interfaces for Systems of Multiple Connected Appliances[END_REF][START_REF] Sottet | Towards Model Driven Engineering of Plastic User Interfaces[END_REF]. Principles and concepts from Model Driven Engineering have however brought some hope into a unifying and systemic approach to the problem of UI plasticity. But is MDE good enough and/or used appropriately?

In this article, we will analyze the contribution of Model-Driven Engineering to HCI as well as its limitations in the light of UI plasticity. From our early experience with MDE and UI plasticity, we will show how to exploit models at run time to obtain maximum flexibility and quality. We will conclude with recommendations for a research agenda.

CONTRIBUTIONS OF MDE TO HCI

The primary contributions of MDE to HCI are two simple notions -that of model and meta-model, which, combined with transformations and mappings, constitute a powerful framework for knowledge sharing and technical integration.

A model is a representation of a thing, with a specific purpose. It is "able to answer specific questions in place of the actual thing under study" [START_REF] Bézivin | First Experiments with the ATL Transformation Language: Transforming XSLT into Xquery[END_REF]. A meta-model sets the rules for producing models. A transformation i s t h e production of a set of target models from a set of source models, according to a transformation definition. In turn, a transformation definition is a set of transformation rules that together describe how source models are transformed into target models [START_REF] Mens | A taxonomy or Model Transformations[END_REF]. A transformation expresses an overall dependency between source and target models. However, experience shows that a finer grain of correspondence needs to be expressed. Typically, the incremental modification of one source element should be propagated easily into the corresponding target element(s) and vice versa. The need for traceability between source and target models is expressed as mappings between source and target elements of these models 1 .

The HCI community has a long experience with models and meta-models, long before MDE existed as a field. In the 1980's, grammars (meta-models) were the formal basis for generating textual and graphical user interfaces [START_REF] Hayes | Design alternatives for user interface management systems based on experience with COUSIN[END_REF]. Until recently, transformation rules were implemented as code within UI generators offering very little to no control over the resulting user interface. In addition, mappings were limited to the expression of correspondence ( b i n d i n g s ) between elements of the user interface with the API of the functional core (i.e. the business code).

MDE has helped the HCI community to promote transformation rules as models. "Transformations as models" has three notable advantages -which, so far, has not been fully exploited by the HCI community:

1. It opens the way to knowledge capitalization and reuse:

frequent transformations can serve as patterns in libraries, which in turn, provide handles for intra-and inter-UI consistency.

2. Comparative evaluations of UI'Ss can be performed in a controlled way, and UI'Ss can be (re)targeted for different contexts of use using d i f f e r e n t transformations.

3. Most notably, transformations can be transformed, offering a powerful formal recursive mechanism for supporting UI plasticity.

To our best knowledge, no research has been conducted on transforming transformations for UI plasticity. On the other hand, patterns are emerging [START_REF] Taleb | Interactive Systems Engineering: A Pattern-Oriented and Model-Driven Architecture[END_REF] and early work has been initiated on UI'Ss generated with different sets of transformation rules to support different usability criteria [START_REF] Gajos | Improving the performance of motor-impaired users with automatically-generated, ability-based interfaces[END_REF][START_REF] Sottet | A Model-Driven Engineering Approach for the Usability of User Interfaces[END_REF]. 1 In mathematics, a mapping is "a rule of correspondence established between two sets that associates each member of the first set with a single member of the second" [The American Heritage Dictionary of the English Language, 1970, p. 797]

Considering the big picture, MDE has provided the HCI community with a vocabulary and a framework to express its own conceptual generic framework for the development of plastic user interfaces: the CAMELEON reference framework [START_REF] Calvary | Plasticity of User Interfaces: A Revised Reference Framework[END_REF]. As shown in Figure 1, the CAMELEON reference framework makes explicit a set of models (e.g., task model, Abstract UI, Concrete UI, Final UI) that serves as a common vocabulary within the HCI community t o discuss and express different perspectives on a user interface. Again, the notion of transformation borrowed from MDE, is used to combine t h e s e m o d e l s into d i s t i n c t d e v e l o p m e n t processes. For example, conventional UI generation operates by the way of top-down vertical transformations.

Typically, an abstract UI (AUI) is derived from the domaindependent concepts and task models. In turn, the AUI is transformed into a concrete UI (CUI), followed by the final executable UI (FUI). At the opposite, a reverse engineering process infers abstract models from more concrete ones using vertical bottom-up transformations. Translations are horizontal transformations that maintain the same level of abstraction between the source and target models.

Figure 1. The CAMELEON reference framework for the development of plastic user interfaces (adapted from [9]).

Unlike the process initiated in the 1980's, which contained one entry point only at a high level of abstraction, the CAMELEON framework authorizes entry points at any level of abstraction from which any combination of horizontal and vertical bottom-up and top-down transformations can be applied. This theoretical flexibility means that the stakeholders involved in the development of an interactive system can use the development process that best suits their practice or the case at handIn short, the CAMELEON reference framework is an MDE-compliant conceptual generic structuring tool for the development of plastic UI's:

• As a structuring reference framework, it federates the HCI community around a consensus.

• As a conceptual generic tool, it sets a vast agenda for technical research.

• As an MDE-compliant framework, it is still unclear in practice that modeling is the only way to go in HCI. This issue is discussed next.

MDE AND HCI IN PRACTICE

The CAMELEON reference framework brings together the "right models" but the HCI community is far from having the "models right". The profusion of initiatives and User Interface Description Languages (UIDL) is symptomatic of the need -and difficulty, to define a coherent set of nonambiguous and easy to understand meta-models capable of covering the problem space of plastic UI's. The UsiXML2 consortium is putting significant effort in this direction, but has not reached its objectives yet. In my opinion, two metamodels (at least) deserve particular attention: transformations and Concrete UI's.

As stated a b o v e , t r a n s f o r m a t i o n s offer an elegant mechanism for f u l l f l exibility a n d technical integration. However, transformations are hard to express (QVT and ATL are not languages for naïve developers). In addition, usability rules are even harder to convey formally [START_REF] Sottet | A Model-Driven Engineering Approach for the Usability of User Interfaces[END_REF]. More importantly, inverse t r a n s f o r m a t i o n s cannot be automatically derived for any source transformations. This is a fundamental flaw that may result in inconsistent models as transformations are performed up and down iteratively during the life cycle of a system, breaking d o w n t h e flexibility of the solution space envisioned by t h e CAMELEON reference framework.

At the C U I l e v e l , meta-modeling, not only lags b e h i n d innovation, but bridles creativity. UIDL's f o r t h e expression of concrete user interfaces are technology-driven instead of leaving rooms

for n e w f o r m s o f i n t e r a c t i o n techniques. Although the CARE properties [START_REF] Coutaz | Four easy pieces for assessing the usability of multimodal interaction: the CARE properties[END_REF] have been devised 15 years ago, CUI languages have hardly scratched the surface of multimodal interaction. We are still unable to generate the paradigmatic "put-that-there" multimodal user interface introduced more than 25 years ago [START_REF] Bolt | Put That There": Voice and gesture at the graphics interface[END_REF]. We do however generate simplistic m u l t i m o d a l U I 's based on XHTML+VoiceXML but with very l i m i t e d m i c r odialogues for interaction repair [START_REF] Berti | Migratory multimodal interfaces in multidevice environments[END_REF]. Actually, CUI-level UIDL's are still struggling with the description of conventional GUI's for desktop computing. Meanwhile:

• New forms of "constructable" computers such as the MIT shiftables 3 and the CMU toy blocks 4 are put on the market;

• Novel interaction techniques are proliferating whether it be for supporting mobility (e.g., SixSense [START_REF] Mistry | SixthSense -A Wearable Gestural Interface[END_REF]), for 3D interaction (where gesture and 3D screens are becoming predominant), or even for graphical tabletops and multisurface interaction [START_REF] Balakrishnan | Special Issue on Ubiquitous Multi-Display Environments[END_REF];

• New requirements are emerging: design is s w i t c h i n g from the development of useful and usable systems for people with precise goals, to engaging a n d inspired interaction spaces whose users can easily switch from a consumer mind to the creator mode.

In short, CUI meta-models need to capture the unbound vibrant convergence of physicality with "digitality". Perhaps, meta-modeling is, by essence, the wrong approach to CUI's: a model, which represents a thing, is necessarily a simplification, therefore a reduction, of the real thing. In these conditions, the subtle aspects of interaction, which make all the differences between constrained and inspired design, are better expressed using code directly in place of an abstraction of this code. However this assertion should be mitigated by the following findings: designers excel at sketching pictures to specify concrete rendering. On the other hand, they find it difficult to express the dynamics, forcing them to use natural language [START_REF] Myers | How designers Design and Program Interactive Behaviors[END_REF]. One way to fill the gap between designers'practice and productive models is to revive work à-la-Peridot [START_REF] Myers | Creating User Interfaces using programming by example, visual programming, and constraints[END_REF] such as SketchiXML [START_REF] Coyette | SketchiXML: towards a multi-agent design tool for sketching user interfaces based on USIXML[END_REF] where drawings are retro-engineered into machinecomputable rendering. As for inferring b e h a v i o r f r o m examples, the promising "Watch What I Do" paradigm initiated in the late 1970's (cf. Dave Smith's PYGMALION system [START_REF] Smith | Pygmalion: An executable Electronic Blackboard. Chapter1 In "Watch What I Do[END_REF]) is still an opened question.

Model Driven Engineering, as a software development methodology, has favored the dichotomy between the design stages and the run time phase, resulting in three major drawbacks:

• Over time, models may get out of sync with the running code.

• Design tools are intended for software professionals, not for "the people". As a result, end-users are doomed to consume what software designers have decided to be good for their hypothetic target users.

• Run time adaptation is limited to the changes of context identified as key by the developers. Again, the envelope for end-users' activities is constrained by design.

Applied to UI development, the dichotomy between design and run time phases means that UI generation from a task model cannot cope with ambient computing where task arrangement may be highly opportunistic and unpredictable. On the other hand, because the task model is not available at run time, the links between the FUI and its original task model are lost. It is then difficult, not to say impossible, t o a r t i c u l a t e r u n -time adaptation based on semantically rich design-time descriptions. As a result, a FUI cannot be remolded beyond its cosmetic surface as supported by the CSS.

Blurring the distinction between the design stage and the run time phase is a promising approach. This idea i s emerging in main stream middleware [START_REF] Ferry | Models at Runtime: Service for Device Composition and Adaptation[END_REF] as well as in HCI. The middleware community, however, d o e s n o t necessarily address e n d -user concerns. Typically, a "sloppy" dynamic reconfiguration at the middleware level is good enough if it preserves system autonomy. It is not "observable" to the end-user whereas UI re-molding and UI redistribution are! Thus, UI plasticity puts additional constraints. In particular, it becomes necessary to make explicit the transition between the source and the target UI's so that, in Norman's terms, end-users can evaluate the new state. We need to pay attention to transition UI's in generic terms, not on a case per case basis.

In the following section, I illustrate the combination of models and code at run time with the work we have done for addressing the problem of plastic UI's at run time.

MODELS AT RUN TIME

Early experience in the development of plastic UI's can be summarized as "one does not fit all". The following three principles show why.

Principle #1: Close-adaptiveness must cooperate with open-adaptiveness [START_REF] Oreizy | An Architecture-Based Approach to Self-Adaptive Software[END_REF]. As discussed above, by design, an interactive system has an "innate domain of plasticity": it is close-adaptive for the set of contexts of use for which this system/component can adapt on its own. For unplanned contexts of use, the system is forced to go beyond its domain of plasticity. It must be open-adaptive so that a tier infrastructure (a middleware) can take over the adaptation process.

Figure 2. A typical functional decomposition for Ambient interactive spaces [2].

As shown in Figure 2, the functional decomposition of the middleware that supports open adaptation includes:

• A context infrastructure t h a t b u i l d s a n d m a i n t a i n s a model of the context of use [START_REF] Reignier | Contexte-Aware Environments: from Specification to Implementation[END_REF].

• A situation synthesizer that computes the situation and possibly informs an evolution engine of the occurrence of a new situation.

• An evolution engine that elaborates a reaction in response to the new situation.

• An Adaptation producer that implements the adaptation plan produced by the evolution engine.

Such a decomposition is commonly used for the development of autonomic systems. To adapt this decomposition for plastic UI's, we propose the following improvements:

• The end-user is kept in the loop: the reaction to a new situation may be a mix of specifications provided by developers or learnt by the evolution engine. In addition, it may call upon end-users' advice by the way of a meta-UI [START_REF] Coutaz | Meta User Interfaces for Ambient Spaces[END_REF]. I see this meta-UI as an end-user development environment.

• The components referred to in the action plan do not necessarily exist as executable code. This is where Principle #2 comes into play. Principle #2: At run time, an interactive system is a set of graphs of models that express different aspects of the system at multiple levels of abstraction. As advocated by the CAMELEON framework, these models are related by mappings and transformations. As a result, an interactive system is not limited to a set of linked pieces of code. Models developed at design-time, which convey high-level design decision, are still available at runtime for performing rational adaptation beyond cosmetic changes. When a component retrieved by the component manager is a highlevel description such as a task model, the configurator relies on reificators to produce executable code as in Digymes [START_REF] Coninx | Dygimes: Dynamically Generating Interfaces for Mobile Computing Devices and Embedded Systems[END_REF] and iCrafter [START_REF] Ponnekanti | Icrafter: a Service Framework for Ubiquitous Computing Environments[END_REF]. A retrieved component may be executable, but may not fit the requirements. Ideally, it can be reversed-engineered through abstractors, then transformed by translators and reified again into executable code [START_REF] Bouillon | Recovering Alternatives Presentation Models of a Web Page with Vaquita[END_REF].

Principle #3: By analogy with the slinky meta-model of Arch, the software developer can play with principles #1 and #2. At one extreme, the interactive system may exist as one single task model linked to one single AUI graph, linked to a single CUI graph, etc. (see Figure 3). This application of Principle #1 does not indeed leave much flexibility to cope with unpredictable situations unless it relies completely on the tier middleware infrastructure that can modify any of these models on the fly, then triggers the appropriate transformations to update the Final UI.

Alternatively, the various perspectives of the system (task models, AUI, FUI, context model, etc.) as well as the adaptation mechanisms o f t h e t i e r i n f r a s t r u c t u r e are distributed across distinct UI service-oriented components, each one covering a small task grain that can be run in different contexts of use. We have adopted this approach to implement the Comet toolkit [START_REF] Demeure | COMET(s), a Software Architecture Style and an Interactors Toolkit for Plastic user Interfaces[END_REF]. Basically, a Comet is a plastic micro-interactive system whose architecture pushes forward the separation of concerns advocated by PAC and MVC. The functional coverage of a comet is left open (from a plastic widget such as a control panel, to a complete system such as a powerpoint-like slide viewer). Each Comet embeds its own task model, its own adaptation algorithm, as well as multiple CUI's and FUI's, each one adapted to a particular context of use. FUI's a r e h a n d -coded possibly using different toolkits to satisfy our requirements for finegrained personalization and heterogeneity. From the infrastructure point of view, a Comet is a service that can be discovered, deployed and integrated dynamically into the configuration that constitutes an interactive environment.

The COTS [START_REF] Bourguin | Defining Task Oriented Component[END_REF], whose executable UI code is meta-described with the task they support, are based on similar ideas.

Figures 4 and5 show another application of principles #1 and #2 for the implementation of Photo-browser. The FUI of Photo-browser is dynamically composed of:

• a Tcl-Tk component running on a multi-point interactive surface (Fig. 4-d),

• a Java component that shows a list of the image names (Fig. 4-b),

• and an HTML-based browser to navigate through the images set (Fig. 4-c). Photo-browser is implemented on top of a tier middleware infrastructure (called Ethylene) that covers the evolution engine, the component manager as well as the adaptation producer of Figure 2 [START_REF] Balme | Interfaces homme-machine plastiques : une approche par composants dynamiques[END_REF]. Ethylene is a distributed system composed of Ethylene factories each one running on possibly different processors (IP devices). The role of an Ethylene factory is to manage the life cycle of a set of components that reside on the same IP device as this factory and that have been registered to this factory. When residing on storage space, a component is meta-described using EthylenXML, an extension of WSDL. This metadescription includes the human task that the component supports, the resources it requires, and whether it is executable code or transformable code. In the latter case, it may be a task model, an AUI, a CUI, or even a graph of these models. For example, the HTML-based component (Fig 4 .c) is a CUI expressed in a variation of HTML. It must be transformed on the fly to be interpreted by an HTML renderer. The Tcl-Tk multi-point UI and the Java list are executable code. T h e i r E t h y l e n e X M L m e t adescription specifies that they support image browsing and image selection tasks, that they need such and such interaction resources (e.g., a Tcl-Tk interpreter and a Diamond Touch) for proper execution, and that they require such and such communication protocol to be interconnected with other components. The GPhone UI component of Figure 5 is an executable Gphone app that supports the next-previous browsing tasks. Interconnection between components is initiated by the factories.

As these examples show, the engineering community of HCI has focused its attention on run time adaptation of the UI portion of an interactive system, not on the dynamic adaptation of the interactive system as a whole. The software engineering community is developing several approaches to enable dynamic bindings for service-oriented architectures. For example, Canfora et al. propose the dynamic composition of web services based on BPEL4People (that expresses a task-like model) as well as an extension of WSDL to meta-describe the services and using these two descriptions to generate the corresponding user interface [START_REF] Canfora | Dynamic Composition of Web Applications in Human centered Processes[END_REF]. Although bindings can be performed at run time, users are confined within the workflow designed by the software developers. In addition, the generated UI's are limited to conventional GUI.

One promising approach to support flexibility at run time, is to consider the functional core components as well as UI components as services. In Ethylene, UI components adhere to this philosophy. They can be implemented in very different technologies, they can be discovered and recruited on the fly based on their meta-description, they can be transformed on the fly. On the other hand, the business logic side of interactive systems is left opened. CRUISe [START_REF] Pietschmann | Dynamic Composition of Service-Oriented Web User Interfaces[END_REF] aims at supporting both sides in a uniform way, but applies to the dynamic composition of web services and UI composition for the web [START_REF] Yu | A Framework for rapid Integration of Presentation Components[END_REF].

CONCLUSION

Model-Driven Engineering has provided the HCI community with useful concepts for f r a m i n g i t s o w n research agenda. Additional research is required for the definition of meta-models, transformations and mappings provided that h i g h -level descriptions can take full advantage of the latest innovations at the FUI level. Models at design time should not disappear at run time, but should be available to go beyond cosmetic adaptation. D e s i g n phase and run-time phase equal "même combat!"

Maximum flexibility and quality should be attainable by modeling the business logic as well as the user interface as services w i t h t h e i r o w n d o m a i n o f p l a s t i c i t y . UI components should not be pure executable code. They have to be meta-described to express their exact nature and contracts with a human-centered perspective. They can be retrieved, transformed, and recomposed on the fly thanks to a tier middleware infrastructure. This middleware, which supports context, dynamic discovery as well as the dynamic (re)composition of business logic and of transformable UI components, will permit interactive systems to go beyond their domain of plasticity. However, we must be careful at keeping the user in the loop while being able to produce transition user interfaces automatically.

The risk is that this wonderful apparatus will be designed for the specialists. We need to put the power in the people's hands and explore the potential from social programming. The success of the Apple App Store is a good indication for this. Mash-up t o o l s have also started t h i s t r e n d f o r composing web-based applications (e.g., Google Gadgets or Yahoo! Widgets). More collaboration should be developed with the "cloud computing crowd". After all, an interactive space is a mini-cloud. If interaction resources were virtualized as memory, network and computing resources are currently envisioned by the "systemers", then this would simplify enormously the development of user interfaces. IAM [START_REF] Hayes | Design alternatives for user interface management systems based on experience with COUSIN[END_REF] was an early attempt in this direction.

In short, MDE is an important tool for adaptation as long as it does not block creativity.

Figure 3 .

 3 Figure 3. An interactive system as a graph of models available at run time. These models are related by mappings and transformations.

Figure 4 .

 4 Figure 4. The Photo-browser application [1]: a dynamic composition of executable and transformable components, managed by a dynamic set of interconnected factories running on different platforms (Windows, MacOS X, and Android).

Figure 5 .

 5 Figure 5. (Left) Connecting a Gphone to the interactive space by laying it down on the interactive table. (Right) Using the Gphone as a remote-controller to browse photos displayed by the HTML UI component of fig. 3c and video-projected on the wall. (In the current implementation, the contact of the Gphone with the Diamond Touch is simulated as a new situation event for interpretation by Ethylene).

http://www.usixml.org

http://sifteo.com/

http://www.modrobotics.com/

ACKNOWLEDGMENTS

I thank the ANR CONTINUUM project (ANR-08-VERS-005) as well as the ITEA 0 8 0 2 6 U s i X M L p r o j e c t f o r supporting this work. Special thanks to Alexandre Demeure for the implementation of the Comets, and to Lionel Balme for the implementation of Ethylene.


Joëlle Coutaz
email: joelle.coutaz@imag.fr

Emeric Fontaine
email: emeric.fontaine@imag.fr

Nadine Mandran
email: nadine.mandran@imag.fr

Alexandre Demeure
email: alexandre.demeure@inrialpes.fr

DisQo : A user needs analysis method for smart home

Keywords: End-User composition, smart artifacts coupling, smart home, ubiquitous computing, service-oriented computing. ACM Classification H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous

How can people identify the services that they might expect from their smart home when they have little to no knowledge about novel technologies? This paper reports on a user needs analysis method designed to answer this question: DisQo. We have recruited 17 families and used a combination of interviews and playful cultural probes. Results show that families are willing to couple smart objects to improve their lives.

INTRODUCTION

Ubiquitous

computing promises unprecedented empowerment from the flexible and robust combination of software services with the physical world. Software researchers assimilate this promise as system autonomy where users are "conveniently" kept out of the loop. Their hypothesis is that services, such as music playback and calendars, are developed by service providers and preassembled by software designers to form new service frontends. Their scientific challenge is then to develop secure, multiscale, multi-layered, virtualized infrastructures that guarantee service front-end continuity. Although service continuity is desirable in many circumstances, end users, with this interpretation of ubiquitous computing, are doomed to behave as mere consumers, just like with conventional desktop computing.

Our hypothesis is that end users are willing to shape their own interactive spaces by coupling smart artifacts, building imaginative new functionalities that were not anticipated by system designers. A number of tools and techniques have been developed to support this view including the Jigsaw editor [START_REF] Humble | Playing with the bits" user-configuration of ubiquitous domestic environments[END_REF], CAMP [START_REF] Truong | CAMP: A Magnetic Poetry Interface for End-User Programming of Capture Applications for the Home[END_REF], iCAP [START_REF] Dey | iCAP: Interactive prototyping of context-aware applications[END_REF], or Newman's work on end user composition with OSCAR [START_REF] Newman | Providing an integrated user experience of networked media, devices, and services through end-user composition[END_REF]. The major focus of this prior work is on exploring novel interaction techniques and on technical frameworks. In this paper, we are concerned with the fundamental meaning (and human needs) of building confederation of interoperating smart artifacts.

In ubiquitous computing, we are unable to predict which artifacts of everyday life end users would be willing to couple (and decouple) to obtain new services. In addition, would such coupling be commutative, associative, distributive over some other operation? In other word, is it possible to define an algebra over smart artifacts so that we can generalize the problem and reason at a high level of abstraction in a rigorous manner? Human sciences offer tools and methods for exploring human needs and behavior. However, the novelty of our problem requires to solicit participants imagination while at the same time controlling the experimentation and respecting the privacy of their intimate home. In this paper, we analyze representative methods that have been developed for analyzing domestic activities. Based on these previous experiences, we propose our own improvement followed by the detailed presentation of the experimental material. Our findings are discussed in the last section.

EXISTING NEED'S ANALYSIS METHOD FOR HOME

The domestic's environment is an uncontrolled field study. Researchers have to respect privacy without imposing temporal and cognitive constraints to the inhabitants. To achieve this goal, the use of only one technique is inefficient. Indeed, interview [START_REF] Davidoff | Principles of Smart Home Control[END_REF][START_REF] Dey | iCAP: Interactive prototyping of context-aware applications[END_REF], journal studies [START_REF] Rode | The domestic economy: A broader unit of analysis for end user programming[END_REF] and playful cultural probe [START_REF] Bernhaupt | Playful Probing: Making Probing more Fun[END_REF], do not respond to all our criteria. Moreover, they induce drawbacks such as the limitation of imagination, the observer's bias or the social conformity. To avoid these limitations, we have to combine techniques.

Rode et al [START_REF] Rode | The fuzzy felt ethnography-understanding the programming patterns of domestic appliances[END_REF] use a three step method. The first step consists in dining with the inhabitants. The goal is to turn the observers into guests in order to promote convivial mood and free talk. The second step consists in visiting the home with the inhabitants. The observers have to ask inhabitants about their use of the technological equipments and electrical appliances. Last, inhabitants are invited to place pieces (representing rooms, peoples, artifacts and activities) on top of a board. This method fit all our criteria but the privacy. Indeed, observers somehow impose themselves to the inhabitants' intimacy.

Davidoff et al. [START_REF] Davidoff | Principles of Smart Home Control[END_REF] also proposes that observers visit the appliances of the home with inhabitants. However, observers have to focus their questions on key moments of the day such as: "morning" and "come back home". At the end of the visit, a cultural probe [START_REF] Gaver | Design: Cultural Probes[END_REF] is left to the family for collecting complementary data. With this probe, the privacy is respected, but the times constraint imposed to the family is extended to one or two weeks.

THE EXPERIMENTAL STUDY

Our method, DisQo, is preceded by a study of industrial solutions proposed on the market. In particular, we want to know why people accept the proposed technologies or not. Due to lack of space, we do not detail this part of the study here. Generally speaking, this study helps us to measure the gap between industrial point of view and the users' ones and to guide our questions to the participants.

Participants

Drawing on Davidoff's et al. experiment and conclusions (i.e. "families want more control of their lives" [START_REF] Davidoff | Principles of Smart Home Control[END_REF]), we focused on "busy" families. The participants have been solicited through bulletin board advertisements, email, as well as from personal relationships.

We have recruited 17 families representing a total of 40 persons (35 adults and 5 children), all living in the area of Grenoble (France). Of the 17 families, 12 were dual income families, 1 was single parent, 2 were house mates, and 2 were retired couples. All families were well educated with medium to high standard of living.

Method

DisQo was designed to reconcile the following requirements: to collect meaningful data in a minimum of time while respecting privacy. As presented above, we are interested in determining how far people are ready to envision the interconnection of everyday devices to improve their lives, and to which extent coupling objects is commutative, associative and distributive. For so doing, we have used a combination of interview [START_REF] Silverman | Introducing qualitative research[END_REF] (good for clarification), playful cultural probe (appropriate for respecting privacy and for improving subjects involvement [START_REF] Bernhaupt | Playful Probing: Making Probing more Fun[END_REF]). The presence of the experimental team (ourselves, from 1 to 3 persons) was limited to 1h30 per family home. Fieldwork was structured as a four-step process: photographing, interview, game, and debriefing.

Step 1: Photographing. Using digital cameras provided by the experimental team, two volunteer family members were asked to take pictures of 10 objects at the rate of 2 objects per room. For each of the 5 rooms of their choice, they were asked to take a picture of one object that they considered to be necessary in their everyday life or that would help them in organizing their lives, as well as a picture of one object that they considered to be superfluous but valuable (typically, a painting). The volunteers (in general, the parents) were not supposed to be in the same room at the same time so that they would not know which pictures the other member had taken. Meanwhile, the experimental team would wait sitting at a place indicated by the parents (typically, the living room where they usually meet with friends and visitors).

Step 2: Interview. We then conducted an interview with all the family members, using the pictures as input material.

Questions were directed at understanding the reasons for their choices, the value attached to the objects or the services provided in daily use. Special attention was given to the (many) remote controller(s) typically found in the household environment. We progressively oriented our questions towards novel uses of smart artifacts. In particular, we asked which objects of the house (including those on the pictures) they would qualify as "programmable" (e.g., TV's, washing machines, alarm clocks), "communicating" (e.g., computers, mobile phones), or emotional (i.e. carrying intimate value). This was used as a means to elicit routines and exceptional needs as well as to prepare the game developed in Step 3.

Step 3: Association game. The association game drew on people creativity using the pictures as play cards. Pictures were sorted randomly and presented two at a time (then, three at a time) on a tablet PC. Family members were asked to imagine which service(s) and value(s) these two (or three) objects coupled together would provide them with (figure 1). Random coupling was designed to solicit imagination in unexpected ways as solutions creativity grows with the semantic distance between elements [START_REF] Mednick | The associative basis of the creative process[END_REF]. Another raison for this random coupling is to get hints about the existence of a "natural" algebra over smart artifacts.

Figure 1: Associations' game user interface: which services useful or not, could result from a communication/cooperation between your TV and your washing machine?

Step 4. Debriefing and informal discussion. The last stage was dedicated to debriefing, including opened friendly discussions.

Overall, we have collected comments and objective data for 349 couplings for a total duration of 25 hours of our presence in the 17 family homes.

DATA ANALYSIS

Interviews and debriefings helped us to identify recurring facts between home families such as key moments during weekdays for which families would expect support from a smart home, or attitudes with regard to "programming the home". Data from the association game as well as from the interviews were used to find answers to our theoretical questions. More specifically, we classified the objects that have been photographed into four categories: objects that have been denoted as "programmable" by the subjects, objects that have been declared as "communicating", objects that support both capabilities, and objects that have none of these two properties. Using the Chi-square test, we have been able to find strong significance between the abilities of the subjects to envision (or not) services depending on the capabilities of the assembled objects. In particular, the communication capability allows people to more easily imagine new services from the assembled object.

FINDINGS

Our experiment has led to three types of results: recurring facts across families, early answers to our theoretical questions, as well as insights about our method.

Recurring Facts

We found a number of facts that are quite consistent with the results reported in prior literature:

1. "Wake-up" time, "on-the-way-to-home" and "arrivinghome" times are key moments to people. To save time and improve efficiency, activities are organized into wellpolished procedures. As a result, exceptions to these routine tasks are sources of stress. Support for avoiding or for solving exceptions is one class of services expected from a smart home. This includes the management of possessions (laundry to be launched because of a business trip planned in a couple of days, food on the point to be missing, medicine close to expiration date), decision-making (what to buy, what to wear today), reminders (doctor appointment), security (door properly locked), resources consumption and resource sharing among family members (typically, hot water and bathroom occupation in the morning), etc. 2. With regard to programming, attitudes range from "I do not want to be assisted" to "It will work 99% of the time, but it will be hell for the other 1%". Motivation for programming is systematically grounded on a clear straight forward observable benefit. Next section presents results obtained about the coupling of everyday life objects. We believe that these results are original with respect to the prior literature.

About Coupling

Our data from the association game shows two important results: (1) Family members are prone to envision new services when coupling involves one "communicating" object, or one "programmable" object, at least. (2) The "communicating" capability has more impact than "programmability" on the capacity of family members to imagine new services. However, 78 of the 349 couplings resulted in service finding although none of the objects were programmable or communicating. For example, the couple "bed-shower", whose objects had not been classified as programmable nor communicating, suggested that "getting up from the bed in the morning would turn the shower on in order to provide water at the right temperature when coming back from the toilet". This means that there is a large body of potentiality for novel services based on mundane everyday objects.

The services suggested by our family members fall into four categories. We illustrate them with the most typical examples drawn from our fieldwork.

Service substitution. People have observed that, for the same (sport) events, commentaries on radio broadcasts are richer than those provided by TV. As a result, they would like to replace the TV sound service with that of the radio to improve the overall quality of the informational experience. Another example is User Interface substitution: some people are quite skill at setting up alarms on their mobile phone, but they do not know how to do this for their physical home alarm clock. As a consequence, they would find it quite convenient to replace the user interface of their alarm clock with that of their phone thanks to a convenient opportunistic coupling of the phone with the alarm clock.

Service improvement. Some household appliances such as washing machines and cupboards, do not provide any convenient way to control and monitor their current internal state. Appliances than are not sufficiently equipped could be improved by coupling them with additional input and output facilities such as those of the TV set.

Service chaining. Service chaining is intended to improve comfort, wellbeing as well as resources for the routine, but hectic, activities. For example, "picking up the towel after the shower would trigger the coffee machine so that coffee would be ready just in time, at the right temperature, along with the radio turned on in the kitchen broadcasting the news using the appropriate sound level". Service "starter". We have observed that some appliances serve as triggers for services that are expected to be precomposed to support routine activities. The towel and the bed mentioned above, play this role, implicitly. Not surprisingly, people also want to have an explicit and reliable control over the home (cf. the worry that 1% of the time, the house would turn into hell). Some people came up with the "morning starter push button" conveniently located close to the bed that would gently "wake up" the house when pushed. We found some results about algebra's properties of couplings. The need for chains of services underpins some form of associativity. For example, one family qualified the "towel-coffee machine" coupling as a "morning package". During the discussion, our family members thought of adding the "morning starter push button" to get a controllable chain.

Coupling for service improvement entails some form of distributivity. Typically, the TV has often been mentioned as a way to observe and control the state of a number of appliances such as the washing machine or the oven.

Commutativity between two coupled artifacts has generally been satisfied with notable exceptions when there is a causality relationship between the artifacts.

About the Experimental Method

The "Snapshots taking" of our fieldwork has multiple advantages: (1) It serves as an ice breaking between the family members and the experimental team; (2) Family members "reveal their house" naturally while we, the experimenters, do not intrude their private spots. (3) Family members get truly involved (and intrigued by what will come next). ( 4) As opposed to playful probing proposed by R. Berhaupt [START_REF] Bernhaupt | Playful Probing: Making Probing more Fun[END_REF], our game uses images of intimate objects, not of generic entities. This increases the interest and imagination of the participants while improving the meaningfulness of the data collected. In future work we will explore the benefits of providing them with a journal for a duration of two weeks in order to test whether their creativity can be increased after the interview [START_REF] Drazin | Multilevel theorizing about creativity in organizations: a sense making perspective[END_REF].

CONCLUSION

We have presented DisQo, a method for investigating the fundamental meaning of building confederation of interoperating artifacts for future smart homes. DisQo combines several techniques to reach a satisfying balance between experimental control, privacy issues, and ecological validity. The key element of the method is for observers to able to "visit" people's homes through the pictures of intimate objects taken by the participants themselves and to use these pictures as playful cultural probes to envision future use.

DisQo has provided us with preliminary answers to our problem of finding an algebra for coupled artifacts. Although additional investigations are necessary, early results from our fieldwork support our hypothesis.

ACKNOWLEDGMENTS

This work has been supported by the ANR CONTINUUM project reference ANR-08-VERS-005.


Frédéric Jourde
email: frederic.jourde@imag.fr

Yann Laurillau
email: yann.laurillau@imag.fr

Laurence Nigay
email: laurence.nigay@imag.fr

COMM Notation for Specifying Collaborative and MultiModal Interactive Systems

Keywords: Groupware, multimodal interaction, specification notation. ACM Classification H.5.3. Group and Organization Interfaces: Computersupported supported cooperative work. H.5.2. User Interfaces: Input devices and strategies. D.2.2. Design Tools and Techniques: User interfaces Human Factors, Design, Algorithms

Multi-user multimodal interactive systems involve multiple users that can use multiple interaction modalities. Although multi-user multimodal systems are becoming more prevalent (especially multimodal systems involving multitouch surfaces), their design is still ad-hoc without properly keeping track of the design process. Addressing this issue of lack of design tools for multi-user multimodal systems, we present the COMM (Collaborative and MultiModal) notation and its on-line editor for specifying multi-user multimodal interactive systems. Extending the CTT notation, the salient features of the COMM notation include the concepts of interactive role and modal task as well as a refinement of the temporal operators applied to tasks using the Allen relationships. A multimodal military command post for the control of unmanned aerial vehicles (UAV) by two operators is used to illustrate the discussion.

INTRODUCTION

We present the COMM (Collaborative and MultiModal) notation dedicated to the specification of multi-user multimodal interactive systems. An interactive system is multimodal if users can interact with the system using at least two (input or output) interaction modalities either operating in parallel or not. Several multi-user interactive systems (e.g. groupware) are multimodal such as the interactive systems described in [START_REF] Brown | Lessons from the lighthouse: collaboration in a shared mixed reality system[END_REF][START_REF] Cohen | QuickSet: multimodal interaction for distributed applications[END_REF][START_REF] Etter | A Tangible User Interface for Multi-User Awareness Systems[END_REF]12,[START_REF] Sallnäs | Supporting Presence in Collaborative Environments by Haptic Force Feedback[END_REF][START_REF] Voida | Challenges in the Analysis of Multimodal Messaging[END_REF]. Furthermore, there is a growing interest in the CSCW community for multi-user multimodal interactive systems based on interactive multitouch surfaces [START_REF] Pauchet | Mutual Awareness in Collocated and Distant Collaborative Tasks Using Shared Interfaces[END_REF][START_REF] Piper | SIDES: a Cooperative Tabletop Computer Game for Social Skills Development[END_REF][START_REF] Tang | Collaborative Coupling over TableTop Displays[END_REF][START_REF] Tse | Interaction with Single User Applications through Speech and Gestures on a Multi-User Tabletop[END_REF][START_REF] Tse | Exploring True Multi-User Multimodal Interaction over a Digital Table[END_REF]. Although multi-user multimodal systems are becoming more prevalent, their design is still ad-hoc without properly keeping track of the design process. The design of such interactive systems gives rise to new challenges due to the multiplicity of users with roles as well as the availability of different modalities and combinations of them. Addressing this issue of a lack of design tools for multi-user multimodal systems, we present the COMM notation and its editor for specifying interactive systems that are both multi-user and multimodal.

The specification step is one of the fundamental steps in any software development process regarding the waterfall [START_REF] Royce | Managing the development of large software systems[END_REF] or the V lifecycle models [START_REF] Mcdermid | Life Cycle Support in the ADA environment[END_REF]. It is essential to have specification tools that help the designer to precisely analyze and describe the user interface and to specify the interaction process [START_REF] Jacob | A Specification Language for Direct Manipulation User Interface[END_REF]. A specification should also help the designer to detect usability problems at an early stage before implementing the interactive systems.

For specifying multi-user multimodal systems, we adopt a task-based notation. As pointed out in [START_REF] Van Der Veer | Introduction to Groupware Task Analysis[END_REF], task analysis is used in various steps of the development process and we focus here on the specification of a system to be then developed. In a previous study [START_REF] Jourde | Towards Specifying Multimodal Collaborative User Interfaces: A Comparison of Collaboration Notations[END_REF], we have reviewed several task-based notations, such as CTT [START_REF] Mori | CTTE : Support for Developing and Analyzing Task Models for Interactive System Design[END_REF], GTA [START_REF] Van Der Veer | Task Based Groupware Design: Putting Theory into Practice[END_REF][START_REF] Van Der Veer | Introduction to Groupware Task Analysis[END_REF], MABTA [START_REF] Lim | Task models for groupware and multitasking: Multiple aspect based task analysis (MABTA) for user requirements gathering in highly-contextualized interactive system design[END_REF], CUA [21] or UML-G [24] and we have highlighted the lack of support to describe the multimodal aspects of a multi-user interface. Addressing this issue, we present a notation, named COMM, that enables the designer to specify collaborative and multimodal aspects of an interactive system. Rather than creating a brand new notation from scratch, using a completely new graphical language to be learned, the COMM notation reuses and coherently articulates concepts of existing notations, such as CTT [START_REF] Mori | CTTE : Support for Developing and Analyzing Task Models for Interactive System Design[END_REF]. To describe the multimodal aspects of the interaction, we base the notation on the OpenInterface conceptual model [START_REF] Serrano | A Three-dimensional Characterization Space of Software Components for Rapidly Developing Multimodal Interfaces[END_REF] for multimodality (definition of a modality [START_REF] Nigay | A Generic Platform for Addressing the Multimodal Challenge[END_REF] and temporal constraints [START_REF] Vernier | A Framework for the Combination and Characterization of Output Modalities[END_REF]). collaborative activities; [START_REF] Allen | Maintaining Knowledge about Temporal Intervals[END_REF] to explicitly describe the link between the abstract User Interface (UI) specified as group tasks and the concrete UI involving multiple interaction modalities. In order to satisfy the first requirement, we introduce the concept of interactive role, in contrast to the concept of role usually used in the CSCW literature, that we will denote in this paper as business role 1 for clarity. To satisfy the second requirement: first, we introduce the concept of modal task, which denotes an atomic user action performed on a physical device; second, we refine CTT temporal operators using the temporal relationships between modalities defined in [START_REF] Vernier | A Framework for the Combination and Characterization of Output Modalities[END_REF] and based on Allen relationships [START_REF] Allen | Maintaining Knowledge about Temporal Intervals[END_REF]. The paper is organized as follows: we first present a state of the art of the existing notations and we focus on their ability to specify concrete user interaction. We then describe the main elements of the COMM notation before presenting the e-COMM editor, an on-line application for specifying interactive systems using the COMM notation. We illustrate our notation by considering the design of a multi-user multimodal military command post. Indeed the COMM notation is currently used in the context of a French Army Research Department project, in partnership with the firms BERTIN, SAGEM, EADS and PY Automation, to design military command posts for the control of two unmanned aerial vehicles (UAV) by two operators, as shown in Figure 1.

RELATED WORK

"Task analysis is the process of understanding the user's task thoroughly enough to help design a computer system that will effectively support users in doing the task." [START_REF] Kieras | Task Analysis and the Design of Functionality[END_REF]. Beyond analyzing current or envisioning user's tasks, taskbased notations are also commonly used to design and specify the user interface (UI) [START_REF] Van Der Veer | Introduction to Groupware Task Analysis[END_REF]. Finally some taskbased specifications are also used as inputs for UI generation. In our study, we focus on task-based specification of the user interface and we do not cover the 1 Business role: Representing role in the organization UI generation aspects. Therefore we do not consider UMLbased notations, such as UML-G [24], which are best suited for software implementation and UI generation or transformation.

Several task-based notations have inspired our work. We reviewed the ones dedicated to the specification of multiuser interactive systems. In the field of group work task analysis, there exist several notations dedicated to the description of group activities such as GTA [START_REF] Van Der Veer | Task Based Groupware Design: Putting Theory into Practice[END_REF], MAD [START_REF] Scapin | Towards a method for task description[END_REF], MABTA [START_REF] Lim | Task models for groupware and multitasking: Multiple aspect based task analysis (MABTA) for user requirements gathering in highly-contextualized interactive system design[END_REF] or CTT [START_REF] Mori | CTTE : Support for Developing and Analyzing Task Models for Interactive System Design[END_REF]. Some of these notations, mostly CTT and CTT-based notations such as CIAN [START_REF] Molina | A conceptual and methodological framework for modelling interactive groupware applications[END_REF], TaskMODL [START_REF] Treatteberg | Model-based User Interface Design[END_REF] or UserView [START_REF] Penichet | Collaborative Social Structures and Task Modelling Integration[END_REF], are used to design and specify the user interface. We analyse these notations in light of our two requirements exposed in the introduction section: capacity to support the specification of (1) cooperative and collaborative activities (2) multimodal interaction.

First the notation should support the specification of both kinds of group work task: collaborative and cooperative. Dillenbourg et al. [START_REF] Dillenbourg | The evolution of research on collaborative learning[END_REF] define the two types of tasks as follows:

"Collaboration" is distinguished from "cooperation" in that cooperative work "... is accomplished by the division of labor among participants, as an activity where each person is responsible for a portion of the problem solving...", whereas collaboration involves the "... mutual engagement of participants in a coordinated effort to solve the problem together."

For multi-user multimodal interaction, this requirement is important because cooperative versus collaborative tasks lead to design of different concrete multimodal UIs. Indeed, cooperative tasks imply a predefined task allocation to business roles (division of labor). At the concrete UI level, it implies that, for each business role, the designer assigns a predefined set of modalities and therefore decides at design time which interaction modalities should be used. In contrast, for collaborative activities, task allocation is performed at runtime. Consequently, when the designer specifies the concrete UI, s/he cannot assign modalities to each business role. Thus, GTA, MAD, MABTA, CTT support only cooperative tasks. Furthermore, considering CTT, a group work task is described with a cooperative task tree and a set of individual task trees for each involved business role. Only CIAN and CUA explicitly identify the two types of tasks, cooperative and collaborative, in addition to the existing types of tasks: user, system, interactive. Nevertheless CIAN and CUA focus on abstract interaction only.

The second requirement is to support the description of concrete multimodal interaction. Existing notations mainly focus on abstract UI: As stated in [START_REF] Molina | A Review of Notations for Conceptual Modeling of Groupware Systems[END_REF], the notations in particular any CTT-based notations such as CIAN [START_REF] Molina | A conceptual and methodological framework for modelling interactive groupware applications[END_REF], enable the specification of abstract UI and may serve as input for designing a concrete/final UI. Nevertheless we note that GTA is based on (N)UAN [START_REF] Hix | Developing User Interfaces: Ensuring Usability Through Product & Process[END_REF] which is well suited to specify concrete interaction in a textual/tabular form while MABTA makes references to a graphical sketch of the UI.

A notation should support a seamless transition from the specification of the abstract UI to the specification of the concrete multimodal UI and vice-versa. Moreover multimodal interaction requires the specification of temporal constraints between modalities [START_REF] Vernier | A Framework for the Combination and Characterization of Output Modalities[END_REF]. Temporal operators such as the ones in CTT-based notations (e.g., sequence, parallelism, interleaving) cannot express such qualitative temporal constraints at a fine grain. For example, we consider a multimodal interaction for navigating in a map; a user may say "here", using voice recognition, while pointing to the map. The temporal constraints can be defined by the designer in several ways: For instance (1) the user must first speak and, when finished, point to the map; or (2) the user must start to point to the map and, after say "here", and then finally release her/his finger. For the first example, modalities are used in a sequential way. For the second example, based on Allen relationships [START_REF] Allen | Maintaining Knowledge about Temporal Intervals[END_REF] there is a temporal coincidence between modalities as defined in [START_REF] Vernier | A Framework for the Combination and Characterization of Output Modalities[END_REF]: Indeed the speech modality is used in the context of the direct-manipulation modality. This is a particular case of parallelism, not covered by CTT operators.

Motivated by the fact that existing notations only partly satisfy the two above requirements for specifying multiuser multimodal interaction, we define the COMM (Collaborative and MultiModal) notation, an extension of the CTT notation. We chose to reuse CTT because it is a popular and widely used notation for UI specification. Therefore usual concepts, such as business role, object, agent, task or event, are included in the COMM notation. In the following sections, we first describe the key features of the COMM notation before presenting the e-COMM editor that supports the COMM notation. The COMM notation extends CTT by introducing two concepts, namely interactive role and modal task, and by considering the five Allen relationships as temporal operators applied to concrete tasks.

COMM NOTATION

• An interactive role differs from the usual concept of (business) role in the CSCW literature. As schematized in Figure 2, an interactive role identifies users that interact with the system using modalities, while a business role denotes a role in an organisation such as a technician, an operator, etc.

• A modal task is an additional type of task corresponding to the concrete interaction level: A modal task denotes an atomic user action performed on a physical device.

• Allen relationships are used to refine existing temporal operators applied to tasks in order to specify multimodal concrete interaction. Since it is a refinement of CTT temporal operators for abstract interaction, COMM supports a seamless transition from abstract UI to concrete multimodal UI and viceversa.

We detail these features of the COMM notation by considering an existing multi-user multimodal interactive system presented in [START_REF] Tse | Interaction with Single User Applications through Speech and Gestures on a Multi-User Tabletop[END_REF] as an illustrative example: colocated players are involved in a team play by interacting with a multitouch table and by issuing voice commands.

The game is to play Warcraft: two players cooperate to control troop units in order to win the game.

Cooperation vs collaboration

Let us first consider the following cooperative multimodal task implemented in the Warcraft multi-user system [START_REF] Tse | Interaction with Single User Applications through Speech and Gestures on a Multi-User Tabletop[END_REF]: moving a group of troop units. This is a cooperative task since two business roles are clearly identified: the first business role, named sergeant, has the responsibility of grouping troop units while the second business role, named chief, has to move the group to the newly specified location. As shown in Figure 3, this cooperative task is represented with CTT by two individual interactive task trees, one per business role, and one cooperative task tree. Using COMM, as shown in Figure 4(a), the tasks are represented using a unique task tree: a label decorates the root node to indicate which business roles are involved (i.e., Chief, Sergeant). Similarly, the interactive task is decorated with labels that indicate which business role (i.e., Chief or Sergeant) must perform the concrete task.

Let us now envision the following collaborative task, based on an existing task available in Warcraft, and inspired from [START_REF] Tse | Interaction with Single User Applications through Speech and Gestures on a Multi-User Tabletop[END_REF]: create a unit such as a knight. For performing this task, the first user selects the mode [barrack] (i.e., a building dedicated to the creation of units) and then selects the kind of units to be created; the second user has to confirm the creation of the unit. Such a task cannot be described with CTT since it only supports the description of cooperative tasks. Indeed specifying the constraint that at least two users involved in this collaborative task must interact with different modalities will be complex using existing notations. This will lead to a non-compact and therefore not easily readable specification. .

Cooperative part

Mutimodal part

Collaborative part

Mutimodal part

Using COMM, as shown in Figure 4(b), we use the concept of interactive role (i.e., <ir1> and <ir2> in Figure 4(b)). An interactive role is associated with each elementary task that composes the unit creation task. We associate the role <ir1> to the two first subtasks and <ir2> to the subtask of confirming the unit creation. This implies that the same role should perform the two first subtasks (specify the unit creation) while the third subtask (confirm the unit creation) should be performed by a different role. Furthermore this specifies that both interactive roles are instantiated at runtime and should be performed by any role (sergeant or chief). To fully describe the collaborative concrete task, the subtasks labelled by interactive roles must then be described in terms of interaction modalities: in the example of Figure 4(b), the two modalities -direct manipulation on the multitouch table and speech commands -for <ir1> and speech commands for <ir2>. For clarity, due to a limited space, we have only represented input modalities.

Interactive roles allow us to specify how tasks are dynamically allocated to business roles and how such tasks can be performed using multiple modalities (i.e., direct manipulation and speech). An interactive role is an ephemeral role played by an actor, dynamically assigned to a business role until the task is completed. As shown in Figure 2, while a business role rules the division of labor, which corresponds to a top-down approach (abstract to concrete), an interactive role is a means to specify how group activities are influenced by user interaction constraints, which corresponds to a bottom-up approach (concrete to abstract). In other words, business and interactive roles make explicit the tension between group work activity requirements and concrete interaction requirements.

Specifying concrete multimodal interaction

In order to fully specify the concrete multimodal interaction, elementary tasks (leaves of the task tree) must be refined in order to describe which modalities are used and how the concrete interaction is performed. To do so, we decompose an elementary task as a composition of modal tasks (or monomodal tasks). Based on the definition an interaction modality as the coupling of a physical device with an interaction language [START_REF] Nigay | A Generic Platform for Addressing the Multimodal Challenge[END_REF], an atomic task is defined as an association of an interactive physical device with an atomic user action that is covered by the interaction language.

In the COMM notation, as shown in Figure 5, a modal task is symbolized by two overlapping rounded rectangles (white and green), each containing a textual label. The first label describes the interaction device and the second one describes the user action (interaction language) performed on the device. For example, the elementary concrete subtask select the mode [barrack] of Figure 4(b) is made of the modal task that corresponds to the modality, put her/his finger on the touch table. This elementary concrete subtask is defined by the modal task (<Augmented table, point to barrack>). Another choice of modalities and in particular the choice of devices (for instance a mouse) could lead one to define different modal tasks for each of the subtasks. task performed using multiple modalities is described as a combination of modal tasks. This is the case of the task create unit in Figure 4(b). When multiple modalities are at the same time, the temporal constraints between the modalities must be specified. This temporal constraint drives the fusion mechanism in order to detect when fusion must be performed. We consider the task of moving troop units of Figure 4(a). While one user (sergeant) can group by sliding her/his hands on the touch table (left/right), another user (chief) can move another group in parallel. To do so the user (chief) will select an existing group of units will then point to the new location on the table while saying <go here>. In the latter case, the temporal constraint that links these two concrete subtasks is specified by a Coincidence operator [START_REF] Vernier | A Framework for the Combination and Characterization of Output Modalities[END_REF] inherited from the Allen [START_REF] Allen | Maintaining Knowledge about Temporal Intervals[END_REF]: Indeed the speech modality is used in the context of the direct-manipulation modality. Such an operator [START_REF] Vernier | A Framework for the Combination and Characterization of Output Modalities[END_REF] is used to specify a temporal constraint between modal tasks at a fine grain of description, while we use the parallel operator of CTT to specify temporal relationships between non-modal tasks as in Figure 4(b). We therefore keep CTT operators (i.e., parallelism and sequence) to describe the relationships between tasks at a higher level of abstraction. To specify the temporal combinations of modal tasks, we use the five temporal operators [START_REF] Vernier | A Framework for the Combination and Characterization of Output Modalities[END_REF] described in Figure 6(a). Figure 6(b) shows how the temporal relationships can be specified between two modal tasks using the e-COMM editor. In Figure 6(a), each line corresponds to the time interval in which a modality is used. Sequential and parallel combinations are used in CTT. We add here three new temporal combinations described in [START_REF] Vernier | A Framework for the Combination and Characterization of Output Modalities[END_REF]. Two modalities are combined anachronously if there is a temporal gap between their usage. Two modalities are concomitant when one modality replaces another one with a time interval during which the two modalities coexist. The coincidence of two modalities is when one modality is only used in the context of another one. This is the case of the example of Figure 4(a), when speech modality is used in the context of (a) (b)

the direct-manipulation modality. The two CTT operators (i.e., parallelism and sequence) are then refined by one of the five temporal relationships defined in [START_REF] Vernier | A Framework for the Combination and Characterization of Output Modalities[END_REF]. This enables us to ensure continuity between the description of the abstract UI, independent of the modalities and the description of the concrete multimodal UI.

Other elements of the COMM notation

In comparison with CTT, additional features are included in the COMM notation. In particular we consider new types of tasks:

• User interactive task (Figure 7-5): A User interactive task implies that interaction occurs through a combination of input and output modalities that are described at the level of the modal tasks (Figure 7-6).

Each user interactive task can be decorated with a role (business role or interactive role). For example in Figure 7-4, the user interactive task is decorated by an interactive role <ir 1> which means that this task can be exclusively performed by the Pilot or the Payload operator (see our example of UAV command post).

We also consider the Action, Presentation and System types of tasks. The first one is based on input modalities only without feedback, the second one is based on output modalities only. The last one denotes an internal computation without feedback.

• Group interactive task (Figure 7-1): This type of task includes the Cooperation task of CTT. But, as explained above, a group task may be cooperative or collaborative. This type of implies multiple User interactive assigned to different roles. A Group interactive task is decorated (Figure 7-2) with the list of business roles that could perform the task and with the minimal number of users required to perform the task; this number may be greater than the number of roles: for example, a task may be performed by two pilots and one payload operator (see our example of UAV command post).

• Group task: This type of task extends the User task of CTT. A User task denotes internal and cognitive action by one user in CTT. We generalize it to Group task that denotes a decision/discussion by the group without interacting with the system.

Finally, and in addition to the refinement of the CTT temporal operators described previously, we also extend the iteration operator of CTT. As shown in Figure 7-3, the iteration operation, represented by a *, is further detailed in order to specify how the iteration occurs. The iteration operation is preceded by a || or a |> symbol which means that the iteration can be performed in parallel or sequentially.

e-COMM: COMM EDITOR

As highlighted by van der Veer et al [START_REF] Van Der Veer | Task Based Groupware Design: Putting Theory into Practice[END_REF] and by Molina et al. [START_REF] Molina | A Review of Notations for Conceptual Modeling of Groupware Systems[END_REF], a notation is fully usable if it exists as a tool for the designer that supports the notation. For instance, we may say that CTTE (CTT editor) has contributed to the success of CTT. Towards this goal, we have developed the e-COMM application that supports the COMM notation. This application is implemented as an on-line editor (iihm.imag.fr/demo/ecomm) developed in C# and based on the Microsoft SilverLight technology. The source code will be publicly available and released under the GPL Licence. The e-COMM graphical user interface (Figure 7) emphasizes direct manipulation of the COMM concepts in order to avoid forms, menus or toolbars as much as possible. Indeed, we think that the plethora of filling form popup windows makes a tool less usable and does not encourage focus on the specification under development. Using e-COMM, when a new task tree is created, there is only one node, the root node. Clicking on the (+) symbol (Figure 7-8) leads to the creation of a new subtask. By default, the sequential operator is selected to link the newly created subtasks. The operator can be modified by clicking on the operator symbol and selecting another one: to do when clicking on the operator, the operator wheel (Figure 7-7) appears and shows the available operators. The same interaction technique is used to change the attributes of a task: name, iteration operator and type. Multiple task types are available such as the ones available in CTT and the multiuser task types. Furthermore in order to efficiently navigate in a large task tree the zoom (in/out) action is controlled by the mouse wheel. It is then easy and smooth to focus on a particular part of the tree or to have a global view of the tree. The resulting COMM task tree specification can be saved in different formats including XML, pdf and image formats in order to be easily included in any specification document. In the following section, the trees shown in Figure 8 and 9 have been defined using the e-COMM editor.

CONCRETE EXAMPLE: UAV COMMAND POST

In the context of a French Army Research Department project, in partnership with the firms BERTIN, SAGEM, EADS and PY Automation, the COMM notation is used to specify military command posts for the control of two unmanned aerial vehicles (UAV) by two operators (Figure 1). In this project we adopted an iterative design of the multiuser multimodal interaction to envision the future command posts, and we illustrate the COMM notation by some specifications done for the first iteration using the e-COMM editor. Two operators are involved in the control of UAVs. The first operator has to control the UAV (navigation, guidance, speed, etc). The second one controls embedded devices (payload) such as an infrared camera or a 360º degree camera. In the following, we denote the first operator by the Pilot role and the second one by the operator role. We present three of the specified tasks along with the designed multimodal interaction: (1) to modify a flight plan dynamically and cooperatively; (2) to collaboratively manage tactical layers; (3) to control a multimodal zoom.

Cooperative modification of a flight plan

We consider the first interactive task. During the flight, if required, the flight plan may be modified. Indeed, the Payload operator may observe an event, such as enemy movements along the trajectory, detected by the embedded devices, which implies redefining of the flight plan for safety reasons. A flight plan is defined as a set of waypoints linked with straight lines. Then, the Payload operator interactively defines a new trajectory by selecting and moving existing waypoints on a map. When finished, the modified flight plan is submitted to the Pilot. He/she may accept or not the new flight plan. If the new plan is not accepted, the current flight plan is still active and the new plan is cancelled. If the new plan is accepted, the effective UAV trajectory is modified in order to follow the new flight plan. As shown in Figure 8, we describe this interaction by a cooperative task. Indeed, there is a clear separation of work between the two roles. The cooperative task is composed of two individual sub tasks. The first one is performed by the Payload operator: the main interaction is the composition of three modal tasks (selecting a waypoint, moving the waypoint and dropping the waypoint on a map) that must be executed sequentially. The modal tasks are performed using a mouse. When the operator decides to terminate the task, it triggers the individual task confirm flight plan an object as a parameter, an instance of the flight plan class. Then, the Pilot must first select the new flight plan and validate or not the modification.

Collaborative management of tactical layers

For the second interactive task, we consider a set of stacked layers that contains geographical or military information represented on a map such as tanks, buildings or landmarks. For simplicity, we consider that there is only one layer that contains items representing military vehicles or buildings. The content of the layer can be modified by both roles (Payload operator and Pilot). The information contained in these layers are used by the system to compute new trajectories as a function of the environmental context such as enemy vehicle movements. As shown in Figure 9, the two main interactive subtasks are: add a new item and manage an existing item. Both subtasks may be executed by both operators (the task is decorated with the label among {Pilot, Payload}) but the execution is exclusive. Indeed, the first subtask must be performed only by the first interactive role <ir1> while the second one must be performed by the interactive role <ir2>. It means that if the Pilot decides to perform the task add a new item, the interactive role <ir1> is instantiated with the Pilot role. It also means that if the Payload operator role decides to perform the task Manage tactical layers, only the task manage existing items is available until the Pilot has terminated her/his task. The corresponding modal tasks are based on finger-based selection of buttons and items displayed on a touch table that offers user identification (e.g., DiamondTouch). In this design, the two users can work in parallel but at a given time, they cannot perform the same task. One user can create a new item by [START_REF] Ajaj | Navigation Modes For Combined Table/Screen 3D Scene Rendering[END_REF] selecting the type of item in a palette and then (2) selecting a location on the map, while the second user is moving an existing item. But the two users cannot work in parallel on the same task: for instance one user cannot select the item to be created in the palette while the other one is selecting the location of the new item on the map. Such a design solution is concisely and precisely specified using COMM by using interactive roles. The last example focuses on the task of zooming in/out in the map described in the previous section. As shown in Figure 10, the task is specified as a combination of two elementary tasks that involve different modal tasks: one modality is based on a pedal which must be pressed to control the zoom speed, while the other one is based on the touch table to specify the zoom center. The modality based on the pedal is active only if the zoom center is specified, in other words only if the user is touching the touch table using her/his finger. Such a temporal constraint is expressed by a Coincidence [START_REF] Vernier | A Framework for the Combination and Characterization of Output Modalities[END_REF] (C label) operator. Since no interactive role is specified, the two modalities can be used in parallel by the two operators, one using the pedal while the other one is specified with her/his finger the zoom center.

Multimodal zoom

We have presented the (Collaborative and MultiModal) COMM notation, an extension of CTT for specifying multiuser multimodal interaction. The salient features of the notation are: (1) the concepts of interactive role and (2) modal task, (3) a refinement of the temporal operators applied to tasks using the Allen relationships. CTT temporal operators are used for abstract tasks and are translated in terms of the five relationships between modalities identified in [START_REF] Vernier | A Framework for the Combination and Characterization of Output Modalities[END_REF] and based on Allen relationships, when describing temporal constraints between modality dependent tasks, i.e, modal tasks. This introduces a smooth continuity between the specification of the abstract UI and that of the concrete multimodal UI. Furthermore e-COMM is a fully operational on-line editor that is used in a military project involving industrial partners (BERTIN, EADS, SAGEM and PY automation). The workflow specification is also implemented in the e-COMM editor but not described in this paper.

As future work, we plan to further evaluate the COMM notation and its editor.

Empirical validation, i.e. by experience, is the first type of validation we assessed. The model has been used to specify existing multi-user multimodal systems such as the ones described in [START_REF] Ajaj | Navigation Modes For Combined Table/Screen 3D Scene Rendering[END_REF][START_REF] Tse | Interaction with Single User Applications through Speech and Gestures on a Multi-User Tabletop[END_REF]. Furthermore we are using the notation for specifying new multi-user multimodal systems as part of the on-going military project. The iterative approach adopted in this project provides a means to compare the result of the design step specified using COMM with the implementation done by another partner, and, then, to evaluate the precision and power of expression of the notation. Moreover in the project, the COMM specifications are resulting from a joint effort from different partners, not only the authors of the notation.

Conceptual validation is the second kind of validation we started to assess. To do so we consider the Notational Dimensions of the Cognitive Dimensions of Notations [7]. For instance low viscosity (resistance to change in the specification) can be studied when a designer changes a modality. In addition to changing the corresponding modal task, s/he only has to adapt the neighboring modal tasks, only if they are parts of a multimodal task (e.g., changing the temporal constraints). Furthermore interactive roles may be impacted by changing a modality. We plan to further evaluate the COMM notation based on the Cognitive Dimensions of Notations analysis framework and in particular the Low premature commitment and the high visibility dimensions by conducting experiments with 16 master students (Master Course on Groupware) using the e-COMM editor.

Figure 1 :

 1 Figure 1: Two military operators controlling UAVs.

Figure 2 :

 2 Figure 2: Business Role / Interactive Role.

Figure 3 :

 3 Figure 3: CTT cooperative task.

Figure 4 :

 4 Figure 4 : (a) COMM cooperative task.

Figure 4 :

 4 Figure 4: (b) COMM collaborative task.Figure 5: Modal task.

Figure 6 :

 6 Figure 6: (a) Temporal combinations between modalities. (b) Palette of temporal combinations in the e-COMM editor.

Figure 7 :

 7 Figure 7: Screenshot of the e-COMM editor.

Figure 8 :

 8 Figure 8: Cooperative task for the modification of a flight plan.

Figure 9 :

 9 Figure 9: Collaborative task for managing tactical layers.

Figure 10 :

 10 Figure 10: Concrete multimodal zoom task.

ACKNOWLEDGMENTS

The work presented in the article is partly funded by DGA (French Army Research Dept.) under contract PEA FH/PA. Special thanks to G. Serghiou for reviewing the article.