Víctor López-Jaquero

Elena Navarro
email: enavarro@dsi.uclm.es

Francisco Montero
email: fmontero@dsi.uclm.es

Pascual González
email: pgonzalez@dsi.uclm.es

Metamodels Infrastructure and Heuristics for Metamodel-Driven Multi-touch Interaction

Keywords: model-driven development, multi-touch interaction, heuristics, Acquired Brain Injury (ABI)

niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

The exploitation of models during the software development process is a valuable tool for stakeholders to convey their ideas about design, needs or requirements of the system-to-be. Developers, designers or software architects are used to employ graphical node-link notations for the manipulation of models. UML is a clear example of this approach which is widely used, for instance, to specify class diagrams used at different stages of the development process.

However, it is frequently the case that users and/or clients of the system-to-be are required to manipulate models that are used during the development of the system to convey their expertise in the problem domain. This scenario can arise easily in the user-centered design, where the final users of the application are involved in the development. This can be a challenging, or even overwhelming, task for them as they usually do not have the required abilities to tackle modeling activities. This has been the problem we had to face during the development of HABITAT [START_REF] Navarro | HABITAT: a Web Supported Treatment for Adquired Brain Injured[END_REF], a system to support the relearning process of people with Acquired Brain Injury (ABI). One of the main cornerstones of this system is its functionality to design new types of relearning activities so that the relearning process can be fully customized according to the specific needs of the people with ABI. During the development of this functionality, the exploitation of models emerged as a suitable solution. However, our users (psychologists, speech therapists, physiotherapists, etc) did not have the necessary abilities for manipulating models, although they did have the knowledge we needed about the problem domain. This was the challenge we had to face with the ideas presented in this work: can we provide users, who are novice users, with a tool for manipulating models that hides the complexity behind this task? The solution to overcome this complexity has been to exploit the benefits provided by the integration of two well-known approaches: Multi-touch interfaces [START_REF]Tabletops -Horizontal Interactive Displays[END_REF] and Model-Driven Development (MDD, [START_REF] Hailpern | Model-driven development: The good, the bad, and the ugly[END_REF][START_REF] Selic | The pragmatics of model-driven development[END_REF]).

Multi-touch interfaces provide users with attractive and innovative facilities for the manipulation of applications by means of touch gestures. Thus, the interaction can be performed in a natural and intuitive way. Several works, such as [START_REF] Ciocca | Browsing museum image collections on a multi-touch table[END_REF] [START_REF] Luebbe | Tangible Media in Process Modeling -A Controlled Experiment[END_REF], have shown in the experiments performed that this approach is actually suitable for novice users, as they perceived the interaction with multi-touch applications as more attractive and interesting than traditional applications. Therefore, its exploitation in this work emerged in a natural way, since cooperation with novice users is our main goal.

MDD is not only becoming increasingly popular among researchers, but also among practitioners. It has proved to have a positive influence on the reliability and productivity of the software development process due to several reasons, such as, the exploitation of techniques for the automatic generation of code or the use of models as drivers of the development process. Both reasons led us to its consideration in this work; first to introduce the necessary facilities for the manipulation of models and second to generate model-manipulation multi-touch interfaces in an automatic way. The generation of multi-touch interfaces is performed by automating a set of heuristics, which are presented in this work, that exploit the structure and semantics of the primitives used for domain modeling by using Ecore metamodels [START_REF]EMF[END_REF].

The paper is organized as follows. Section 2 provides an overview of ABI, by describing the target population and how this work was conducted. Section 3 presents and analyzes the related previous work. Section 4 describes the metamodels infrastructure to support our proposal. Section 5 describes the heuristics developed. Section 6 presents the initial results of the conducted exploratory evaluation. Finally, Section 0 rounds off the paper by presenting the conclusions drawn and some future work.

Case study: treatment of Acquired Brain Injury

People with Acquired-Brain Injury (ABI) have suffered "damage to the brain that occurs after birth and which is not related to congenital disorders, developmental disabilities, or processes that progressively damage the brain" [START_REF]Toronto Acquired Brain Injury Network[END_REF]. There are several causes of ABI, such as cerebral vascular pathology, skull-brain trauma due to accidents, meningitis, brain tumours, etc. Therefore, it can be stated that just about everybody is exposed to this risk in its daily life. Cases affected by this disability are becoming increasingly common. According to the JCCM Health Council [START_REF]Conjunto mínimo de datos básicos de la Región de Castilla La Mancha[END_REF] 4 out of every 1000 persons suffer some kind of ABI at some time in their lives. Although people of all ages can be affected, it is more frequent among the younger and older members of the population, as they are more prone to accidents.

According to the experts, the process of integral ABI relearning must include cognitive treatment, in addition to physical and occupational therapies. It should also be emphasized that ABI associations, such as ADACE, which we have collaborated with in different projects, highlight that ABI victims should be provided with a proper treatment as soon as possible, since there is increasing evidence of its effectiveness during the first stages after injury [START_REF] Cicerone | Evidence-based cognitive relearning: Recommendations for clinical practice[END_REF]. However, identifying the proper treatment for each person is a difficult and time-consuming task, since brain injury has dramatically varied effects and no two people can expect the same resulting difficulties. This means that an individualized relearning process must be identified for each person. In this context, providing specialists with tools to create and customize the activities and tasks that they use in the processes of recovery [START_REF] Montero | Computer-aided relearning activity patterns for people with acquired brain injury[END_REF] is a must. This is the aim of our system: HABITAT [START_REF] Navarro | HABITAT: a Web Supported Treatment for Adquired Brain Injured[END_REF].

HABITAT enables ADACE specialists to create relearning activities by instantiating the implemented relearning patterns [START_REF] Montero | Computer-aided relearning activity patterns for people with acquired brain injury[END_REF]. These relearning patterns were validated by the specialists and were put into practice thanks to the implementation we made in HABITAT. However, the specialists highlighted the need to customize these patterns and this led us to define the relearning pattern metamodel described in Section 4.1 and its implementation in HABITAT. This metamodel had to be instantiated and used by the specialists to define and modify the special relearning patterns they needed. Nevertheless, as these people were not used to manipulating models by using node-link representations, the alternative was to provide them with multi-touch User Interfaces (UI), which are automatically generated by means of the heuristics presented in Section 5 and using the metamodels infrastructure offered in Section 4.

Related work

The generation of UIs or visual metaphors to edit models is not a new trend. All casetools offer a means of manipulating models by providing a visual notation. Nevertheless, generating a UI to manipulate a domain model is not commonly available. One example of generation of UIs out of a domain model is the Graphical Modeling Framework (GMF, [START_REF]GMF[END_REF]) and Eclipse Modeling Framework (EMF, [START_REF]EMF[END_REF]). By using EMF a developer can create a domain metamodel by means of Ecore and then generate a graphical editor by using the GMF framework. The features of this editor are specified by means of a set of models. Another example is Executable UML [START_REF] Milisev | Model Driven Development with Executable UML[END_REF] which aims at generating UIs out of a UML specification. The UIs generated are for standard desktop application, and the interaction is mostly based on drag&drop interaction in a tree. The author provides some guidelines regarding how UI is generated, the so called interactive manifestations. This approach is not intended for metamodel manipulation, but introduces interesting ideas regarding the generation of UI out of UML models. Nevertheless, the generated UI is not appropriate for novice users and the user has no feedback regarding multiplicities in the specification of the cardinalities of the relationships. Another approach pursuing the generation of UI out of object-oriented specifications is Naked Objects [START_REF] Pawson | Naked Objects[END_REF]. In Naked Objects, the applications are specified solely by using domain entity objects. A direct matching between domain objects and presentation is proposed. Unfortunately, the heuristics to generate the presentation are not described. As for Executable UML, this approach is not aimed at manipulating metamodels, but at supporting whole application user interface generation. In Naked Objects, the presentation cannot be customized; therefore it cannot be adapted to different user skills or preferences. There is no guidance regarding the order the tasks should be carried out, the target platform is desktop applications and it is not designed for novice users.

There are also other similar approaches from the human-computer interaction community. Model-based User Interface Development Environments (Mb-UIDE) [START_REF] Puerta | A model-based interface development environment[END_REF] provide a mechanism to design the UI by means of a number of declarative models, which are latter translated into code directly executable on a specific platform or into an intermediate language (usually XML-based). Mb-UIDE has been in use since the beginning of the 90s and it is becoming increasingly integrated into the MDD approach [START_REF] Vanderdonckt | A MDA-Compliant Environment for Developing User Interfaces of Information Systems[END_REF].

In Mb-UIDE, the domain model represents the information required by the user to carry out the tasks through the UI. To express these models, different notations have been used, but undoubtedly, the most commonly used are entity-relationship notation and UML class diagrams. Some of the MB-UIDEs using class diagrams are OVID [START_REF] Roberts | Designing for the User with OVID: Bridging User Interface Design and Software Engineering[END_REF], Janus [START_REF] Balzert | From OOA to GUI -The JANUS-System Human-Computer Interaction[END_REF], AME [START_REF] Märtin | Software Life Cycle Automation for Interactive Applications: The AME Desing Environment[END_REF], Teallach [START_REF] Barclay | The Teallach Tool: Using Models for Flexible User Interface Design[END_REF], OO-H [START_REF] Cachero | Una expresión a los métodos OO para el modelado y generación automática de interfaces hipermediales[END_REF] and IdealXML [START_REF] Montero | IdealXML: An Interaction Design Tool and a Task-Based Approach to User Interface Design[END_REF]. Two of the Mb-UIDEs that use entity-relationship notation for domain modeling are Trident [START_REF] Bodart | Computer-Aided Window Identification in TRIDENT[END_REF] and Genius [START_REF] Janssen | Generating User Interfaces from Data Models and Dialogue Net Specification[END_REF]. Just-UI [START_REF] Molina | User Interface Conceptual Patterns, Interactive Systems. Design, Specification, and Verification[END_REF] is a MB-UIDE that provides a set of patterns to generate a UI out of a domain model for standard desktop applications. Although these approaches are not aimed at generating UI for the manipulation/edition of a domain model, they provide interesting insights into what should be modeled to automatically generate a UI out of models.

We took inspiration from these approaches to identify the key features that should be modeled to generate a UI, including the requirements to have some extra models apart from the domain model to generate usable UIs to manipulate the models. These extra models enable the modeling of, for instance, the aesthetics of the user interface to be generated. These extra models are described in depth in section 4.

Since our goal is to be able to generate a UI for the manipulation of domain models by novice users, with no experience in using software design tools, the proposal used should not be the one used in most CASE tools. Therefore, some research was carried out on the different interaction techniques available. This study showed that multitouch user interfaces were the most intuitive ones for novice users, such as ABI specialists [START_REF] Montero | Computer-aided relearning activity patterns for people with acquired brain injury[END_REF]. The most important reason why ABI specialists chose multi-touch interaction was because they felt it was natural, and also because they were used to the techniques involved in multi-touch smart phones. Nowadays, multi-touch interaction and gestures are being widely used in many kinds of portable and fixed devices [START_REF]Tabletops -Horizontal Interactive Displays[END_REF].

The concept of interaction style refers to the different ways the user can communicate or otherwise interact with an artifact. There are different interaction styles, for instance: command language, form filling, menu selection or direct manipulation. In our context, direct manipulation was chosen because it offers several advantages such as [START_REF] Schneiderman | Direct Manipulation for Comprehensible, Predictable and Controllable User Interfaces[END_REF]: visually presenting task concepts, it is easy to learn, errors can be avoided more easily and recognition memory, as opposed to cued or free recall memory, can be emphasized. There are currently many electronic devices in which touch and motion-based gestures are supported and there are de facto standards related to this type of interaction, such as the Apple Human Interface guidelines [START_REF]Apple. iOS Human Interface Guidelines[END_REF]. These guidelines have been used for the definition of the heuristics presented in Section 5.

Metamodels infrastructure

Several approaches have emerged to date describing the guidelines to execute the MDD paradigm. Perhaps, the most commonly known is the Model-Driven Architecture (MDA, [START_REF] Miller | MDA Guide Version 1.0.1[END_REF]), an initiative of the Object Management Group (OMG). MDA promotes the separation of the domain model from the underlying technology to facilitate higher flexibility while designing and evolving software systems. One of the key elements of the MDA initiative is the Meta-Object Facility (MOF, [START_REF] Omg | Meta-Object Facility (MOF)[END_REF]), a four-layer architecture used for the definition of the metamodels and models involved in the software development process, as shown on the right part of Fig. 1.

Fig. 1. Stakeholders, software products and MOF architecture in our approach

Fig. 1 shows the workflow of our proposal. The key element is the instantiation process: metamodels (M2) are instances of Ecore (M3), models (M1) are instances of metamodels, and domain instances (M0) are instances of models (this level is shown in gray to indicate it is out of the scope of this paper due to space limitations). In this instantiation process each stakeholder, specifically developers and ABI specialists, are focused on their interests and abilities. Developers are responsible for the specification of metamodels and the development of a flexible tool for relearning activity patterns. However, ABI specialists are focused on the definition of relearning activity patterns by using the tool with a high level of usability created by the developers. The UI of this tool was generated automatically by the developers, using the set of heuristics that are presented in Section 5. These heuristics were automated by using XPAND [START_REF]XPAND[END_REF], a specialized language for code generation out of Ecore models. Moreover, the editor model was automatically created thanks to the capabilities offered by the Eclipse Modeling Framework (EMF, [START_REF]EMF[END_REF]). As Fig. 1 shows, the MOF architecture has been used to describe both the domain metamodels (M2) and the models for the systems under development (M1). The domain metamodels are used to describe the core processes and domain concepts that are to be used by developers to convert design into code. According to our case study, the domain metamodel is the relearning pattern metamodel. This architecture has been used as follows:

 Meta-metamodel (M3) level. It offers a collection of primitives to define metamodels at level M2, that is, it is a meta-metamodel to describe metamodels. In this proposal, Ecore from EMF was used as meta-metamodel.  Metamodel (M2) level. The elements in this metamodel are used to describe the elements of the model at level M1. ABI specialists demand software for the creation of relearning activities for people with ABI that is customizable enough.

Although, these activities are known by specialists, they unfortunately cannot implement them. HABITAT provides a computer-based tool for relearning activity patterns specification for ABI specialists. With this aim, in this M2 level, developers created the domain metamodels that were used jointly with the heuristics presented in Section 5 to create a multi-touch tool in an automatic way. This tool is used by ABI specialists to manipulate the domain model, that is, the model for specifying relearning activity patterns.  Model (M1) level is defined by instantiating the M2 metamodel, which is used to define the relearning patterns that enable the specialists to create relearning activities. ABI specialists work, at this level, with the computer-based tool developed by the developers. ABI specialists create the relearning activity patterns to provide tools for people with ABI. At this level, concrete interaction styles, deficits and resources are specified.  Instances (M0) level. At this level, the instances of the domain model are created.

In our case, the relearning activities are defined by the specialists and used by the people with ABI, as instances of the M1 model. ABI specialist instantiate and create different tools for people with ABI where different deficits and interaction possibilities are considered. People with ABI use specific software designed by ABI specialists for their treatment.

Fig. 2. Metamodels Infrastructure

Although it is not shown in Fig. 1, at M2 level, developers identify and specify three metamodels: domain, mapping and presentation. As it can be observed in Fig. 2, these metamodels are generic, so that they can be used for any general purpose domain. Next, all these three metamodels are described:

Domain Metamodel Mapping Metamodel

 Domain metamodel. This metamodel supports the specification of the facilities and services required for the target domain. As it can be observed, the main element of Ecore, EClass, has been used to facilitate that any domain metamodel defined as instance of Ecore can be used in this proposal. In our case study, developers are aimed at providing an environment for ABI specialists supporting the specification of any number of relearning activities patterns. Later, in Section 4.1 we present the HABITAT relearning activity pattern metamodel as an example of a concrete domain metamodel that can be used.  Presentation metamodel. This metamodel is used to support the specification of the presentation details for each domain metamodel element. Section 4.2 describes the presentation metamodel developed in this proposal.  Mapping metamodel. This metamodel is a mediator entity between domain and presentation metamodels. It relates both metamodels supporting loose coupling between the domain metamodel and the presentation metamodel.

Domain Metamodel: HABITAT relearning pattern metamodel

At the metamodel level (M2), developers defined a specification useful for relearning activity patterns specification. These patterns constitute the domain to be defined by our novice users, ABI specialists. At the beginning, many ABI specialists, for instance in the ADACE association [START_REF] Montero | Computer-aided relearning activity patterns for people with acquired brain injury[END_REF], documented the relearning activities using cards. These relearning activity descriptions had different elements or sections that were analyzed and abstracted away to create the metamodel shown in Fig. 3 (for the sake of clarity, class attributes are not shown in the figure). Purposes and descriptions of the elements of relearning activity patterns are the following:

Fig. 3. Relearning pattern metamodel

 Model. This is a general-purpose element for organizing the domain metamodel. The definition of this kind of element is a constraint imposed by Ecore.  ModelElement. This element was defined to abstract away some attributes that are shared by several metamodel elements.  Pattern. This element represents a relearning activity pattern at a conceptual level, that is, concrete relearning activity patterns will be instances of this element. It represents an abstraction of a type of treatment or activity for people with ABI. It will be instantiated by ABI specialists.  Relation. This element represents the relationships between relearning activity patterns. Few patterns live in isolation. Typically, they introduce new, hopefully smaller and more tractable activities which will lead you to other relearning activities. Or, there may be other patterns that treat the same set of deficits.  Parameter. This element is used to represent the information of each pattern.

Instances of this element that ABI specialists can specify are related to evaluation criteria, timing, etc.  ParameterSet. This element represents sets of parameters with a common purpose.  Deficit. Each relearning activity pattern is addressed to treat a specific set of deficits (e.g.: physical, cognitive and/or emotional/behavioral).  RelearningGroup. In order to facilitate the treatment process, different groups were defined by ADACE, each one being characterized by a set of specific deficits.  MultimediaResource. This element is used to associate multimedia resources to relearning activity patterns.  InteractionStyle. This element represents the interaction style that the ABI specialists wants each person with ABI to use when he/she is doing a relearning activity.  ActivityInteraction. This element represents the kind or type of activity that ABI specialist wants to associate with a relearning activity pattern. For instance, association or puzzle activities are instances of this element.

By using the previous elements developers provide ABI specialists with a software tool. Then specialists can document relearning activity patterns. Customization and personalization of this software tool can be achieved by the presentation metamodel that will be described in the next section.

Presentation metamodel

This metamodel was defined by developers to achieve flexibility in the look and feel of the computer-based tool provided to the ABI specialists. Each domain model will be related to at least one presentation model, that is, an instance of the presentation metamodel illustrated in Fig. 4.

The presentation metamodel elements are described as follows:

 PresentationModel: This is a general-purpose element for organizing the presentation metamodel. It has a similar purpose to Model in the domain metamodel (see Fig. 3).  PresentationElement. Each time a domain element is instantiated, a

PresentationElement will be created to allow the customization of its presentation.

As can be observed in Fig. 4, it helps in describing different parameters, such as background color, tooltip, or label.  Font. This element is used to allow the font customization of domain elements. It is related to alignment, size, etc.  Icon. This element was included to associate each domain element with an icon.  Border. This element is used to support border customization in domain elements.

Fig. 4. Presentation Metamodel

Once domain and presentation metamodels were introduced we will describe in the following section the heuristics identified for generating multi-touch UIs that allow novice users to manipulate models.

Heuristics for model-driven multi-touch interaction

Heuristics are experience-based techniques for problem solving, learning, and discovery. These methods are used to speed up the process of finding a satisfactory solution, where an exhaustive search is impractical or conditioned in a certain way. Examples of this method include using a "rule of thumb" or common sense. When these heuristics are repeatedly used, and thoroughly tested, they can become patterns. We were able to identify heuristics by applying our previous experience in similar multi-touch interaction UIs developed in collaboration with ADACE. These UIs pursued hiding the complexity usually found when novice users have to manipulate metamodels in a MDD environment.

As shown in Fig. 3, after the developer has specified the domain metamodel, the tool that will be used by the novice users is generated. This tool is generated automatically by means of EMF and XPAND frameworks, and the heuristics proposed in this paper.

The heuristics gathered in this section proved useful for UI development out of a set of models. These UIs were suggested and evaluated by ABI specialists, who were our end-users. Multi-touch and direct manipulation were the interaction styles chosen to support the application. The inputs for the definition of these heuristics were the specialist's requirements. Our activities were driven by the metamodeling principles underlying Ecore metamodel.

One of the main elements considered by the heuristics presented in this work is the semantics of the relationships that can established in Ecore. To make easier understanding the heuristics proposed, the semantics of UML relationships [START_REF] Booch | Unified Modeling Language User Guide[END_REF], and how they can be expressed in Ecore, is described:  Association: An association relationship is a structural relationship between two model elements that shows that objects of one classifier connect and can navigate to objects of another classifier. Even in bidirectional relationships, an association connects two classifiers, the primary (supplier) and secondary (client). This relationship is specified in Ecore by establishing an EReference between two EClasses. If a bidirectional relationship is needed, then two EReferences have to be created, one for each direction, and their EOpposite attribute must be set to specify that one is opposite of the other one.  Composition: a composition relationship represents a whole-part relationship, and it is a specific type of aggregation, which is another type of UML relationship. An aggregation relationship depicts a classifier as a part of, or subordinate to, another classifier. A composition relationship specifies that the lifetime of the part classifier is dependent on the lifetime of the whole classifier. To specify this type of relationship in Ecore, an EReference between two EClasses is created. Besides, its attribute containment must be set to true.  Generalization: A generalization relationship denotes that a specialized (child) model element is based on a general (parent) model element. Although the parent model element can have one or more children, and any child model element can have one or more parents, typically is the case that a single parent has multiple children. This relationship type is described in Ecore by setting the attribute ESuperType of the child EClass to its parent EClass.

Other important attributes used while describing Association and Composition relationships in Ecore are LowerBound and UpperBound. They are used to describe the cardinality, that is, how many instances from one of the entities is related to each instance in the other entity involved in the relationship.

While the ABI domain metamodel was being developed, and UIs were being discussed with ABI specialists, we identified empirically a set of heuristics correlating the semantics and structure found in the metamodel and the user interaction to manipulate it. These heuristics were gathered in specific scenarios, but we found that they can be reused again and again in other scenarios. In our proposal, multi-touch and motion-based gestures are used to design the following heuristics:

 Heuristic #1 -[Root]: all the derivation of the user interface starts from the root of the model as this is a constraint imposed by Ecore. For instance, Model was the selected root for the Relearning pattern metamodel shown in Fig. 2.  Heuristic #2 -[Recursion]: as the user browses the elements of the domain model to create and manipulate an instance, all the heuristics are applied recursively. That is, the root where all the heuristics are being applied is the current element the user is browsing.

 Heuristic #3 -[Association (2-directional) -Drag&Drop]. A bidirectional
association relationship is used, for example, to specify that a pattern can be related to several relearning groups, and that a relearning group can be related to several patterns (see Fig. 3). Drag&Drop is the interaction style to be used whenever an association of the domain model has to be manipulated. As shown in Fig. 5, by dragging one instance from the lower part, and dropping it into the upper part, the user can easily create the relationship between a pattern and a relearning group. To provide more guidance to the user, and prevent errors, feedback related to the number of elements that the user can or must drag is provided. If the user can relate or not the elements (optional relationship), then a suggestion written in grey text is shown in the upper part. On the other hand, if it is mandatory for the user to create the relationship (mandatory relationship) then this text is written in red. Note that depending on the path followed to reach an element in the metamodel, this heuristic represents one direction or the other of the bidirectional relationship.

Always the direction represented in the UI is the one whose origin is the current element the user is manipulating. . This association is used, for instance, to describe the unidirectional relationship between activity interaction and multimedia resources (see Fig. 3). As Fig. 6 shows, to specify a multimedia resource for a relearning pattern, an activity interaction must be previously defined. That is, unidirectional relationships describe dependencies in the order that instances from the metamodel can be created. Disable is the interaction used to reflect the dependencies created by unidirectional relationships, those elements having an incoming unidirectional relationship will be disabled in the user interfaces until the dependency is fulfilled. In the example of Fig. 6, once an activity interaction is defined, multimedia resources can be created. Once at least a multimedia resource has been created a UI similar to that presented in Fig. 5 will be offered to the user to relate it to an activity interaction.  Heuristic #5 -[Generalization -Tap]. In the example, different kinds of ABI deficits are shown, such as cognitive, physic or emotional/behavioral. In the UI generated from the metamodel, the user should be able to navigate through the types of deficits and to specify special deficits for each relearning pattern. Tap is the gesture used to navigate the generalization hierarchies. In our example, the user navigates the deficits types of our metamodel. Once a particular deficit type is selected (see Fig. 7), the user can manipulate its instances. . This relationship is used, for instance, in the ABI domain metamodel to specify that model is composed of 0 or many ActivityInteraction. This means that the user should be supported in creating, editing or deleting activity interactions instances. Press is the interaction style to be used for the manipulation of these relationships. A press is a touch in a surface for an extended period of time that is used to select the element to be modified, and then it can be drag to the bottom side of the UI for its edition or deletion. Fig. 8 shows how an activity interaction is manipulated. If a new instance has to be created, then the user presses on the free area of the UI so that a UI to specify the new instance is shown by the system. For instance, when the user presses on the free area of the Fig. 8 then the UI depicted in It is worth noting that although the ABI domain has been used as guiding example for the explanation of the proposed heuristics, they are totally independent of any domain. No information or relation to the domain has been considered while they were identified and automated by means of XPAND, so that they can be reused in different contexts by only feeding the generation tool with new metamodels. In addition, this proposal has a direct impact on productivity, as most MDD proposals do. This means that as the domain metamodel evolves, the multi-touch UI changes in an automatic as well without coding effort.

RelearningGroup

Exploratory evaluation

Both the heuristics and the metamodel infrastructure were validated and refined during the development of the HABITAT project by ABI specialists. This project aims at helping people with ABI. ABI patients have brain injuries caused by damage to the brain after birth that can involve the loss of cognitive, physical and/or emotional capabilities. The relearning process to improve their quality of life and help them in recovering their lost capabilities is a long and hard road.

This relearning process should be flexible since every single patient has specific needs to which the relearning process should be tailored as far as possible. This personalization should be achieved by using the specialists' normal vocabulary so that both the knowledge and the experience gathered during the relearning process can be reused. The issue of common vocabulary has already been addressed in previous studies [START_REF] Montero | Computer-aided relearning activity patterns for people with acquired brain injury[END_REF].

The exploratory evaluation was conducted in collaboration with three ABI specialists. An iterative process was used for refining the heuristics on the basis of the feedback collected from the subjects. First, we met with the specialist to detect their needs in terms of modelling concepts. After several meetings, a metamodel was created that gathers the knowledge about the problem that is used to support the ABI specialists in creating their own relearning activity patterns to help during the relearning process of people with ABI. Second, a first version of the heuristics was defined, and they were used to create the first prototype that was tested by the ABI specialist. After several iterations the final version of the heuristics presented in this paper was produced. In each iteration, the cognitive walkthrough usability testing technique was used to detect possible usability issues in the prototype created by applying the heuristics.

Then, the engine to automate the generation of the user interface by applying the heuristics was developed. This engine was used to generate the multi-touch user interfaces for the ABI metamodel. This tool was used by the ABI specialists to instantiate the metamodel to create new relearning patterns.

The interviews with the ABI specialists revealed positive results, since all the specialists told that they were able to create the intended relearning pattern by using the generated user interface. Nevertheless, a thorough evaluation is required to fully validate all the heuristics, involving subjects from different domains.

Conclusions and further work

Model-Driven Development is a powerful tool to develop software. Nevertheless, some of the models involved in a model-driven development are actually dependent on a specific domain, where specific experts are actually the ones that should manipulate them. Nevertheless, usually domain experts are novice users. By providing a multi-touch based interaction, fully driven by the underlying metamodel, we are supporting novice users in the manipulation of metamodels. Our experience with ABI specialist have thrown successful results, supporting the specialist in creating relearning patterns [START_REF] Montero | Computer-aided relearning activity patterns for people with acquired brain injury[END_REF] out of UI automatically derived from the domain metamodel.

In this paper a metamodels infrastructure and a set of heuristics are proposed. The heuristics drive the generation of multi-touch user interfaces that support the manipulation of metamodels to manipulate domain models. These heuristics exploit the semantics and structure of any metamodel specified by using Ecore to generate these multi-touch user interfaces.

The multi-touch user interfaces are generated so novice users can create models from metamodels, bridging the gap between the developer's language and what novice users understand. Furthermore, a metamodel infrastructure is also introduced to support the customization of the presentation of the different metamodel elements. Thus, icons or other aesthetics features can be specified to make the generated user interfaces more attractive to the user. Generating automatically a multi-touch user interface from metamodels also helps in improving certain basic usability principles, such as: visual presentation of task concepts, easy to learn, error prevention (by supporting the user in performing only valid actions), consistency (since the same tasks, such as adding items to an aggregation, is always carried out in the same way) or presentation structuring (by using the relationships in the metamodel to group related concepts).

As further work, we are applying these heuristics to other domains to gain further experience and improve our current tools for the generation of multi-touch user interfaces from domain metamodels.

Another future work is related to the evaluation of the proposal. Although the exploratory evaluation threw positive results, a more thorough evaluation is required to fully validate the heuristics presented. Currently several experiments are being designed, involving users from different domains.

Fig. 5 .

 5 Fig. 5. Generated UIs and gesture according to Heuristic #3

Fig. 6 . 4 

 64 Fig. 6. Generated UIs and gestures for Heuristic #4

Fig. 7 . 5 

 75 Fig. 7. Generated UIs and gestures for Heuristic #5

Fig. 9 Fig. 8 .Fig. 9 .

 989 Fig. 8. Generated UIs and gestures for Heuristic #6 while editing or deleting an activity interaction

Acknowledgements. We would like to thank the ADACE Association (http://www.adaceclm.org) for their cooperation in our research on ABI. This work has been partially supported by the grant insPIre (TIN2012-34003) from the Spanish Government Department of Science and Innovation and the grant AVANZA TSI-020400-2011-20 from the Ministry of Industry, Tourism and Commerce of Spain.


Felix Kistler

Elisabeth André

User-Defined Body Gestures for an Interactive Storytelling Scenario

Keywords: User Defined Gestures, Kinect, Full Body Tracking, Depth Sensor, Interaction, Interactive Storytelling

For improving full body interaction in an interactive storytelling scenario, we conducted a study to get a user-defined gesture set. 22 users performed 251 gestures while running through the story script with real interaction disabled, but with hints of what set of actions was currently requested by the application. We describe our interaction design process, starting with the conduction of the study, continuing with the analysis of the recorded data including the creation of gesture taxonomy and the selection of gesture candidates, and ending with the integration of the gestures in our application.

Introduction

Creating intuitive interaction has always been a difficult, but important task for developers of games or interactive storytelling applications. As depth cameras have become broadly available consumer products with the Microsoft Kinect sensor 1In this paper we adapt the method for creating a user-defined gesture set for full body gestures in an interactive storytelling scenario. We describe the full process from conducting the user study, analysis of the recorded data, to the integration of the gesture set in our application.

, opportunities emerge for such systems to make use of novel full body interaction techniques. This in turn poses new challenges for the interaction designer. Various researchers have already started to integrate this new kind of interaction in their interactive storytelling system [START_REF] Kistler | Full body gestures enhancing a game book for interactive story telling[END_REF][START_REF] Álvarez | Exploring body language as narrative interface[END_REF], but usually the gesture set for interacting with the system is chosen by the developers themselves according to their imagination and preferences. However, a gesture that is intuitive for the developers does not necessarily have to be intuitive for the majority of users. A different approach that creates a userdefined gesture set has been addressed by several other domains, such as surface computing [START_REF] Wobbrock | User-defined gestures for surface computing[END_REF], public displays [START_REF] Kurdyukova | Studying user-defined iPad gestures for interaction in multi-display environment[END_REF], or human robot interaction [START_REF] Obaid | User-defined body gestures for navigational control of a humanoid robot[END_REF].

1.1

In-Game Actions and Input Gestures

In computer games and interactive storytelling exist several types of in-game actions users can trigger via interaction. Two common action types are: navigation and dialogue. Basic navigation includes changing the position and orientation. Dialogue actions are usually involved when embodied conversational agents exist in the scenario that users can speak with. There are also other action types, e.g. manipulation actions which change the properties of an object, but our work will focus on navigation and dialogue. Furthermore, we focus on full body gestures that users need to perform for triggering those two types of actions. For navigation, this might be a quite straightforward choice, as body movements are also used for navigation in real-life. For dialogue the main interaction modality in real-life is speech and it therefore might seem a bit awkward to use body gestures for it. However, conversational gestures are used in real-life for emphasizing or enhancing speech utterances, and sometimes even to replace them, e.g. when performing a head nod instead of saying "yes". In addition, we want to keep the variety of different input modalities as small as possible because full body interaction itself can be quite difficult for users that are used to mouse and keyboard interaction and Kurdyukova et al. [START_REF] Kurdyukova | Introducing multiple interaction devices to interactive storytelling: Experiences from practice[END_REF] have also shown that too many different input modalities can be more distracting than engaging for users. Overall, interactive storytelling applications can include a huge variety of specific actions for navigation and dialogue. It therefore seems quite impossible to find a generic set of actions for those two types. Instead, we investigate an action set created for our specific scenario. Nevertheless, it should represent a combination of actions typical to interactive storytelling scenarios and we have the hope that our findings also apply to other scenarios without major differences.

Related Work

In the following, we first describe work on classifying human gestures that helps us build our gesture taxonomy. Afterwards, we summarize an influential approach for acquiring a user-defined gesture set that we will adapt and use for our own work. The last area of research we analyze is that of full body gesture recognition, as we eventually aim to implement the recognition of our user-defined gesture set.

Human gestures

There exist different taxonomies for the classification of conversational human gestures. One of the first was introduced by Efron [START_REF] Efron | Gesture and Environment[END_REF] who presented five categories: physiographics, kinetographics, ideographics, deictics, and batons. Further, Kendon [START_REF] Kendon | How gestures can become like words[END_REF] tried to link his gesture taxonomy to the relation with speech and defined the following categories: gesticulation, language-like gestures, pantomime, emblems, and sign language. Another popular taxonomy was proposed by McNeill [START_REF] Mcneill | Head and Mind: What Gestures Reveal About Thought[END_REF] who presented five types of gestures: iconic, metaphoric, deictic, cohesive, and beat gestures.

The three taxonomies have a considerable overlap in their covered concepts, and they all focus on hand gestures during a conversation. However, the properties they de-scribe can also be used to categorize full body gestures for human computer interaction as in our case. None of the three taxonomies perfectly suits this purpose, but we have decided to use the terminology by McNeill and therefore describe the items of this taxonomy in more detail in the following. Iconic and metaphoric gestures both try to convey information by visually depicting an icon. However, they do this at different abstraction layers: iconic gestures are more concrete and directly represent a physical, spatial or temporal property of a real-world referent, e.g. when moving two fingers to indicate somebody is walking. Metaphoric gestures refer to abstract properties of a referent. For example, somebody might depict a container to refer to the contents of a story. Deictic gestures are pointing gestures that indicate a position or direction.

The last two types of gestures do not convey meaning but accompany speech to emphasize parts of it (beat gestures) or to keep up the continuity (cohesive gestures). Therefore, those two types are less suited for direct interaction, whereas the other three categories seem to be good candidates for representing in-game actions. That is why we use them in our later described taxonomy. McNeill also defined three phases of a gesture: preparation, stroke, and retraction. Preparation is the phase in which the body is brought from its rest to a position that is suitable for executing the gesture.

The stroke phase contains the main part of the gesture, while in the retraction phase the body is brought back to its rest. In terms of gesture recognition, the stroke phase is the most important part of a gesture as it contains the actual information.

User defined gestures

In the last couple of years, gestural interfaces have become more and more popular, ranging from small multitouch phones to large interactive tables or walls, but also to freehand and full body interaction using a depth sensor as the Microsoft Kinect. Nevertheless, gesture sets are often designed without sufficiently taking into accounts the preferences, habits, and needs of the actual users. For this reason, several researchers started to involve the user into the design process of the gestural interaction. Wobbrock et al. [START_REF] Wobbrock | User-defined gestures for surface computing[END_REF] presented a seminal approach to develop intuitive gestures for surface interfaces. They gathered a gesture set by presenting the wanted effect within the system to the user, and then asking the user to perform a gesture that should trigger this effect. After that, they determined gesture candidates by looking at all gestures performed by the users for a specific effect, and calculating an agreement score based on how often the same gestures were used. Researchers already adopted this process for other areas, e.g. Kurdyukova et al. [START_REF] Kurdyukova | Studying user-defined iPad gestures for interaction in multi-display environment[END_REF] used it to design gestures for transferring data between tablet computers and a multi-display environment, and Obaid et al. [START_REF] Obaid | User-defined body gestures for navigational control of a humanoid robot[END_REF] applied the process to create a full body gesture set for the navigational control of humanoid robots. In this paper, we adapt the process by Wobbrock et al. to identify intuitive gestures for an interactive storytelling scenario.

Full Body Gesture Recognition

A lot of work was already done in the field of gesture recognition in general, and different approaches were developed and extensively tested, including statistical clas-sifiers [START_REF] Rubine | Specifying gestures by example[END_REF], Hidden Markov Models [START_REF] Sezgin | HMM-based efficient sketch recognition[END_REF], dynamic programming [START_REF] Myers | A comparative study of several dynamic time-warping algorithms for connected-word recognition[END_REF], and many more. Recognition algorithms of this kind were used in various application areas, although most of those methods are quite complicated to implement, are computational expensive, or need to be trained with a lot of example data [START_REF] Wobbrock | Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes[END_REF]. An effort to make gesture recognition more accessible to developers without a strong background in pattern recognition is the $1 recognizer presented by [START_REF] Wobbrock | Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes[END_REF]. However, this approach was -as the aforementioned ones -still targeted at gestures defined by the movement of a single point in 2D space. It also relied on manual data segmentation, so the start and end of a gesture had to be indicated to the algorithm, either in an implicit way, e.g. by touching the surface of a PDA with the pen [START_REF] Wobbrock | Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes[END_REF], or in an explicit way, e.g. by pressing a button on a Wiimote [START_REF] Kratz | A $3 gesture recognizer: simple gesture recognition for devices equipped with 3D acceleration sensors[END_REF]. There have also been approaches to get rid of those shortcomings by porting them to 3D space [START_REF] Kratz | A $3 gesture recognizer: simple gesture recognition for devices equipped with 3D acceleration sensors[END_REF], or removing the need for manual data segmentation [START_REF] Zinnen | A new approach to enable gesture recognition in continuous data streams[END_REF]. However, the research for those methods is partly still in an early phase, and they are quite complicated to apply for practitioners as well.

The release of the Kinect sensor again motivated researchers to investigate gesture recognition for the more complex motion capturing data provided by full body tracking. The challenge with this data is that it contains multi-point data in 3D space (position and orientation of multiple important joints of one or more users). In addition, there is no obvious way to apply manual data segmentation (no device in the users' hands to press a button on), and the data itself also is rather noisy [START_REF] Khoshelham | Accuracy and resolution of Kinect depth data for indoor mapping applications[END_REF]. On the other hand, this means that the data itself already contains more information as in other interaction modalities. In this way, one single data frame for one tracked user already can be seen as a gesture, or more precisely a posture, as it defines a specific configuration of the user's skeleton. For this reason, researchers developed easy to use techniques for full body gesture recognition in application prototypes. One of the first was the Flexible Action and Articulated Skeleton Toolkit (FAAST) [START_REF] Suma | FAAST: The flexible action and articulated skeleton toolkit[END_REF] that bound gestures defined by simple text scripts to key and mouse events, and therefore enabled to control arbitrary applications via full body interaction. We developed a similar approach in our FUBI framework 24 of which an earlier version was already presented in [START_REF] Kistler | Natural interaction with culturally adaptive virtual characters[END_REF]. It tries to achieve more powerful gesture recognition by giving more complex configuration options in an XML-based definition language. We use this framework to implement the user-defined body gestures in section .

Interactive Storytelling Scenario and Gesture Study

In this section, we describe the scenario of our application and its intended user interaction. We further explain our interaction study and present the results of its analysis.

Scenario and User Interaction

Our interactive storytelling scenario aims to provide intercultural training for young adults (18-25 year olds). The users learn by participating actively in the narrative in which they have to interact with virtual characters from different cultures. However, the characters do not represent real cultures, but synthetic ones as defined by Hofstede [START_REF] Hofstede | Role playing with synthetic cultures: the evasive rules of the game. Experimental Interactive Learning in Industrial Management: New approaches to Learning, Studying and Teaching[END_REF]. The users adopt the role of a character that has not traveled too much for most of his life. The scenario starts at the café of the character's grandmother, in which he receives a letter from his deceased grandfather. In this letter, the grandfather, who liked to travel the world, promises the grandson a "lost treasure" that he should find in a journey through different countries. In each country the grandson has to interact with locals in so-called critical incidents to progress. To be successful, the users have to select the correct interaction options depending on the agents' simulated synthetic culture. The selection itself should eventually be done by performing a corresponding full body gesture. In the final country the users will find out that the promised treasure is the experience that the grandfather had while travelling. The scenario is implemented using the cross-platform game engine Unity3D3

We conducted our interaction study for the introduction in the café and the first country of our scenario which at this state included two critical incidents. The users' first task was to find out the way to his hotel by interacting with people in a bar (first critical incident, cf. and an agent architecture for culturally adaptable behaviors [START_REF] Dias | Fatima modular: Towards an agent architecture with a generic appraisal framework[END_REF]. Fig. 1 left-hand side). In the subsequent incident users had to find the responsible supervisor in a nearby museum in order to receive entry permission for a park (second critical incident, cf. Fig. 1 right-hand side). The scene in the grandmother's café and the mentioned two critical incidents together included the following in-game actions to be triggered by the users: yes, no, sit at bar and wait, approach group, ask for directions, leave the bar, ask about supervisor, ask guard to talk to supervisor, approach supervisor, ask permission.

Study Setup, Procedure, and Participants

The experiment was arranged in a room of about 3 meters width and 6.5 meters depth. The participants were standing at a distance of about 2.5 meters in front of a 50 inch plasma display. A camera was placed in a height of about 1.5 meters left of the display to record the users' front from a slightly tilted view. The participants were told that they should place themselves at the initial position, but that they were still allowed to freely move within the camera's field of view during the study. The experimenter was sitting to the left of the participant and controlling the application running on the display via mouse and keyboard. After a short introduction and a demographic questionnaire that also included a question about the users' experience with body gesture based interaction, the experimenter explained the participants their role in the study. The experimenter ran through the story script of the application and as soon as a user input would have been requested by the application, text boxes with the currently available in-game actions were displayed as overlays on the virtual scene as depicted in Fig. 1. At this point, the participants' task was to invent and perform a gesture for each displayed action, one after the other. The participants were told that they were allowed to use their full body for gesturing, but that the gesture itself should mainly be intuitive for them to trigger the requested action. It should, however, have a semantic relation to the action and not consist of simply pointing towards the action label on screen. To keep the process as reproducible as possible, the experimenter always spoke out the action that the user should investigate next and also gave a short explanation about the meaning of the action to avoid misunderstandings. After performing their invented gesture, the participants should indicate on a questionnaire how easy it was for them to come up with that gesture on a 7-point Likert scale.

22 participants took part in the study including 4 females and 18 males. Their age ranged from 22 to 35 with an average of 26.23 (SD 3.80). All except for one were right-handed. The participants were recruited from our university campus and therefore all had a computer science background. They stated themselves a medium experience with body gesture based interaction of 2.18 (SD 0.85) on a scale from 0 (no experience) to 4 (practically daily usage).

Results

The next chapters depict the results of our study, including our gesture taxonomy, a description of the gesture set, user ratings, agreement scores and time performances. The gesture visually depicts an icon and describes the ingame action in an abstract way.

Gesture Taxonomy.

The recorded videos were analyzed and annotated using the ELAN annotation tools 4 Table 1 to extract the stroke phases of all gestures performed by the study participants for each in-game action. We manually classified all performed gestures according to three dimensions: form, gesture type, and (involved) body parts. Each dimension consisted of multiple items as shown in . The dimensions were based on the taxonomy we had used in [START_REF] Obaid | User-defined body gestures for navigational control of a humanoid robot[END_REF], we only left out the view-point dimension, as our interactive storytelling scenario always kept the user in a first-person perspective and there were no changes in the view-point for this reason. Furthermore, we changed the wording of the gesture type dimension to be closer to McNeill [START_REF] Mcneill | Head and Mind: What Gestures Reveal About Thought[END_REF].

Fig. 2 displays the overall taxonomy distribution for the 251 performed gestures. The frequency of static and dynamic gestures was quite similar. Users tended to perform few deictic gestures, but more metaphoric ones. They only seldom chose two hand gestures, but roughly an equal number of one hand and full body gestures. The gestures we categorized as iconic according to McNeill [START_REF] Mcneill | Head and Mind: What Gestures Reveal About Thought[END_REF] in fact were very concrete, which means that most of them were directly miming the meant in-game action, e.g. approach group was often expressed by actually walking a step forward.

Gesture Set.

We selected suitable gesture candidates for each in-game action as follows:

1. For each action 𝑎 we identified a set 𝑀(𝑎) containing all proposed gestures. 2. The gestures in 𝑀(𝑎) were then grouped into subsets of identical gestures 𝑀 𝑖 (𝑎) with 𝑖 ∈ 1. . 𝑛 𝑎 and 𝑛 𝑎 being the total number of identified subsets for action a. 3. The first candidate 𝑐 1 (𝑎) was determined by the largest subset 𝑀 𝑖 (𝑎), i.e.: 𝑐 1 (𝑎) = 𝑀𝐴𝑋 𝑖∈1..𝑛 𝑎 (𝑀 𝑖 (𝑎)).

(1)

4. The second candidate 𝑐 2 (𝑎) was determined by the second largest subset 𝑀 𝑖 (𝑎): (

) 2
There can been more than one first or second candidate if there are multiple largest subsets, but this was not the case for our data. Further, we only took the second candidate into account if the size of the corresponding subset was at least half the size of the subset of the first candidate. This was done to ensure that the considered candidates always had been performed by a relevant percentage of users. 2 summarizes the gesture candidates for all ten in-game actions. The third column includes the percentage of how often this candidate was performed among all gestures proposed for this action, and the last three columns depict the candidate's taxonomy. A second candidate was taken into account only in three cases (leave the bar, ask guard to talk to supervisor, and ask permission).

The gesture candidates are further exemplified by images of users performing them in Table 3. For the actions yes and no, most users chose a head nod or head shake as gestures. The action sit at bar and wait was in most times represented by actually adopting to a sitting position (=sit down). Similarly, we found gesture candidates that represented the action quite directly for approach group, leave bar, and approach supervisor, in which the users did a step backward or a step forward. For the action leave bar a second gesture candidate was turn away which meant the user actually turned around as if going away. Ask permission was additionally expressed by the gesture tip on shoulder that was chosen because the supervisor -that participants should ask for permission to enter a park -stood there with the back to them (cf. Fig. 1 right-hand side: the virtual character at the back), so they assumed they first needed to get his attention. For the ask actions we often got the gesture arms out that always included moving the arms to an outward position with open hands, often accompanied by raising the shoulders. The only action for which the gesture candidates were pointing gestures was ask guard to talk to supervisor. Participants either chose to point in the direction of the supervisor (point to front), or to point at the guard first and only afterwards to the supervisor (point at one after the other). User Ratings. Fig. 3 depicts the average user ratings for the easiness to invent the gestures for the 10 in-game actions on a scale from 0 (very hard) to 6 (very easy). Error bars represent the standard error. The actions are ordered according to their user rating.

Fig. 3. User difficulty ratings of the 10 actions.

A one-way repeated measures ANOVA revealed that the ratings differed significantly between the different actions with F(9, 21) = 15.90, 𝑝 < 0.01, 𝜂 2 = 0.43. Post-hoc tests with Bonferroni correction showed that the more complex conversational actions were perceived as more difficult to invent a gesture for them. Accordingly, all actions that include asking character(s) something were rated as the most difficult ones. In particular, all those actions were rated significantly lower (p < 0.01) than the actions yes and no. Approach supervisor and leave bar, were also rated significantly higher (p < 0.05) than the ask actions, except for ask permission. Approach group was only significantly higher rated (p < 0.05) than ask about supervisor. We found no significant difference between sit at bar and wait and the ask actions, and there was also no significant difference between the ratings of the different ask actions.

Agreement Scores.

To further investigate the level of agreement among the participants, we calculated an agreement score based on the process defined and used by Wobbrock et al. [START_REF] Wobbrock | User-defined gestures for surface computing[END_REF]. For an action a, the agreement score 𝐴𝑆(𝑎) is defined as:

𝐴𝑆(𝑎) = ∑ � |𝑀 𝑖 (𝑎)| |𝑀(𝑎)| � 2 𝑖∈1..𝑛 𝑎 ) (3) 0 1 2 3 4 5
An agreement score 𝐴𝑆(𝑎) for an action a is represented by a number in the range [1 ⁄ |𝑀(𝑎)| , 1], with a higher value corresponding to a higher agreement, and 1 representing a perfect agreement (all participants chose the same gesture for this action). The overall agreement we got for our action set was 0.329 (SD 0.129). Fig. 4 depicts the agreement scores of the different actions. They are ordered from the action with the highest agreement to the one with the lowest agreement. Similar to the user ratings, more complex actions (i.e. all "ask …" actions) caused lower agreement between the users and we got a large number of different gestures for those.

In fact, the results reveal that the level of agreement between the participants strongly correlates to the easiness to invent gestures (Pearson's 𝑟 = 0.812, 𝑝 < 0.01).

When the participants thought it was easy to find a gesture for an in-game action, more participants chose the same gestures, and in opposite, when the participants thought it was difficult to invent a gesture for an action, we got a higher variation of gestures as well.

Timings.

Table 4 depicts the times it took the users to perform one stroke of the gesture candidates. One stroke means e.g. for the head nod gesture that the user moves the head from the resting position upwards, then downwards under the resting position, and upwards to the resting position again. In other words, one stroke consists of the minimal gesture that can be found when dividing the gesture into equal sub gestures. For static gestures, one stroke consists only of a hold phase in which the user holds the relevant posture until moving back to a resting position. The table enlists the average times as well as the standard deviation, the minimum, and maximum among the different gesture performances.

Integration of the Gesture Set

After collecting the data for our gesture set, we aimed to enable the gestural interaction in our application. In the following, we describe the recognition framework we used to achieve this task, the implementation of the gesture recognizers, and how we integrated them into our interactive storytelling scenario.

Recognition Framework

For implementing the recognition of the gesture candidates in our application, we used our framework for full body interaction (FUBI) [START_REF] Kistler | Natural interaction with culturally adaptive virtual characters[END_REF]. The recognition framework uses the Kinect for Windows SDK 51. Joint orientation recognizers are defined by a minimum and/or a maximum angle for a specific joint.

for getting the depth data provided by the Kinect sensor, and for applying full body user tracking on that data. In this way, we get positions and orientations for 20 different user joints. The joint data is analyzed in the recognition framework for detecting gestures that are defined via XML files. Those XML files first can contain three types of basic gesture recognizers: 2. Joint relation recognizers look at the position of a joint in relation to a second one, e.g. whether and how much a joint is above a second one or how large the distance between those two is.

3. Linear movement recognizers are defined by a specific direction and a minimum and/or maximum speed.

In addition, those three types of basic recognizers can be combined to sequences in so-called combination recognizers. A combination recognizer consists of several states that contain sets of the above mentioned basic recognizers. For each state there is a minimum and maximum duration defined that those recognizers have to be fulfilled for the recognition process to get into and stay in this state. For the transition to the next state there is also a maximum duration defined. Fig. 5 depicts the XML definition for a combination recognizer that is meant to recognize a head nod gesture.

Integration of Gestures in the Application

For implementing the recognizers for the gesture candidates, one first has to determine by which sequence of basic recognizers a gesture candidate can be described and how the recognizers' parameters need to be adjusted. This is done by studying sample videos of the gesture in more detail and approximating the parameters. For determining the time constraints of a combination recognizer, the measured timings as depicted in Table 4 can be used. However, it has to be mentioned that in most cases, the timings cannot be used directly, but should rather serve as a basis for understanding a gesture and its temporal variance between the users.

To implement the gesture candidate head nod for the action yes, we used the XML definition as shown in Fig. 5. The definition includes two different joint orientation recognizers that observe the x angle of the head. Those joint orientation recognizers were combined in a sequence of four states. Note that all states do not have a minimum duration, so it would be sufficient if the recognizers of one state are fulfilled only for one frame of the tracking data stream. However, each state has a maximum duration of 1 second and a time for transition to the next state of 0.4 seconds. That means a head nod could include a head tilted upwards for at most 1 second, and then a movement downwards that lasts at most 0.4 seconds follows. The tilted downwards head could be hold for up to 1 second. After that follows an upwards movement for again at most 0.4 seconds, and the whole sequence needs to be repeated a second time. Therefore, a head nod is recognized as soon as two single nods are detected. This was sufficient for our purpose as we were not interested in distinguishing different kinds or numbers of nods.

The rest of the gesture candidates were implemented in a similar way. We used combinations of joint orientation recognizers for the gesture candidates head shake, sit down, and turn away to check the orientations of the joints included in the gesture. For the candidate arms out we used a recognizer that combines two joint relation recognizers in one state. The first joint relation recognizer observed the left hand and shoulder and waited for the hand to be in a height similar to the shoulder (y difference smaller than 30 cm) and that the hand was at least 35 cm left of the shoulder (using the x coordinate). The second recognizer looked for the same properties with the right hand and shoulder. The gesture candidate point to front as well used a joint relation recognizer for checking the hand position. In addition it ensured that the right elbow was not too far away from the line from the shoulder to the hand (at max 12 cm) and that the hand was not moving with more than 0.5 m/s. The last condition was implemented using a linear movement recognizer. The candidate pointing at one after the other was defined in the same way but with an additional state that allowed hand movement in between the two pointing states. We also implemented the second gesture candidate for the action ask permission that was the gesture tip on shoulder. This was basically realized the same way as point to front, but in addition, it waited for a sequence of linear movements in upward and downward direction of the hand. The gesture candidates step forward and step forward were implemented as linear movement recognizers looking at the torso joint and waiting for a movement in z or -z direction after a short phase of standing still.

The integration of the gestures in the application was done similarly to how we conducted the study. Instead of only displaying text boxes with the currently available actions, we additionally displayed symbols that visualized how the gestures for these actions should be performed. The symbols consisted of either a single image that is displayed constantly or multiple images that are displayed as an animation sequence. The Unity3D integration the FUBI framework only needs a recognizer definition that is named the same way as the image (sequence) that is used as a symbol for it. As soon as the symbol is displayed on screen, the recognition framework automatically checks the corresponding recognizer for users tracked with the depth sensor, and -if the recognition has been successful -it triggers an event related to the symbol. The event for each symbol can be defined in the same way as it is done for default interface buttons in Unity3D. Fig. 6 depicts a scene of the first critical incident, now displaying four symbols of the new gesture set that can currently be selected by the user.

Although we implemented the recognizers for all gesture candidates, we had to decide which of the candidates we would actually use in the three cases in which two candidates were determined. As they seemed to fit a bit better to the parallel and surrounding gestures and because of their partly more reliable recognizers, we chose the gesture turn away for the action leave bar, point to front for ask guard to talk to supervisor, and tip on shoulder for ask permission. We preliminarily tested the integration of the new gesture set in an informal study with the same setup as in the previous study, but this time a Kinect sensor was placed centered below the screen to enable the interaction. Three users took part in the study, two males and one female. They had already participated in the study for the creation of the gesture set, thus they were used to the setting. They again ran through the story script, but this time the gestural interaction was enabled, so they always had to perform one of the requested gestures to continue. We had to omit the part in which the two gestures head nod and head shake appeared due to (later solved) technical issues with the Kinect face tracking SDK. For the rest of the gestures, the detection already worked quite robust. Out of 15 performed gestures, 11 were directly recognized by the first attempt of the user, while four of them had to be executed a second time to be recognized by the system. Thus, we had four false negatives and 15 true positives, which can be summarized to a preliminary recognition rate of 79 %. In addition, most of the false negatives were due to a wrong gesture performance by the participant. This was caused by the fact that the gestures were not explained to the participants, but they had to guess how to perform them by looking at the symbols displayed on screen as depicted in Fig. 6. The system never recognized a wrong gesture, so there were no false positives. Overall, we got positive feedback about the integration, and the participants liked to interact by using the new gesture set.

Discussion

As far as the taxonomy distribution of our gesture set is concerned, we got quite different results in comparison to our previous work that investigated user-defined gestures for navigating a humanoid robot [START_REF] Obaid | User-defined body gestures for navigational control of a humanoid robot[END_REF]. We got much less deictic gestures as in the previous work, but more metaphoric ones. This is due to the target of the gestural interaction. While deictic gestures seem to be more suitable for navigational actions, our action set also included conversational actions that need to be described with metaphoric gestures due to their increased semantic complexity. We also had a closer look at the taxonomy distribution of each action itself that revealed that the users never used iconic gestures for the conversational actions (all ask actions plus yes and no) except for the gesture tip on shoulder of the action ask permission. All other actions -which can be categorized as navigational actions -included all three types of gestures, except for leave bar for which we only observed iconic and deictic gestures, but no metaphoric ones. For further increasing the information content of their gestures, the participants more often used other body parts than their hands in opposite to the robot navigation task. However, they used less dynamic gestures, which indicates that full body gestures often provide enough information in a static version. As full body gestures were overall frequently chosen quite often, it can be said that this kind of gestures is worth to be used in interactive storytelling scenarios in general. It seemed that full body gestures are especially intuitive for triggering the in-game actions that occur in this kind of scenarios.

We proposed to select the gesture candidates according to how many users chose a gesture for one in-game action. However, this does not always have to be the best choice. For example, it makes no sense to give the user the choice between two actions represented by the same gesture at the same point in time. In this case it is better to select a less often chosen gesture candidate for at least one of the actions. There are also other cases in which it is helpful to select a different gesture, e.g. if the recognition software is not able to detect the gesture in a robust way. A more specific reason for doing this is also given in our application. As we aim at intercultural training, we want the users, at a later point in our scenario, to be confronted with gestures that are unfamiliar to them, as this can occur when travelling to different countries. For this purpose it might also be worth to conduct the study with participants of different cultural background to get a different gesture set.

Another challenge we faced was the problem of potentially too complex in-game actions, and especially the difficulty to represent verbal actions with non-verbal gestures. This can be seen in the relatively low agreement scores and user ratings we got for all actions that involve asking virtual characters about something. At the state of the study, our scenario never included multiple ask actions in parallel, so we had no problems with their ambiguous gestures, but as soon as this occurs, it might be necessary to refine the actions for making them easier to be related to gestural input. Otherwise, it might also be necessary to include different kinds of interaction, e.g. using a graphical user interface or other modalities such as speech input.

As described in section 4, it was feasible for us to implement a prototypic recognition logic for the gathered gesture set within our recognition framework [START_REF] Kistler | Natural interaction with culturally adaptive virtual characters[END_REF] and to integrate the gestures in our storytelling scenario. The recorded videos of the gesture performances were very useful for this task as well as the measured timings for the gestures' stroke phases. Nevertheless, the creation of a gesture recognizer based on this data is still a challenging task that has to be done in a careful way to realize the gestures at least close to as they were intended by the users.

Conclusion and Future Work

In this paper, we presented how we produced a user-defined gesture set for an interactive storytelling scenario. We described the full process from conducting the study, over analysis of the data, to the implementation of the gesture candidates. During this process we obtained a taxonomy of full body gestures for our interaction set, user ratings and agreement scores for each in-game action, the time performances of all gesture candidates, and we finally integrated the gesture candidates in our applications using our open source full body interaction framework FUBI [START_REF] Kistler | Natural interaction with culturally adaptive virtual characters[END_REF] . A first validation for FUBI according to accuracy and usability was already done with a different interactive storytelling scenario [START_REF] Kistler | Full body gestures enhancing a game book for interactive story telling[END_REF] that included different kinds of iconic gestures. We plan to conduct a similar study with the new scenario in order to provide a more complete validation, also with more abstract metaphorical gestures. At the moment, we investigated the already implemented first part of our scenario. As soon as more of the application is ready, we want to continue the study with the additional in-game actions. In the meantime, it became clear that our scenario will have interactions that sometimes include multiple conversational actions in parallel that we will probably not be able to represent with unambiguous gestures. Therefore, we plan to include an additional interaction with a graphical user interface for those cases in our application.

Fig. 1 .

 1 Fig. 1. Virtual environment of the two investigated critical incidents.

Fig. 2 .

 2 Fig. 2. Overall taxonomy distribution.

4

  http://www.lat-mpi.eu/tools/elan/ 𝑐 2 (𝑎) = 𝑀𝐴𝑋 𝑖∈1..𝑛 𝑎 , 𝑀 𝑖 (𝑎)≠𝑐 1 (𝑎) (𝑀 𝑖 (𝑎)).

Fig. 4 .

 4 Fig. 4. Agreement scores for the 10 actions.

Fig. 5 .

 5 Fig. 5. XML definition of the recognizer for a head nod.

Fig. 6 .

 6 Fig. 6. Gesture symbols in the first critical incident.

Table 1 .

 1 Full body gesture taxonomy for our interactive storytelling scenario.

		static gesture	A static body gesture is held after a preparation phase.
	Form	dynamic gesture	The gesture contains movement of one or more body parts during the stroke phase.
		one hand	The gesture is performed with one hand.
	Body parts	two hands	…with two hands.
		full body	…with at least one other body part than the hands.
		deictic	The gesture is indicating a position or direction.
	Gesture type	iconic	The used gesture visually depicts the meant in-game action or a part of it directly.
		metaphoric

Table 2 .

 2 Gesture candidates for each action.

	In-game action	Gesture candidates	Occurrences Form	Gesture type	Body parts
	yes	head nod	68%	dynamic metaphoric full body
	no	head shake	68%	dynamic metaphoric full body
	sit at bar and wait	sit down	56%	static	iconic	full body
	approach group	step forward	56%	dynamic iconic	full body
	ask for directions	arms out	34%	static	metaphoric two hands
	leave bar	turn away step backward	45% 27%	dynamic iconic dynamic iconic	full body full body
	ask about supervisor arms out	50%	static	metaphoric two hands
	ask guard to talk to supervisor	point at one after the other point to front	38% 21%	static static	deictic deictic	one hand one hand
	approach supervisor step forward	56%	dynamic iconic	full body
	ask permission	arms out tip on shoulder	23% 19%	static dynamic iconic metaphoric two hands one hand
	Table

Table 3 .

 3 User images of the gesture candidates.

	head nod	head shake	sit down

step forward (from left to right) / backward (from right to left) turn away arms out point to front (left image) / to one after the other (both images) tip on shoulder

Table 4 .

 4 Times for one stroke of the gesture candidates.

	in-game action	gesture candidates	mean time	SD	MIN MAX
	yes	head nod	0.822 0.374 0.420 1.560
	no	head shake	0.687 0.233 0.474 1.420
	sit at bar and wait	sit down	0.661 0.635 0.120 1.970
	approach group	step forward	1.508 0.702 0.422 2.770
	ask for directions	arms out	0.707 0.392 0.295 1.375
	leave bar	turn away step backward	1.738 0.908 0.557 3.400 2.041 0.593 1.295 3.000
	ask about supervisor	arms out	0.589 0.341 0.200 1.280
	ask guard to talk to supervisor	point at one after the other 1.013 0.257 0.540 1.410
		point to front	0.560 0.558 0.130 1.625
	approach supervisor	step forward	1.444 0.659 0.517 2.470
	ask permission	arms out tip on shoulder	0.759 0.578 0.190 1.770 0.356 0.066 0.257 0.410

http://www.xbox.com/KINECT

http:/www.hcm-lab.de/fubi.html

http://unity3d.com

http://www.microsoft.com/en-us/kinectforwindows/

Acknowledgments. This work was funded by the European Commission within FP7 under grant agreement eCUTE (FP7-ICT-257666). We would like to thank our project partners for the collaboration, and especially Nick Degens, Hélio Mascarenhas, Samuel Mascarenhas, and André Silva for their work on the storyboard, agent architecture, and asset creation that formed the background for our study.