Luca Corcella
email: l.corcella@isti.cnr.it

Marco Manca
email: m.manca@isti.cnr.it

Fabio Paternò
email: f.paterno@isti.cnr.it

Carmen Santoro
email: c.santoro@isti.cnr.it

A Visual Tool for Analysing IoT Trigger/Action Programming

Keywords: Trigger Action Programming, Visual Analytics, Log User Interaction, Internet of Things Applications

The Trigger-Action programming paradigm has been widely adopted in the last few years, especially in the Internet of Things (IoT) domain because it allows end users without programming experience to describe how their applications should react to the many events that can occur in such very dynamic contexts. Several end user development tools exist, in both the research and industrial fields, which aim to support the increasing need to specify such rules. Thus, it becomes important for application developers and domain experts to enrich such environments with functionalities able to monitor how users actually interact with such rule editors, and show useful information to analyse the end user activity. In this paper, we present a visual tool for monitoring and analysing how users interact with a trigger-action rule editor. The goal is to provide a tool useful to better understand what end users' personalization needs are, how they are expressed, how users actually specify rules, and whether users encounter any issues in interacting with the personalization features offered by the editors. The proposed solution supports the analysis through a dashboard and a set of timelines describing the actual use of the personalization tool, with the possibility to select specific events of interest. It moreover provides data useful for understanding the types of triggers, actions and rules actually composed by users, and whether they effectively exploit the personalization features offered.

Introduction

A consequence of the rapid spread of the Internet of Things (IoT) is that the environments where we live and act are increasingly characterized by the presence of a multitude of interactive devices and smart objects interconnected with each other. Since we interact with our applications in such very dynamic and unpredictable environments, it is not possible to foresee at design time how an application should react to all the possible contextual changes that can occur during its use. Only end users can know the most appropriate ways their applications should react to dynamic contextual events. For such reasons, in order to obtain applications able to adapt to the context of use in an effective way it becomes important to allow end users themselves to 'program' the behaviour of their applications.

In this trend, trigger-action programming has emerged as a useful and intuitive approach. Users can personalise the application behaviour through sets of rules indicating triggers and consequent effects. Triggers can be instantaneous events (corresponding to context changes) and/or conditions that, if satisfied, activate the execution of specific actions. This type of approach has stimulated several contributions both from the research [e.g. 2, 3, 4, 5, 9] and industrial viewpoints (IFTTT, Tasker, Zapier, Resonance AI, …). This approach can be adopted in many domains that share the need for supporting tailoring of applications that would benefit from considering the occurrence of dynamic events (smart retail, remote elderly assistance, smart home, industry 4.0, …). Although seemingly intuitive overall, because it mainly asks users to indicate the relevant events and desired actions, sometimes identifying the relevant concepts and understanding how to specify them using the tool is not always clear to people without a programming background. For example, Huang and Cakmak [START_REF] Huang | Supporting mental model accuracy in trigger-action programming[END_REF] found that users may encounter difficulties interpreting the differences between events and conditions or between action types, and such misunderstandings can cause undesired behaviours.

In this area, IFTTT 1 (If This Then That) has been particularly successful. It has more than 320,000 automation scripts (called "applets") offered by more than 400 service providers. The applets have been installed more than 20 million times, and more than half of IFTTT services are IoT device-related [START_REF] Mi | An empirical characterization of IFTTT: ecosystem, usage, and performance[END_REF]. One large repository of IFTTT rules is even publicly available [START_REF] Ur | Trigger-Action Programming in the Wild: An Analysis of 200,000 IFTTT Recipes[END_REF]. Thus, we can foresee in the near future an increasing interest in environments allowing people to provide many rules to personalize their context-dependent applications.

In this perspective, the availability of tools able to analyse how users actually try to personalise their context-dependent applications with such approaches can become very useful, not only for developers of trigger-action authoring environments, but also for IoT application developers and domain experts. We have thus considered previous work in the area of analytic tools for Web site usability, which often log user interactions in order to support identification of potential usability problems. However, the application of such tools to analyse the use of trigger-action programming would not provide the most relevant information. Indeed, differently from existing tools that exploit log analysis for usability evaluation purposes, in this case the goal is not strictly to understand whether there is some bad user interface design, but rather to see how the personalization needs are expressed by users, and whether they have some conceptual problems in expressing them in terms of trigger-action rules.

In this paper, after discussion of related work, we introduce the trigger-action programming environment considered in this study. Then, we discuss the design requirements for the novel analytic system for IoT programming platforms, describe the functionalities supported by the current version, and report on a first user test. Lastly, we discuss the user test on this tool, draw some conclusions, and provide suggestions for future work.

Related Work

Our work draws from research on trigger-action programming for IoT applications and tools for visualizing logs of Web interactions.

Trigger-Action Programming

Both in research and industrial fields there has been interest in the trigger-action programming to allow users to define their own adaptation rules. From the research perspective, we started our study from the TARE [START_REF] Ghiani | Personalization of Context-Dependent Applications Through Trigger-Action Rules[END_REF] trigger-action rule editor that provides the possibility to create rules more flexibly than IFTTT since they can be created as compositions of multiple triggers and actions. In this area, Desolda et al. [START_REF] Desolda | End-user development for the internet of things: EFESTO and the 5W composition paradigm[END_REF] developed EFESTO, a visual environment that allows users to express rules for controlling smart objects. The followed paradigm is based on the 5W model, which defines some specification constructs (Which, What, When, Where, Why) to build rules coupling multiple events and conditions exposed by smart objects, and for defining temporal and spatial constraints on rule activation and actions execution. Coutaz and al. presented AppsGate [START_REF] Coutaz | A First -Person Experience with End -User Development for Smart Homes[END_REF], an EUD (End-User Development) environment designed to empower people with tools to augment and control their home. AppsGate aims to support different activities such as monitoring the home state and programming its behaviour in a context-dependent manner. Another similar approach is ImAtHome [START_REF] Fogli | ImAtHome: Making trigger-action programming easy and fun[END_REF], an iOS application built over Apple HomeKit allowing home inhabitants without programming skills to control home automation by means of creating scenes and rules for defining the complex behaviour of a smart home. The approach proposed by [START_REF] Nandi | Automatic Trigger Generation for Rule-based Smart Homes[END_REF] aims to address some specific issues of users when writing trigger-action programming rules by a tool, called TriGen, aimed at preventing errors due to too few triggers in the rules by statically analysing a rule's actions to determine which triggers are necessary. Another approach to address some of the issues that users encounter when they specify ECA (Event, Condition, Action) rules is reported in [START_REF] Terrier | CCBL: A Language for Better Supporting Context Centered Programming in the Smart Home[END_REF]. Still in this area Metaxas and Markopoulos [START_REF] Metaxas | Natural Contextual Reasoning for End Users[END_REF] propose a context-range editor supporting end users formulate logical expressions regarding the context, define the concept of affinity regrouping heuristics, and present the mechanisms to apply it throughout the contextual ranges of the involved services. The semantic information that the services disclose lets the editor recognize this affinity and allows it to group terms in logical expressions when they refer to the same aspect (e.g. user's activity).

The monitoring and visualization method proposed in this paper can be useful to analyse the user interactions with these tools as well, since they still support triggeraction rules for IoT applications, and thus their use is characterised by similar usergenerated events such as trigger selection, action selection, rule saved.

Web Analytics Tools

One typical use of the information contained in logs of user interactions is for usability studies. Palmer [START_REF] Palmer | Web site usability, design, and performance metrics[END_REF] presents different metrics for measuring usability, and lists different types of methods to evaluate a user interface. UsaProxy [START_REF] Atterer | Knowing the user's every move: user activity tracking for website usability evaluation and implicit interaction[END_REF] exploits a proxy-based solution to access remote Web pages: the proxy adds some JavaScript code to specify the listeners which log the user interaction with the concerned page(s). The output produced by the proxy is a simple list representing the IP address of the connected device, the visited pages, and some events' description, without any particular visualization able to support their analyses. MUSE [START_REF] Paternò | Customizable Automatic Detection of Bad Usability Smells in Mobile Accessed Web Applications[END_REF] also exploits a proxy server in order to insert in the target Web pages some code to log user interactions. The logged events are shown in a timeline representation in which it is possible to compare a timeline representing the 'optimal' interaction with the one expressing the 'real' user interactions in order to help designers to discover some usability issues in the user interaction. WELFIT [START_REF] Santana | WELFIT: A Remote Evaluation Tool for Identifying Web Usage Patterns through Client-Side Logging[END_REF] is a tool to identify usage patterns based on client-side event logs and by presenting event stream composition characteristics. The system records usage data during real use, identifies usage patterns, and indicates potential user interface design problems. Harms and Grabowsky [START_REF] Harms | Usage-Based Automatic Detection of Usability Smells[END_REF] proposed to transform the recorded user interaction in task trees that are then checked to identify usability issues. The goal of such contributions is to identify a method to record user interactions and then further analyse the logs in order to highlight usability problems. HistoryViewer [START_REF] Segura | HistoryViewer: Instrumenting a Visual Analytics Application to Support Revisiting a Session of Interactive Data Analysis[END_REF] is a system that aims to support exploration of log data obtained from user interactions. In this case the goal is to support final users for communication purposes, and not usability evaluators, by describing the interactions that took place in a way they can recall and communicate their own discoveries about the data, not focused on the interaction mechanisms or on difficulties they may have encountered. Differently from such proposals, in this work we focus on providing designers of trigger-action rule editors and IoT application developers with interactive visualisations supporting exploration and filtering of the logged relevant interaction data, so as to derive higher-level information such as the types of rules that users were interested in creating with the tool, the most popular trigger and action types used, and the types of usage patterns followed by users while interacting with the tool. The visualizations offered also allow them to rapidly identify whether the personalisation rules composition was straightforward or the users had to go through various possibilities before completing their tasks. Moreover, by analysing the users' log it is possible to understand what the personalization needs are, how they are expressed by the users, whether their rules actually support the desired results, if the personalization features offered by the editors are sufficient.

3

The Proposed Approach

In this section we first introduce the trigger-action programming environment that has been considered in this study, and then we report on the initial set of requirements that have been identified for the visualizations to provide. Lastly, we describe how it has been instrumented in order to obtain relevant and meaningful log file for further analysis.

The Trigger-Action Rule Editor

In this study we considered the TARE platform (Trigger Action Rule Environment) [START_REF] Ghiani | Personalization of Context-Dependent Applications Through Trigger-Action Rules[END_REF], which allows users to define their trigger action rules in an intuitive way. The tool is flexible in the order in which users can specify the rules (they can start either from triggers or from actions), they can also re-use a previously defined rule in order to create a new one. Moreover, they can combine multiple triggers by using the Boolean operators AND and OR. The Not operator is also supported to check whether some event has not occurred in a specific period of time. Figure 1 shows the context hierarchy of the Rule Editor: users navigate by first selecting the context dimension to which the considered trigger belongs, and then they go through the associated context categories (and sub-categories), which group together logically related context elements. The leaves of the context hierarchy refer to context entities, and are used to specify triggers and their parameters. For instance, in Figure 1 the trigger is "IF the user is close to the kitchen", and it involves the context entity "user". Also the possible actions are indicated through a classification based on the type of effects they allow users to achieve (changes to the user interface, changes to the state of some appliances, sending reminders or alarms, …). In addition, TARE provides users with the possibility to search for a specific trigger, by specifying a concept to search for in the hierarchy of the possible triggers. On the left side there is also an interactive panel that indicates the main steps in the workflow associated with rule editing, and continuously provides feedback on what has been done, indications of what can be done from the current state, and thus it can be used to control and activate the various parts of the rule editing.

The Requirements for IoT Trigger-Action Programming Visual Analytics

TARE has been validated in three different trials, which involved 58 users in total.

During the tests, users, who had no prior programming experience, had to perform various tasks concerning the specification of personalization rules with different complexity in terms of number of triggers and actions.

In particular, the rule editor has been used in two interdisciplinary projects in the area of Ambient Assisted Living. Such projects involve different types of organizations: application developers, technology providers, medical institutions, elderly organizations. Thus, the projects' meetings and discussions were useful to identify possible requirements for the relevant visual analytics tool. In addition, the papers published in the area of trigger-action programming were considered as well, in particular for the part concerning how they have been empirically tested. Thus, by observing the results in the trials and considering previous work in the area, we identified some features required in an environment to support the analysis of how users define context-dependent personalization rules.

One important general requirement for a visual analyser is the availability of interactive data exploration: the tool should provide users with different zooming levels, as well as the possibility to select individual items and get specific details on demand. In our case, it should also provide different interactive features to enable users to focus on different aspects of trigger-action programming. For the type of tool considered in this study, relevant information includes: the most recurring/frequent context entities used in rules; the most recurring combinations of trigger types and action types used in created rules; the most recurring sequences of usage patterns logged, the most used operator to connect multiple triggers (AND/OR), etc. Indeed, this type of information can be useful to better understand also the aspects that users prefer to consider in the specification of their personalization rules.

Other useful information that the tool should provide, for each user and also across users, is:  The rule part users prefer to start editing the rules from (triggers and then actions or vice versa), in order to provide more flexibility in the rule editing;  The sequence of trigger/action dimensions and entities that users have passed through to reach the trigger/action leaf of interest, in order to assess whether the proposed logical organization of the contextual aspects is intuitive for end users;  The time spent to create a rule (max, min, average);  The time spent to modify a rule (max, min, average);  The number of rules created in each session and in all sessions;  The number of triggers/actions created in each session and in all sessions;  The number of rules/triggers/actions that users started to edit without saving them;  The number of simple/complex rules created by the users (simple rules are rules involving only one trigger, while complex rules are rules which involve a combination of multiple triggers).

We also judged it useful to provide designers with the possibility to filter the results to allow users to configure the list of events they want to focus on, as well as to provide further quantitative information, such as the context dimension and the trigger entity that have been most used in the defined triggers. Such summarized representations of the users' sessions are particularly useful when the number of events becomes very high and difficult to manage.

The Logs

To support the identified requirements, we had to identify the events that were meaningful to log. We decided to exclude some low-level events to log (such as mouse over, mouse out, blur) that were judged not particularly relevant for the type of planned analysis. We have focused only on the interactions with the trigger and the action hierarchy, and on the editor parts which manage the rules. The logging implementation was done by a JavaScript file which appends handlers to the relevant events supported in TARE and related to rule creation, editing, saving. In particular, we found it useful to log user's selections of:

 "New Rule", "Save Rule", "Save Rule as", "Edit Rule" and "Delete Rule" buttons (used to manage rules);  "Triggers" and "Actions" buttons (used to go to the part of the tool dedicated respectively to trigger and action specification);  "AND" and "OR" buttons (used to compose two triggers);



4

The Tool Visualizations

In this section we show and discuss the two types of visual information provided.

The Dashboard

When the tool is accessed, it shows a dashboard (see Figure 2). On the left hand side, it is possible to select a specific user who interacted with the rule editor, then the dashboard shows an overview of the activities carried out by the considered user: the total rules, triggers and actions created in all interaction sessions, the number of rules which had been modified; the rules that have been specified (described in natural language), the context dimensions involved in the rule editing (see the pie chart on the right showing the percentage of triggers for each contextual dimension), the most used triggers and actions grouped by dimensions and the time of each working session. By "session" we mean the interval of time between a user's login and a user's logout from the system (or the system automatically does a logout after two hours of idle time).

In addition, the tool shows information about the number of rules that have been saved and not (see the bar charts visualised in the bottom part and clustered by session). The unsaved rules are those that the user started to edit and never saved. Such bars are interactive and the user can select each bar to get the details of the concerned rules (e.g. the names of the rules saved). For each trigger/action dimension there is a section which shows the name of the context entities and the number of times they have been used in all defined rules. The dashboard also provides indication of how long each session lasted overall and also how long it took to create and save a specific rule.

The Timelines

In order to visualize the sequence of relevant events logged during users' activities in a simple manner, we decided to use a dynamic timeline visualization (see Fig. 4), which provides a time-dependent overview of the relevant events that occurred. In particular, when the Timelines tab is selected, it is possible to see for the currently selected user a set of timelines, each one presenting the list of events recorded in an interactive session.

In the timelines, each event is identified by a label describing it and including the name of the associated event and the corresponding value: such a value is shown only when this is applicable, e.g. for the leaves of the hierarchy. The timeline is thus a monodimensional visualisation, in which the X axis represents time, according to which the events are ordered, thereby making it easy to see when they occurred. Each timeline corresponds to a user's session. It is worth noting that the events are automatically rearranged in a vertical manner by the library used 3 according to the current level of zoom and the number of labels to visualise, in order to avoid overlapping between labels. However, due to the large number of events that can be recorded, the timeline could be difficult to interpret (see for example Figure 4). For this reason, in addition to the possibility to zoom the timeline in and out, the tool provides users with a functionality that allows them to select only the events of interest, so that only specific types of events will be shown in the final visualization (e.g. the events that involve selecting a specific type of context dimension). In addition, in order to allow users to better perceive the differences between the various types of events, they can select a specific color to assign to each of them (see Figure 3). Furthermore, it is worth noting that the list of triggers from which this selection is carried out only shows the actual triggers that occur in the timeline and not all the possible triggers that are potentially available in the system. Another option that was provided in the tool as filtering support is the possibility to save the current configuration of events that are of interest for the user, in order to be ready for later use, also allowing users to name such configuration in a meaningful manner. This is done to enable users to retrieve and load this configuration more quickly and effectively later on. Indeed, sometimes an effective filtering process could involve multiple iterations (e.g. the user progressively refines the current set of events of interest so as to better focus on the information s/he currently judges as important). In addition, different visualisations (obtained through different sets of filters) might have different goals. Therefore, this feature allows users to save time and have the intended visualisations ready for use, instead of doing this process over from scratch whenever they access the tool. Figure 4 focuses on a portion of a timeline obtained when the user created two simple rules: a first rule called "motion-lamp", whose specification is "when there is motion, do turn on the light in the living room", a second rule called "spd-temperature-alarm" whose specification is the following: "when body temperature is more than 30OC, do send an alarm by sms to caregiver". As you can see from Figure 4, the associated timeline becomes very crowded without any filtering: this makes interpretation of the succession of events occurring in the timeline problematic. Nonetheless, in spite of this, the timeline shown in Figure 4 is still able to provide some information that could be relevant for the analyzer. For instance, the timeline highlights the time interval in which the sequence of events associated with a specific rule composition occurred, by displaying an additional red line under the X axis of the timeline. This red line represents a useful cue because it can be used as a reference when performing zooming and filtering operations, and thereby allow users to better focus on how the creation of that rule was actually carried out. Other information that the timeline shown in Figure 4 provides although at a coarse grain is the parts of the timeline in which the interactions were more (or less) close to each other, which could be relevant information for the analyzer. In the example shown in Figure 4, closer interactions occur in the first part of rule building, whereas more distanced interactions occur in the vicinity of critical actions such as saving actions or saving rules (the user takes more time to think about the rule before actually saving it). However, in some cases the user may want to further investigate the succession of events occurred in a specific time interval, and observe them at a finer level of detail. Figure 5 shows a visualization obtained from the timeline shown in Figure 4, by applying some filtering in order to hide events that were judged uninteresting for the user. In particular, the filtering done for obtaining the visualization shown in Figure 5 only displays events associated with TriggerDimension, TriggerEntity, SaveTrigger, Action-Category, ActionEntity, SaveAction and SaveRule events (which are also visualized in Figure 3). This is to better understand the interactions occurring in the timeline visualized in Figure 4 (in particular, for building the first saved rule named "motion lamp").

From Figure 5 it is possible to see that after exploring several context dimensions without saving any trigger, the user focused on the "environment" dimension, and then selected "motion". Then the user, after considering on various action categories ("All", "Kitchen"), focused on the action category "Living room". Within that category the user finally selected the action entity "Hue color light living room". This latter event was followed by a "Save action" event, and then a "SaveRule" event, meaning that the rule was actually saved. Another interesting type of analysis that can be done concerns the use of the search functionality included in the rule editor. For example, frequent use of the search functionality would suggest that the user found it quicker to search than to explore the topdown hierarchy of triggers. Furthermore, it could be useful to compare what users specify in the "search" field and what they actually used afterwards in the rule specification; or analyse whether, after searching for an element, users actually found that element of interest and proceeded with the rule specification or they needed to repeat this process multiple times (and which terms they used in such repetitions). In addition, when a specific trigger element is repeatedly searched for by the same user instead of exploring the hierarchy, this could be a sign that the position of that concept within the hierarchy is not very logical according to the user's model and therefore not easy to locate.

A First User Test

In order to assess the usability of the tool a first user test was organized. In particular, we wanted to investigate whether the tool was able to provide usable representations of the interactions carried out by end users with the TARE rule editor, in order to derive useful information about its usability and usefulness.

Participants

A set of 10 participants fully gender-balanced (5 females) with age ranging between 24 and 49 (mean=30.4, median=26, std. dev.=7.6) were involved in the study. They were volunteers recruited through mailing lists. As for the education of participants, 1 user held a High School degree, 5 a Master Degree, 1 a PhD and 3 a Bachelor. Half users (50%) had already used a visual analytics tool before the test. They mentioned Res-cueTime, Google Fit, Strava, Elevate, WUP, Wireshark, Matlab (visual toolbox), Apache log viewer, Kibana, Google Fusion Tables, Google Analytics, TensorBoard.

Half of users (50%) had some familiarity with developing applications exploiting sensors. One user developed a system providing a geo-referenced visualization of on fine dust particles of polluting nature, and a system providing a visualization for data associated with a remotely operated submarine vehicle. A user developed a domotic application, another one mentioned both a system for motion detection exploiting a video camera, and an application for measuring light brightness (using a photodiode sensor). A cross-device application using a Kinect sensor and an application used for remotely monitoring the activity of elderly people were the applications mentioned by other two users. This type of expertise seems relevant because the visual analytics tool is oriented to analysers interested in investigating how IoT services have been personalized through trigger action rules by end users.

Test Organisation

The test was done in a laboratory. During the test, a moderator observed the participants interacting with the tool, annotating whether and how they carried out tasks and also any further relevant issue and remark. The test was organised in four phases: introduction and motivations, familiarisation, test execution, questionnaire. In the test the users were first introduced to the motivations and main functionalities of this work and of the trigger-action rule editor through a PowerPoint presentation, which also contained a video showing an example of use and application of the TARE tool, which was done to better support people unfamiliar with that tool. Next, users had to create a trigger-action rule of their choice (using the TARE tool) and then, through the log visualizer they had the possibility to get familiar with the corresponding timeline visualization. Then, they had to access the visualizations related to a given user in a particular session. This was selected beforehand for the test and was the same for all users. Users had to accomplish a list of tasks involving the dashboard and the timelines associated with that particular user in that specific session. In particular, the selected session provided data of a user who initially explored the hierarchy of triggers and action without saving any rules and then the user was able to save two rules: the first one involved a time interval longer than the second one, which was characterised by interactions carried out in shorter time range. We judged it more relevant to provide users with relevant log data created by others, which would better reflect more realistic situations of designers/developers analysing the data generated by other users.

Tasks. The tasks to accomplish covered both the information provided by the dashboard (mainly, information about the most used triggers and actions, the most used context dimensions, the number of rules that was successfully saved by users, the time associated with the various user sessions, etc.). In particular, for the dashboard they had to:

 Task1: Provide the name of the session with the highest number of rules created  Task2: Provide the name of the most used context dimension  Task3: Provide the name of the most used action dimension  Task4: Provide the name of the longest session As for the timelines, users had to analyze the time-dependent visualization of the occurred interactions, even using some filtering to better focus only on the most meaningful information. In particular, regarding the timelines users had to:

 Task5: Set a filter to visualise only newRule and saveRule and then answer what they could derive from the visualised information  Task6: Add further filters (also using different colours) involving TriggerDimension, CancelTrigger, SaveTrigger. Then users had to indicate whether and where in the session the user had problems and why.

The test moderator provided users with a PowerPoint slide in which all the tasks were visualised. Users were instructed to solve the tasks by answering the associated questions, by verbally communicating them to the moderator who wrote them down for future analysis. At the end they had to fill in a questionnaire aiming to assess various aspects. They had to provide first some personal information, whether they had previous experiences with visual analytics tools and with applications using sensors. Then, on a 1 (worst) to 7 (best) Likert scale they had to assess usability and utility of the dashboard, the timelines, and the filtering feature. They also had the possibility to provide comments and suggestions, and indicate three positive and three negative aspects in the tool proposed.

The tasks submitted to users were identified in such a way to cover some typical information that could be useful for analysing user interactions with the personalisation editor. In particular, the first four tasks (Task1-Task4) involved the identification of very specific information which was available in the visualisations shown to users: in order to carry out such tasks users had to provide the moderator with an answer to the questions associated with the tasks (e.g. provide the name of the longest session). The last two tasks were more open-ended, and implied analysing and reasoning on the provided data in order to derive more general conclusions. Since Task 6 was mainly a refining of Task5, it was needed to be presented after Task5. For this reason, we decided to follow the same order for all users in presenting all the tasks to them.

Results

Tasks. All the participants were able to successfully provide the correct answer to the first four tasks associated with the information provided by the dashboard. Nonetheless, they provided further suggestions for improving the dashboard. For example, more than one user suggested to exploit a more consistent use of colours in the dashboard, another user suggested including a scrollbar in the panel showing the sessions associated with a specific user in order to avoid bringing about a too long page in case of a user having many sessions associated. Regarding Task5, all users except two were able to identify that there were some rules that were not saved at the beginning of the considered sessions. This was probably due to the fact that the visualisation showed a red line in correspondence with rules that were actually saved and therefore the two users just focused on this highlighted information. Regarding Task6, as a consequence of the further refining asked in this task, all the users were able to identify that the concerned user at the beginning of the session did not save any rule, while at the end of the session the user was able to save two rules. The most interesting part of this task was the different motivations users gave to that behaviour. On the one hand some of them interpreted this behaviour as an actual issue (e.g. "the user did not find what she was looking for and therefore was not able to save any rule"). On the other hand, other users interpreted this behaviour as just the user's need of first exploring the hierarchy for better familiarisation (even saving just some triggers), without necessarily saving any rule. One user also focused on the saved rules (instead of the unsaved ones, as most users did), especially noticing the different time slots needed for saving the two rules and the closer interactions occurring for creating the second rule, deriving that the second rule involved a quicker and more straightforward interaction.

Questionnaire. In the post-task feedback, participants provided positive feedback regarding the tool, and found it easy to perform the tasks. Across all questions, the median ratings were at or above 5.5 on a 7-point Likert-scale (7 = best), as it is possible to see from the following rated aspects:

 Usability of the dashboard (median: 6)  Usability of the timelines (median: 5.5)  Usefulness of information provided in dashboards (median: 7)  Usefulness of information provided in the timelines (median:6)  Usability of the feature for setting filters (median: 7)  Usefulness of the approach for understanding how users exploited the personalisation tool (median: 7)

An overview is presented in the following stacked bar chart. Moreover, participants also answered a series of open-ended questions, whose answers have been detailed in the following.

Do you have suggestion to improve the dashboard usability?

One user suggested grouping rules not by session but per day, in order to have more meaningful information for the user. In addition, another user suggested adding the possibility to hide/show some parts of the dashboard in order to allow users to better focus only on the relevant information. The same user also suggested adding further information for better identifying the sessions (e.g. day, hour), which are currently just subsequently named (e.g. session1, session2).

Do you have suggestion to improve the timeline usability?

One user suggested putting the most recent timeline in the top part of the user interface (whereas now the timelines are visualized following a chronological order, with the most recent ones in the bottom part of the window). Three users complained about the way the labels were vertically visualized. Since the lines going from the labels to the X axis of the timeline are positioned in the center of the label, a user suggested moving such lines in the left-most side of the label (in a sort of a flag-based shape) as a way to better visualize the order of the events while reading the labels. Another user asked for more space dedicated to each single timeline (currently more than one timeline is visualized in the same window). This is to have a better overview of the timeline, while at the same time avoiding scrolling up the visualization vertically, with the risk of losing the overall context of the occurred interactions.

Do you have any suggestion to improve the dashboard utility?

Only a few users provided suggestions for improving this aspect. For instance, one user suggested that the information associated with triggers used should be put under 0,00% 20,00% 40,00% 60,00% 80,00% 100,00%

Usability of dashboards

Usability of timelines

Utility of dashboards

Utility of timelines

Usability of filters

Utility of the approach

Summary of Ratings

Rating=1 Rating=2 Rating=3 Rating=4 Rating=5 Rating=6 Rating=7 the pie chart showing the percentages of used triggers, since it represents a more detailed refinement of the pie-chart -based visualization.

Do you have suggestion to improve the timeline utility?

Several users did not have any particular suggestions for improving this aspect. A couple of users questioned about the utility of showing some information in the labels, for instance the value that was set for a particular trigger (e.g. "Save Trigger 26").

Do you have suggestion to improve the usability of the filtering features?

Most users appreciated this functionality. Just one user noted that it was not possible to move the window supporting the selection of filters, thus this window could cover the underneath timelines.

Is there any information that you did not find in the tool and you would have had? One user suggested having further information about users (e.g. a picture) in order to better identify them. Another user said that it could be better adding the option to have the whole window dedicated to the visualization of a specific timeline of interest, so as to have more space for a more convenient visualization. Another user suggested adding in the dashboard a pie chart for visualizing the used actions, along the same line of the one dedicated to triggers used.

Please indicate three positive aspects you found while using the tool Among the aspects that users mentioned as most appreciated there was the dashboard. It provides the users with an overall view of the interactions done by users, understand where the user found potential difficulties (e.g. in terms of longer time, higher number of interactions). Many users appreciated the possibility of filtering the timelines to better focus on a subset of events of interest.

Please indicate three negative aspects you found while using the tool Some users complained about the clarity of some labels and how some data were presented. Another aspect regarded the limited space devoted to the visualization of the timeline of interest. Another aspect that was mentioned was the need of vertically scrolling the timelines to get all the relevant information. A user complained about a non-perfectly consistent use of colors (for instance the same color was used in different parts of the dashboard although no specific relationship was held between the associated information).

Do you have further suggestion to improve the tool?

A user suggested adding further interactivity in the information visualized in the dashboard, so that it is possible for the user to refine and interactively explore the information at various levels of detail. Another user suggested improving the timeline visualization, even grouping together some data for the goal of obtaining lighter and clearer representations. Another user mentioned providing users with the possibility to order some information according to various criteria. For instance, each session could be ordered in terms of duration but also in terms of number of rules saved, etc.

Discussion. The results of the user test were overall encouraging and promising. Users appreciated the type of information provided by the tool and the supported features, and gave us useful and relevant feedback for further improving them.

In general, the users were able to successfully carry out the submitted tasks, showing that the tool is easy to learn since they used it for the first time and the learning phase was minimal. In addition, it was interesting to note how different users explained in slightly different manners the same information represented in the timelines, especially when carrying out Task6, which was the most open-ended task. Some users did not interpret the unsaved rules as a usability issue but just as a sign of users' need to get familiar with the tool and the hierarchies elements, or that some users preferred to restart from scratch instead of editing the elements appearing in the current rule state. Others, instead, interpret such data as a sign of users not finding what they were looking for in the hierarchies. In this regard, the filtering features seem very useful in effectively supporting further exploration and investigation of the information of interest.

Conclusions

In this paper we present a method and the features of a supporting tool for analysing the users' behaviour when interacting with a trigger-action rule editor for personalising their IoT context-dependent applications. We discuss the most relevant features for this analysis, provide example visualizations that can be supported, and report on an initial user test.

While in this work we applied the approach to a specific tool (TARE), the type of analysis of the users' behaviour presented can be easily extended and applied to other tools supporting trigger-action programming of IoT context-dependent applications. Indeed, all such tools share a number of key logical concepts on which the visualisations are centred (triggers, actions, rules, trigger operators,). Regarding the timelines, by logging the relevant user actions it is possible to derive the usage patterns exploited by users while interacting with the tool. Future work will be dedicated to extending the functionalities of the visual analytics tool taking into account the feedback from its use, and carrying out further, more longitudinal, empirical validations.

Fig. 1 .

 1 Fig. 1. The TARE editor user interface.

Fig. 2 .

 2 Fig. 2. The dashboard presenting summary information.

Fig. 3 .

 3 Fig. 3. The Log Filter: only the events of interest have been selected by the user.

Fig. 4 .

 4 Fig. 4. The timeline associated with the considered example, as initially visualized.

Fig. 5 .

 5 Fig. 5. Timeline obtained by applying filtering to the timeline shown in Figure 4.

Fig. 6 .

 6 Fig. 6. Stacked bar chart showing ratings assigned to various aspects of the tool.

  From the commercial point of view IFTTT is one of the most used application. It provides mechanisms to create rules composed of one trigger and one action. Triggers are events occurring in some connected applications, and cause the execution of associated actions in other applications. The possible applications are grouped according to their intended goal,

i.e. environment control & monitoring, calendars & scheduling, news & information. Ur et al.

[START_REF] Ur | Practical trigger-action programming in the smart home[END_REF]

reported on a 226-participant usability test of trigger-action programming, finding that inexperienced users can quickly learn multiple triggers or actions obtained by extending the IFTTT language. Resonance AI 2 is a tool for developers that aims to automate and personalize applications. It provides contextual awareness services to enhance products and services with real-time understanding and reactions based on the current user's environment. Such data become actionable triggers that developers can use to automate or suggest actions in order to personalize apps and devices behaviour.

  Trigger/Context Dimension elements, to select one specific trigger dimension (User, Environment, Technology, Social);  Action Dimension elements (Update/Distribute UI, Change Appliance State, Activate Functionalities, Alarm, Reminder), to select one specific action dimension;  Trigger type, to select a specific type of trigger within the hierarchy of triggers;  Action type, to select a specific type of actions within the hierarchy of actions; Entering specific Trigger parameters values (to specify the values associated involved in the specification of a trigger);  Save/Update/Cancel Trigger or Save/Update/Cancel Action commands;  Search Trigger Element (to search for a specific trigger in the hierarchy).

	 Action Operator (to select a specific type of action, e.g. turn on-off, open, close);
	 Event/Condition (to specify whether the trigger specification refers to an event or to
	a condition);
	
	 Trigger Operator (to select the operator involved in a trigger specification, e.g. equal,
	different, more, less);

https://ifttt.com

https://www.resonance-ai.com

http://visjs.org/


Dominik Rupprecht

Rainer Blum

Birgit Bomsdorf

Software Support for Coherent Prototyping of 3D Gesture Interactions

Keywords: Gesture development, interaction design, user interface design, model-based development, development support

When prototyping applications that include touchless 3D gesture interaction three design matters must be taken into consideration: the gestures the user must execute, the visual representation and the dialog flow. Ideally, these aspects should be considered in parallel, to achieve a coherent design process, and avoid ineffective extra effort stemming from coordination between them. A flexible changeover of perspectives among the separate matters is needed. This paper proposes a software environment that enables the desired coherent rapid prototyping of applications with 3D gesture interactions. Its core consists of two types of mapping and a so-called co-simulation functionality. The environment facilitates combining existing software tools from industry and literature to cover the three design matters, i.e. specification and simulation of UI prototypes, gestures, and dialog models. It assists developers at design time in the specification of gestures and in binding them to UI prototypes as well as to statecharts used for defining dialog models. Relevant coherency information is used to offer the option to evaluate gestures at runtime in the context of UI prototype and dialog model. The co-simulator manages the synchronized simulation of all relevant artefacts once a gesture event occurs. Therefore, it enables quickly building prototypes that go beyond the capabilities of the individual tools. This paper describes the usage of the proposed environment in form of a case study with several software tools, each covering one of the three design aspects. It also shows its general applicability, meaning that it can be used with other tools too.

Introduction

When developing applications with touchless 3D gestures, three design matters and their mutual dependencies are important [START_REF] Bomsdorf | Early Prototyping of 3D-Gesture Interaction within the Presentation-Gesture-Dialog Design Space[END_REF]:

 The gestures: The movements and/or static poses of the human body or parts of it required to execute an intended interaction with a system must be defined.  The presentation: The user interface must be designed and developed that informs the user about the required gestures and presents the system feedback.  The dialog: The dialog flow specifies the interactions and their sequences, e.g. the conditions and point in time a specific gesture invokes a system functionality.

Bomsdorf and Blum [START_REF] Bomsdorf | Early Prototyping of 3D-Gesture Interaction within the Presentation-Gesture-Dialog Design Space[END_REF] state that the design of gestures, presentation and dialog span a coherent design space, with different involved stakeholders (different project participants like user interaction designers, end users or software developers). Each of them needs specific tool support for one up to all the design matters. Gestures appropriate to interact with the system must be identified and specified in detail as part of this process. They can be bound to the presentation (e.g. starting with simple UI prototypes) and to dialog models. After that connection, the prototype of an application can be evaluated iteratively. The three design aspects must be kept synchronized. Furthermore, an easy switching between the different axes of the design space is needed. For instance, if a designer alters gestures, this could also imply modifications of the presentation, to present suitable affordances to the user. Software support of this coherent, three axes design space may be pursued by two kinds of approaches for enabling an easy switching between the axes of the design space. A single integrated tool can be a reasonable option if it combines functionalities to create UI prototypes and dialog models and bind gestures to them [START_REF] Bomsdorf | Early Prototyping of 3D-Gesture Interaction within the Presentation-Gesture-Dialog Design Space[END_REF]. It has the advantage that users benefit from a consistent terminology and user interface typically present in a single software tool. However, the downside is that prospective users first need to familiarize with such a new design tool and put their usual tools aside. This may impede the acceptance of the new tool. Also, due to the fact, that gesture sensors are not yet usable via plug and play, such single integrated tools must be altered for every new gesture sensor to be added. In addition, as industrial practice and the literature show, state of the art tools for these different design spaces exist and are widespread and established. Therefore, this work proposes a solution that integrates these existing software tools with the aim to support a coherent rapid prototyping of 3D gesture interaction and a flexible change of perspective between the three axes of the design space.

The next section presents related work focusing on the dialog axis and its combination with 3D gestures, as well as on the presentation axis and its combination with 3D gestures. This is followed by related work concerning the concept of co-simulation. Section 3 gives a broad overview of the software environment proposed in this paper and demonstrates the integration of the above-mentioned tools in form of a case study. Section 4 shows how the proposed environment is applicable to other tools or other interaction techniques. The paper concludes in section 5 with a summary and discussion of possible future developments.

2

Related work

Model based approaches for dialog modeling

In model-based approaches the focus is on the systematic development of an interactive systems. According to Meixner et al. [START_REF] Meixner | Past, Present, and Future of Model-Based User Interface Development[END_REF] the "interface model" consists of all used models, divided into task, dialog, and presentation, while the dialog constitutes the central point. The latter describes the interaction of a user with the user interface and the resulting invocation of system functionality -largely independent of a specific technical implementation and look and feel. The concrete presentation is developed later. Often, Harel statecharts are used to specify dialog models (e.g. Feuerstack et al. [START_REF] Feuerstack | Model-based Design and Generation of a Gesture-based User Interface Navigation Control[END_REF]). Their central concepts are dialog states, transitions between these states, triggers set off by user actions that cause these transitions, and conditions to control them. Different modeling tools that implement this kind of statecharts are the Scade Suite1 , the Nutaq Model-Based Design Kit2 , or Yakindu3 , which is utilized in the work described in this paper. These modeling tools help designers to produce valid diagrams via automatic checks and auto-complete functions (cf. Kistner and Nuernberger [START_REF] Kistner | Developing User Interfaces using SCXML Statecharts[END_REF]).

To prove the quality of the workflow of an application, user inputs can be simulated by triggering them in the UI of the modeling tool, while the reactions of the system are visualized with animated diagrams (cf. Pintér et al. [START_REF] Pintér | Supporting Design and Development of Safety Critical Applications by Model Based Tools[END_REF]).

A connection of gestures with dialog models (working in the dialog-gesture design space) can be found in the work of Feuerstack et al. [START_REF] Feuerstack | Model-based Design and Generation of a Gesture-based User Interface Navigation Control[END_REF] and Bomsdorf et al. [START_REF] Bomsdorf | WeBewIn: Rapid Prototyping bewegungsbasierter Interaktionen[END_REF]. Both allow executable models to be controlled by gestures. But, dialog model and gestures are specified separately and involve the extra effort of an explicit modeling step that links the gestures to the executable dialog model, after the gestures have been implemented for a specific gesture recognizer.

With GestIT Spano et al. [START_REF] Spano | GestIT[END_REF] show and categorize the difficulties to a model-based design of applications with gesture interaction. They propose a declarative and compositional framework for multiplatform gesture definitions as a step towards a new model to solve the single-event granularity problem and providing a separation of concerns. But their approach targets at later stages in the development process where applications are already created by programmers. Our goal is to enable prototyping of interactive gesture applications in earlier design phases.

Rapid prototyping of gesture interaction

Rapid prototypes are an effective method if design ideas shall be implemented and evaluated with stakeholders like intended users or customers already in the design phase. These are visual approaches focusing amongst other things on the look and feel (structure and behavior) of the UI, in contrast to the above-mentioned presentation models, which are abstract UI specifications. In early stages authors like van Buskirk and Moroney [START_REF] Van Buskirk | Extending prototyping[END_REF] recommend working with paper prototypes or other low fidelity prototypes to test different ideas fast with little effort. As stated by Van den Berg et al. [START_REF] Van Den Bergh | GRIP[END_REF], once interactive behavior is needed, more sophisticated prototypes (e.g. implemented with HTML) are typically realized with prototyping tools like Balsamiq4 , Axure 5 , or Pidoco 6 . These tools allow the creation of click-through wireframes to simulate an application. Balsamiq and Pidoco support touch gestures to be used in prototypes, but not touchless 3D gestures yet. The latter is one central topic of this work, and we incorporated Pidoco as example for our approach. Rapid prototypes focus on the perceptible presentation, while the dialog is typically given solely implicit in the quite simple form of the implemented sequence of user interface changes that can be caused by user actions.

Co-Simulation

In an iterative, systematic and user-centric development the different design perspectives should be supported to an equal extent by a software tool or an environment.

The interim results of each axis of the design space must be checked and their consequences understood in the other axes of the design space, due to their mutual concurrence representing all the same application (cf. Barboni et al. [START_REF] Barboni | Beyond Modelling: An Integrated Environment Supporting Co-execution of Tasks and Systems Models[END_REF]).

An early approach to that challenge forms NVIDIAs UI Composer Studio 7 . This embedded systems tool, targeted at the automotive industry, allows designers to specify dialogs for digital dashboards and to bind them to the structure of graphical user interfaces. It relies on its own XML schema to bind the transitions of the dialog model to the expected structural ("page") changes in the presentation of the prototypes. According to Kistner and Nuernberger [START_REF] Kistner | Developing User Interfaces using SCXML Statecharts[END_REF] NVIDIA UI Composer Studio eases both, the prototyping and the dialog modeling for complex applications, providing a coherent development while relying on a single tool.

The approach of Martinie et al. [START_REF] Martinie | A generic tool-supported framework for coupling task models and interactive applications[END_REF] faces a similar problem of enabling a coherent design and co-simulation of different separate aspects of an application. It binds an already existing application to a task model which is based on the already implemented workflow of the application. This is achieved through java annotations inserted within the program code of the application (changing the application program code itself). These annotations make it possible to extract the interactive widgets of the application automatically. After defining a systematic correspondence (a mapping) between these widgets and the task model, both, the application and the task model, can be co-simulated, i.e. they are executed synchronously. However, this approach is intended for later stages of the design process compared to our approach and it does not focus on gestures explicitly. In addition, the code of an application to be bound to a task model must be altered as described. But, that work is another example that shows the complexity of a bidirectional combination of models and presentation.

Software Environment

This section introduces our approach to support the required coherent design. The proposed solution utilizes existing, established tools for dialog modeling, rapid prototyping and gesture recognition and connects them with the help of a semantic-driven binding. Its purpose is to overcome the described shortcomings of the side-by-side use of such tools, to take the mutual dependencies of the three design matters gesture, presentation and dialog into account and in sum to enable a coherent development of 3D gesture interactions.

To make the different tools work together closely, a mapping is needed. Mapping means, that semantically corresponded aspects of each tool are put into relation to each other, like the execution of a specific gesture and the associated reaction in a prototype. This forms the central prerequisite for the desired co-simulation. At design time a common set of triggers (called sync triggers further in this paper) is shared between the involved tools to realize the mapping and co-simulation.

Internal or external mapping

Two types of mappings are implemented between the existing design tools to cater for the different cases that exist in practice: Internal and external mapping. If programmatic access to the internals of a program code is available, that can be used to import sync triggers and bind them to the internal control flow of the software, the mapping is implemented natively in the source code of the tool. We call this option internal mapping. Consequently, the actual communication between the tools is made possible via an application programming interface (API), that provides access to the required, mapping functionality. All tools must import a simple text file containing a list of sync triggers (trigger set) and map them to the desired internal functionality. If an event occurs in one tool that is mapped to a sync trigger, this trigger is subsequently called in the other tool via the involved trigger APIs.

For software tools with no or too limited access to the program code of the tool itself a concept called external mapping is used. This approach keeps the necessary changes on the software code as minimal as possible if at all. However, in this case the respective tool must feature an API that permits access to some kind of internal event system. That event system is bound to external events that are handled in other components of our environment using a separate mapping editor. The mapping between two tools is arranged with this editor during the design phase. As a prerequisite, the tool in question must provide a functionality to export information regarding its relevant internal events into a text file (a list of event names). These events are mapped to the sync triggers inside the mapping editor. In addition, for the intended co-execution of all involved tools a co-simulator is needed. During run time it processes the defined mappings by translating events outcoming from a tool into sync triggers and vice versa.

Environment overview

In this section we detail our approach by means of a case study with the following software products as exemplary tools for the different axes of the design space:

 Presentation: Pidoco 6 for rapid prototyping of graphical UI.

Pidoco is a client/server software tool with web front-end using up to date web technologies. It allows quick creation of click-through wireframes and fully interactive prototypes to simulate and test the look and feel of an application concept.

 Dialog: Yakindu 3 for building statecharts.

Yakindu is a set of plugins written in Java within the Eclipse RCP and IDE environment 8 consisting of four main parts: statechart editing, simulation, validation and code generation.  Gestures: Microsoft Kinect and a gesture simulator.

The Kinect (Version 2) 9 is a combination of soft-and hardware for building and recognizing full body gestures. Instead, or in parallel, a gesture simulator can be used, which can trigger gesture events (as if done by the Kinect).

Internal and external mappings had to be realized. The Pidoco company allowed the native extension of their software tool to support internal mapping between sync triggers and the actions of the UI prototypes built with the Pidoco tool. The external mapping approach was used for the dialog modeling inside Yakindu since this tool features an extension API. A direct internal mapping of gestures was not desired by Yakindu. Therefore, this case study featured a combination of internal and external mapping. Fig. 1 gives an overview of the case-specific implementation of our environment.

Fig. 1. Overview of implemented environment

In general, in the design phase a set of sync triggers must be compiled. Our sync triggers consist of a set of gestures, which are expected to occur, and they serve as a key data structure since all involved software tools are bound to it. For this implementation, this set of sync triggers (simple JSON file) was imported into the Kinect recognizer and in parallel into a gesture simulator software. This enabled the two tools to trigger gesture recognition events within the environment (resulting in the invocation of corresponding sync triggers).

Pidoco's native extension can import a sync trigger set (same JSON file), providing functionality to bind single sync triggers to reactions inside a UI prototype and, thus, to react to received triggers. Yakindu was extended with an export functionality for statechart events and variables via the extension of its existing API. When working on a concrete prototype, the user must import this statechart information (also a JSON file) into the mapping editor, where it must be bound to the sync trigger set by the user.

These mappings saved either inside a tool (internal mapping; in this case with the Pidoco tool) or externally in the mapping editor (external mapping; Yakindu, Kinect, gesture simulator), are subsequently used in the simulation stage for the synchronized execution of the tools. In our case study implementation, if a gesture is recognized by the Kinect gesture recognizer or is triggered manually within the gesture simulator, it is sent over through the shared communication channel. The Pidoco software tool listens for these events and then checks, if a prototype transition is mapped to an incoming sync trigger. It then reacts accordingly by carrying out the determined reaction. Each change during runtime within the UI prototype simulation caused by a user (e.g. a button click) is in turn propagated over the shared communication channel to all applications.

Based on the defined mapping between gestures and sync triggers, both structures are translated into one another by our co-simulator. A gesture, whether triggered by the Kinect recognizer (or, alternatively, the gesture simulator) is translated into statechart events and then sent to Yakindu. Each event occurring inside the statechart tool is sent to the network respectively. It is then translated back into gestures, respectively sync triggers by the co-simulator for the other tools to react.

In our case study the possibility to test and evaluate the interplay of prototype artefacts from two or three axes of the design space in parallel is demonstrated, using either real gestures or simulated ones. The next sections provide more details about the purpose of all parts of the environment including the implementation of required additional software components.

Gesture processing

Gesture recognition was accomplished with a tool chain using Microsoft's Kinect (version 2). Typically, gestures are first recorded with the Kinect Studio10 and then processed with the Visual Gesture Builder11 resulting in a set of gestures. For our case study, these gesture sets were then extended with the required meta data including name, description of involved body parts and additional input values for the gesture recognizer. This editing task was implemented with the Kinect SDK in form of a separate component called gesture editor, where the sync trigger (JSON) file is created.

With our case study implementation, it is also possible to create gesture sets inside this gesture editor without recording actual gestures. These "virtual" gestures are composed of the mentioned meta data and a descriptive video demonstration. A gesture simulator can use both the "virtual" gesture sets, and the sets recorded with the Kinect to trigger sync triggers.

Mapping

Pidoco. As mentioned above, for the design of the presentation, i.e. a UI prototype, the mapping between gestures respectively sync triggers and actions within prototypes was implemented inside Pidoco (internal mapping). Actions like mouse clicks, touch gestures or motion of a mobile device can be configured as triggers inside Pidoco for user interface reactions without requiring any programming. These reactions range from UI page changes to playing sounds or highlighting areas. Each interaction consists of a user action and a system reaction. These interactions can be added via a context menu (see Fig. 2) to every interactive element on the screen. To support the internal mapping approach the Pidoco company extended this context menu, so that the array of assignable actions includes gestures as additional actions (see Fig. 2,①).

To specify gestures as actions within a prototype a gesture set (a set of sync triggers) must be loaded via file import into Pidoco first (another functionality that was also added to the Pidoco software). The gestures are then available as actions in the aforementioned context menu. After selecting an action (see Fig. 2,②) it can be bound to the desired reaction (see Fig. 2,③). Yakindu. For the design of the dialog, mappings must be created in the external mapping editor (external mapping). It maps gestures onto events within model statecharts. First, events are to be assigned to state transitions inside Yakindu. They trigger the transitions, representing dialog changes, if all defined conditions are met. The basic version of the Yakindu software is open source and provides a flexible extension interface, enabling the extraction and processing of design and runtime information.

To create the external mapping, information about the internal events and variables of a specific statechart must be exported. This export functionality was developed as a so-called decoupled code extension using the built-in features of the Yakindu API. The extension builds a JSON file automatically. All the events of a Yakindu statechart (see Fig. 3,①) are represented in this JSON file. To establish the binding the user must import the file into the mapping editor inside our environment. The mapping editor then extracts the trigger information. Afterwards the Yakindu events (see Fig. 3,②) must be mapped to the sync triggers (see Fig. 3,③) by the user.

Co-Simulation

Consequentially, both internal and external mapping were considered during the simulation stage to synchronize events and triggers between the involved tools. All three or just two of the tools can run synchronized this way. Whatever change is occurring in one of the tools, it is transferred to the others so that these can react accordingly. Triggers are sent to the shared communication channel, which was implemented as an event bus. The Kinect gesture recognizer and the gesture simulator simply send the sync triggers that correspond to the current gesture recognition. In Pidoco each incoming trigger is checked if it is valid for the current page. This is true if a reaction is specified for the trigger. In that case, the reaction is executed in the UI prototype.

For the Yakindu tool the co-simulator of our environment uses the mapping table of the mapping editor to translate Yakindu events to sync triggers and vice versa. This mechanism works continuously while in simulation mode, forwarding the messages between the participating software tools. If an incoming event meets the conditions of a transition as defined in the statechart inside Yakindu the simulation of the statechart continues.

Generalizability of the approach

Our environment is applicable to other combinations of tools, as well as to other interaction techniques, like touch or mouse control. As a prerequisite it must be possible to define a common set of sync triggers. Then, the different involved applications can be controlled using either the internal or the external mapping and can interact via the event bus, without changes in our environment.

If a tool provides a message interface and the option to retrieve the states (and changes to these states) of the objects of the application under development, the mapping can be done within the external mapping editor with minimal changes to the existing code of the tool. In that case, to realize the connection to our environment, the set of relevant events must be exported from the target application as a JSON file. It is then bound to the sync triggers inside our mapping editor. During runtime the cosimulator sends the corresponding events to the tool, which can react accordingly. Only both functionalities, the export and the receiving of external events would need to be added to a tool, in case it does not provide it originally.

If a software tool is open to modifications, the mapping of sync triggers to tool functionality may alternatively be implemented within the tool itself. This results in more flexibility to react to specific semantics of gesture bindings within a particular software and is advantageous if for example the UI design options provided by the tool shall specifically reflect the additional concept of gestures.

The mapping concept presented here can be extended to more than the three regarded design spaces (i.e., gestures, presentation and dialog model) and may, for example, additionally include task models or other models to describe the interactive behavior of software applications like the Interaction Flow Modeling Language (IFML)12 . For that purpose, it may be necessary to extend our mapping editor and cosimulator to support new types of events.

Even an alternative gesture recognizer can be used, e.g. the Leap Motion sensor13 . We plan to support it in the future, because of its ability to recognize hand and finger gestures. After the usual binding, as explained above, the Leap gestures would be processed by our environment in the same way as the Kinect gestures from our case study.

5

Discussion and future work

In this paper, we present our work on an environment for rapid prototyping of 3D gesture interactions. To support the early phases in the design of 3D gesture interactions we consider three design matters: gestures, dialog model and presentation. Rather than developing completely new software to cover the tremendous breadth of required functionality we decided to combine already known and established software tools that developers are familiar with. We show a case study on how to combine Microsoft Kinect gestures with Yakindu statecharts (for dialog modeling) and Pidoco rapid UI prototypes within a novel, integrated environment that serves a coherent design process. It allows detailed consideration of the mutual dependencies of these three design matters and enables designers to change perspectives by switching between them flexibly during the design and evaluation of applications with 3D gesture interactions.

We designed the functionality of the proposed environment based on a thorough user-centered requirements engineering with industrial suppliers of software tools and with partners that hold relevant use cases. An evaluation of the described resulting implementation with real users, to find out how beneficial our approach is for designers, is still under work. Parts of the environment as implemented in the described case study (with Pidoco and the gesture simulator) were used by computer science students of our university as part of their study projects building and evaluating gesture interaction applications. We found out, that they could work effectively with the separate tools and the combination of both even without any previous training. Only minor issues concerning the usability of the UI of the environment (e.g. naming of buttons) occurred but were considered for the further development of our environment.

The core parts of the proposed environment are, first, two kinds of mapping as ways to bind gestures to prototypes and to statecharts, and, second, a co-simulator. If existing software is open for extension to import sets of gestures and bind them to their control flow, mapping can be done inside the existing tool (internal mapping). If changes to existing software must be kept to a minimum, instead an API is required to trigger and export events to bind them to gestures in a separate mapping editor that was developed especially for this purpose (external mapping). Both types of mapping are used at runtime to evaluate the gestures in the context of the UI prototype and dialog model respectively. The co-simulator guarantees the synchronized execution of prototype and dialog model once a gesture is recognized or its respective event is triggered.

The two proposed approaches for mapping in combination with the implemented event bus for bi-directional communication provide flexibility for the integration of different types of software tools into the environment and ensure high interchangeability of implementations. Alternative tools only need to support one of the two described mappings and then can be combined with our environment.

However, using different software tools can also cause inconsistency in the semantics of the overall application design. With this work a synchronized simulation envi-ronment is proposed, but not a synchronized model. This necessitates future work that ensures consistency between presentation and dialog.

In addition, gestures are currently translated simply into sync triggers. But there is more information that could be exchanged meaningfully between applications, like values of variables and attributes from the dialog model to the presentation. For example, counter variables held in Yakindu could be reflected in a Pidoco prototype to enrich the prototyped UI with some functionality provided by the dialog model. Or gestures can consist of several events (cf. the granularity problem in Spano et al. [START_REF] Spano | GestIT[END_REF]) or must be traced continuously, resulting in more complex sync triggers. Our environment will be expanded to support those increased sync triggers as well.

Fig. 2 .

 2 Fig. 2. Mapping in Pidoco (extract)

Fig. 3 .

 3 Fig. 3. External Mapping in Yakindu ① and the mapping table in the mapping editor (② and ③) (extracts)

http://www.esterel-technologies.com/products/scade-suite/

http://www.nutaq.com/software/model-based-design-kit/

https://www.itemis.com/en/yakindu/statechart-tools/

https://balsamiq.com/products/mockups/

https://www.axure.com/

https://pidoco.com/en

http://uicomposer.nvidia.com/

https://www.eclipse.org/downloads/packages/eclipse-rcp-and-rap-developers/heliosr

https://developer.microsoft.com/en-us/windows/kinect

https://msdn.microsoft.com/de-de/library/dn785306.aspx

https://msdn.microsoft.com/de-de/library/dn785304.aspx

http://www.ifml.org/

https://www.leapmotion.com/

Acknowledgments

This research was financially supported by the German Federal Ministry of Education and Research within the framework of the program Forschung an Fachhochschulen -IngenieurNachwuchs (project no. 03FH007IX5).


Jonathan Tolle
email: jonathan.tolle@thalesaleniaspace.com

Early incremental user testing design approach validation for satellite command center's application

Keywords: User testing, Incremental software design, User centered design, Return on experience

Thales Alenia Space has recently widely deployed user centered design process in their software product conception. This standard breaks today technology and data centric approach by integrating end-users all along the iterative design stages : context and usage understanding, end-user need specification, quick mockups and end-user validation. This paper is a return on experience. It describes and investigates a dirty UCD methodology relevance based on prototyping and user testing only (skipping user research first activities). This process is made to fit project which needs front-end requirements at day one.

Introduction

User centered design is a well-known software conception standard in ergonomics and human factors fields (ISO 9241-210) [START_REF]Ergonomics of human-system interaction --Part 210: Human-centred design for interactive systems[END_REF]. To guarantee system's utility and usability, it implies four stages where end-user is integrated within many ergonomics methods such as focus group [START_REF] Caplan | Using focus group methodology for ergonomic design[END_REF], participatory design [START_REF] Muller | Participatory design[END_REF], interviews, job task-model based analysis [START_REF] Martinie | Task-model based assessment of automation levels: application to space ground segments[END_REF], persona specification [START_REF] Bornet | La méthode des personas: principes, intérêts et limites[END_REF] and so on. In some constrained industrial context, deploying every practices cannot be possible. Thales Alenia Space believes that every process can be tailored and scaled to fit every project. For our new satellite command center (SCC) procedure executor named PRISM, we try a dirty UCD approach based on incremental mockups and user testing only. As user requirements were taken from our previous product version, this process enables earlier front-end development than classical ergonomics methodology.

This testimonial explains first how we integrated UCD activities within agile development process. Then we briefly present our study case PRISM and its incremental mockups. Next, we inspect our process relevance from usability metrics and user feedbacks. Does this approach lead to solution convergence ? Does it bring high quality interaction and high user satisfaction ? Does it allow continuous interaction improvement ? From these problematic, the paper concludes on a preliminary analysis of this dirty UCD approach (ongoing work). Please note this article doesn't list any SCC user needs. It only aims to analyze the process relevance from ergonomics point of view.

Process overview

Industry often works with subcontractors while developing products. When interaction designers are not involved in project planning phase, it may results to a development-first approach without much consideration for ergonomics activities [START_REF] Holtzblatt | Rapid user centered design techniques: Challenges and solutions[END_REF]. It can be hard to break this convenient design culture as subcontractor is employing software engineers in tight timeframe. In this context, we try a new incremental process to deal with early front-end development requirements.

Fig. 1. 'Early incremental user testing design approach' activities

To ensure usable and useful software, we empirically plan 3 user test increments with a minimum of 5 participants per end-user test [START_REF] Nielsen | Why you only need to test with 5 users[END_REF]. These tests are based on the walkthrough ergonomics methodology [START_REF] Wharton | The cognitive walkthrough method: A practitioner's guide[END_REF] by using fictive operational scenarios. Scenario's increments are built from simple and non-risky interactions (increment 1) to complex widget and data visualization interactions (increment 2 and 3). Scenario coverage (e.g. "1+2+3") between increments allows us to compare if previous frontend modifications bring better ergonomics satisfaction through usability metrics. At the end of each increment, user observations are discussed with the project manager before integrating them to subcontractor's agile development: considering usability gain over development planning, feasibility and effort. Thales Alenia Space has specified a graphical charter for every web applications and lot of reusable widgets have already been designed for other solutions (such as button, table, list, interactive menu, and so on). Therefore we decided to directly work with high fidelity mockups rather than having a first low fidelity prototype's iteration. Prototypes were made using AXURE software. This tool creates interactive interfaces that can simulate expected scenario's solution behavior and be used in our user testing sessions.

Process inspection

User testing protocol

To investigate process relevance, it is important to set up a testing protocol. It allows us to compare data between increments. At the moment, two end-user test iterations have been performed (the third and last iteration will be done before product release). 10 distinct end-users participate to this analysis : 5 at first iteration and 7 at the second one (2 persons from increment 1 were involved as well in increment 2). Participants are all PRISM final end-users and come from different departments with different level of expertise. Each individual test session lasts 1 hour and a half and follows the same walkthrough methodology [START_REF] Wharton | The cognitive walkthrough method: A practitioner's guide[END_REF] :

1. Welcome participant, Informed consent form signature and present user centered approach 2. Scenario reading and mockup limitation explanations (example : few auto scroll interactions could not be simulated) 3. Participant interaction with the prototype performing scenario goals without any help from the ergonomist. If the user seems stuck after many tries, the ergonomist gives the solution but then the interaction is tagged as failed (0% or 100% in Fig. 7 and Fig. 8). 4. System usability scale (SUS) survey to measure global software satisfaction 5. Task satisfaction scale form and user remark debriefing 6. Open questions and participant acknowledgments

User satisfaction

System usability scale.

At the end of each test, participants fill SUS form. SUS gives us the global satisfaction of the prototype. Rather than "Think aloud protocol", participants were not allowed to give any observation before this step. This protocol aims to avoid any ergonomist's explanation bias from participant satisfaction opinion. From increment 1 and 2, we can observe a big satisfaction deterioration : losing 17 points on SUS average score. The main empirical hypothesis raised from this drop is the higher complexity and the higher number of widgets provided between increments (see Fig. 3 and Fig. 4). For the next iteration, our SUS goal is to reach a score higher to 72/100 by moving secondary widgets out of first end-user's eye exploration (and placing them into visible / hidden panel).

Task satisfaction scale. SUS doesn't extract which technical task lower the satisfaction. So we decided to add a dirty homemade survey (rating from 1 to 5). This survey aims to prioritize which part of the software front-end need to be improved for the next iteration. A proper survey wasn't investigated as it would have taken too much time in the test protocol. . From an empiric decision, every score below 3,4/5 is marked "has to be improved"

From increment 1 and 2, five equivalent tasks were covered (e.g. "ouverture d'une procedure") and four tasks were totally new. We can notice that modifications made between increments did bring better satisfaction for three of them (while the others stayed stable). By contrast with SUS score, it gives us a good feedback for our dirty UCD approach relevance. This chart will also be used to compare this analysis to iteration 3 and check task satisfaction progress.

Interaction quality

Interaction quality is inspected through two usability metrics :

 Effectiveness : does your participant succeeded or not to complete the interaction ? (column "goal reached")  Efficiency : does your participant make errors before completing his goal ? And how long does he take to finish it ? (column "error rate" and "time") The tables above list every interaction needed to complete the scenario at increment 1 and 2. The table colorization shows which operation need to be improved or modify in priority for the next increment. For instance, between mockup 1 and 2, a big frontend modification has been designed to open and initialize a procedure : passing from native browser "file explorer" to "custom procedure explorer" (interactions represented by #1.02 to #1.04 of Fig. 7 and #1.02 to #1.09 of Fig. 8). Surprisingly even if task satisfaction grew significantly (Fig. 9), the newer interface didn't lower interaction error rate: 1,6 at iteration 1 and 3,5 at iteration.

On other hand, time measures didn't bring any comparative added-value on interaction quality. Overall time evaluation must be changed to task-oriented time evaluation. For the next iteration, time should be sampled by task (column "scenario part") and task should be kept consistent on next iterations. Also by using keystroke-level ergonomics method [START_REF] Card | The keystroke-level model for user performance time with interactive systems[END_REF], we could analyze and compare the interaction performance to what an expert would have done.

Solution convergence

Solution convergence is analyzed from the user remark number at each end of increment (see activity 4 Fig. 1).

Fig. 10. User remark list sample. Each remark is tagged with the number of people who ask for it, from which department the participant comes from, when it has been identify (increment number), what is the empirical perceived ergonomics gain if integrated and what is the perceived complexity to implement it.

At the first increment, we report 97 user remarks on the tested prototype. At the second iteration, we count only 51 observations (approximately -50%) for a prototype which cover more complexity and tasks. This reduction shows a first solution convergence to the final user requirements. This preliminary analysis will be corroborated with the next increment results.

Continuous improvement

Another important point measured in our process is the continuous improvement of PRISM interactions at each increments. From remark listing (Fig. 10), every user feedback is labeled by his integration status (according to interaction designer and project manager discussion) : Accepted (green) meaning "sent directly to development requirements" -To be specified (blue) meaning "related to other software system specifications" -Delayed (purple) meaning "to be investigated at next iteration" -To be discussed (orange) meaning "to be discussed with the chef architect for feasibility investigation" -Rejected (red) meaning "remark is not relevant or in contradiction with previous taken decision". From the two increment statements, we can deduce that the closer the project gets to the deadline, the fewer remarks are integrated into development (-54% accepted, +58% rejected and +80% to be discussed).

Conclusion

This paper presents the ongoing work of an early incremental user testing design approach. This dirty user centered design process allows front-end development at day one (as the first increment was prototyped and tested within 10 days). To assess this methodology relevance, we base our preliminary analysis on utility, usability and traceability metrics from two incremental user testing iterations.

As more complex features are integrated at each iteration, this approach doesn't seem relevant to keep SUS score consistently high. Also from interaction efficiency and effectiveness metrics, no significant improvements were noticed. These results are mainly due to the incremental approach where designer has to focus on next increment requirements rather than improving previous ones. From user remark traceability statement, this process doesn't as well give the opportunity to make major modifications as the development team gets closer to the release deadline.

However, homemade task satisfaction progress and user remark number analysis give a good feedback on the convergence of user needs. Furthermore, by integrating user in prototype testing loops, we identify one "game changer" idea which will be developed and test at iteration 3.

If development-first approach is requested and mandatory, this incremental user testing design approach can be applied as a dirty UCD process. Despite the lack of user research activities, our return on experience shows solution convergence and great feedback from participants but low capabilities on continuous improvement (interaction usability and remark integration) and some risks of user satisfaction drops.

Fig. 2 .

 2 Fig. 2. PRISM simplified eco-system

Fig. 3 .Fig. 4 .

 34 Fig. 3. Increment 1: main screen which displays direct feedback of the procedure execution Fig. 4. Increment 2: main screen which displays direct feedback of the procedure execution and provides complementary services to end-user

Fig. 5 .

 5 Fig. 5. French translated SUS result from the 5 participants of increment 1.

Fig. 6 .

 6 Fig. 6. French translated SUS result from the 7 participants of increment 2. Note that one participant results have been removed from average score due to inconsistent answers.

Fig. 7 .

 7 Fig. 7. Bar chart high-level functionality satisfaction result from both iterations (1 & 2). Froman empiric decision, every score below 3,4/5 is marked "has to be improved"

Fig. 8 .Fig. 9 .

 89 Fig. 8. Efficiency and effectiveness report from iteration 1 (5 participants)

Fig. 11 . 1 Fig. 12 .

 11112 Fig. 11. Remark traceability chart iteration 1