Sebastian Feuerstack
email: feuerstack@offis.de

Bertram Wortelen
email: wortelen@offis.de

Revealing Differences in Designers' and Users' Perspectives: A Tool-supported Process for Visual Attention Prediction for Designing HMIs for Maritime Monitoring Tasks

Keywords: Visual attention, HMI analysis, Monitoring task

Monitoring complex systems includes scanning, aggregating and processing data from various sources. The design of graphical interfaces for monitoring tasks involves a fine-grained exploration of the importance and expected frequency of events that an operator needs to be informed about. The Human Efficiency Evaluator is a tool for the prediction of human behavior. We extended it to predict the distribution of operator's attention while monitoring interfaces. The prediction is based on the SEEV model. We show that our tool can be used by experts with different backgrounds to generate predictions following a structured, semi-automated process. In a qualitative study with subject matter experts, we analyzed different HMI designs for a navigation task in the maritime domain. We evaluated their modeling time, tested different prediction result visualizations, and investigated in the model differences between the subjects. Different to what we originally expected, the study revealed that the models created by the subjects substantially differ depending on their perspectives. Heat maps visualizing the predicted attention allocation were appreciated by the subjects and enabled them to argue about their perspective.

Introduction

Controlling safety-critical systems often includes several monitoring tasks to observe the current system state and to predict future system states that might affect the controlling activities. A ship bridge is an example for such a safety-critical system. Modern bridge systems offer a broad range of automation of routine tasks and most of the navigation decisions (passage planning for instance) can be performed prior to the voyage. Therefore most of the time on a ship bridge is spent on the navigation monitoring task. This includes observing the ships status, its navigation path and watching out for future events that require adjusting the planned route of the vessel's by changing its speed and heading to prevent dangerous situations.

Studies show that in between 75%-80% of accidents in ship navigation happen because the human operator had not access to information that could have prevented the accident [START_REF]Annex 24 -MCA guidance notes for voyage planning[END_REF]. A study that investigated the lack of situation awareness of mariners revealed the importance of situation awareness for the decision making process in the maritime domain. From the 177 maritime accident reports analyzed, 71% percent of the human errors were situation awareness related. 58.5% of those could be classified as caused by failures in correctly perceiving information [START_REF] Michelle R Grech | Human error in maritime operations: Analyses of accident reports using the leximancer tool[END_REF]. These types of failures are caused by data not being available, data not being easy to discriminate, misperceptions or failures in monitoring or observing data [START_REF] Michelle R Grech | Human error in maritime operations: Analyses of accident reports using the leximancer tool[END_REF].

Electronic integrated bridge concepts are driving future navigation system planning [START_REF] Bowditch | The American practical navigator : An Epitome of navigation[END_REF]. Such systems aggregate data from various ship sensors and support the vessel navigation task with a map-centric view augmented with current and future navigation paths of the own vessel and real-time information of other vessels in the current area of the ship [START_REF] Bowditch | The American practical navigator : An Epitome of navigation[END_REF]. Monitoring such complex systems demands intermittently for the human operator's attention to observe and interpret several information sources [START_REF] Hollnagel | Joint cognitive systems : foundations of cognitive systems engineering[END_REF].

This contribution presents a tool, the Human Efficiency Evaluator (HEE), which supports subject matter experts (SMEs), such as HMI designers and domain experts, with no background in cognitive modeling to generate and benefit from attention predictions. With the HEE predictions can be performed by a cognitive human operator simulation in a very early design phase where only HMI design images or sketches of future interfaces are available. The HEE uses the Adaptive Information Expectancy model (AIE model) [START_REF] Wortelen | Dynamic simulation and prediction of drivers' attention distribution[END_REF], which is based on the SEEV model [START_REF] Wickens | Attentional models of multitask pilot performance using advanced display technology[END_REF] and is a dynamic simulation model of attention distribution.

The following section discusses related work. Thereafter we detail the theoretical background of our approach, the process and the underlying models in section 3. Section 4 describes how the process is implemented in the HEE. Section 5 illustrates a use case in the maritime domain that we used to evaluate our approach. This qualitative study with SMEs in section 6 reports about the evaluation of the HEE, which was focused on analyzing the modeling differences between different potential users of the HEE: domain experts, HMI designers, cognitive modelers, and situation awareness experts. Different to what we originally expected, we found much more differences in modeling than similarities between the different SMEs. We close by discussing and summarizing the study results and elaborate a list of hypothesizes derived from observations made during the study. Finally, section 8 concludes and states future work.

Related Work

The Adaptive Information Expectancy (AIE) model is a predictive simulation model of attention distribution [START_REF] Wortelen | Das Adaptive-Information-Expectancy-Modell zur Aufmerksamkeitssimulation eines kognitiven Fahrermodells[END_REF][START_REF] Wortelen | Adaptive simulation of monitoring behavior[END_REF]. It is an integral part of the CASCaS (Cognitive Architecture for Safety Critical Task Simulation) architecture [START_REF] Lüdtke | Multi-criteria evaluation of aircraft cockpit systems by model-based simulation of pilot performance[END_REF]. It simulates the attention distribution of a human operator based on two sets of input parameters: Expectancy and Task Value. They describe how often new information can be expected from an information source (IS) and how valuable the information is for accomplishing the tasks of the human operator. These factors are relevant for showing optimal monitor-ing behavior and have been shown to be the main drivers for skilled operators like pilots and drivers [START_REF] Wickens | Attention-Situation Awareness A-SA Model of Pilot Error[END_REF]. An expectancy coefficient 𝑢 𝑔 and a value coefficient 𝑣 𝑔 is assigned to each goal 𝑔 of a cognitive model executed in CASCaS.

A cognitive architecture like CASCaS can be understood as a generic interpreter that executes formalized procedures of a human operator in a psychological plausible way. An overview of cognitive computational models like ACT-R, MIDAS and others is provided in [START_REF]Cognitive Systems: Human Cognitive Models in Systems Design[END_REF][START_REF] Wickens | Modeling and evaluating pilot performance in nextgen: Review of and recommendations regarding pilot modeling efforts, architectures, and validation studies[END_REF]. Computational cognitive models have got the potential to automate parts of human factor analyses during system design. In order to leverage this potential the models have to be embedded in a design tool which can be readily applied by design experts.

Monitoring involves detecting and reacting to events and is composed of a set of monitoring goals. To execute such a goal the human operator looks to the IS that can signal the event. Upon event detection the operator utilizes the perceived information to react to this event. If no event is detected, the operator's attention shifts to another monitoring goal probabilistically based on the expectancy and value coefficients. The probability of switching to goal 𝑔 among a set 𝐺 of monitoring goals is defined as (cf. [START_REF] Wortelen | Das Adaptive-Information-Expectancy-Modell zur Aufmerksamkeitssimulation eines kognitiven Fahrermodells[END_REF]):

𝑃(𝑔) = 𝑢 𝑔 ∑ 𝑢 𝑔 𝑖 𝑔 𝑖 ∈𝐺 • 𝑣 𝑔 ∑ 𝑣 𝑔 𝑖 𝑔 𝑖 ∈𝐺
Several tools already support cognitive model creation. CogTool [START_REF] John | Predictive human performance modeling made easy[END_REF] supports the generation of ACT-R [START_REF]The Atomic Components of Thought[END_REF] models implementing deterministic sequences of actions. These models are based on GOMS and KLM and therefore CogTool targets on evaluating Windows-, Icons, Menus, and Pointer (WIMP) user interfaces but also considers speech-commands and basic gestures. For monitoring simulations the HEE generates a probabilistic sequence of actions. Based on CASCaS and the AIE model prediction of average values (percentage dwell times, gaze frequencies) and prediction of distributions (reaction times, duration of diversion) are possible. MIDAS is a system developed by NASA [START_REF] Corker | An architecture and model for cognitive engineering simulation analysis: Application to advanced aviation automation[END_REF], which uses cognitive models to analyze tasks and interfaces in the aerospace domain. It simulates visual attention distribution similar to CASCaS, but uses the SEEV model of Wickens et al. [START_REF] Wickens | Attentional models of multitask pilot performance using advanced display technology[END_REF] instead of the AIE model. Both models are strongly related, but differ in some aspects [START_REF] Wortelen | Das Adaptive-Information-Expectancy-Modell zur Aufmerksamkeitssimulation eines kognitiven Fahrermodells[END_REF].

Approach

Cognitive models require input from an expert to predict the attention. In our case these inputs are the identification and location of ISs on a graphical display. Because we use the AIE model (see Section 2) for prediction of attention distribution, for each Information Source (IS) an expectancy coefficient and a value coefficient need to be defined by an expert. Fig. 1 illustrates this. We explain the theory of how to derive the expectancy and value coefficients from the experts' inputs and formalize it for both in the two upcoming subsections. Based on these information and the identification of one IS that most probably make the expert aware of a specific unexpected event (subsection 3.4) the cognitive operator model (subsection 3.5) and the HMI model are generated. After the theoretical foundation, section 4 explains how forms and visuali-zations support the coefficient derivation, which is then feed into the Human Efficiency Evaluator for running the simulation and generating the predictions.

Identification of Information Sources

Information sources (IS) are regions of a display or on a design sketch that communicate a single piece of information to the operator. The process requires that an expert collects all information that can be extracted from a graphical display. Each information is marked as an IS by specifying as exactly as possible its graphical position, its size and by unambiguously naming the IS so that it can be referred to by its name.

Definition of Expectancy Coefficients

Earlier AIE model applications extracted the expectancy coefficients during the simulation of the cognitive model in interaction with realistic environment simulations [START_REF] Wortelen | Dynamic simulation and prediction of drivers' attention distribution[END_REF], [START_REF] Wortelen | Adaptive simulation of monitoring behavior[END_REF]. Since this approach is focused on evaluating design sketches, we derive them manually using the lowest ordinal heuristic for this purpose as proposed by Wickens et al. [START_REF] Wickens | Pilot task management: Testing an attentional expected value model of visual scanning[END_REF]. For this heuristic the expert needs to order all IS based on the amount of new information expected. The rank of an IS in this order determines its expectancy coefficient. The heuristic does not require a total order on the IS. It is sufficient to specify a partial order between the IS across all designs. Partial orders are typically visualized by Hasse diagrams. Following, we formally describe the notation that the expert can use to define the partial order. Let 𝐷 be the set of all considered designs, 𝑆 be the set of all defined IS, and 𝑆 𝐷 : 𝐷 → 𝒫(S) the function that gives for each design the IS that are defined on it. The same IS can be defined on several designs. The tuple (𝑠|𝑑) specifies the IS 𝑠 defined on design 𝑑. By defining relations between different IS such a partial order is established. Relations are formally defined by statements, such as (𝑠 𝑖 |𝑑 𝑗 ) > (𝑠 𝑘 |𝑑 𝑙 ). Which states that IS 𝑠 𝑖 of design 𝑑 𝑗 provides relevant events with a higher frequency than IS 𝑠 𝑘 of design 𝑑 𝑙 . Such statements can relate IS by the operators ">", "<", or "=". Further on, several statements can be condensed into a single one to specify relations time efficiently by comma separating IS and designs and also by using quantors. To support an automatic transformation of the Fig. 1. Main activities of the process to generate the models that are then feed into a simulation to predict reaction times and the attention allocation of the operator.

Comma separated IS.

Several comma-separated IS and/or designs can be listed on the left hand side and/or right hand side of the relation. Such statements create a set of relationships by using the Cartesian product of the entries on the left hand side and on the right hand side of the statement, e.g.:

(𝑠 1 , 𝑠 2 |𝑑) > (𝑠 3 , 𝑠 4 |𝑑) ≡ {(𝑠 𝑖 |𝑑) > (𝑠 𝑘 |𝑑) | 𝑠 𝑖 ∈ {𝑠 1 , 𝑠 2 }, 𝑠 𝑘 ∈ {𝑠 3 , 𝑠 4 }} ≡ {(𝑠 1 |𝑑) > (𝑠 3 |𝑑), (𝑠 1 |𝑑) > (𝑠 4 |𝑑), (𝑠 2 |𝑑) > (𝑠 3 |𝑑), (𝑠 2 |𝑑) > (𝑠 4 |𝑑)}
Quantors .

An Asterix is used as quantor to describe that a relation holds for all IS of a specific design ( * |𝑑) or for a specific IS in all designs (𝑠| * ). The exact semantic depends on whether a quantor is used on both sides of the relation. Quantor on both sides means Cartesian product of all IS respectively designs. The following specifies that 𝑠 1 provides information more frequently than 𝑠 2 regardless of a specific design.

(𝑠 1 | * ) > (𝑠 2 | * ) ≡ {(𝑠 1 |𝑑 𝑖 ) > (𝑠 2 |𝑑 𝑘 ) | 𝑑 𝑖 , 𝑑 𝑘 ∈ 𝐷}
If only one side contains a quantor, it stands for all information sources or designs, except the ones listed on the respective other side. The following specifies that IS 𝑠 1 on design 𝑑 provides information more frequently than all other IS on that design.

(𝑠 1 |𝑑) > ( * |𝑑) ≡ {(𝑠 1 |𝑑) > (𝑠|𝑑) | 𝑠 ∈ 𝑆 𝐷 (𝑑)\𝑠 1 }
A variable following a quantor can be used to bind two quantors on both sides to the same variable. It prevents that the full Cartesian product is created. The following specifies that for each design 𝑠 1 provides information more frequently than 𝑠 2 , but this relation cannot be drawn across different designs.

(𝑠 1 | * 𝑌) > (𝑠 2 | * 𝑌) ≡ {(𝑠 1 |𝑑) > (𝑠 2 |𝑑) | 𝑑 ∈ 𝐷}
The expectancy coefficient for (𝑠|𝑑) is determined by the order of (𝑠|𝑑) within the partial order. In the Hasse diagram of the relation this is the greatest length of a path from any minimal element to (𝑠|𝑑) plus 1. More formally: Let ≤ be the transitive reduction of the defined partial order and ∆ ≤ be the distance between two elements: expectancy(𝑠, 𝑑) = max 𝑑 𝑘 ∈𝐷,𝑠 𝑖 ∈𝑆 𝐷 (𝑑 𝑘 ) (∆ ≤ ((𝑠 𝑖 |𝑑 𝑘 ), (𝑠|𝑑))) + 1

Definition of Value Coefficients

Monitoring is typically performed to collect information relevant for tasks a human operator has to perform. The value of an IS depends on the values of the tasks for which it provides information. To obtain the value coefficients for the IS, in a first step the importance of all human operator tasks is rated by the lowest ordinal algorithms as shown above for the expectancy coefficient. Second, the user fills out a relevance matrix 𝑅 specifying the relevancy R for each IS and each task (cf. [START_REF] Wickens | Attentional models of multitask pilot performance using advanced display technology[END_REF]). 𝑅(𝑠, 𝑡) = 0 states, that IS 𝑠 is not relevant for task 𝑡, 𝑅(𝑠, 𝑡) = 0.5 defines that 𝑠 supports 𝑡, but is not mandatory for t, and 𝑅(𝑠, 𝑡) = 1 means, that 𝑠 is mandatory for 𝑡.

The value of an IS 𝑠 is determined by (cf. [START_REF] Wickens | Attentional models of multitask pilot performance using advanced display technology[END_REF]):

value(𝑠) = ∑ 𝑅(𝑠, 𝑡) • value(𝑡) 𝑡∈𝑇

Event Definition

Monitoring is performed in order to maintain suitable situation awareness and to detect events in a timely manner and to react to events if necessary. The reaction time depends on the amount of attention directed to the IS that displays the event. When simulating the allocation of attention, the reaction time to events can also be predicted. An event 𝑒 = (𝑠, 𝑡) is defined by the user by specifying the time 𝑡 of occurrence and the IS 𝑠, that most likely will make the operator aware of the occurrence of this specific event. This might vary between designs. Therefor the IS has to be specified for each design.

Generation of Monitoring Operator Model

The operator model specifying the monitoring is automatically created based on the IS markups and the expectancy and the value coefficients. For the overall monitoring, a top level goal 𝑔 𝑡 is created, which has only one associated procedural rule. This rule activates a set of sub-goalsone for monitoring each IS. This structure is reflected by rule R1 in Fig. 2. The sub-goals are annotated with the expectancy and value coefficients as defined above. Rule R2 is executed, if an event is shown by IS 𝑠 𝑖 . If no event is shown, R3 is executed. However, as the operator usually does not know the current state of information on 𝑠 𝑖 the conditions of R2 and R3 cannot be evaluated, because this information is unknown. This state triggers rule R4, which commands the perception and motor component of the cognitive architecture to move the gaze to 𝑠 𝑖 . After the information about the event is perceived, the model either triggers R2, in case there is an event, or triggers R3, if there is none. The resulting model, will constantly switch between the monitoring sub-goals to move visual attention between the IS based on the probability calculated with the expectancy and value coefficients.

Tool-supported Attention Prediction with the HEE

The Human Efficiency Evaluator (HEE) is based on CogTool because of two reasons. First, we share the idea of supporting an evaluation of interfaces based by annotating design sketches at a very early stage. Second, the results are also predictions based on a simulation. But there are several fundamental differences between both. CogTool focuses on performance prediction of WIMP (Windows, Icons, Menus, Pointer) interfaces. Therefore, the annotation of design sketches is based on a fixed palette of the most common WIMP widgets like buttons, menu bars, and radio buttons. The annotation process to construct a user interface model is therefore straight-forward by identifying and marking widgets exactly as they are depicted in the design sketch. However, nautical maps have no fixed widgets (even though there are standards, e.g. color) and new design proposals often intentionally break with some of already existing concepts. As we show later, the identification and markup of IS substantially depends on the background and experience of the tool user and also often on interpretations. With CogTool one annotates what is depicted, with HEE one also marks what one knows. Different to CogTool, that inherently ends up with similar HMI models fed into the simulation, map-centric monitoring HMI models for the same design can substantially differ between users and therefore the resulting predictions as well.

Therefore, we re-implemented the entire backend of CogTool to use our cognitive architecture CASCaS (Cognitive Architecture for Safety Critical Task Simulation) [START_REF] Lüdtke | Multi-criteria evaluation of aircraft cockpit systems by model-based simulation of pilot performance[END_REF], [START_REF] Lüdtke | Modeling pilot and driver behavior for human error simulation[END_REF][START_REF] Frische | Simulating visual attention allocation of pilots in an advanced cockpit environment[END_REF] and integrated support for generating operator models. These models support the Adaptive Information Expectancy (AIE) model [START_REF] Wortelen | Dynamic simulation and prediction of drivers' attention distribution[END_REF] to predict the distribution of attention and the average reaction time to a certain event. Further on, we exchanged the hard-coded widgets palette of CogTool to annotate the designs with a model-based backend that enables us to define new annotation options (like for the IS in this case) without recompiling the tool.

The HEE currently supports users with the IS identification, automates the operator model generation and the generation of visualizations of the predicted attention distribution and the computed average reaction times (cf. Fig. 1).

Fig. 3 shows the HMI design modeling perspective of the HEE. The left vertical bar contains a palette of widgets that can be used to annotate a design. For attention allocation prediction, one widget ("i") is used to mark relevant sources of information for each design. The main area of Fig. 3 displays an already annotated design where several IS have been already marked by the user. Further widgets are used from the palette to identify the operator's initial line of sight, the operator's distance to the monitor, and to set the physical dimensions of the monitor.

For specifying the expectancy and value coefficients we use paper forms. We realized that those give the user more freedom by using quantors (c.f. subsection 3.1) to aggregating relations over several IS and designs or by introducing abbreviations for IS that the user feels comfortable with. The expectancy is specified by relations like described in subsection 3.1. The value is defined by filling in a printed relevance matrix form (c.f. subsection 3.2). Both are then transcribed to an interactive excel sheet (see Fig. 4). To ease writing down the relations, we used abbreviations for the IS names, e.g., 'AH' for 'Vessel A Heading'. The sheet is then processed by a script that solves the quantors (c.f. subsection 3.1) and generates two Hasse diagrams: one for the expectancy (c.f. Fig. 5) and another one for the value coefficient. The graph ranks (c.f. Fig. 5) correspond to the coefficient values and then are entered in the HEE tool. After the user has chosen an IS to fire an event at a certain moment in time (c.f. subsection 3.3) the operator model for CASCaS can be automatically generated. It consists of:  An HMI model defining the physical dimensions and spatial locations of the simulated operator and all IS (which we call topology in CASCaS). The HMI model is automatically created based on the rectangular definitions of information sources. The 2-dimensional coordinates are projected onto a 3-dimensonial plane, which is placed at a specific distance in front of the head of the simulated operator.  An operator model composed of the operator's monitoring goals as a procedural knowledge specified as rules (c.f. subsection 3.4).

Both models are fed into CASCaS and executed as a Monte-Carlo simulation. Based on the desired visualization the HEE processes the simulation results and generates a heat map for each design (see Fig. 7). It is also possible to feed the data into interactive excel sheets that currently can generate a diagram with the average attention allocation per IS in percent as a pie chart and display the reaction time to an event while monitoring the interface as a histogram (see Fig. 8).

Use Case

Electronic Chart Display and Information Systems depicting nautical maps enriched by navigation-relevant information are one of the main sources of information of shipmasters and navigation officers and need to be continuously monitored. Within the Cooperative Shipping and Navigation on Sea project, COSINUS1 , several new display designs have been proposed. Some of them are depicted in Fig. 6. Fig. 6 a) shows a state of the art chart display and Fig. 6 b) the corresponding design sketch, that highlights speed vectors (SV) as the current state of the art concept to display vessel behaviors with arrows and a red line illustrating the route of the own ship. All sketches illustrate a prediction. Therefore the speed vectors point to locations of the future vessels' locations in 10 minutes (with the assumption that they do not change their course and speed).

Fig. 6 c) depicts the design G: the entire routes of all surroundings vessels together with the maximum tolerance for deviations (gates), which have been set beforehand. Regions where the temporal and spatial distance to other vessels are very small are marked red. The design TCPA of Fig. 6 d) does not present the entire route, but only the future routes of the surrounding vessels. These routes end with displaying the vessels' orientation and position in 10 minutes. Furthermore the time to closest point of approach (TCPA) is annotated at the vessels position.

Evaluation

We evaluated the attention prediction process with the HEE by a qualitative study 2 .

The main objective was to get first insights into how well the HEE tool can be operated by users, and how helpful the visualizations that HEE creates are for the users. However, this is the first evaluation made for HEE. It therefore has a strong explorative character. Based on its findings, we plan to perform more extensive studies afterwards. The current study had a fixed, scripted procedure and implemented a within-subject design with four subject matter experts. The procedure contained explorative questions and pre-scripted help text (elaborated during a pre-test) that were planned to be read in case a subject is not able to perform a task. We designed the study to test three main hypotheses that were supported by several sub-hypotheses:

H 1 :"Users without specific prior knowledge are able to use the HEE and end up with results in a reasonable amount of time."

H 2 :"The variations between the models specified by the participants are small."

H 3 :"The result visualization of the HEE is clear." H 3a :"A pie chart is an easy understandable visualization of the average attention allocation prediction." H 3b :"A histogram is an easy to understand visualization of reaction time prediction." H 3c :"Heatmaps are an easy understandable visualization of the average attention distribution prediction."

2 A video documentation of the study can be found online at http://multi-access.de/512 Fig. 8. Histogram of the Average Reaction Time. With a probability of 80% the average reaction time of the operator to a certain event took between 0 and 19s.

6.1

Participants

Four subject matter experts participated in the study: a cognitive modeling expert (Cog), an interface designer (HMI) that created the TCPA design sketch, and an expert for the maritime domain (Exp), and an expert for situation awareness (SA). All were between 30 and 55 years and neither had prior experience with using the HEE nor performed any of the three modeling tasks of the study before.

Apparatus

Participants were sat comfortably in front of a computer screen, and the experimenter sat to the side. A current state of the art map display, the three design variants (c.f. Fig. 6), and three basic tasks of a ship master (avoid low water sections; avoid dangerous vessel approaches; follow your planned route) were presented as slides to the subjects. The explanation was read from a manuscript.

Software

The HEE was opened with a project were the three designs variants have been already added. The subjects received an explanation of the modeling functions: marking an IS, resizing, moving, deleting, and naming it.

Procedure

The experiment was composed of the four expert activities of the overall prediction process (c.f. Fig. 1): (1) IS annotation with the HEE, (2) specifying the expected information update frequency, (3) task ranking by impact and defining the relevancy of each IS for each task and design variant, (4) identifying the IS of a predefined event.

Further on, (5) the three resulting visualizations were analyzed. None of the participants was informed about what kind of results the HEE produces.

Between the first four activities that took each participant between 120-180 minutes in total and the fifth activity that took around 30-60 minutes there was a break. The break was required by the experimenters to calculate and prepare the visualizations.

Results

Based on the video recordings and notes taken during the study we evaluated the data with respect to each sub-hypothesis.

H 1 :"Users without specific prior knowledge are able to use the HEE and end up with results in a reasonable amount of time."

Table 1 shows the overall duration of the entire study for each participant and the total time for performing the three modeling steps. None of the subjects had used the tool before. Thus, for each step we gave a scripted introduction and allowed questions before each modeling activity started. The instruction time is listed in the first column of each modeling step. The mean of the modeling time is 2:02 hours. The Cog felt most familiar with all modeling steps and intuitively decided to mark the fewest IS (18) of all participants which reduced the work in the proceeding steps substantially.

The SA had the most IS marked (47). We all asked the participants to comment on each IS identified, which Exp did in detail. It explains the relative high amount of time for the IS identification step (90min) of Exp.

H 2: "The variations between the models specified by the participants are small."

As a first step we compared the IS defined by participants. All participants together defined 130 IS over all designs. During phase 1 participants explained which information they marked with an IS. We created a list of all this information and checked for each participant if it was marked. An excerpt of this list is shown in Table 2. The list contains 68 information elements. Sometimes the classification is not clear, because one participant marked an IS containing several information elements, while another participant marked one IS for each of these information elements. Therefore we distinguished between marking identical information, marking similar information and not marking the information. Only four information elements are marked identical by all participants: 3 out of 4 beacons, the position of ownship and an area with higher danger of collision around the strait in design TCPA. Furthermore 8 information elements are marked by all participants, but not in the same way. These are headings and routes of vessels A, B and C and positions of vessels B and C. In contrast, 26 IS are only marked by one of the participants. Thus the IS models show a high variance.

There are not only differences in the number and kind of information that is marked. Even if the same information is marked by two participants, the marked regions of their IS can be very different. Fig. 9 shows for all four subjects the marked IS on design G. It is easy to visually identify differences between the four models. For example it can be seen, that the expert user marked only few small regions in the center of the sketch, while the HMI designer has covered the entire sketch with a focus on the center. As a measure of the similarity between rectangular IS we use the root integrated squared distance (RISD), which is sensitive to differences in size and position of two rectangles. Based on Table 2 the mean RISD between two IS from different participants marking identical information is far smaller (𝑀 ̅ =66.8 mm), than for two IS marking similar information (𝑀 ̅ =208.2 mm). For every information in Table 2 that was marked at least by two participants, we calculated the mean RISD between the IS. Not surprisingly it turned out, that the 10 most similarly marked information elements (mean RISD ≤ 28.7 mm) are displayed as small icons with clear boundaries: vessel positions, beacon positions, TCPA speech bubbles and labels. However, the 11 th -most similar information is an area with high danger of collision in design TCPA. This is surprising, because there is no clear symbol or display on the screen that gives a boundary for this area. All participants marked it solely based on interpretation and aggregation of several other information elements, like predicted vessel positions, routes and strait. It is also one of the very few information elements, which was marked by all four participants. It seems that the TCPA design easily supports the recognition of critical regions, without directly highlighting them.

As a conclusion, we did not find support for the hypothesis, because the number of defined IS greatly varies and also the kind of marked information elements. Furthermore, even the same information was marked differently. However, this difference varies strongly between the information elements. H 3a :"A pie chart is an easy understandable visualization of the average attention allocation prediction."

Confronted with the average attention allocation prediction, visualized as a pie chart (Fig. 7a) and without knowing which pie chart reflects which design, all subjects raised the question of how an optimal attention distribution can be identified. Three subjects assumed that most attention should be focused on a few IS only, only HMI argued for a "balanced distribution". Cog assumes "reduced scanning effort" for few big chunks although mentioned that huge chunks might identify IS which require high effort to derive specific information. SA stated that huge chunks combined with only a low amount of IS identify less complex information spaces, but in this case it remains questionable if a small information space still offers all information to perform a task. Different to the others, only Exp analyzed the specific IS that received most attention. Exp considered the own vessel direction and speed as most important IS and preferred to have the attention focused on them. Based on observing the four subject matter experts (SMEs) the pure pie chart did not support H 3a . The biggest problem was the lack of measure to judge about the goodness of overall attention allocation.

H 3b :"A histogram is a simple to understand visualization of a reaction time prediction."

Although each participant was given an example on how the cumulative distribution function is read (e.g. "In 80% of the simulated cases the reaction time was below 10 seconds"), they focused on the frequency distribution shown as bar diagram, whereas Cog required additional support while initially focusing on the cumulative frequency distribution (line chart). Based on the four subjects we could not found support for H 3b .

H 3c :"Heatmaps are an easy understandable visualization of the average attention distribution prediction."

All participants stated that the heat map-based visualization of the attention distribution matches their expectancies. Only for design G HMI missed a hot spot for the crossing gates, Cog missed attention for one vessel and SA could not explain strong focus on the own ship in one design. While looking at the heat map all participants found arguments for their preferred design. The decision for one design based on the heat map visualization was difficult for three participants. SA decided based on the reduction in ergonomic effort (spots are located close to each other) and also looked for equally distributed hotspots of the most relevant IS. HMI based the decision on looking for hotspots covering the most relevant IS. Cog and Exp took their decision instantly based on the design that hot spotted the high traffic area best while Exp also considered the reliability of the IS. We could found support for H 3c since all SMEs understood the visualization, stated that it presents expected hotspots and were able to argue based on the visualization.

Discussion

In earlier studies we used the HEE to predict task performance and operator workload for HMI designs in the aeronautics domain and the Adaptive Information Expectancy (AIE) model to predict attention distribution in the automotive domain. The analysis of map-centric monitoring tasks is based on different prerequisites. The model creation figured out to be much more based on individual experiences and background unlike for other HMIs: In an automotive HMI or an airplane cockpit instruments communicate clearly defined information and each instrument has a fixed position. Monitoring a nautical map involves interpretation and the amount, size and position of ISs considered as relevant differ greatly between the roles that participate in the design process. This has an effect on the attention predictions. The SMEs modeled their perspective and understanding of the HMI quite differently and with different inputs the attention predictions results differ as well. Interestingly, this does seem to affect the users' expectancy: None of our participants was really surprised by the results. Especially for the heat maps it could be observed that the predictions offered only few inconsistencies from what was expected and all subjects were able to analyze and argue about the designs (H 3c ) based on their results. Furthermore the overlay of IS on the design sketches were used by participants during arguing to point at exactly the information they were talking about.

We summarize our observations into the following hypotheses:

Hp : "Using the HEE for modeling map-based monitoring tasks makes expert knowledge explicit and can be used as a basis for argumentation of a role specific perspective."

Hp a :"The resulting models differ between different roles." Hp b :"Role-specific IS can be identified and visually communicated." Hp c :"The definition and naming of IS provides a common vocabulary, which supports the discussion between different perspectives."

The average reaction time visualization as a histogram figured out to be complicated to understand (H 3b ). It was hard for the subjects to identify the shortest reaction time.

The probability and their respective cumulation were difficult to understand for the subjects. The visualization of the attention allocation as a pie chart did not motivate the subjects to reflect the percentage share of specific IS of the overall attention allocation. Rather the subjects were unsure about a concrete measurement to argue for or against big chunks or for comparing slide sizes (H 3a ).

Conclusion and Future Work

In this paper we presented a tool-supported, semi-automated process to predict human attention allocation that enables non-experts in cognitive modeling to analyze nautical chart displays. In a qualitative study experts with different backgrounds were able to understand and successfully perform the process and generate HMI-models of three different HMI design variants in a reasonable amount of time.

We initially assumed that the HEE tool helps non-human factor experts to create valid predictions of human attention distribution to map-based HMIs. In contrast this study revealed that different HEE users created very different models, resulting in different predictions. The subjects identified a total amount of 130 information sources, only 4 are equal, 8 more are similar but marked differently. 26 were only identified by one of the participants. We also tested different types of result visualizations. The visualization of the predicted reaction time distribution by a histogram figured out to be too complicated to understand. Similar, pie-charts depicting the average attention allocation offered only little use for most of the subjects. However, heat maps of attention distributions were understood by all subjects. It turned out that information depicted in the heat maps was in the range of what was expected or learnt by the participants during modeling. We further observed that HEE users easily articulate their respective expert knowledge while referencing their marked IS on the heat map.

It is technically easy to identify differences between models created by different users. We assume that showing these differences with a suitable visualization can help to exchange knowledge between experts with different background and also differences in their mental models can be revealed. Furthermore the explicit definition of IS could help experts of different areas to communicate with a shared vocabulary. These assumptions will be investigated in future studies.

  attention prediction process relations with quantors into a partial order (visualized as a Hasse diagram) we specified them formally.

Fig. 2 .

 2 Fig. 2. Generic structure of rules for simulation of the operator's monitoring behavior in CASCaS-like pseudo code.

Fig. 3 .

 3 Fig. 3. Design Modeling Perspective of the HEE tool.

Fig. 4 .

 4 Fig. 4. Transcription of the Expectancy relation to an Excel sheet.

Fig. 5 .

 5 Fig. 5. Section of a Hasse diagram depicting the partial ordered graph with ranks on the left side. Node labels contain the abbreviation of an IS and a design name, separated by underscore.

Fig. 6 .

 6 Fig. 6. Different variants of showing future vessel positions on a map display: (a) Current version, (b) Abstracted design of the map display: Speed Vectors, (c) Gates, (d) TCPA and position prediction.

Fig. 7 .

 7 Fig. 7. Pie chart of the average attention allocation (a) and a heat map illustrating the monitoring behavior of the simulated operator (b).

H 3 :Fig. 9 .

 39 Fig. 9. IS marked by the subjects for design G. Darkness of the regions corresponds to the calculated expectancy coefficients.

Table 1 .

 1 Amount of time required to explain and to perform each modelling task in minutes.

	Subject	IS		Expectancy	Relevance	Entire	Entire
		Identification					Modelling Time	study
	Cog	3	23	6	16	1	3	42	94
	SA	6	70	3	55	1	7	132	112
	HMI	5	52	2	53	1	7	112	105
	Exp	6	90	3	54	1	16	160	231

Table 2 .

 2 Comparison of IS between participants (excerpt).

	Information	Cog	SA	HMI	Exp
		identical similar identical similar identical similar identical similar
	Position beacon center-south	LF1	L1	LF3	L4
	Position beacon south-east		L4		L3
	Ownship Position	ESP	E	POwn	E
	Vessel A Position				RPLAB A
	Vessel B Position	BP	V3		RPLAB B

http://www.emaritime.de/projects/cosinus, last checked 04/15/15

Acknowledgments. The HEE is developed in the EU ARTEMIS JU project HoliDes (http://www.holides.eu/) SP-8, GA No.: 332933. Any contents herein reflect only the authors' views. The ARTEMIS JU is not liable for any use that may be made of the information contained herein.


Changhyeon Lee

Jaedong Lee
email: jdlee@korea.ac.kr

Gerard J Kim
email: gjkim@korea.ac.kr

Touch Skin: Proprioceptive Input for Small Screen Devices

Keywords: Touch screen, proprioception, smart watch

The smart watch, increasingly gaining popularity, has limited input and output capabilities due to its size and thus mostly used as a surrogate device to the smart phone. In this poster, we propose "Touch Skin (TS)" that enlarges the interaction space of the smart watch using the hand (or skin) surface and proprioceptive sense. While the input interface is displayed on the small smart watch screen, the interaction is carried out by touching on the larger hand surface to which the input interface elements (e.g. graphical buttons and keys) are mapped. We hypothesize that even though the display and interaction surface are separated, the humans are nevertheless able to perform competently based on one's proprioceptive sense. While sensing for finger touch positions on the hand/skin surface remains to be a technical hurdle, we first explore whether our hypothesis is valid through an enactment study comparing the performance the Touch Skin input to that of the nominal smart phones.

Introduction

The smart watch has limited input and output capabilities due to its form factor. As such they are used more as of a fashion item and a surrogate slave device to the larger smart phones. For example, the touch screen on the smart watch can only display about four to six easily touchable icons (e.g. on a 2x2 grid) and touching in letters or digits, selecting an application out of many menu items can be very difficult.

Improving the interactivity of the smart watch can make it more independent and selfcontained device. With this motivation, we propose in this poster the concept of "Touch Skin (TS)," that enlarges the interaction space of the smart watch using the large hand (or skin) surface area and proprioceptive sense. Proprioception refers to the human's capability of sensing of the relative position of neighboring parts of the body and strength of effort being employed in movement [START_REF]Wikipedia, Proprioception[END_REF]. In humans, it is provided by proprioceptors in skeletal striated muscles and in joints. With the TS, while the input interface is displayed on the small smart watch screen, the interaction is carried out on the hand surface to which the input interface elements (e.g. graphical buttons and keys) are mapped. We hypothesize that even though the display and interaction surfaces are separated, the humans are projected to nevertheless perform very well based on the proprioceptive sense. For example a human user might be able to pinpoint (with one's finger) to a particular part on the hand (e.g. palm, back, fingers) up to some accuracy and resolution, e.g. up to par with similar touch screen devices.

The concept of TS is well illustrated in Figure 1.

To realize "Touch Skin", sensing movement or position on skin surface and installing such a capability on the smart watch will be the major technical challenge. We believe that, for example, a smart watch equipped with a depth or ultrasonic sensor on its side can detect finger position on the palm at sufficient resolution to realize our proposal (see Figure 1). In this study, however, as a first step, we explore whether the proposed interaction scheme will perform sufficiently or comparable to that of the smart phone (for certain major tasks) through an enactment study (i.e. pretending as if the finger/skin tracking is operational).

Related Work

Proprioception has been applied to HCI in few occasions. Mine et al. introduced the concept of the "scale-down grab" in which a remote object manipulation was carried out by scaling down the entire scene and bringing the target object within reach and rely on the proprioceptive sense to manipulate it well [START_REF] Mine | Exploiting Proprioception in Virtual-Environment Interaction[END_REF]. Gustafson et al. presented the notion of "imaginary" interface in which one might interact on an arbitrary object well even a without display (e.g. on the palm) as a convenient substitute. While similar to our approach, theirs was not based on the notion of proprioception, but rather similarity in the physical mapping and familiarity [START_REF] Gustafson | Imaginary Interfaces: Spatial Interaction with Empty Hands and Without Visual Feedback[END_REF]. Using the skin (e.g. palm of the hand) for interaction surface is an increasingly popular idea. In the context of wearable augmented reality, it is an attractive option where a interface elements can be augmented onto the hand using a projector or through a see-through display [START_REF] Mistry | SixthSense : A Wearable Gestural Interface[END_REF] and where the fingers/hands are tracked using a vision based approach. Skinput, on the other hand, by Harrison et al., sought to transform the skin into a touch screen like surface by analyzing the mechanical vibrations that propagate throughout the body [START_REF] Harrison | Skinput: Appropriating the Body as an Input Surface[END_REF]. Laput et al. presented an augmented input method for smart watches, by projecting icons on the nearby user skin and making them touch sensitive [START_REF] Laput | Skin Buttons: Cheap, Small, Lowpowered and Clickable Fixed-icon Laser Projectors[END_REF]. Our concept is similar except that we claim that the interaction surface can be relatively far from the small screen device (like the smart watch) without any display.

Experiment

Experimental Procedure

The purpose of the "enactment" pilot experiment was just to quickly and partially validate the feasibility of our approach, that is, whether the proposed proprioceptive interaction can produce input performance competitive to that of the usual smart phone. Five paid subjects, 4 men and 1 women with a mean age of 25 (one left handed), participated in the experiment. The experiment tested four input methods: (1) regular smartphone input, (2) TS with keys mapped to the palm, (3) TS with keys mapped to the back of the hand, and ( 4) TS with keys mapped to the fingers (see Figure 2). The dependent variables were the input completion time, error, and answers to a usability questionnaire (omitted here for lack of space).

The task was to enter fifteen 10 digit numbers as quickly and accurately as possible using the respective input methods tried out in balanced order. The experiment was an enactment because detecting of the skin touch was not implemented, thus no sensor error nor delay was considered. The subject performance was recorded by a video camera and measured manually by the experimenter. When the TS was used, the input key layout was displayed on a separate monitor screen in a small scale (as small of a smart watch screen). The subject wore a glove with the input key mapping drawn on it just to aid the experimenter manually measure for the input error while reviewing the recorded video footage. The subjects were instructed to enter the numbers as fast and accurately as possible. When testing for the three TS conditions, they were also instructed to only watch the monitor screen rather than their hands.

Results

The three TS conditions did not differ in the task completion time but TS with the mapping on the back of the hand showed a lower accuracy with a statistical significance. When compared to the smartphone, the TS finger and TS palm were less accurate but actually slightly faster in performance (see Figure 2). The survey showed that while the smartphone touch screen interface was still superior in most categories, TS still received a very positive response and even preferred over the smartphone. Overall, we can see that given a stable skin event detection method, the TS would exhibit reasonable input performance with satisfactory usability.

Discussion and Conclusion

In this poster, we propose "Touch Skin (TS)" that enlarges the interaction space of the smart watch using the hand (or skin) surface and proprioceptive sense. While the input interface is displayed on the small smart watch screen, the interaction is carried out by touching on the larger hand surface to which the input interface elements (e.g. graphical buttons and keys) are mapped. The pilot enactment study showed promising results of input performance and usability competent to that of the smart phone. We plan to implement and mount a finger detection sensor on the smart watch and conduct a more formal and full-fledged usability experiment to further validate our approach. Ease of Use Intuitiveness Ease of Learning Familiarity Preference SmartPhone Palm Finger Back

Fig. 1 .

 1 Fig. 1. The concept of the proprioception based "Touch Skin" input method for a small screen device.

Fig. 2 .

 2 Fig. 2. Top: Performance results between TS palm, TS finger and smartphone: task completion time (left) and accuracy (right). Bottom: Usability survey results.

Acknowledgements

. This research was funded by the Forensic Research Program of the National Forensic Service (NFS), Ministry of Government Administration and Home Affairs, Korea. (NFS-2015-DIGITAL-04), and the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT & Future Planning (No. 2011-0030079)


Euan Freeman
email: euan.freeman@glasgow.ac.uk

Stephen Brewster
email: stephen.brewster@glasgow.ac.uk

Vuokko Lantz
email: vuokko.lantz@nokia.com

Interactive Light Feedback: Illuminating Above-Device Gesture Interfaces

Keywords: Above-Device Interaction, Gesture Feedback, Gesture Interaction, Interactive Light Feedback, Mobile Devices

In-air hand gestures allow users to interact with mobile phones without reaching out and touching them. Users need helpful and meaningful feedback while they gesture, although mobile phones have limited feedback capabilities because of their small screen sizes. Interactive light feedback illuminates the surface surrounding a mobile phone, giving users visual feedback over a larger area and without affecting on-screen content. We explore the design space for interactive light and our demonstration shows how we can use this output modality for gesture feedback.

Introduction

Above-device interfaces allow users to interact with their mobile phones using inair hand gestures over the device. Gestures can be used when touchscreen input is unavailable or inconvenient; for example, users could gesture to follow a recipe on their phone without getting the touchscreen messy or having to wash their hands between using the phone and handling food. Gestures also allow more expressive forms of input than touch, allowing users to do more than simply touch the screen.

Users need helpful feedback when gesturing to be able to interact confidently and effectively. However, mobile phones have limited display capabilities because of their small size. Giving visual feedback about gesture interaction would affect on-screen content and may require redesigning application interfaces for gesture and non-gesture use. Audio and tactile feedback have been considered instead, although lacks the expressive, spatial characteristics of visual feedback. We propose giving visual feedback in the space surrounding the device, using light to illuminate surrounding table surfaces while users gesture (as in Figure 1). We call this interactive light feedback.

Mobile phones have already used light to illuminate table surfaces for other purposes. For example, Samsung's Galaxy S6 Edge [START_REF]Samsung Galaxy S[END_REF] has a curved screen which illuminates the table surface when placed face-down, glowing different colours to show who is calling. Some of Sony's Xperia phones feature an Illumination Bar [START_REF]Sony Illumination[END_REF] along one edge, which is used for visualisations during media playback. Research prototypes have also used lights embedded within the device bezel [START_REF] Müller | Sparkle: An Ambient Light Display for Dynamic Off-Screen Points of Interest[END_REF] and case [START_REF] Qin | Dynamic Ambient Lighting for Mobile Devices[END_REF] to visualise off-screen content. We build on these examples by looking at how interactive light can be used for dynamic gesture feedback instead.

In this paper we look at the design space for interactive light feedback. We share ideas about how light can be used to create gesture feedback and our demonstration allows others to experience these designs.

Related Work

Users require plentiful feedback when gesturing. However, mobile phones have small displays which limits the amount of feedback they can give. Others have suggested using other modalities instead for gesture feedback. Freeman et al. [START_REF] Freeman | Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions[END_REF] looked at tactile feedback for in-air gestures. They evaluated a variety of methods of giving tactile feedback, including ultrasound haptics and vibration from smartwatches and other wearables. They found tactile feedback effective at enhancing visual feedback, although it lacked necessary bandwidth to replace it.

Others have used audio feedback for in-air interactions while phones are in a pocket, where visual feedback would go unnoticed. Imaginary Phone [START_REF] Gustafson | Imaginary Phone: Learning Imaginary Interfaces by Transferring Spatial Memory from a Familiar Device[END_REF] and Nenya [START_REF] Ashbrook | Nenya: Subtle and Eyes-Free Mobile Input with a Magnetically-Tracked Finger Ring[END_REF] read selected menu items aloud as users gestured, either in front of their body (for Imaginary Phone) or by moving a ring on their finger (for Nenya). In these cases, audio feedback came after gestures; users received no cues during interaction to help them. Our research looks at continuous interactive light feedback during gesture interaction, giving users cues as they interact.

In similar work to ours, Qin et al. [START_REF] Qin | Dynamic Ambient Lighting for Mobile Devices[END_REF] presented a prototype mobile phone which had LEDs embedded in its case. They presented an example application where users could respond to phone calls by touching the table on the left (illuminated red) or right (illuminated green) side of the phone. We build on this idea by using interactive light for dynamic feedback during gestures, rather than static feedforward before interaction. Rainbowfish [START_REF] Grosse-Puppendahl | Ambient Gesture-Recognizing Surfaces with Visual Feedback[END_REF] used a grid of LEDs to give feedback about gestures over a proximity-sensing surface, acting as a display. In our work, light is used to illuminate the area around a display, instead.

Interactive Light Feedback

Earlier we defined interactive light feedback as visual feedback given by illuminating the space surrounding a device. In our prototype (Figure 1), we use LEDs around the device edge to present interactive light feedback, although light sources could be enclosed within the bezel itself, as in Sparkle [START_REF] Müller | Sparkle: An Ambient Light Display for Dynamic Off-Screen Points of Interest[END_REF]. We use LEDs as they are small, cheap and have low power requirements. Our prototype consists of a flexible strip of 60 LEDs1 (approximately 7mm apart) affixed to a mobile phone and controlled by an Arduino microcontroller. Each LED is independently controllable and can vary its own hue and brightness.

Each LED has two design properties: brightness, which is always variable, and hue, which is fixed in some LEDs. These basic properties of light become more expressive when varied over time and location. Harrison et al. [START_REF] Harrison | Unlocking the Expressivity of Point Lights[END_REF] showed how expressive and informative a single LED can be when change in brightness is animated over time. They presented a set of "light behaviours" which could be used to communicate information from a single light.

We have found dynamic change in brightness to be especially effective for gesture feedback, as users immediately see a response to their hand movements. One of our feedback designs uses brightness to show how well users' hands and gestures can be sensed, using a visibility metaphor: when light is easier to see (that is, brightness is greater), it is because hands can be more easily seen (Figure 1). If light is less visible, it is because users are not as easily sensed; for example, their hands may be too far from the sensor. Brightness can also be varied at different locations to create spatial patterns. For example, lights could be illuminated (that is, brightness greater than zero) to show hand position relative to the mobile phone (as in Figure 2). This is similar to the "shadow" feedback design in Rainbowfish [START_REF] Grosse-Puppendahl | Ambient Gesture-Recognizing Surfaces with Visual Feedback[END_REF], although feedback does not occlude on-screen content, since it is around the device instead. Spatial and temporal changes in brightness could also be combined to create rich feedback metaphors; for example, a progress bar could "fill up" over time as users dwell to make a selection (as with the gestures in [START_REF] Freeman | Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions[END_REF]).

Hue can also be changed by some LEDs, allowing colourful feedback designs. Like brightness, hue can be varied over time and location to create expressive and informative feedback. Colours often have iconic meanings, making them useful for presenting state or static information. For example, green and red light could be used to show if a gesture was recognised or not. Hue could also be used for application-specific feedback metaphors; for example, using blue and red light to show temperature for thermostat control.

Summary

In this paper we described interactive light feedback, a technique which allows phones to give visual feedback about above-device gestures without affecting on-screen content. We discussed how brightness and hue allow rich information presentation and described how these could be used for gesture feedback. Our demonstration allows others to interact with a gesture interface and experience these designs themselves, to see how interactive light feedback can be used to informatively illuminate in-air interfaces.

Fig. 1 .

 1 Fig. 1. Interactive light feedback around a mobile phone. Here, brightness changes dynamically to show how users how well their hands can be sensed.

Fig. 2 .

 2 Fig. 2. Varying brightness at different locations to show hand position.

Adafruit NeoPixel LED Strip: http://www.adafruit.com/products/1506

Acknowledgements

This research was part-funded by Nokia Technologies, Finland.


Shiroq Al-Megren

Ahmed Kharrufa
email: ahmed.kharrufa@ncl.ac.uk

Jonathan Hook
email: jonathan.hook@york.ac.uk

Amey Holden
email: ameyholden3@gmail.com

Selina Sutton
email: selina.sutton@ncl.ac.uk

Patrick Olivier
email: patrick.olivier@ncl.ac.uk

Comparing Fatigue when Using Large Horizontal and Vertical Multi-Touch Interaction Displays

Keywords: Large displays, interaction, tabletops, fatigue, ergonomics

We report on a user study that compared muscle fatigue experienced when using a large multi-touch display in horizontal and vertical configurations over a one-hour period. Muscle fatigue is recognized as the reduction in a muscle's capacity to generate force or power output and was measured objectively and subjectively before and after a puzzle-solving task. While subjective measures showed a significant level of overall arm muscle fatigue after the task for both configurations, objective measures showed a significant level of muscle fatigue on the middle deltoids and the non-dominant extensor digitorum for the vertical configuration only. We discuss the design implications of these findings and suggest relevant future areas of investigation.

Introduction

Muscle fatigue is the transitory decrease in a muscle's capacity to contract and perform actions, which leads to the development of musculoskeletal imbalance if it persists for longer stretches of time and overruns the body's recovery system [START_REF] Vøllestad | Measurement of human muscle fatigue[END_REF][START_REF] Wahlström | Ergonomics, musculoskeletal disorders and computer work[END_REF]. Previous work has reported on subjective perceptions of fatigue [START_REF] Meyer | Device comparisons for goal-directed drawing tasks[END_REF][START_REF] Sears | High precision touchscreens: design strategies and comparisons with a mouse[END_REF], which are deemed unreliable for assessing physical demands [START_REF] Barriera-Viruet | Questionnaires vs observational and direct measurements: a systematic review[END_REF], and objective measures of muscle fatigue [START_REF] Young | Wrist and shoulder posture and muscle activity during touch-screen tablet use[END_REF][START_REF] Lozano | The impact on musculoskeletal system during multitouch tablet interactions[END_REF] on tablet sized vertical touchscreens. Past research has also explored how display angle affects a range of different aspects of users' interaction with large interactive displays (e.g. [START_REF] Ichino | Effects of the display angle in museums on user's cognition, behavior, and subjective responses[END_REF][START_REF] Muller-Tomfelde | Tilted tabletops: In between horizontal and vertical workspaces[END_REF]). However, relatively little attention has been paid to the impact that display angle has on users' physiology over time and consequential musculoskeletal damage. The few studies that have investigated this issue have required subjects to perform a constrained task repeatedly for a relatively short time, rather than exploring the typical varied and longitudinal usage that designers envisage, such as teachers' interaction with interactive boards (e.g., [START_REF] Zerpa | The effect of common teaching tools on upper extremity muscle activity[END_REF] who looked at vertical boards only). Other studies, such as [START_REF] Morris | A field study of knowledge workers'; use of interactive horizontal displays[END_REF][START_REF] Wigdor | Living with a tabletop: Analysis and observations of long term office use of a multi-touch table[END_REF], have reported informal and conflicting insights for longitudinal usage of large interactive displays.

We present a study that measured muscle fatigue objectively and subjectively for both horizontal and vertical large interactive displays using an unconstrained task, which required users to perform gestures commonly used in multi-touch interaction. Our results suggest that when designing multi-touch applications that will be used for extended periods, designs that consider fatigue should, in general, favor horizontal over vertical displays, and in both cases should aim at locating interactions at closely accessible locations on the display whenever possible. Our results also suggest that designers should pay close attention to objective measures of fatigue when evaluating the appropriateness of their designs to users' physiology, rather than relying on subjective measures alone.

Study Design

We performed the study on a Microsoft PixelSense multi-touch tabletop with a 30" rear projected display (21" high, 27" deep, and 42.5" wide). The task was to complete a series of 25-piece puzzles, using the tabletop's Puzzle application, which was adapted to include scaling of pieces. Pieces could be translated, scaled and rotated, or an integrated combination of two or more of these actions, using one or more fingers, uni-or bi-manually. Initially, the pieces of each puzzle were displayed at random orientations, sizes and locations. Two pieces could only connect to form one larger piece if they were a similar size (within 20%) and orientation (within 30°). Once completed the puzzle was replaced with another. This exercise was repeated for onehour. The puzzles were displayed in a random order and subjects completed five puzzles on average. This task was chosen because it required participants to complete common types of interactions used when performing different tasks using large interactive displays, along with intermittent periods of no activity.

Procedure

Surface Electromyography (SEMG) was used to monitor three muscles on each arm before, during and after the task. These muscles are: middle deltoid, bicep brachii, and extensor digitorum muscles (see Figure 1a). Lozano et al. [START_REF] Lozano | The impact on musculoskeletal system during multitouch tablet interactions[END_REF] suggested these muscles are appropriate sites for measuring gestural interaction and fatigue on a multi-touch tablet when using SEMG. The deltoid placement was altered, from anterior to middle, to consider both shoulder abduction and flexion. To locate the placement of the SEMG sensor for each muscle, a specific location on the arm was palpated as the subject performed a movement activating the target muscle. The skin was then shaved, wiped with alcohol and the sensors placed parallel to their prospective muscle at 2cm apart. We followed Criswell's guidelines [START_REF] Criswell | Cram's Introduction to Surface Electromyography[END_REF] to ensure correct placement. Two measures were then collected: maximal voluntary isometric contractions (MVIC) and ratings using the Borg scale [START_REF] Borg | A new generation for scaling methods: Level-anchored ratio scaling[END_REF] both before and after the task (as described below). The subjects were then asked to sit and familiarize themselves with the task as the possible interaction techniques were described verbally. They were then asked to wait for the start and the end of the session to be announced. During the session, ten electromyograms (EMG) were recorded for muscle activity analysis. Participants were all males to limit interferences due to anatomical differences [START_REF] Cioni | Sex differences in surface EMG interference pattern power spectrum[END_REF].

Experiment 1: Horizontal Configuration

Eighteen males, who were familiar with multi-touch technology and were free of musculoskeletal disorders, took part. The mean age was 26.4 (±4.6). Three were lefthanded. The tabletop was raised to a height of 26" using wooden panels to allow comfort and participants sat on a chair 18" in height (Figure 1a).

Experiment 2: Vertical Configuration

A second participant group of eighteen males took part; all were right-handed, familiar with multi-touch technology and free of musculoskeletal disorder, with an average age of 28.8 (±4.8). The tabletop was placed on its side on top of a desk and participants were sat on a chair 18" in height (Figure 1b).

Objective Measures

Muscle fatigue has objectively been detected non-invasively using sonomyography, near-infrared spectroscopy, mechanomyogram, and SEMG. Recent research has also proposed a novel approach, "Consumed Endurance" [START_REF] Hincapi-Ramos | Consumed endurance: a metric to quantify arm fatigue of mid-air interactions[END_REF], which does not necessitate the employment of specialized equipment. A survey carried out by Al-Mulla et al. [START_REF] Al-Mulla | A review of non-invasive techniques to detect and predict localised muscle fatigue[END_REF] found SEMG to be the most suited measurement for the detection and quantification of fatigue. Previous applications of SEMG for the assessment of fatigue include interacting with software on a traditional setup [START_REF] Peres | Software ergonomics: relating subjective and objective measures[END_REF] and a multi-touch table [START_REF] Young | Wrist and shoulder posture and muscle activity during touch-screen tablet use[END_REF][START_REF] Lozano | The impact on musculoskeletal system during multitouch tablet interactions[END_REF]. Accordingly, SEMG was used to detect and measure physiological changes to skeletal muscles due to contraction. Processed EMG data can provide information about localized fatigue and force. A MVIC, where the muscle's tension changes while its length remains the same [START_REF] Sherwood | Human Physiology: From Cells to Systems[END_REF], is used to quantify force and localized muscle fatigue. To gain a MVIC the subject is instructed to achieve the greatest possible force of contraction, constantly, for a short period. To do this our subjects were asked to remain seated and maintain a posture for 10 seconds while holding a 2.5kg weight. The postures were; i) Middle deltoid: arm elevated at 90° in the frontal plane; ii) Bicep brachii: arm held close to the body while elevating the forearm at 90° in the sagittal plane; iii) Extensor digitorum: the forearm resting on a desk with the wrist resting on the edge of the table. A MVIC is a baseline and so future readings may be greater than 100%.

Localized Fatigue. For fatigue indexing, we adopted the median power frequency (MPF) of the MVIC in the time domain as a reference point [START_REF] Phinyomark | The Usefulness of Mean and Median Frequencies in Electromyography Analysis[END_REF]. Essentially, a decreasing MPF signal indicates that muscle fatigue is increasing. Previous research corroborates the reliability and consistency of this method of analysis (e.g. [START_REF] Murata | Evaluation of shoulder muscular fatigue induced during mouse operation in a VDT task[END_REF]) unlike using the amplitude in the time domain, where the literature reports significant contradictions (e.g. [START_REF] Lozano | The impact on musculoskeletal system during multitouch tablet interactions[END_REF], [START_REF] Murata | Evaluation of shoulder muscular fatigue induced during mouse operation in a VDT task[END_REF]).

Force. 'Force' quantifies a muscle's electrical activity during contraction and is described as a percentage of a MVIC. To extract force information we integrated a rectified EMG, a long-established technique due to their linear relationship [START_REF] Lawrence | Myoelectric signal versus force relationship in different human muscles[END_REF].

The SEMG used was ZeroWire, a wireless system with six surface channels, with Biosense's bio-logic disposable press-stud electrodes (Ag-AgCI). This system operates using light autonomous signal processing and power transmission units, each weighing 10gm. Each channel provides a bandwidth of 10-1000 Hz for a signal sampled at 2000 sample/sec. The transmitters wirelessly transfer the signals captured with the electrodes to the main unit, which is directly connected to a computer running the ZeroWire software suite. This minimizes the restriction of a user's movements. Matlab and Microsoft Excel were used to process the EMGs and analyze the results.

Localized Fatigue Analysis

The 10-second MVIC collected before and after the task was divided into 10 segments with a 50% overlap. This led to 3000 data points for each segment from the original 2000. The last segment was excluded from the analysis because of the overlap. Each segment was then rectified and passed through a low pass fourth-order Butterworth filter with a cut-off frequency of 500Hz. A fast Fourier transform was then performed to calculate the power spectrum of each segment from which the MPF is obtained. Matlab was used to process the EMG using functions provided by the Signal Processing Toolbox (DSP) and the Biomechanics et al. Toolbox (BEAT) [23].

For normalization (to eliminate variations between the subjects such as age and muscle mass) the data recorded before the task was averaged and used as a reference value for the data collected after the task. These values were averaged to produce singular values representing the MPF of each muscle. A paired t-test was then carried out between the two data sets.

Localized Fatigue Results

Horizontal. While some muscles did show a decrease of MPF indicating some level of fatigue, none of them showed a significant evidence for increased fatigue. The dominant and non-dominant extensor digitorum showed a decrease to 80% and 63% respectively, while the dominant and non-dominant bicep brachii showed a decrease to 88% and 95% respectively. Moreover, while the dominant middle deltoid also showed evidence of significant increase of MPF indicating decreased fatigue (119% and t(17)=1.74, p=0.01), no change was noted for the non-dominant middle deltoid.

Vertical. The middle deltoid showed a significant decrease of MPF indicating increased fatigue (63% and t(17)=1.74, p=0.003 for the dominant, and 71%, t(17)=1.74, p=0.02 for the non-dominant side). As for the extensor digitorum only the non-dominant side showed a significant decrease of MPF indicating increased fatigue (75% and t(17)=1.74, p=0.01) with no noted decrease for the dominant hand. The brachii showed non-significant decrease of MPF for the non-dominant hand to 79% and no-decrease for the dominant hand.

Force Analysis

Ten 1-minute EMGs were collected during the experimental task at 5-minute intervals. Each EMG was passed through a low-pass fourth-order Butterworth filter with a cut-off frequency of 500Hz. The averaged root mean square was then calculated using BEAT, which was then averaged. Normalization was carried out using the averaged MVICs collected before the start of the task. The normalized values were then averaged to represent the muscle activity as a percentage of the MVIC (see Figure 2). A one-way repeated measure analysis of variance (ANOVA) was then carried out.

Fig. 2 Force results for 1-minute readings collected every 5 minutes for both configurations

Force Results

Horizontal. A one-way repeated ANOVA for the 10 measures showed that only the dominant extensor digitorum showed significant inconsistency of muscle activity throughout the task (a change of 13% and F(17, 9)=3.09, p=0.002). No significant evidence was found against the other muscles proving their activity to be consistent. Nevertheless, the non-dominant extensor digitorum showed an increase and decrease of up to 8% and the bicep brachii showed a change of up to 10% and 14% for the dominant and non-dominant sides respectively. The middle deltoids' activity proved relatively stable with activations ranging from 8-11% for both sides. Vertical. A one-way repeated ANOVA for the 10 measures showed significant inconsistency of muscle activity throughout the task for the dominant extensor digi-torum (a change of 13% and F(17, 9)=2.85, p=0.004). No significant evidence was found against the other muscles proving their activity to be consistent. Nevertheless, both biceps brachii showed an increase and decrease in activation of up to 13%, while both middle deltoid muscles showed relatively consistent activation of up to 18%. Table 1 marks the muscles that showed significant evidence of increase in level of fatigue after the trial session for both configurations. It also marks significant inconsistencies in activation of force during task interactions.

 Force × × × × Bicep Brachii Fatigue × × × × Force × × × × Extensor digitorum Fatigue × × ×  Force  ×  × 5

Subjective Measures

When examining fatigue, subjective ratings are commonly used to assess perceived physical exertion and supplement objective measures. We used the Borg CR100 scale [START_REF] Borg | A new generation for scaling methods: Level-anchored ratio scaling[END_REF], which is a fine graded ratio scaling method that estimates the level of exertion and determines the ratio relationship between perceptual responses. The scale contains subjective dynamic ranges with values ranging from 0 to 120 annotated with verbal anchors ranging from "nothing at all" to "absolute maximum". This scale was chosen as the most appropriate after critical assessment and comparison of four other scales. The subjects were asked to rate their perceived muscle exertion after the MVIC had been recorded before and after the experiment. The average of the scores collected was analyzed for statistical significance using a paired t-test. For the horizontal configuration, the averaged value of the ratings was significantly greater after the task (t(17)=1.74, p=0.0003). Similarly for the vertical configuration, the averaged value was significantly greater after the task (t(17)=1.74, p=0.0005). This indicates an overall increased perceived level of fatigue for both configurations (see Table 2).

Discussion and Conclusion

The main aim of the study was to compare the muscle fatigue experienced when using large multi-touch interactive displays in horizontal and vertical configurations, over a one-hour period.

Our findings from the objective measures provide statistically significant evidence for the presence of potentially damaging fatigue for the middle deltoids for the vertical case only and not the horizontal. This can be due to the elevation required of the arms when interacting with the vertical display (i.e. gorilla-arm effect [START_REF]Gorilla arm[END_REF]) and the employment of larger force activation percentages. The presence of fatigue was also noted for the non-dominant extensor digitorum despite consistent activation of force. This can be due to the regular use of the non-dominant muscle (either uni-or bimanually) in the vertical configuration, as can be derived from the significantly inconsistent activation of the dominant extensor digitorum. The inconsistent activation of dominant extensor digitorum suggests that the users were alternating between dominant and non-dominant sides. Unlike the objective measures, the results of the subjective measures showed that the subjects perceived the presence of fatigue in both the vertical and horizontal configurations with statistical significance. This emphasizes the unreliability of a person's perception of fatigue, where objective results found significant levels of muscle fatigue that can be damaging only in the vertical case.

These results have clear implications on interaction design for large interactive displays. Vertical interactive displays, while suitable for intermittent use over short periods of time, are not as suitable for frequent longer useour study showed presence of fatigue for a one-hour task, but it might occur in shorter durations. Prolonged activities leading to muscle fatigue, which has previously been reported anecdotally, have the potential to lead to musculoskeletal disorders [START_REF] Vøllestad | Measurement of human muscle fatigue[END_REF][START_REF] Wahlström | Ergonomics, musculoskeletal disorders and computer work[END_REF]. For tasks that require frequent interactions in the range of one or more hours, designers should choose a horizontal display where possible to minimize damage to the musculoskeletal system, modify the design of interaction techniques to reduce the need for continuous interaction, or to change the location of distant interactions when possible to more accessible locations on the display. Moreover, subjective measures showed a significant level of fatigue for both horizontal and vertical cases, indicating that for either case, designers should locate points of interaction in spaces easily accessible to the user whenever possible. As a simple example, for large displays, the use of contextual commands that are located based on users' interaction location is recommended over the traditional desktop model of showing commands on upper or side toolbars and menus. However, identifying the display space that helps reduce fatigue, whether on horizontal or vertical displays, is the subject of further investigation.

This study has several limitations that point towards future work. We studied fatigue for only male participants to overcome physiological differences. Additional studies with female participants would give insight on sex differences and its effect on measured fatigue when interacting with large interactive displays. Furthermore, our study only considered a small subset of the muscles' activated when interacting with the display and only two angles of interaction. In the future, we plan to assess the fatigability of other muscles (e.g. capitis muscles) at additional angles of interaction.

Fig. 1 .

 1 Fig. 1. The experimental set up for the horizontal (a) and vertical (b) configurations

Table 1 .

 1 Objective measures summary of significant increase in fatigue and force inconsistency

				Significant Evidence
			Horizontal		Vertical
			Dom.	Non-Dom.	Dom.	Non-Dom.
	Middle Deltoid	Fatigue	×	×	

Table 2 .

 2 Subjective mean and standard deviation results for the horizontal and vertical configurations

	Configuration	Before	After
	Horizontal	3.97 (±7.96)	15.44 (±14.93)
	Vertical	3.17 (±5.40)	12.44 (±11.92)


José Abdelnour-Nocera

Chris Csikszentmihályi

Torkil Clemmensen

Christian Sturm
email: c.sturm@arolis.com

Jose Abdelnour-Nocera
email: jose.abdelnour-nocera@uwl.ac.uk

Design, Innovation and Respect in the Global South

Keywords: interaction design, innovation, development, global south

Panel Topics and Objective

The aim of this panel is to explore the current global discourse on development and situate these in terms of the real challenges faced by interaction designers 'on the ground'. Through their different perspectives and experiences panelists, will articulate the different ontological, epistemological and methodological dimensions of the concept and practice of interaction design in the Global South. This means questioning (1) 'what' we mean by design; (2) the relationships 'between' designers, what is designed and the intended users of those designs; and (3) 'how' we design. At different levels the panelists will question the development discourse with a particular focus on the United Nations new universal agenda for post-2015 development goals set in 2013 by a high-level panel of presidents and foreign ministers: To what extent should the field of interaction design and international development (IDID) be underpinned by this agenda? Are development discourses such as the post 2015 agenda limiting innovation and sustainability in the Global South? How can designers in the Global South untangle the cultural and political scripts embedded in imported ID practices and tools? How can the Global South create their own design practices outside of these development discourses?

The panel will provoke different answers and a useful debate around these issues. in particular psychology as a science of design. The focus of his research is on cultural psychological perspectives on usability and user experience. As Danish representative in IFIP (International Federation of Information Processing) TC 13(Technical committee on Human-Computer Interaction), and chair (2008)(2009)(2010)(2011)(2012)(2013)(2014) of Working Group 13.6 on Human Work Interaction Design (HWID), he co-organizes a series of international working conferences on work analysis and usability/user experiences in organisational, human, social and cultural contexts..

2.3

Christian Sturm: Respect-Based Design "It is not the economy -"It's the respect, stupid!" Contrary to the common perspective that economic development is driving the well-being of people around the world, it is the missing respect that is responsible for the lack of it: lack of respect from employers toward employees, lack of respect from employees toward their employers, lack of respect from politicians toward citizens and vice versa, lack of respect from researchers toward study participants, from developers and designers toward us-ers…the list is almost infinite. I argue that teaching and practicing respect together with raising the level of self-awareness and awareness of others is the closest we can get contributing to the "Post-2015 Development Agenda". Let's think of respectbased design approaches we could develop to get started.

Christian is professor at the Hamm-Lippstadt University of Applied Sciences in Germany. He holds a bachelor in computer science from Furtwangen University and a PhD in cognitive psychology (major), cultural anthropology (minor) and telematics (minor) from the University of Freiburg. His research interests include cross-cultural aspects of HCI, UX and entrepreneurship, based on his experience living for many years in Mexico, Spain and Egypt.

2.4

Jose Abdelnour-Nocera: IDID under a new Development Agenda

"In May 2013 a United Nations High-level panel released a new universal agenda for post-2015 development goals. This agenda sets new goals for poverty eradication and sustainable development. As a sociotechnical designer, I will discuss the relevance, validity and viability of these goals in terms of concrete design experiences with Namibian farmers, South African townships and Venezuelan rural doctors. "

José is Associate Professor in Sociotechnical Design and Head of the Sociotechnical Centre for Innovation and User Experience at the University of West London. He is the current Chair for IFIP TC 13.8 working group in Interaction Design for International Development as well as Chair for the British Computer Society Sociotechnical Specialist Group. His interests lie in the sociotechnical and cultural aspects of systems design, development and use. In pursuing these interests, he has been involved as researcher and consultant in several projects in the UK and overseas in the domains of mHealth, e-learning, social development, e-commerce, e-governance and enterprise resource planning systems. Dr. Abdelnour-Nocera gained an MSc in Social Psychology from Simon Bolivar University, Venezuela and a PhD in Computing from The Open University, UK.

Panelists 2.1 Chris Csikszentmihályi: Innovating in the Global South

  He has been a professor at colleges, universities, and institutes, including Distinguished Visiting Professor of Art and Design Research at Parsons the New School for Design. He cofounded and directed the MIT Center for Future Civic Media (C4), which was dedicated to developing technologies that strengthen communities. He also founded the MIT Media Lab's Computing Culture group, which worked to create unique media technologies for cultural and political applications. Trained as an artist, he has worked in the intersection of new technologies, media, and the arts for 16 years, lecturing, showing new media work, and presenting installations on five continents and one subcontinent. He was a 2005 Rockefeller New Media Fellow, and a 2007-2008 fellow at Harvard's Radcliffe Institute for Advanced Study, and has taught at the UC San Diego, Rensselaer Polytechnic Institute, and Turku University.

	"As a humanist technologist, most of my work has been developed in the US for
	American users and situations. I have recently taken up the challenge of developing
	new technologies in the context of Uganda; we hope that what we are creating there
	might also be useful in much of the rest of the world. But Uganda is awash with the
	rubric of "development," a fraught set of actors, agendas, and assumptions, and in-
	deed one of the largest and most ubiquitous industries in this and many other coun-

tries. So even though my current project, RootIO seeks to create a platform for intracommunity dialog in rural areas, when most professionals of any sort working in Uganda are exposed to it, they immediately state that it would be a great platform for development messaging and behavior change. These different goals, while not wholly exclusive, are very different. In this panel I hope to describe why I am less interested in development messaging than in intra-community media, and how we are designing for the latter but constantly forced into the mold of the former. I'll bring in some of the post-colonial theory that informs our work, and how we feel that there is a difference between how users are defined in the Global South versus the North."

Chris Csikszentmihalyi is the current ERA chair at Madeira Interactive Technologies Institute.

2.2 Torkil Clemmensen: Reframing HCI Concepts and Tools "

  If we are to support a sustainable development, we need to integrate social, economic and environmental dimensions in our development and use of IT. I want to focus on the importance of reframing HCI from local perspectives by integrating cultural perspectives on HCI research and practice. In particular we need a) new concepts of usability and UX which are grounded in the practices of local people and local compa-

	nies, b) a more diverse toolbox of HCI methods that better covers the diversity of
	human life, and c) to develop the UX profession itself (certification, specialization,
	community building). Paradoxically, to reframe HCI from local perspectives not only
	requires more empirical studies, but also reconsidering what theories that we use."
	Torkil Clemmensen, PhD, Professor mso at Department of IT Management, Co-
	penhagen Business School, Denmark. His interest is in Human-Computer Interaction,


Sketching User Experiences Tutorial

Nicolai Marquardt

University College London, UCL Interaction Centre Gower Street, London, UK nicolai.marquardt@acm.org

Abstract. When designing novel user interfaces, paper-pencil sketches can support the design thinking process and are valuable for communicating design ideas to others. In this hands-on tutorial we will demonstrate how to integrate sketching into researchers' and interaction designers' everyday practicewith a particular focus on the design of novel user experiences. Participants will learn essential sketching strategies, apply these in practice during many hands-on exercises, and learn the various ways of using sketches as a tool during all stages of the HCI research and design process. Our emphasis is on quick, easy to learn, and easy to apply methods for generating and refining ideas.

Tutorial Structure

Creating hand drawn paper-pencil sketches [START_REF] Baskinger | Pencils before pixels: a primer in hand-generated sketching[END_REF] can be a valuable tool for finding the right design; before later refining the work and getting the design right [START_REF] Buxton | Sketching User Experiences: Getting the Design Right and the Right Design[END_REF]. This hands-on tutorial will demonstrate how to integrate sketching into researchers' and interaction designers' everyday practice.

In this half-day tutorial we will guide participants through selected sketching techniques and strategies. These techniques are partially based on the Sketching User Experiences Workbook [START_REF] Greenberg | Sketching User Experiences: The Workbook[END_REF], but also include other techniques and examples not covered in the book. Live sketching demonstrations and step-by-step instructions will illustrate a basic toolset for getting started sketching when working on HCI research projects. In particular, the demonstrated techniques include, for example (also see techniques in [START_REF] Buxton | Sketching User Experiences: Getting the Design Right and the Right Design[END_REF][START_REF] Greenberg | Sketching User Experiences: The Workbook[END_REF]):

 Sketching vocabulary: learning to quickly draw objects, people, and activities  Rapid sketching of people, emotions, gestures, and objects: learning sketching shortcuts and strategies to rapidly sketch common elements of sketches in HCI  10 plus 10 design funnel: developing 10 different ideas and refinements of selected ideas  Photo tracing: create collections of sketch outlines that form the basis of composed sketches  Hybrid sketches: combining sketches with photos  Storyboards for interaction sequences: creating visual illustrations of an interaction sequence and telling a story about use and context over time  Sketch boards: sharing and discussing sketches with others; running critiques With a series of hands-on exercises during the tutorial and different provided templates, the participants of the tutorial can directly apply the learned techniques in practice (see activities in Figure 1). We will demonstrate many best practices and sketching shortcuts, and involve all participants in joint sharing and discussion sessions of the sketches created during the different hands-on activities (see examples in Figure 2). We will demonstrate how we used sketching techniques in our recent research projects when designing interactive systems, and highlight how to apply the learned sketching techniques during all stages of the design and research process. We end the tutorial with an overview of additional resources and books about sketching techniques, and also possible software and hardware for digital sketching.

This tutorial is open for everyone and does not require any previous drawing expertise. We will provide sketching materials, but please feel free to bring your own sketches to share, sketchbooks, pens or paper.

Fig. 1 .

 1 Fig. 1. Activities from previous sketching tutorials and workshops (from top-left to bottomright): live sketching demonstrations, learning photo tracing, participant's sketch of interactive system, sharing and discussing participants sketches, an rapidly sketching wireframes.

Fig. 2 .

 2 Fig. 2. Sketching techniques covered in the course (from top left to bottom right): sketching vocabulary, photo tracing, templates sketches, hybrid sketches, sketching storyboards, and creating photo-based storyboards [3].