Henna Heikkilä
email: henna.heikkila@sis.uta.fi

Tools for a Gaze-controlled Drawing Application -Comparing Gaze Gestures against Dwell Buttons

Keywords: gaze interaction, eye tracking, drawing with gaze, gaze gestures

We designed and implemented a gaze-controlled drawing application that utilizes modifiable and movable shapes. Moving and resizing tools were implemented with gaze gestures. Our gaze gestures are simple one-segment gestures that end outside the screen. Also, we use the closure of the eyes to stop actions in the drawing application. We carried out an experiment to compare gaze gestures with a dwell-based implementation of the tools. Results showed that, in terms of performance, gaze gestures were an equally good input method as dwell buttons. Furthermore, more than 40% of the participants gave better ratings for gaze gestures than for the dwell-based implementation, and under 20% preferred dwell over gestures. Our study shows that gaze gestures can be a feasible alternative for dwell-based interaction when they are designed properly and implemented in the appropriate application area.

Introduction

Eye trackers and gaze-controlled applications enable many disabled users to join the information society independently. Gaze-controlled applications are controlled via eye gaze through an eye tracker. Before use, the eye tracker is calibrated to the user's eyes. Then, during use, the eye tracker follows the user's gaze point and delivers the data to applications. The applications use the data to determine the user's point of interest or intentions. Often, the applications have dwell buttons that the user clicks to control the application. A dwell button is "clicked" when the user's gaze point has remained on the button for a predetermined time (usually 200-500 milliseconds). Over 30 years, eye tracking research has concentrated mostly on communication. In the last decade, the focus has shifted towards leisure applications, such as games and online communities. Many researchers work to enable disabled users to use applications similar to those that able-bodied users already use. Among these are writing applications, Internet browsers, drawing applications, and games of various types. We have concentrated on drawing applications, and our goal is to implement a drawing application that is easy to use and enables the user to correct their drawing mistakes easily.

Even when fixating on a target, the eye does not stay still. The small natural movements occurring during fixation are called microsaccades. Microsaccades usually stay within one degree of movement, which translates to remaining within one square centimeter on the screen when the user is sitting at arm's length from the monitor. The jitter caused by the microsaccades makes it difficult to hit small objects. Also, small calibration errors are common, especially after one has been using the tracker for a while. To avoid the problems caused by microsaccades and calibration errors, the controls, such as buttons and menus, need to be larger in gaze-controlled applications than normally. Larger controls take room from the screen, particularly in drawing applications, where most of the screen space is needed for the drawing area.

In gaze-controlled applications, the user uses the eyes to study the feedback the application is giving and to control the application. If these two actions are not well separated from each other, the "Midas Touch problem" may arise: wherever the user looks, a command is issued [START_REF] Jakob | The Use of Eye Movements in Human-Computer Interaction Techniques: What You Look At is What You Get[END_REF]. This problem is often rectified by extending the dwell time used for determining whether a command has been issued or not.

At first, buttons clicked through dwell time and blinking were the most commonly used input method for issuing commands. A little over 10 years ago, the first gesture-like input methods for gaze-controlled applications were introduced, see [START_REF] Heikkilä | Speed and Accuracy of Gaze Gestures[END_REF]. The concept of gaze gestures was first put forward five years ago [START_REF] Drewes | Interacting with the Computer Using Gaze Gestures[END_REF], and it has evolved since. Fundamentally, gaze gestures are predetermined gaze paths, or patterns of eye movements, that are interpreted as commands to the application. Gaze gestures can be short or long, simple or more complex, location-bound or location-independent.

Next, we give an overview of research on gaze-controlled drawing applications and on gaze gestures that are relevant to our research. We then present our gaze-controlled drawing application, called EyeSketch, and show how gaze gestures are used in its tools. Towards the end of the piece, we present the study wherein we compared dwell time to gaze gestures, and we discuss the results. We conclude by considering the work done so far and discussing some future directions for our drawing application.

Related Research

To our knowledge, five drawing applications controlled via eye gaze have been presented previously. Four of them utilize only gaze, and the other combines gaze with voice commands. The first eye drawing application, Eye Painting (also known as EaglePaint), was presented 16 years ago by Gips and Olivieri [START_REF] Gips | EagleEyes: An Eye Control System for Persons with Disabilities[END_REF]. Eye Painting was one of the applications for their EagleEyes, an EOG-based eye tracking technology. In Eye Painting, the user was able to draw colored lines on the screen by moving the head and eyes.

Eye Painting utilizes so-called free-eye drawing [START_REF] Tchalenko | Free-Eye Drawing[END_REF], in which the line of drawing appears wherever the user looks and the person drawing has no way of lifting the pen from the drawing canvas. Thus, every shape is connected to the next by the line. A related problem with free-eye drawing is the lack of separation between drawing and looking around. When gaze is used to control a technology, usually the same channel is used for input and for examining the output. If users want to look around and examine the drawing, they probably want to pause the drawing process for the time being.

To solve the aforementioned problems with free-eye drawing, Hornof et al. [START_REF] Hornof | EyeDraw: Enabling Children with Severe Motor Impairments to Draw with Their Eyes[END_REF][START_REF] Hornof | Eyedraw: A System for Drawing Pictures with Eye Movements[END_REF][START_REF] Hornof | EyeDraw: A System for Drawing Pictures with the Eyes[END_REF]] created an application called EyeDraw. In EyeDraw, looking and drawing are separated by two 500-millisecond dwell-time spans; drawing of a shape starts only when the gaze point has stayed relatively still for a second [START_REF] Hornof | Eyedraw: A System for Drawing Pictures with Eye Movements[END_REF]. To end the drawing of the shape, the user needs to dwell for a second at the end point. Instead of free-eye drawing, EyeDraw utilizes shapes. In the first version, the user was able to draw only lines and ellipses, but the shape collection later grew to include rectangles and predefined stamps too. EyeArt [START_REF] Meyer | Conception and Development of an Accessible Application for Producing Images by Gaze Interaction -EyeArt[END_REF] resembles EyeDraw in many respects, but it has a wider range of drawing tools, including seven different shapes, a text tool, and an eraser tool. In addition, the user can adjust the border thickness of a soon-to-be-drawn shape and fill drawn shapes with color by using the paint can tool.

Van der Kamp and Sundstedt [START_REF] Van Der Kamp | Gaze and Voice Controlled Drawing[END_REF] presented a drawing application that combines gaze and voice input. A voice command is used in place of dwelling to control drawing and to access the tools and their properties. These authors claim that their solution solves two problems that plagued the previous applications. First, they wanted to remove the need to dwell, since, they said, the use of dwell frustrates users. Second, they hid the tool menus to free screen space for the drawing canvas and to prevent unintended selections from the menus during drawing or looking. In their solution, tool menus appear only through a voice command.

The three drawing applications discussed above utilize separation of looking and drawing. They share the problem that the position or size of the shape drawn cannot be adjusted, which means that the user needs to draw the shape in precisely the right place and in exactly the right size or else undo/erase it and start again. Yeo and Chiu [START_REF] Yeo | Gaze Estimation Model for Eye Drawing[END_REF] introduced a third technique when designing their gaze-estimation model that constitutes an attempt to separate looking (or thinking and searching) from drawing through examination of gaze patterns. Their model assumes that when the gaze points are close together, the user wants to draw, and that when the gaze points are mostly far from each other, the user is thinking or searching for something. When the gaze points cluster within an area, that area is determined to be the "area of interest." If a fixation longer than 500 milliseconds falls within the area of interest, the centroid of the gaze points is calculated and drawing begins at that point.

In our drawing application, gaze gestures are used to move and resize shapes. Our gaze gestures utilize simple, one-segment gestures and off-screen space. Next, we present the three gaze-gesture implementations that are relevant for our study. Møllenbach et al. [START_REF] Møllenbach | Single Gaze Gestures[END_REF] created simple, single-segment gaze gestures. In their one-segment gaze gestures, the gesture was made across the screen: it started from what they called the gesture area and ended in another gesture area, on the opposite side of the screen. The assortment of these Single Gaze Gestures, as the authors call them, is small, but they can be used for simple tasks, such as top-level navigation of applications or controlling one's environment.

Isokoski [START_REF] Isokoski | Text Input Methods for Eye Trackers Using Off-Screen Targets[END_REF] used off-screen targets in his eye writing application. He used five off-screen targets attached to the monitor frame. The user's gaze was tracked with a head-mounted SMI EyeLink tracker, which was able to track the gaze beyond the screen area when the user was seated 100 centimeters away from the monitor. A short dwell time, 100 milliseconds, was used as the threshold for determining whether the user's gaze actually stopped over an off-screen target or just wandered over it.

Another application using off-screen space is Snap Clutch, by Istance et al. [START_REF] Istance | Snap Clutch, a Moded Approach to Solving the Midas Touch Problem[END_REF]. In Snap Clutch, the user can switch mode in the application by looking outside the screen space. The authors claim that quick glances of this sort are a fast and effortless way to control their application.

In our application, we use the closure of the eyes also. The closure of both eyes for a longer time is a rarely used input method, although it is easier to recognize as intentional than are the more frequently used blinks and winks (closure of one or both eyes for only a short time). Especially in the case of blinks, it is difficult to determine which are intentional and which involuntary -reflexive actions that occur when the eyes are getting dry, as the eyes often do more readily when one is looking at a computer screen. As far as we know, only Hemmert et al. [START_REF] Hemmert | Spoken Words: Activating Text-To-Speech through Eye Closure[END_REF][START_REF] Hemmert | Perspective Change: A System for Switching Between On-screen Views by Closing one Eye[END_REF] have used the closure of one and both eyes to control applications. By closing both eyes, the user was able to activate text-to-speech functionality in a writing application, and closing just one eye allowed users to switch between modes in a first-person shooter game and to filter other than the most recently used icons from the desktop.

EyeSketch: A Gaze-controlled Drawing Application

Our motivation in the design of our drawing application is twofold. First, we wanted to create a drawing application with which the user can produce pleasing pictures without unintentional gaps between shapes or accidental overlapping of shapes. Second, we wanted to create a new kind of drawing application. Of the preexisting drawing applications, none used objects that can be modified, yet able-bodied users have their choice of many such applications. Moreover, we believe that modifiable objects can solve the positioning problem in addition.

The application

We chose an approach wherein the shapes, or objects, drawn can be moved or resized, and in which their other properties can be modified after these are drawn.

For the first version of our application, we implemented basic drawing tools, tools for modifying the shapes drawn, and tools for saving and opening pictures drawn with the application. Tool buttons were placed around the drawing canvas and implemented as dwell buttons. We used 80 × 80 pixels as the size for a tool button. These buttons are selected when the gaze point has stayed on the button for 400 milliseconds.

Our basic drawing tools include tools for drawing rectangles, ellipses, and lines. Before and after drawing of the shapes, their fill color, border color, and border thickness can be changed. To aid in creation of the drawing, a grid is implemented behind the drawing canvas. The user can choose whether to display the grid or not.

For later modification of a shape, we have a Select tool. When a shape is selected, its color and line thickness can be changed, and it can be removed with the Undo tool.

The Move, Nudge, and Resize tools are implemented with gaze gestures. With the Nudge tool, the shape moves one grid square in the direction of the gaze gesture made. With the Move tool, the shape starts to move towards the gesture's direction and stops when the user closes the eyes or when the moving shape hits the edge of the drawing canvas. The Resize tool causes resizing handles to appear at the sides of the selected shape (see Fig. 1). The size of each handle is 50 × 50 pixels. A handle is selected by dwell first (100 milliseconds), followed by a gesture; depending on the gesture direction (inwards or outwards), the shape shrinks or grows from the side on which the handle is attached.

We integrated the COGAIN ETU Driver1 into the drawing application to deliver the eye tracking data from the eye tracker to our drawing application. The ETU Driver supports several makes of eye tracker. Therefore, the application can be used with multiple eye trackers, since the ETU Driver makes sure that the gaze data will be delivered to the application in the same form regardless of the eye tracker used.

Gaze gestures

We chose gaze gestures for this use since they are less vulnerable to calibration errors and the jitter in the eyes. Our gaze gestures are simple one-segment gestures that end outside the screen. We also use the closure of both eyes to stop a moving shape. The gaze gesture used with the Move and Nudge tools always starts on top of a shape already drawn, proceeds in one of the eight directions (toward a side or corner of the screen), and ends outside the screen (see Fig. 2). We named the eight directions after the cardinal and ordinal directions, with north being toward the top of the screen, northeast toward the upper right-hand corner of the screen, east toward the right side of the screen, etc.

With the Resize tool, the gaze gesture starts on top of a resize handle attached to the side of the selected shape; proceeds left/right or up/down, depending on the side on which the handle is attached; and ends outside the screen area. To be able to start the gaze gesture, the gaze has to stay on the shape or resize handle for 100 milliseconds for it to become selected. The user can start the gesture when the gaze cursor changes its color from black to orange. As the user makes the gaze gesture, three gaze points must fall into the same segment (see Fig. 3) in the direction of the gesture before exiting the screen area. Since the 60 Hz eye trackers take a gaze-point sample once every 16th millisecond, the move from the shape to outside the screen must take at least 64 milliseconds. If it takes more than 1,500 milliseconds, the gesture process stops. When the gaze has remained outside the screen area for 100 milliseconds, the gesture-recognition process ends and the command is issued. A feedback sound is played to the user when they can return the gaze to the screen without canceling the action.

In use of the Move tool, the gaze gesture makes the shape move in the direction of the gesture. Closing both eyes for 300 milliseconds stops the movement. While the eyes are closed, the shape keeps moving until the threshold time has been reached. Once that time has elapsed, the shape returns to where it was when the eyes were closed. A feedback sound is played to the user when the eyes may be opened.

Gaze Gestures vs. Dwell -an Experiment

To find out whether our gaze gestures would be a feasible input method for the Move, Nudge, and Resize tools, we designed an experiment in which the gaze gestures and often-used dwell buttons were compared.

Participants

Twelve participants, seven male and five female, volunteered for the tests. Their ages ranged from 18 to 38 years (mean: 24.3 years). Only one of the participants wore eyeglasses during the test. None of the participants had prior experience of eye tracking. Nine participants were familiar with the concept of gestures, and five of them had tried gestures in some form; they reported having used them with cell phones (hand/finger gestures) and with video-game consoles (bodily gestures).

Apparatus

We used a Tobii T60 eye tracker with a sampling rate of 60 Hz to track the participant's gaze. The resolution of the screen was set to 1280 × 1024 pixels (17-inch LCD screen with a width of 338 mm and height of 272 mm).

The test application was a light version of our drawing application: only the Move, Nudge, Resize, Undo, and Look around tools were available. In each task, the object to be moved or resized was already drawn in the drawing area. The target size and position were indicated through a similar object with a thick red border (see Fig. 4).

Dwell Implementation

The dwell implementation differs from our gaze-gesture implementation described above only in terms of the implementation of the Move, Nudge, and Resize tools.

With the Move and Nudge tools, eight dwell buttons, with arrows showing the direction appear, around the selected shape (as shown in Fig. 5). The participant needs to keep the gaze on the button for 400 milliseconds for it to be clicked. The Move tool makes the shape start to move in the given direction when the arrow button is clicked. The movement is stopped in the same way as in the gesture implementation: by closing of both eyes. In the implementation of the Nudge tool, the shape moves one grid step in the given direction and stops automatically.

For the Resize tool, a pair of dwell buttons appears on each side of the selected shape (see Fig. 6). The button closer to the shape makes the shape smaller, and the one further from the shape increases the shape's size. When the participant has fixated on the dwell button for 400 milliseconds, it will be clicked and the size of the shape decreases or increases by one step.

The 400-millisecond threshold for dwell time was selected on the basis of literature from the field of eye typing, wherein dwell time is used to select letters from an on-screen keyboard. Majaranta and Räihä [START_REF] Majaranta | Text Entry by Gaze: Utilizing Eye-Tracking[END_REF] concluded in their review that in eye typing studies with novice users, dwell times ranged from 450 to 1000 milliseconds. Majaranta et al. [START_REF] Majaranta | Fast Gaze Typing with an Adjustable Dwell Time[END_REF] performed a longitudinal eye typing study wherein participants were able to adjust the dwell time. In their study, none of the participants used a dwell time of 400 milliseconds or less during the first session. After five sessions (that is, 75 minutes' practice), most participants had decreased the dwell time to 400 milliseconds or less. For novice users, any dwell time shorter than 400 milliseconds would cause significantly more unintended commands.

Tasks

We asked participants to perform four tasks with each style of input. The tasks were the following: 1. Move the object drawn to the target area.

2. Use already-drawn objects to build a house in the target area.

3. Resize the object drawn until it matches the target area. 4. Use drawn objects to build a house in the target area. Resize the objects when necessary.

For Tasks 1 and 2, the participant had only movement tools, Move and Nudge, available. For Task 3, only the Resize tool was active. For the fourth task, the participant was able to use both the movement tools and the resizing tool. Tasks 1 and 3 were used to train the participants in use of the new tools. Task 2 was selected to reveal possible difficulties when there are several objects in the drawing area. With Task 4, we wanted to see how well switching from one tool to another works.

Procedure

Each test took 40-60 minutes. At the beginning of the test, the participant was asked to fill in a questionnaire form, for background information. Then the purpose and the procedure of the test were introduced, and informed consent was requested from the participant. The test had two parts, each using one of the two input styles. The two parts followed the same procedure; only the input style was different. The order of the input styles was counterbalanced. At the beginning of each part, the experimenter demonstrated the input style with a mouse. Then the participant was seated in front of the eye tracker, at arm's length from the monitor, and the eye tracker was calibrated. After calibration, the experimenter started the testing software and the participant performed the four tasks. After completing the tasks, the participant was asked to fill in a user-satisfaction form. Meanwhile, the experimenter restarted the eye tracker. After a short break, the second part of the test was started. After the second part and the associated user-satisfaction form, the participant was briefly interviewed about the experiences during the test.

Results and Discussion

We calculated task-completion times, times for completing an action, and the number of unnecessary actions from the data collected. To test the statistical significance of our results, we used repeated-measures ANOVAs with Greenhouse-Geisser correction and paired-sample t-tests for post hoc pairwise comparisons.

Task-completion times. In comparison of the mean times for completion of a full task (see Fig. 7), the dwell implementation was revealed to be faster for tasks 1 and 2, wherein the participants only had to move objects. In tasks 3 and 4, which involved the need to resize the objects in addition, the two implementation types took equally long for completion, on average. This means that the resizing actions take so much longer to complete with the dwell implementation that the advantage gained in the moving actions is lost. Only the main effect for the task was significant (F 2,19 = 72.45, p < .001), as can be expected from the nature of the tasks. When examining the times to complete the first task with the two implementation types, we found that the first performance of Task 1 took significantly more time than the second one (F 1,10 = 5.95, p < .05). This was independent of the implementation type used (F 1,10 = 1.47, p > .05). The results demonstrate that it always takes time to figure out how to use one's eyes to control an application when a gaze-controlled application is used for the first time.

Completion time for an action. On average, performance of an action was almost equally fast in the two implementation types (see Fig. 8) for all actions. Statistical tests showed that task had a significant effect on the time taken per action (F 1,15 = 8.97, p < .01). It also had an interaction effect with the implementation type on the completion times (F 1,15 = 7.96, p < .01). Implementation type on its own did not have a statistically significant effect on completion times (F 1,11 = 1.19, p > .05). The exception is that in Task 1, the actions with the dwell implementation took almost twice as long as those performed with the gesture implementation. The implementation type had a significant effect on completion time for Task 1 (F 1,10 = 5.67, p < .05), and it did not matter which implementation type the participant used first (F 1,10 = 0.00, ns). We believe this result reflects the fact that the participants tried out the gaze gestures more than the dwell buttons before starting to perform the task. We observed in the tests that many participants made several gestures to get a shape to move and then stopped the movement before the shape had moved more than a couple of steps. With the dwell implementation, there was less behavior of this kind.

Excess actions. We calculated the optimal number of actions for each task. Optimal performance in tasks 1-4 involved 2, 12, 10, and 54 actions, respectively. Only four times during the tests did a participant manage to complete a task optimally. When the task was only to move the shapes (tasks 1 and 2), the participants used more actions in the gesture implementation than in the dwell implementation. However, when the tasks included resizing of the shapes (tasks 3 and 4), more actions were employed in the dwell implementation than in the gesture implementation.

The data support our observations during the tests: the participants had difficulties in resizing the shapes with the dwell implementation, because the dwell buttons to make the shape smaller and larger were next to each other. Because of jitter in the gaze and small calibration errors, the participants often accidentally clicked the wrong dwell button. Then another action was needed to reverse this wrong action. Therefore, to complete one successful action, the participant had to perform three actions. As expected, task had a significant main effect on unnecessary actions (F 2,23 = 16.83, p < .001), since the tasks were very different in how many actions were needed for their completion. Implementation type did not have a main effect on the number of excess actions (F 1,11 = 0.005, ns), since whether a given implementation type fared better or worse varied from one task to the next. Statistical testing revealed that task and implementation type had a significant interaction effect (F 2,20 = 4.73, p < .05). This supports what is visible in Figure 9. The result means that one implementation type is better for certain tasks than the other, and vice versa.

The participants used more actions than needed in their very first task, no matter which implementation type they started the test with. When the participants started with the gesture implementation, they used, on average, 11.3 actions more than needed for completion of the first task. When facing the same task in the dwell implementation later, they used only 2.3 actions more than the optimum. The participants who started with the dwell implementation performed 5.0 actions more than the number needed in their very first task, and only 5.5 actions more when they later completed the task with the gesture implementation.

The statistical tests showed that there was a statistically significant difference between the two types of implementation in the number of unnecessary actions for Task 1 (F 1,10 = 6.10, p < .05). That is, for Task 1, the participants made more unnecessary actions with one of the implementation types (the gesture implementation) than in the other. As described above with regard to completion time per action, we observed during the tests that, with the first task, the participants tried out the use of gaze gestures more than they did the use of dwell buttons. Our results suggest that the use of gaze gestures needs more training in the beginning than that of dwell buttons.

Subjective impressions. We asked the participants to evaluate their use experience on a seven-point Likert scale (with 1 indicating "strongly disagree" and 7 standing for "strongly agree"). They evaluated their experience on seven dimensions after using both implementation types. When one looks at the average scores, the gesture implementation appears better or at least equally good on all dimensions (see Fig. 10). The largest differences emerged for ease of resizing objects and in interaction speed. For ease of moving objects and on the natural-interaction dimension, the two implementation types were equally good. None of the differences was shown to be statistically significant in Wilcoxon signed-rank testing. All participants who started with the dwell implementation rated the gesture implementation better than the dwell implementation or equally good. Participants who started with the gesture implementation were less unanimous with their scores. When we asked about the preference between the two in the interviews, seven participants preferred gaze gestures, four preferred the dwell implementation, and one was undecided. In particular, the difficulty in hitting the correct resizing dwell button in tasks 3 and 4 tipped the scale to gesture implementation. If we had increased the dwell-button size from 50 × 50 pixels or left space between the dwell buttons, such problems might have been avoided. However, seeing the drawing and accessing the objects drawn are essential to drawing applications. Also, in that solution, when several objects are placed close to each other and the user could readily select the wrong one by accident, the resize buttons for the wrongly selected object might hide the intended object and cause a dilemma. In our study, we already gave twice as much space from the drawing canvas to the dwell buttons as in the gesture implementation. Had we given them even more space, the situation would have been neither comparable to the gaze-gesture implementation nor appropriate for drawing applications anymore.

Conclusions

As described at the start of the paper, our motivation is to solve problems in existing gaze-controlled drawing applications by creating a new kind of drawing application that utilizes movable, resizable, and modifiable objects. The first step was to establish a functional way to move and resize the objects. For this purpose, we selected gaze gestures, since we wanted to keep the drawing canvas as free of buttons as possible. The next step was to test whether the gaze gestures could work as well as the dwell buttons that are the traditional input style. The results from the experiment described in this paper are very encouraging. Although the dwell buttons were the better input style for moving shapes, in resizing tasks the gaze gestures proved to be an even better input style, solving all the problems from which the dwell implementation suffered. Furthermore, the participants were able to move and resize the shapes with gaze gestures even when the drawing canvas was half-filled with various shapes. We were excited to learn that most of the participants in our experiment felt that the gaze gestures worked well and that they preferred the gaze gestures to the dwell-button implementation.

In terms of time to issue a command, gaze gestures may never beat dwell buttons. The real advantage of gaze gestures is their ability to remain functional despite calibration errors and low accuracy of the eye tracker. Our resizing task showed how vulnerable the dwell-time input is to even small accuracy problems. Overall, our results showed that the simple, one-segment gaze gestures can be used for tasks other than switching between modes. The only limitation for one-segment gaze gestures is the small size of the gesture vocabulary. However, with adequate planning, the use cases could be numerous.

We have implemented a very usable way to move and resize shapes in a drawing application. Our next two steps are user tests with users from our target user groupi.e., with disabled users -and releasing the drawing application for the public.

Fig. 1 .

 1 Fig. 1. The resizing handles for the Resize tool appear around the selected shape.

Fig. 2 .

 2 Fig. 2. The gaze gesture starts on top of a drawn shape (or a resizing handle) and ends outside the screen.

Fig. 3 .

 3 Fig. 3. At least three gaze points must fall into the same gesture segment before the gesture can be completed.

Fig. 4 .

 4 Fig. 4. Test application showing Task 4: Use drawn objects to build a house in the target area. Resize the objects when necessary.

Fig. 5 .

 5 Fig. 5. The dwell buttons for the Move tool appear around the selected shape in the dwell implementation.

Fig. 6 .

 6 Fig. 6. The dwell buttons for the Resize tool appear around the selected shape in the dwell implementation.

Fig. 7 .

 7 Fig. 7. Task-completion times, in seconds, for the two implementations. The error bars show the standard deviations of the means.

Fig. 8 .

 8 Fig. 8. Completion time per action, in seconds, for the two implementations. The error bars show the standard deviations of the means.

Fig. 9 .

 9 Fig. 9. Number of excess actions per task in the two implementations. The error bars show the standard deviations of the means.

Fig. 10 .

 10 Fig. 10. Subjective impressions gathered after each implementation type. The error bars show the standard deviations of the means.

The COGAIN ETU Driver (i.e., Eye-Tracking Universal Driver) can be downloaded from http://www.sis.uta.fi/~csolsp/downloads.php.

Acknowledgements

This work was supported by the Finnish Doctoral Program in User-Centered Information Technology (UCIT).


Tomi Heimonen
email: tomi.heimonen@sis.uta.fi

Jaakko Hakulinen
email: jaakko.hakulinen@sis.uta.fi

Markku Turunen
email: markku.turunen@sis.uta.fi

Jussi P P Jokinen
email: jussi.p.p.jokinen@jyu.fi

Tuuli Keskinen
email: tuuli.keskinen@sis.uta.fi

Roope Raisamo
email: roope.raisamo@sis.uta.fi

Designing Gesture-Based Control for Factory Automation

Keywords: Gesture-based interaction, emotions, user experience, field study

We report the development and evaluation of a gesture-based interaction prototype for controlling the loading station of a factory automation system. In this context, gesture-based interaction has the potential to free users from the tedious physical controls but it must also account for safety considerations and users' perceptions. We evaluated the gesture interaction concept in the field to understand its applicability to industrial settings. Our findings suggest that gesture-based interaction is an emotional, physically charged experience that has the potential to enhance the work process. Participants' feedback also highlighted challenges related to the reliability of gesture recognition technology in the workplace, the perceived professionalism of gesture-based interaction, and the role of physical feedback in promoting feeling of control. Our results inform the development of gesture-based interaction for similar contexts.

Introduction

Gesture-based interaction could provide many benefits as an alternative to physical controls in industrial settings, where the manual operation of machinery can be ergonomically challenging and hazardous or is governed by safety regulations. Several characteristics of gesture-based systems affect their usability in such settings, including technical aspects such as latency of operation, stability, resolution, and precision [START_REF] Crowley | Perceptual User Interfaces: Things That See[END_REF], as well as human-centered aspects such as intuitiveness, ergonomics, and ease of gesture performance and recall [START_REF] Nielsen | A Procedure for Developing Intuitive and Ergonomic Gesture Interfaces for HCI[END_REF]. Appropriately designed systems must balance these sometimes contradictory concerns. We report the findings from a year-long design and development process of a gesture-based interaction concept, focusing on the results of evaluations conducted in real context of use. The overall goal of the research was to create novel interaction concepts for metal workshops and to study their potential in real work tasks. Our research was aimed at answering the following questions:

─ Which factors affect the acceptability of gesture control in industrial settings? ─ What is the user experience of gesture control in industrial settings, especially with respect to its emotional aspects?

2 Related Work

Gesture-Based Interaction

Gesture tracking technologies can be broadly divided into perceptual (utilizing, e.g., vision and audio) and non-perceptual (based on, e.g., mouse or touch surfaces) [START_REF] Karam | A Taxonomy of Gestures in Human Computer Interactions[END_REF].

The taxonomy proposed by Karam and schraefel [START_REF] Karam | A Taxonomy of Gestures in Human Computer Interactions[END_REF] divides gesture styles between deictic (pointing), manipulative (tight coupling between the controlled object and hand/arm movement), semaphoric (symbolic), gesticulation (combination of gesture and speech), and language gestures (e.g., sign languages). Due to the prevalence of symbolic gesturing in the real world (e.g., police and military hand signals), we chose it as the starting point for our design. Empty-handed, or touchless, gestural interaction has been proposed to be advantageous in various real world contexts, such as medical environments and environments where physical input devices could be vandalized [START_REF] De La Barré | Touchless Interaction -Novel Chances and Challenges[END_REF]. In our context of use, the situation is similar to the latter, as introducing input devices into the factory environment is not viable. For example, touchscreens could be smeared with residues, and remote controls could easily be damaged or misplaced. Examples of perceptual gesture-based interaction in a context of use resembling ours can be found in human-robot interaction [START_REF] Burger | Two-handed Gesture Recognition and Fusion with Speech to Command a Robot[END_REF][START_REF] Cerlinca | Hand Posture Recognition for Human-Robot Interaction[END_REF][START_REF] Suay | Humanoid Robot Control Using Depth Camera[END_REF]. However, to our knowledge, our work is one of the first user-centered studies of designing gesture-based control to operate factory automation systems.

Designing Gesture Interactions

Combining the technical and user-centered requirements makes the design of gesture interaction challenging. Although it may be possible to design an efficient and easy to recognize gesture vocabulary for human-machine interaction, one needs to also consider the social implications of gesturing in the workplace. Previous research suggests that gestures that are based on subtle movements and are similar to existing technology and every day actions are preferred to gestures that look uncommon, are physically uncomfortable, or interfere with face-to-face conversations [START_REF] Rico | Usable Gestures for Mobile Interfaces: Evaluating Social Acceptability[END_REF]. Additionally, the general meaning of the gesture should be understandable to both the user and bystanders to be socially acceptable [START_REF] Montero | Would You Do That?: Understanding Social Acceptance of Gestural Interfaces[END_REF]. Early stage development that looks past technical restrictions is one way to avoid expending effort on gestures that are not acceptable [START_REF] Rico | Usable Gestures for Mobile Interfaces: Evaluating Social Acceptability[END_REF]. However, as Wobbrock et al. [START_REF] Wobbrock | User-defined Gestures for Surface Computing[END_REF] point out, reliability of recognition is nevertheless an important criterion for early prototypes. Our design approach is a hybrid of the technology-oriented and human-centered styles; we aimed for reliable recognition through designed gestures, but carried out a series of design workshops to ensure the gestures are also acceptable.

Gesture-Based Interaction for Factory Automation

The context of use for the proposed gesture interaction concept is the loading station area, which is a part of a manufacturing automation system (Figure 1). The operator's task is to attach raw materials to a pallet using various tools, which the automation system then moves to machining, and subsequently to remove the machined parts.

Using a button panel situated to the side of the loading station, the operator can open and close the door separating the automated storage from the operating area, drive the pallet in or out of the storage, and rotate the pallet in either horizontal direction. The emergency stop can also be used at any time. The loading station controls are placed away from the pallet due to safety regulations.

In crowded workshop conditions, the controls can be hard to reach, and their operation requires constant movement from the pallet to the controls and back. These issues could be alleviated with gestures, as the loading station could be controlled by simply taking a step back to a safe distance from the machine and performing the appropriate gesture, even while grasping tools. The use of gestures would naturally also require strong safeguards, such as multiple means of tracking the operator's location with respect to the machine.

Gesture Vocabulary for Loading Station Control

The gesture vocabulary was designed in a series of workshops with representatives from factory automation manufacturing industry to ensure its validity in the realistic industrial context. This collaboration was grounded on prior user observations in industrial environments. During the design process we iterated different versions of the gesture vocabulary and visual feedback to ensure the operation was robust and the feedback informative. This process resulted in a gesture set of seven poses (Figure 2). The gestures were designed to be distinctive from one another, so that their recognition would be as reliable as possible using the template-based recognition algorithm implemented for the prototype. The poses also attempt to provide intuitive mappings to the functioning of the loading station through body-centric associations. For example, driving the pallet in and out is mapped to the user's horizontally extended arm, while door operation is mapped to vertically extended arm, both matching the direction of the resulting movement.

Loading Station Simulator

A simulator application was implemented to demonstrate the concept. It contains a 3D model of the loading station and an instruction screen, which would be visible on a separate display in a real world set up. Figure 3 shows the screens as displayed during the field tests. The instruction screen has icons, which depict the pose for each operation and indicate their activation. It provides continuous feedback on system state and gesture recognition, and proved valuable in letting users know that the system is successfully tracking their movements. We utilized the Microsoft Kinect sensor via the OpenNI framework (http://www.openni.org) in early versions of the prototype, and the official Kinect SDK in later iterations.

Evaluating Gesture-Based Interaction in Real Contexts of Use

The emotional user experience and acceptance of the concept were evaluated in real workshop environment. The first study was conducted at a factory automation manufacturing site and the second at a local metal workshop, where loading stations similar to the simulated one are used daily. Externally, the prototype was similar in both studies, but internal data processing was updated to improve stability.

Method

Participants were opportunistically recruited on-site. A total of 22 participants, 2 female and 20 male, took part in the evaluation. They had an average work experience with factory automation loading stations of 6.0 years (range = 0-25 years, s.d. = 6.9). Participant backgrounds ranged from factory automation product designers to assembly line workers and metal workshop laborers. After a demonstration of the gesture commands by a researcher, the participants were able to briefly practice the gestures. Next, they were asked to perform each of the gestures twice according to a script cued by a researcher, simulating the operation of a real loading station. Interaction took place in front of the simulator, which was presented on a 37-inch high definition television next to the loading station cell.

User experience was measured with a questionnaire, which contained a Likert scale of eleven questions concerning emotional experiences including alertness, anxiety, delight, desperation, determination, excellence, frustration, pride, skillfulness, success, and surprise. In addition, an item concerning the feeling of control was included. Another Likert scale of five statements concerned the acceptability of the gestures: their practicality, necessity, professionalism, intelligence, and physical demand. The questionnaire was filled after the participants had tested the prototype. In addition, short interviews were conducted with nine of the participants.

Results

Emotional User Experience. The responses to the questionnaire were analyzed using multidimensional scaling (MDS). The resulting configuration shows which emotions are related to each other (Figure 4). The assumption is that similar emotional items are correlated with the same underlying user experience factor, which can be calculated as their sum variable. The items were separated by their emotional valence into pleasant (α = .862) and unpleasant (α = .804) emotional groups, along the x-axis. Cronbach alphas over 0.8 were taken as a confirmation that the items were reliable indicators of the underlying emotional factor.

The results indicate the emergence of two distinct emotional user experience factors: competence [START_REF] Gravill | Experience Effects on the Accuracy of Selfassessed User Competence[END_REF] and frustration [START_REF] Juutinen | Emotional Obstacles of E-learning[END_REF]. Competence, or self-efficacy, is linked to the participants being pleasantly surprised by the new suggested interface (Spearman ρ = .771, p < .001), and associating feeling of control with very positive emotional experiences, such as pride and success. In addition, a positive association between having a pleasant user experience and feeling in control was observed (ρ = .508, p = .019). Frustration was evident in even slightly negative, or unpleasant, user experience being strongly associated with not accepting the gestures. Accordingly, negative correlations were observed between feeling frustration and perceiving the gestures as intelligent (ρ = -0.542, p = 0.011), professional (ρ = -0.587, p = 0.005), safe (ρ = -0.671, p = 0.001), and needful (ρ = -0.509, p = 0.018). Interview Feedback. The interviews were conducted with metal workshop workers (4 respondents) that operate loading stations daily, and factory automation mechanics (5) that assemble the loading stations. Most felt that the new scheme of interaction was a welcome introduction to the usual work routine. The notable benefits mentioned were the increased freedom of movement, speeding up the work process, improving safety, and easing the learning process of the control interface due to easily learnable gestures. However, many also suggested that the current interface is more reliable, and the shift to non-physical controls would introduce errors without making them easier. Four notable items of doubt were extracted from the interviews:

1. Can the operator trust the delicate sensory in a factory environment? 2. How easily does the sensor break or get dirty? 3. Is it safe to perform elaborate gestures in factory environment? 4. The control interface is already easy. Would it truly be easier to perform gestures instead of walking to the control station to press a button?

When asked about the perceived professionalism of the proposed user interface, the participants made references to attributes not welcome in industrial environment such as "games" and "fooling around". Mostly these references were associated with the gestures themselves: moving arms and body around was perceived as being something not suitable in a factory environment, which suggests gesture control as currently designed might break social norms at the workplace. Physical controls also seem to significantly affect the feeling of professionalism. The most important factor associated with feeling of control was the lack of physical contact with the interface. Comments such as "it felt strange that I couldn't touch it", "I'm used to getting direct physical feedback from the button", and "I feel more in control with the traditional physical buttons" indicated this to be one of the critical factors influencing the acceptance of the new gesture-based interface.

Discussion

Our research contributes to the development of gesture-based interaction for industrial workplaces. First, we found that the proposed gesture interface evoked clear emotional experiences as distinct groups of pleasant and unpleasant emotions were observed. These were shown to be associated with feeling of control and acceptance of the concept. In designing gesture-based interaction, it is thus critical to actively support the feelings of self-efficacy and minimize the experienced frustration. Second, we identified doubts related to the professionalism of gesture-based interaction that are strongly linked to the absence of physical control and the social acceptability of gesturing in the workplace. The first aspect is closely associated with feeling self-efficant: is it possible to feel competent and in control without physical interaction? This resonates with findings from previous research, which suggest that it is important to understand the relationship between the physicality of control and the system being used, and how this affects the users' preferences [START_REF] Freeman | The Role of Physical Controllers in Motion Video Gaming[END_REF]. In this respect, an obvious shortcoming of touchless interaction is the lack of tactile feedback, and one must provide a form of feedback that promotes the feeling of control when physical controls are absent. With respect to acceptability, it should be noted that the focus of the research was to develop and study novel interaction concepts for future work settings. It is an open question to what extent the concerns with professionalism apply to new generations of users who already have experiences with using gestures in games and with mobile devices.

Finally, results from studies that are not a part of the users' work practice need to be interpreted with care. Benefits and drawbacks with respect to existing interfaces are difficult to estimate without experiencing gesture-based interaction in the context of real work tasks. For example, would the audiovisual feedback from a real loading station compensate for the lack of physical control? Longitudinal, in situ studies of gesture-based production systems are needed to address the above issues.

Conclusion

Gesture-based interaction in industrial setting is a novel and physically charged activity that is also an emotional experience. When designing gesture-based interaction for industrial environments, attention should be paid to creating a system that promotes self-efficant experiences. An optimal interaction is challenging but practical in order to make the interaction an emotionally pleasurable and novel experience. Challenges to address in future work are related to studying the perceived robustness of the gesture tracking technology in real work tasks, designing the physical gesture movements to fit the work context, and compensating for the lack of physical control with alternative feedback methods.

Fig. 1 .

 1 Fig. 1. User attaching materials to the pallet (original image courtesy of Fastems).

Fig. 2 .

 2 Fig. 2. The proposed gesture set consisting of static hand and arm poses.

Fig. 3 .

 3 Fig. 3. Loading station simulator (left) and the gesture instruction screen (right).

Fig. 4 .

 4 Fig. 4. Multidimensional scaling solution of experienced emotions. The configuration shows that two groups are formed along the x-axis, which indicates the valence of the emotions.

Acknowledgements. This research was carried out as part of the Finnish Metals and Engineering Competence Cluster (FIMECC)'s UXUS program. Appreciation is extended to Matti Nurminen and colleagues from Fastems Oy Ab, Tuomas Kiviluoto from the Agora Center at the University of Jyväskylä, and Mika Käki from Idean.