Masatomo Kobayashi

Atsushi Hiyama
email: atsushi@cyber.t.u-tokyo.ac.jp

Takahiro Miura
email: miura@human.rcast.u-tokyo.ac.jp

Chieko Asakawa

Michitaka Hirose
email: hirose@cyber.t.u-tokyo.ac.jp

Tohru Ifukube
email: ifukube@human.rcast.u-tokyo.ac.jp

Elderly User Evaluation of Mobile Touchscreen Interactions

Keywords: Mobile, Smartphones, Touchscreens, Gestures, Aging, Elderly, Senior Citizens, User Evaluation

Smartphones with touchscreen-based interfaces are increasingly used by non-technical groups including the elderly. However, application developers have little understanding of how senior users interact with their products and of how to design senior-friendly interfaces. As an initial study to assess standard mobile touchscreen interfaces for the elderly, we conducted performance measurements and observational evaluations of 20 elderly participants. The tasks included performing basic gestures such as taps, drags, and pinching motions and using basic interactive components such as software keyboards and photo viewers. We found that mobile touchscreens were generally easy for the elderly to use and a week's experience generally improved their proficiency. However, careful observations identified several typical problems that should be addressed in future interfaces. We discuss the implications of our experiments, seeking to provide informal guidelines for application developers to design better interfaces for elderly people.

Introduction

Mobile phones are filling essential roles in today's societies both for the elderly and younger groups. The Ministry of Internal Affairs and Communications in Japan reported that approximately 75% of people aged six or older had their own mobile phones in 2009 [1]. At the same time, more than 70% of people aged 60-69 and 40% of people aged 70-79 used mobile phones.

It is known that senior citizens tend to use their mobile phones for relatively limited purposes (e.g., [START_REF]Use of ICT at Home[END_REF]). The limiting factors include hard-to-see displays, hard-to-press buttons, and hard-to-learn procedures. Special mobile phones for the elderly (such as the Raku-Raku Phone from NTT DoCoMo [START_REF]Universal Design Initiatives[END_REF]) have been developed and are widely used, and they provide such features as simplified interfaces with larger buttons. However, their limited functionality may increase the digital divide between the older and younger generations.

Touchscreen-based smartphones have the potential to address these problems. First, the finger-based intuitive interactions can be useful regardless of the age of the user. Second, touchscreen-based interfaces allow us to offer senior-friendly interfaces through software-level adaptations. Senior citizens can use the same hardware devices as younger people and also access online services and communities. This increases the social inclusion of the elderly, improving the quality of their lives and making a more sustainable society by benefiting from the power of senior citizens.

The current penetration rate of touchscreen-based smartphones for the elderly is estimated to be low. However, in the next few years they are expected to become the majority of mobile phones, since many of the new mobile phones are smartphones and the major smartphone operating systems all support touchscreens. The market of smartphone applications for senior citizens is expected to grow due to the rapidly increasing numbers of elderly mobile phone users in developed countries. Our informal preliminary survey supports this expectation, indicating that at least some senior citizens, even those who usually feel hesitation about new technologies, were interested in the latest touchscreen-based smartphones and enjoyed using gestures on their screens. Given the explosive growth of mobile phones including smartphones (even in emerging countries [4]), senior-friendly smartphone applications will be strongly desired throughout the world.

A major problem is that application developers currently have little understanding of how to design better touchscreen interfaces for elderly users since the de facto standards of basic operations on touchscreen-based smartphones, which consist of tapping, dragging, and pinching, have only been adopted in the last few years. We need frameworks and guidelines supported by empirical evidence to help develop senior-friendly interfaces. As an initial step, we need to know how the elderly interact with current touchscreen interfaces as a basis for the development of frameworks and guidelines.

We observed and measured the actions of elderly people using touchscreen smartphones. Based on previous research on senior citizens' use of traditional buttonbased mobile interfaces (e.g., [START_REF] Siek | Fat Finger Worries: How Older and Younger Users Physically Interact with PDAs[END_REF] [START_REF] Ueda | Usability Evaluation of Mobile Phone for the Elderly: The Effect of Properties of the Button[END_REF]) and gesture-based commands on touchscreens (e.g., [START_REF] Stößel | Gestural Interfaces for Elderly Users: Help or Hindrance? In: Gesture in Embodied Communication and Human-Computer Interaction[END_REF][8]), we focused on typical finger-based operations on standard devices to assess the practical use of mobile touchscreen interfaces. The goal of this study is to determine the trends and problems of mobile touchscreen interfaces that elderly users actually encounter. The tasks included: (1) controlling basic operations with gestures such as taps, drags, and pinching motions, for which we measured task completion times, analyzed their behaviors while making the motions, and asked about the users' preferences; and (2) using interactive components operated with basic gestures such as software keyboards and photo viewers, for which we simply observed their behaviors and asked for user comments. Based on the quantitative and qualitative results, we try to provide informal guidelines for application developers to design senior-friendly interfaces. Our initial experiments also suggested future research directions and new experiments.

The remainder of this paper is organized as follows. First, in Section 2 we summarize related work. Next we describe the methods of our experiments. Then we present the results of the performance measurements and review the observational analyses and the subjective feedback. We also discuss the implications of the experiments and possible future research. Finally, we summarize this work.

Related Work

Much research has been done to evaluate the usability for elderly people of desktop or laptop user interfaces [START_REF] Arias-Torres | The Design and Evaluation of a Pen-Based Computer Interface for Novice Older Users[END_REF] [START_REF] Mahmud | Involving Psychometric Tests for Input Device Evaluation with Older People[END_REF]. Guidelines for designing accessible Web user interfaces for elderly people have been proposed [START_REF]Web Accessibility and Older People: Meeting the Needs of Ageing Web Users[END_REF]. In contrast, developing design guidelines for the elderly using such new interfaces as mobile terminals with touchscreen interfaces is ongoing work. Design guidelines for general touchscreen-based mobile interfaces have only recently been adopted and guidelines that consider elderly users are yet to be investigated [START_REF]iOS Human Interface Guidelines[END_REF].

There have been a number of user evaluation studies of touchscreen interfaces. Leonardi et al. designed a tabletop touch panel interface and found that the direct interaction metaphor was easy to understand and had a pleasing effect that attracted and motivated elderly participants in their study [START_REF] Leonardi | An Exploratory Study of a Touch-Based Gestural Interface for Elderly[END_REF]. Lepicard et al. found twohanded touchscreen input was difficult for elderly users [START_REF] Lepicard | Touch Screen User Interfaces for Older Subjects: Effect of the Targets Number and the Two Hands Use[END_REF]. Stößel et al. compared old users to young users in 42 different gesture inputs for touch surfaces and measured their speed and accuracy. They found that older users are a little slower but there was no significant difference in accuracy and suggested that older adults favor accuracy over speed [START_REF] Stößel | Gestural Interfaces for Elderly Users: Help or Hindrance? In: Gesture in Embodied Communication and Human-Computer Interaction[END_REF].

Many studies of elderly users and mobile terminals have also been conducted. Seik et al. compared the performance using PDA applications between younger adults and older adults. Their results showed that both older and younger participants performed at the same level [START_REF] Siek | Fat Finger Worries: How Older and Younger Users Physically Interact with PDAs[END_REF]. Darroch et al. examined the preferred font sizes on PDA screens, comparing older and younger adults. There were no significant differences in reading performance and accuracy between the older and younger adults, but the preferred size of the font was slightly larger for the older participants [START_REF] Darroch | The Effect of Age and Font Size on Reading Text on Handheld Computers[END_REF]. Kurniawan investigated the problems that older people face when using mobile phones and assessed some characteristics of a mobile phone for the elderly [START_REF] Kurniawan | Older People and Mobile Phones: A Multi-Method Investigation[END_REF]. Older people are relatively passive adopters with fears of the consequences of using unfamiliar technologies, such as reduced face to face communications, or of accidents caused by careless use, such as by talking while driving. They like functions that support their declining functional capabilities.

Usability studies for new smartphones with touchscreen interfaces have just begun. Stone proposed a special free text input method for elderly people [START_REF] Stone | Mobile Touch Interfaces for the Elderly[END_REF]. There have been some accessibility-related evaluations of mobile touchscreens for visuallyimpaired or motor-impaired users [START_REF] Guerreiro | Towards Accessible Touch Interfaces[END_REF] [START_REF] Mcgookin | Investigating Touchscreen Accessibility for People with Visual Impairments[END_REF]. Stößel et al. compared older users and younger users in how they interacted with touchscreens including multi-touch systems. They focused on symbolic gestures and direct manipulations. Their results showed that older users prefer direct manipulations. There were no age-related differences in direct manipulations, but considerable age differences in the use of symbolic gestures.

Symbolic gestures are relatively more accepted by older users. Also, they found no age-related differences in single-finger gestures, but younger users were more likely to use double-fingers gestures [START_REF] Stößel | Mobile Device Interaction Gestures for Older Users[END_REF].

These studies give us valuable information regarding various perspectives of touchscreen interfaces for elderly people. However, few studies have focused on the up-to-date, de facto standard set of touchscreen operations, which consists of simple gestures such as tapping for selecting items, dragging for scrolling, and pinching for zooming. To derive general guidelines for designing senior-friendly interfaces on a common device, we assessed the elder's trends in performance and behaviors in the basic set of touchscreen operations.

Methodology

Participants

Twenty elderly Japanese in their 60s and 70s (14 females and 6 males) were paid to participate in the experiments. Their profiles are summarized in Table 1.

All of the participants had at least two years of prior experience with personal computers, except for one participant whose experience was not recorded (P15). Twelve of the participants had at least two years of prior experience with mobile phones, one was not a mobile phone user (P9), and two did not report their mobile phone experience (P1 and P15). Four of the participants had up to two years of experience with touchscreen-based mobile devices (P1, P4, P5, and P14). Given their relatively longer experience with traditional information technologies, the results of our experiments are expected to reflect the problems specific to mobile touchscreen interfaces rather than those due to the lack of general IT skills and knowledge.

Only one participant was left-handed (P18), but she interacted with the touchscreen primarily with her right hand, as did all of the right-handed participants. Some stated that they had difficulties in their daily lives due to age-related problems with their eyes (P1, P4, and P9) or age-related hearing loss (P5).

Apparatus

We used an iPad (the "large device" with a 9.7-inch, 147.8×197.1 mm multi-touch screen with 768×1024 pixel resolution, weighing 680 g) and an iPod touch (the "small device" with a 3.5-inch, 49.3×74.0 mm multi-touch screen with 640×960 pixel resolution, weighing 101 g) running iOS 4.2 in the experiments. The large device was enclosed in a case while the small device was used without a case. The experimental software described below was implemented as a native application for the small device running in the low resolution mode (320×460 pixels without a status bar at the top of the screen) at 163 ppi. On the large device, it was run in the small device emulation mode (double size) at 66 ppi. We tested the small device as a representative of mobile devices that have similar size touchscreens. We included the large device in the experiments believing that the larger touchscreen would be especially helpful for the elderly participants, since they may have visual and motor limitations that would be eased by the larger screen.

The participants sat on a chair in front of a table during the experiments. We asked them to hold the devices in a comfortable manner. As a result, the participants who used a large device generally put it on the table while the other participants who used a small device generally held it with their left hands.

Procedure

The participants performed the series of experimental procedures twice, separated by one week, except for one participant (P11) who could not return for the second session. Each session consisted of (a) performance measurements for gesture operations and (b) realistic uses of interface components. Each session took about two hours, including a post-experiment interview. During the one-week period, each participant was asked to practice the gesture operations using the experimental software at least once per day. They were also allowed to freely use the pre-installed applications.

Performance measurements. Each participant was asked to perform four standard touchscreen operations: tapping, dragging, pinching without panning, and pinching with panning. Fig. 1 shows images of each task.

Each tapping trial started when a target, a white rounded-square button, appeared at a random location on the screen. It ended when the target was successfully tapped, meaning the finger was briefly placed on and quickly removed from the target. For each trial, the target was randomly selected from three sizes: 30, 50, and 70 pixels. The target became blue as the finger was touching the target. Once a target had been dismissed, the next target appeared until the participant had completed 30 tapping trials. Before the timed trials, the participant was allowed to have some practice trials.

We used 30, 50, and 70 pixels as the typical sizes of the keys in a software keyboard, general buttons, and icons on the home screen, respectively. We controlled the logical size in pixels, instead of the physical size, aiming to identify the problems that elder users may face in the actual use of standard mobile interfaces.

Each dragging trial started when an image (200×200 pixels) appeared at a random location on the screen. It ended when the image was moved into the target using dragging motions. The target was a green square (also 200×200 pixels) with 20-pixelwide borders, which was always displayed at the center of the screen. The width of the borders of the target rectangle represented the tolerance, so errors up to 10 pixels in any direction were permitted for the final image location when the trial ended. The participants had some practice trials and 30 timed trials, just as with the tapping trials.

Each pinching-without-panning trial started when a variable-size square image appeared at the center of the screen. It ended when the image was adjusted to match the target by using two-finger pinching or spreading motions. The initial image was 50, 100, 300, or 400 pixels. The target was the same 200-pixel target used in the dragging trials. The final image size could range from 190 to 210 pixels. The participants again had some practice trials but only 20 timed trials.

Each pinching-with-panning trial started when a square image appeared at a random location on the screen. It ended when the image's size and location had been adjusted to match the same target rectangle. This required two-finger pinching and spreading motions and one-or two-finger dragging. The initial image sizes and the target size were the same as in the "without panning" condition. The participants performed some practice trials and 20 timed trials. Realistic use. The participants were asked to use two standard interactive components: photo viewer and software keyboard. We chose these two components as representative of fundamental tasks in online communications: browsing and entering information. Also, they require the use of all of the tested gestures. First each participant was briefly shown how to use each component. Then they were asked to use the component for about 20 minutes with informal tasks such as finding photos that contain more than ten people and copying some printed reference sentences. We also briefly introduced them to some frequently used applications such as cameras, music players, e-book readers (for magazines and newspapers), and games.

Design

For the performance measurements, the device (large or small device) was a betweenparticipant factor, where P0-P9 used the large device and P10-P19 used the small device. The week (1 or 2) was a within-participant factor. Other within-participant factors included the target size for tapping tasks and initial size for pinching tasks. We measured the task completion time as the primary dependent variable. We also investigated the participants' behaviors in depth and asked about their preferences.

For the realistic use experiment, we observed the participants' behaviors and then asked them for comments.

Hypotheses

Here are the main hypotheses we sought to test in this study:

1. Basic touchscreen operations are easy for the elderly to perform without training. 2. A week's practice will improve the performance. 3. The simplest gesture, tapping, is the easiest to perform. 4. Pinching with panning is the most difficult to perform because of its complexity. 5. The larger screen is preferred and more efficient for tapping, because the targets are larger, but the smaller screen is better for dragging and pinching operations, requiring smaller finger movements. 6. For the interactive components, the participants face problems similar to those of traditional user interfaces, such as unclear instructions and unclear indicators of the current state.

Basic Gesture Performances

We found no significant effects on the task completion times due to the age, gender, dominant hand, prior experience, or difficulties in vision/hearing. This does not prove that these factors cannot affect the performance because we did not control for them in this experiment. Their precise effects or irrelevance should be assessed in future research.

Taps

Fig. 2 shows the average task completion times for tapping operations for each target size. For the first week, the average times were 0.77 seconds and 1.02 seconds for large and small devices, respectively. For the second week, they were 0.72 seconds and 0.94 seconds. A week's practice reduced the task completion times by 7% and 9% for large and small devices, respectively. Analysis of variance showed significant main effects on the task completion times due to the device (F 1,1158 = 14.43, p < .001) and the target size (F 2,1158 = 28.81, p < .001). It also showed a significant interaction effect between the device and the target size (F 2,1158 = 8.96, p < .001). There were no significant main effects of the week's practice (F 1,1158 = 1.12, p > .05). A post hoc analysis found that tapping 30pixel targets on the small device was significantly more difficult than the other conditions. It took about twice as long to tap the 30-pixel target compared to the 50or 70-pixel targets.

Drag

Fig. 3 shows the average task completion times for dragging operations. For the first week, the average times were 2.09 seconds and 2.17 seconds for the large and small devices, respectively. For the second week, they were 1.59 seconds and 1.77 seconds. A week's practice reduced the task completion times by 24% and 18% for the large and small devices, respectively.

Analysis of variance showed significant main effects from the week's practice (F 1,1138 = 50.83, p < .001) and for the device (F 1,1138 = 4.52, p < .05) on the task completion times. There were no significant interaction effects.

Pinching and spreading without panning

Fig. 4 shows the average task completion times for zooming in and out operations for each initial image size. In the first week, the average times were 2.69 seconds and 2.75 seconds for the large and small devices, respectively. For the second week, they were 1.92 seconds and 1.99 seconds. A week's practice reduced the task completion times by 29% and 28% for the large and small devices, respectively.

Analysis of variance showed significant main effects on the task completion time of the week's practice (F 1,753 = 100.15, p < .001) and from the initial size (F 3,753 = 20.87, p < .001). It also showed a significant interaction effect between the device and the initial size (F 3,753 = 6.51, p < .001). There were no significant main effects of the device (F 1,753 = 0.63, p > .05). A post hoc analysis found that spreading motions were more difficult than pinching motions, especially on the small device. It took 3.13 seconds to zoom in to resize the image from 50 pixels to 200 pixels while it took only 2.19 seconds to zoom out from 400 pixels to 200 pixels, even though the required amount of finger movement on the screen was larger for zooming out. Some participants commented that spreading (zooming-in) motions were more difficult than pinching (zooming-out) motions, reinforcing these results.

Pinching and spreading with panning

Fig. 5 shows the average task completion times for zooming with panning operations for each initial image size. In the first week, the average times were 4.57 seconds and 5.01 seconds for the large and small devices, respectively. In the second week, they were 3.20 seconds and 3.60 seconds. A week's practice reduced the task completion times by 30% and 29% for large and small devices, respectively. The times were approximately the total of the times for separately dragging and pinching without panning. This seems to confirm that the participants often used a 2-phase approach, where they first dragged the image to the center of the screen and then zoomed in or out to fit the target rectangle.

Analysis of variance showed significant main effects on the task completion times of the week (F 1,738 = 76.71, p < .001), the device (F 1,738 = 11.92, p < .005), and the initial size (F 3,738 = 18.85, p < .001). It also showed a significant interaction effect between the week and the initial size (F 3,738 = 5.09, p < .005). A post hoc analysis found that a week's practice greatly improved the performance, especially for zooming in from 50 pixels to 200 pixels (6.38 seconds and 4.06 seconds for weeks 1 and 2, respectively).

Observational Findings

Why is tapping a small target so difficult?

To help explain why tapping 30-pixel targets showed much worse performance on the small device, Fig. 6 shows the locations where the participants actually touched in relative coordinates with respect to the target location. The average x-locations were off center by +12.9 (SD = 10.0) and +8.28 (SD = 24.2) pixels for Weeks 1 and 2, respectively. In contrast, the corresponding values for the y-axis were only +2.0 (SD = 9.7) and +1.8 (SD = 31.2) pixels. The participants clearly tended to touch the right side of the target. The size and location of the target did not affect this tendency. The gap between intended and actual touch locations could be explained by the fact that all of the participants used their right hands to perform the operations. The gap in the (x, y) range was similar for the 50-and 70-pixel targets on the small device while it was smaller for the large device. The touch locations mostly ranged from (-25, -25) to (+25, +25). As a result, the 30-pixel targets frequently caused errors and increased the task completion times, while the larger targets had few errors. For 30-pixel targets on the small device, the error rate was 39%, where an error means the participant tapped the screen outside of the target at least once during the trial. This error rate was much worse than the acceptable 4% value used in most human-computer interaction (HCI) literature. The error rates were 6.5% and 5.6% for 50-and 70-pixel targets, respectively, on the small device. The respective values were 13.6%, 1.4%, and 1.7% for 30-, 50-, and 70-pixel targets on the large device.

The distribution of touch locations also indicates why the performance was little improved after a week's practice. As shown in Fig. 6, the variance of the locations in Week 2 was actually larger than in Week 1. Our observations and the participants' comments showed that this was caused by a change in the participants' error correction strategy. Due to the gap between the intended and actual touch locations, the participants often tapped outside the target and saw that nothing happened. In that case, in Week 1 they tended to tap the same place until they received the expected feedback. In Week 2, they seemed to change their strategy and tapped at different places, now being aware of the gap between the intended and actual touch locations and trying to correct for it. However, this strategy often resulted in an error by overshooting the target, especially for small 30-pixel targets on the small device (Fig. 7-a). In addition, we found another type of strategy change. In Week 2, some participants tried to vary the speed and pressure by slowing down or speeding up their finger movements if they failed to tap the target on the first attempt. The average contact times with the screen when tapping were 105 ms (SD = 43 ms) and 114 ms (SD = 76 ms) for Weeks 1 and 2, respectively. This increased-pressure strategy often caused other problems. First, an overly slow tap can be recognized as a "hold" operation by many applications. Second, exerting more pressure on the screen during a slow motion can cause the finger to draw a random path, which is then recognized as a drag (Fig. 7-b). The low visibility of the visual feedback seemed to make these problems even worse, since a dynamic change of the target color was often obscured by the finger when tapping a small (30-pixel) target. Due to these factors, five of the participants (P7 who used the large device and P12, P13, P15, and P16 who used the small device) actually became slower after a week's practice.

Fig. 6. The distribution of touch locations relative to the location of the 30-pixel target on the small device before and after a week's practice. The target area is shown as a rounded square.

The participants tended to tap the right side of the target. The variation of touch locations was larger after the week's practice.

Problems in applications

The problems with tapping small targets were frequently observed during the use of the practical interactive components. For example, the participants failed to tap the proper key on the software keyboard or failed in moving the cursor to a specific position in a text input area. In addition, we found that the participants faced general user interface problems, which seem to be common with traditional user interfaces. First, the participants were often confused due to unclear instructions. A typical gesture-based interface on a touchscreen lacks textual labels and menus to show "what is possible with this application". It also provides few explicit instructions for each gesture. Even though an interface is carefully designed so that it is natural and intuitive for users to control many of its functions, the participants tended to stick to the few functions that the experimenters had explained. They were sometimes confused when they invoked an unexpected function with an unintentional gesture. A participant commented that he wanted standard menus as with a personal computer (PC). Though the participants were experienced PC users and familiar with standard QWERTY keyboards, they often could not find the backspace and shift keys on the software keyboard because those keys did not have the textual labels "Backspace" and "Shift".

The participants were also confused when an application has multiple modes without an indication of the current mode. For example, the photo viewer provides multiple thumbnail views such as by folder, by event, and by person, but it was difficult to tell which view was active. The software keyboard was more problematic. Since Japanese needs to input thousands of Chinese-related characters as well as characters using three alphabetic sets, Arabic digits, and symbols, the interface requires frequent mode-switching between predictive input [START_REF] Masui | POBox: An Efficient Text Input Method for Handheld and Ubiquitous Computers[END_REF], 5-touch input [START_REF] Macdonald | The UD Phenomenon in Japan: Product Innovation through Universal Design[END_REF], and a QWERTY keyboard. The participants frequently lost track of their current mode, especially when another mode had a similar look-and-feel. One participant explicitly complained that there was no textual indication of the current mode.

Subjective Feedback

In contrast to our expectations, the participants commented that dragging and pinching were preferred and easier to do than tapping. In particular, most of the participants stated that dragging was the easiest operation to perform. They found that pinching was also easy, though some reported that pinching with panning was somewhat difficult.

As regards the device, the tradeoff between the size and weight was mentioned. Many participants said that the large device was too heavy to carry while the small device was too small to read. Some female participants wanted a middle-sized, 7-inch device that would fit into their handbags.

When it came to applications, the participants wanted to use various types of applications, such as cameras and photo viewers, newspapers, e-books, and games. Many of them stated that they especially wanted smartphones when they had a few extra minutes, such as when riding the train.

The participants frequently commented that using touchscreen interfaces was "enjoyable". For example: These comments confirm the findings of previous research [START_REF] Chikawa | Usability Evaluation of Touch Screen Mobile Phone for the Elderly[END_REF]. The enjoyability is one of the most important factors in encouraging the elderly to continue participating in society via information technologies. We need more investigation of the factors influencing why they enjoyed using these devices. Perhaps it is part of the nature of touchscreen-based interfaces or the participants were excited by using new devices.

It

Discussion

The experiments partly confirmed and partly rejected our hypotheses. The first hypothesis was generally confirmed. The participants, even those who had never used touchscreens, performed the gesture-based operations reasonably well, except for tapping on small targets. The second hypothesis was partly confirmed. A week's practice significantly improved the performance for dragging and pinching operations, though there were no significant effects for tapping. The third and fourth hypotheses were rejected. The participants stated that dragging and pinching were easier and more comfortable than tapping. Though some of them found that pinching with panning was a little difficult, their performance was not seriously hindered by the difficulty. The fifth hypothesis was partly confirmed. It was surprising that even for dragging and pinching, as well as for tapping, the larger screen outperformed the smaller screen, even though it required more than twice the amount of finger movement on the screen. However it was reported that the device with the large screen was too heavy to carry and use outdoors. The last hypothesis was also confirmed.

Design Implications

Based on the experimental findings, we suggest these informal guidelines for application developers seeking to design better interfaces for the elderly.

Use larger targets (8 mm or larger in size).

The elderly tend to make errors when tapping a small target, such as a 30-pixel button on a small screen. In the experiment, the touch locations were mostly distributed within 8 mm on the physical screen regardless of the device and the target size. Thus, interactive objects such as buttons, icons, and clickable text should at a minimum be larger than 8 mm. Note that some common components in the latest smartphone interfaces, such as keys in a software keyboard and a "back" button in the upper-left corner of a small device, conflict with this guideline. With targets located close to each other without spacing (e.g., software keyboard), the size should be much larger to avoid invoking adjacent targets. This guideline confirms the result of previous work [START_REF] Lee | The Performance of Touch Screen Soft Buttons[END_REF], which found that the speed and error rates were much worse when tapping buttons smaller than approximately 8 mm.

Address the gap between intended and actual touch locations. The elderly tend to miss their intended targets due to parallax and the large contact area of each finger. Providing a calibration mechanism might be a solution. However, practical calibration is difficult because of the nature of handheld devices. Users frequently tilt and rotate their devices depending on their situations. Another solution might include providing appropriate visual feedback to indicate where the users touched the screen, even when they have missed all of the interactive targets.

Consider using drag and pinch gestures rather than taps. The elderly tend to prefer dragging and pinching operations over tapping. This is a new finding that was not observed in an earlier experiment on multi-touch interactions for elder users [START_REF] Stößel | Mobile Device Interaction Gestures for Older Users[END_REF]. It is also the opposite of the result for motor-impaired users [START_REF] Guerreiro | Towards Accessible Touch Interfaces[END_REF]. Though applications need not avoid using drag and pinch operations, the application should provide instructions and clear visual clues to show which gesture invokes which function. Since typical functions invoked by dragging and pinching provide no visual clues, such as flipping and zooming photos, most participants in our experiment could not find those functions without guidance.

Explicitly display the current mode. Some elderly users are less likely to notice changes of the modes and can become confused. Applications should avoid multimode interfaces as much as possible. If a multi-mode design is needed, then there should be explicit feedback about mode-changes and a display of the current mode. The indicators of mode changes should be persistent and large, since the elderly users may fail to notice short alerts or small changes in the look-and-feel. Also, the feedback should be readable as they tend to have difficulty in interpreting the meanings of symbolic representations.

Future Work

A limitation of this study is that we did not have younger participants as a reference group. We focused on investigating how the elderly actually interact with mobile touchscreens and how practice improves their proficiency, as a first step towards guidelines and frameworks to develop senior-friendly interfaces. Hence in-depth analyses of aging effects were beyond our scope. An informal experiment we conducted with two young participants (2 females, 24 and 31 years old), in which they performed touchscreen operations 3%-49% faster than the senior participants, indicated that younger users interact more rapidly with touchscreen interfaces. However, the general trends were similar. For example, considering the result of previous research [START_REF] Lee | The Performance of Touch Screen Soft Buttons[END_REF], the minimal target size that a user can tap with good speed and accuracy seems to be around 8 mm for both younger and older people. Additional investigation of the similarity and differences between younger and older users in their performances, behaviors, and preferences would help in designing more universal interfaces that satisfy users of all ages.

In addition to the comparison with younger users' performance, more investigation is needed to understand the trends and problems we encountered. Are they caused by physical and cognitive disabilities such as attention and memory loss related to aging? How do the experiences with other technologies affect performance? A study with more control of the variables such as age, disability, and experience with technologies may discover clues towards more practical solutions for specific user groups.

Conclusion

We conducted performance measurements and observational evaluations to assess standard mobile touchscreen interfaces when used by the elderly. Our participants were 20 Japanese in their 60s and 70s. The tasks included (1) controlling basic operations with gestures such as taps, drags, and pinching motions and (2) using basic interactive components such as software keyboards and photo viewers. The results show that touchscreen mobile interfaces are preferred and not too difficult to use, even by the elderly. A week's practice significantly improved the performance in dragging and pinching, but did not significantly affect tapping. We identified several typical problems, such as mismatches between the user's visual target and the touched position as detected by the sensor. We discussed the implications of the experiments on the design of better interfaces for the elderly and considered future research directions. This and future studies will provide the basis for the development of senior-friendly user interfaces.

Fig. 1 .

 1 Fig. 1. Our experimental software was designed to practice and assess four standard touchscreen operations: (a) tapping, (b) dragging, (c) pinching without panning, and (d) pinching with panning. The text, lines, and arrows are annotations, not displayed in the actual experiments.

Fig. 2 .

 2 Fig. 2. Tapping task completion times for each target size with standard errors before and after a week's practice.

Fig. 3 .

 3 Fig.3. Task completion times for dragging with standard errors before and after a week's practice.

Fig. 4 .

 4 Fig. 4. Task completion times of pinching without panning with standard errors before and after a week's practice.

Fig. 5 .

 5 Fig. 5. Task completion times for pinching with panning with standard errors before and after a week's practice.

Fig. 7 .

 7 Fig. 7. (a) Due to repetitive overshooting, a participant touched the screen 32 times to tap a small target (P12, Week 2). (b) Adding extra pressure (i.e., "pressing") on the screen while tapping can cause an irregular trajectory (P13, Week 2).

  is fun to flip through photos [in the photo viewer]. (P5) I want to do nothing but use this [a touchscreen]. (P14)

Table 1 .

 1 Twenty senior citizens participated in the experiments. P0-P9 tested a large screen device while P10-P19 tested a small screen device.

	#	Age	Gender	Dominant hand	PC	Prior experiences Mobile Touch-screen	Difficulties in vision/hearing
	P0	70s	Female	Right	10 years	6 years	none	-
	P1	70s	Male	Right	10 years	-	1 year	Vision
	P2	70s	Female	Right	10 years	6 years	none	-
	P3	70s	Male	Right	2 years	5 years	none	-
	P4	70s	Female	Right	10 years	5 years	once	Vision
	P5	70s	Female	Right	8 years	2 years	2 years	Hearing
	P6	70s	Male	Right	15 years	3 years	none	-
	P7	70s	Male	Right	23 years 15 years	none	-
	P8	70s	Female	Right	6 years	3 years	none	-
	P9	60s	Female	Right	6 years	none	none	Vision
	P10	60s	Female	Right	7 years	8 years	none	-
	P11	60s	Female	Right	10 years 10 years	none	-
	P12	70s	Female	Right	8 years	3 years	none	-
	P13	70s	Male	Right	10 years	5 years	none	-
	P14	70s	Male	Right	15 years	8 years	once	-
	P15	60s	Female	Right	-		-	none	-
	P16	60s	Female	Right	6 years	6 years	none	-
	P17	60s	Female	Right	15 years 10 years	none	-
	P18	70s	Female	Left	10 years 11 years	none	-
	P19	60s	Female	Right	10 years	5 years	none	-

Acknowledgments. This research was partially supported by the Japan Science and Technology Agency (JST) under the Strategic Promotion of Innovative Research and Development Program. We thank Ms. Yoriko Ohbayashi, Mr. Sternly K. Simon, Mr. Daisuke Ochiai, Mr. Daisuke Sato, and Dr. Susumu Harada for their assistance. We also thank all of the participants in our experiments.


Mike Bennett
email: mikemb@stanford.edu

Kevin Mccarthy

Sile O'modhrain

Barry Smyth

SimpleFlow: Enhancing Gestural Interaction With Gesture Prediction, Abbreviation And Autocompletion

Gestural interfaces are now a familiar mode of user interaction and gestural input is an important part of the way that users can interact with such interfaces. However, entering gestures accurately and efficiently can be challenging. In this paper we present two styles of visual gesture autocompletion for 2D predictive gesture entry. Both styles enable users to abbreviate gestures. We experimentally evaluate and compare both styles of visual autocompletion against each other and against non-predictive gesture entry. The best performing visual autocompletion is referred to as SimpleFlow. Our findings establish that users of SimpleFlow take significant advantage of gesture autocompletion by entering partial gestures rather than whole gestures. Compared to nonpredictive gesture entry, users enter partial gestures that are 41% shorter than the complete gestures, while simultaneously improving the accuracy (+13%, from 68% to 81%) and speed (+10%) of their gesture input. The results provide insights into why SimpleFlow leads to significantly enhanced performance, while showing how predictive gestures with simple visual autocompletion impacts upon the gesture abbreviation, accuracy, speed and cognitive load of 2D predictive gesture entry.

Introduction

Gestural interfaces are now a familiar mode of user interaction and gestural input is an important part of the way that users can interact with su ch interfaces. Gestural input accommodates a style of interaction that goes beyond the point -and-click of traditional WIMP-based interfaces and allows users to interact in a more intuitive and efficient [START_REF] Rubine | Specifying gestures by examp le[END_REF][START_REF] Ward | Dasher -a gesture-driven data entry interface for mobile computing[END_REF][START_REF] Cao | Evaluation of an on-line adaptive gesture interface with command prediction[END_REF][START_REF] Harrison | Skinput: Appropriating the body as an input surface[END_REF]. For example, simple swiping motions (whether mouse pointer, trackpad, or direct screen-contact) can be used as more intuitive navigation co ntrols, circumventing the precision that is demanded by more traditional point -and-click interactions [START_REF] Yatani | Escape: A target selection technique using visually-cued gestures[END_REF]. A range of more sophisticated gestures, such as a two-fingered 'pinch' for zooming, are now an increasingly familiar part of gesture-based interaction dictionaries [START_REF] Freeman | Shadowguides:Visualizations for in-situ learning of multi-touch and whole-hand gestures[END_REF][START_REF] Gustafson | Imaginary interfaces: Spatial interaction with empty hands and without visual feedback[END_REF]. Ultimately, gesture-based interaction promises to provide users with a more intu itive and richer interaction vocabulary, offering greater interaction bandwidth for lo wer input effort [START_REF] Yatani | Escape: A target selection technique using visually-cued gestures[END_REF][START_REF] Collomb | Improving drag-and-drop on wall-size displays[END_REF]. However, this is not always the case and certainly the ques t for more expressive gestures can lead to suboptimal gestures that are difficult for u sers to produce reliably, leading to interaction failures and frustration. One way to help users as they produce gestures is to provide autocompletion sugges tions as the gesture unfolds [START_REF] Arvo | Fluid sketches: Continuous recognition and morphing of simple hand-drawn shapes[END_REF][START_REF] Tandler | Using incremental gesture recognition to provide immediate feedback while drawing pen gestures[END_REF][START_REF] Bau | Octopocus: A dynamic guide for learning gesture-based command sets[END_REF][START_REF] Freeman | Shadowguides:Visualizations for in-situ learning of multi-touch and whole-hand gestures[END_REF]. If this can be done efficiently and accurately then the user will benefit in a number of ways. For example, by accepting a gesture suggestion the user can complete their gesture early and so gestural input can be made more efficient [START_REF] Soukoreff | Text entry for mobile computing: M odels and methods, theory and practice[END_REF][START_REF] Collomb | Improving drag-and-drop on wall-size displays[END_REF][START_REF] Baudisch | Drag-and-Pop and Drag-and-Pick: Techniques for accessing remote screen content on touch-and pen-operated systems[END_REF]. Moreover, by encouraging early gesture completion it removes the risk of e rrors that might have occurred if the gesture had to be manually completed. At the same time there may be additional costs for the user, especially if they fin d the gesture completions to be difficult to accommodate into the work-flow; if completion suggestions are inaccurate, for example, then users will quickly become frustrated by such inconvenient interruptions.

In this paper we describe two approaches to gesture autocompletion (we refer to these as predictive gestures), both of which present the user with visual feedback as a gesture is entered (Figure 1 & 2). Both also provide the user with an option to autocomplete the target gesture early. In addition, we describe the results of a detailed user evaluation, in which the predictive gestures are compared to manual (nonpredictive) gesture input, focusing on the gesture shortening, accuracy, speed and cognitive load characteristics of these techniques. In particular we demonstrate and explain how predictive gesture techniques can lead users to significantly abbreviate gestures while improving the speed and accuracy of their gesture input; along with S impleFlow: Enhancing Gestural Interaction With Gesture Prediction, Abbreviation And Autocompletion 3 users stating a clear preference for the predictive gesture input over non-predictive entry.

The contributions of this work are: 1. Based on the visual autocompletion styles in this paper, our results establish that users will shorten predictive gestures by a significant amount. For predictive text entry it is well established that users will significantly shorten words during text entry. Previous to this it had not been established whether users of predictive ge sture entry systems will enter shorter gestures, nor had it been established by how much they will shorten the gestures by. 2. Does allowing gesture abbreviation and showing visual autocompletion during ge sture entry increase the cognitive demands on users? Unexpectedly, we find that the cognitive load does not increase for the autocompletion styles. 3. We introduce SimpleFlow, an effective style of visual autocompletion, which su btly but for users importantly differs from the simplest form of visual feedback (Scale Free Dynamic Paths). 4. Are there significant trade-offs between predictive versus non-predictive gesture entry and the speed and accuracy with which gestures are entered and recognised?

We establish interactions and tradeoffs between gesture autocompletion, gesture abbreviation, input speed and input accuracy for SimpleFlow and Scale Free Dynamic Paths.

Related Work

Entering gestures accurately and quickly can be challenging [START_REF] Appert | Using strokes as command shortcuts: Cognitive benefits and toolkit support[END_REF][START_REF] Wobbrock | Gestures without libraries, toolkits or training: A $1 recognizer for user interface prototypes[END_REF][START_REF] Long | Visual similarity of pen gestures[END_REF]. Some of the challenges arise because of the need for algorithms that accurately recognise gestures [START_REF] Wobbrock | Gestures without libraries, toolkits or training: A $1 recognizer for user interface prototypes[END_REF][START_REF] Li | Protractor: A fast and accurate gesture recognizer[END_REF]. Other challenges include questions about what interface designs pro vide beneficial visual feedback before, during and after gesture entry [START_REF] Bau | Octopocus: A dynamic guide for learning gesture-based command sets[END_REF][START_REF] Appert | Using strokes as command shortcuts: Cognitive benefits and toolkit support[END_REF].

Visual Feedback For Predictions

Va rious styles of visual feedback have been proposed for gesture entry [START_REF] Freeman | Shadowguides:Visualizations for in-situ learning of multi-touch and whole-hand gestures[END_REF][START_REF] Ward | Dasher -a gesture-driven data entry interface for mobile computing[END_REF][START_REF] Bau | Octopocus: A dynamic guide for learning gesture-based command sets[END_REF][START_REF] Igarashi | Interactive beautification: A technique for rapid geometric design[END_REF][START_REF] Agar | Polygon recognition in sketch-based interfaces with immediate and continuous feedback[END_REF][START_REF] Tandler | Using incremental gesture recognition to provide immediate feedback while drawing pen gestures[END_REF][START_REF] Yatani | Escape: A target selection technique using visually-cued gestures[END_REF][START_REF] Kristensson | Command strokes with and without preview: Using pen gestures on keyboard for command selection[END_REF]. Often the feedback styles are for enhancing pre-and post-gesture entry.

Research on pre-gesture feedback aims to help users know and remember what set of gestures are available, while post-gesture feedback helps users understand whether they successfully entered a gesture, and if not what went wrong during gesture entry. ShadowGuides [START_REF] Freeman | Shadowguides:Visualizations for in-situ learning of multi-touch and whole-hand gestures[END_REF], OctoPocus [START_REF] Bau | Octopocus: A dynamic guide for learning gesture-based command sets[END_REF], Fluid Sketches [START_REF] Arvo | Fluid sketches: Continuous recognition and morphing of simple hand-drawn shapes[END_REF] and others [START_REF] Igarashi | Interactive beautification: A technique for rapid geometric design[END_REF][START_REF] Li | Sketch recognition with continuous feedback based on incremental intention extraction[END_REF][START_REF] Agar | Polygon recognition in sketch-based interfaces with immediate and continuous feedback[END_REF] are examples of interaction techniques which provide real-time visual feedback during gesture entry. These aim to help users learn the gestures better and help them unde rstand how well they are entering the gestures. Other forms of real-time feedback help the user understand what the results of the gesture will be, such as telling them what actions will be performed, e.g. show the user where a dragged icon will end up [START_REF] Baudisch | Drag-and-Pop and Drag-and-Pick: Techniques for accessing remote screen content on touch-and pen-operated systems[END_REF], or what action will be performed [START_REF] Bau | Octopocus: A dynamic guide for learning gesture-based command sets[END_REF].

Bau et al. [START_REF] Bau | Octopocus: A dynamic guide for learning gesture-based command sets[END_REF] propose a very useful classification framework for the various styles of visual feedback. Within their classification framework SimpleFlow is a dynamic

Mike Bennett1,2, Kevin McCarthy2, S ile O'Modhrain3, and Barry S myth2

guide, with feedback that has a continuous update rate, a real recognition value, a gesture filter and a gesture representation.

Applying Gestures

Recently there has been a surge in applications of gestures without visual feedback and in more novel contexts, such as using human skin as a gesture input surface [START_REF] Harrison | Skinput: Appropriating the body as an input surface[END_REF], or moving your hands freely in the air as a form of bimanual gesture entry [START_REF] Gustafson | Imaginary interfaces: Spatial interaction with empty hands and without visual feedback[END_REF].

Other styles of gesture input which are less obviously gesture prediction include Drag-and-Pop / Drag-and-Pick [START_REF] Baudisch | Drag-and-Pop and Drag-and-Pick: Techniques for accessing remote screen content on touch-and pen-operated systems[END_REF], Push-and-Pop [START_REF] Collomb | Improving drag-and-drop on wall-size displays[END_REF], Escape [START_REF] Yatani | Escape: A target selection technique using visually-cued gestures[END_REF], Dasher [START_REF] Ward | Dasher -a gesture-driven data entry interface for mobile computing[END_REF] and ma rking menus [START_REF] Kurtenbach | The limits of expert performance using hierarchic marking menus[END_REF]. In those examples the effects of users' physical actions are enhanced, so they can perform shorter and less actions than normally required for the tasks, which can be thought of as forms of gesture prediction and abbreviation.

Modeling Gesture Interaction

Models of human performance when following trajectories have also been investiga ted [START_REF] Accot | Beyond Fitts' Law: M odels for trajectory-based HCI tasks[END_REF], and could prove useful for modeling gesture input [START_REF] Cao | M odeling human performance of pen stroke gestures[END_REF][START_REF] Isokoski | M odel for unistroke writing time[END_REF], as models and metrics of human performance have proven useful for predictive text in put [START_REF] Soukoreff | Text entry for mobile computing: M odels and methods, theory and practice[END_REF][START_REF] Soukoreff | M etrics for text entry research: An evaluation of msd and kspc, and a new unified error metric[END_REF]. Related to modeling gestures is research around measuring and modeling the complexity of gestures [START_REF] Long | Visual similarity of pen gestures[END_REF], which could be applied to quantifying the predictability of gestures.

Visual Autocompletion & Predictive Gestures

In this section we present1 two styles of predictive visual feedback, which are shown during gesture entry. Both styles show visual predictions of the target gesture a user is inputting. In this way they allow the autocompletion o f gestures and thus encourage quicker and more efficient gesture input. We also outline the non -predictive visual feedback commonly found in many gesture entry applications [START_REF] Doozan | Strokeit gestures for Windows[END_REF][START_REF] Bennett | wayv: Gestures for Linux[END_REF][START_REF] Gomita | Firegestures: Gesture control for Firefox web browser[END_REF], which we experimentally compare the predictive visual feedback against.

Visual Autocompletion

We implemented two styles of visual autocompletion for predictive gestures. Both styles are closely related and subtly but importantly differ in how they provide the real-time visual feedback. The first style of visual autocompletion is called SimpleFlow, as shown in Figure 1, and the second style is called Scale Free Dynamic Paths (SF-Path), as shown in Figure 2.

The purpose of SimpleFlow and SF-Path is to keep the user informed during gesture input, such that they know as early as possible which of the trained gestures (examples in Figure 4) matches their input. Both styles are designed to enhance user performance during gesture entry, by improving the speed and accuracy with which gestures are entered. Both are also d e-S impleFlow: Enhancing Gestural Interaction With Gesture Prediction, Abbreviation And Autocompletion 5 signed to enable users to abbreviate gestures. The ability to enter abbreviated gestures resembles predictive text entry systems -where users can enter complete words by only typing the first few letters of the words [START_REF] Soukoreff | Text entry for mobile computing: M odels and methods, theory and practice[END_REF][START_REF] Soukoreff | M etrics for text entry research: An evaluation of msd and kspc, and a new unified error metric[END_REF]. Figures 1 &2 show examples of what users see on-screen while using SimpleFlow and SF-Path. The stars and arrow-heads in these figures are for illustrative purposes only; they indicate the start and current end points of the gesture being entered. Ge sture entry starts when the user presses the mouse button and begins moving the mouse. When entering predictive gestures, users are simultaneously shown two forms of visual feedback. First, they see the gesture path they have entered so far, i.e. the gesture ink. Secondly, they see the gesture prediction drawn in red. The gesture prediction is automatically scaled to match the shape and scale of the gesture ink. If a u ser chooses to stop entering the gesture at any time (by releasing the mouse button), then the currently predicted gesture is entered -as though the user drew the full gesture themselves.

Unlike the OctoPocus [START_REF] Bau | Octopocus: A dynamic guide for learning gesture-based command sets[END_REF] and ShadowGuides [START_REF] Freeman | Shadowguides:Visualizations for in-situ learning of multi-touch and whole-hand gestures[END_REF] systems, SimpleFlow and SF-Path only show one gesture prediction at a time. If another prediction is more suitable, then the existing prediction is instantly switched out for the new prediction. Only one gesture prediction is shown because participants in our initial pilot studies voiced strong concerns about the high level of visual complexity that occurs when showing multiple simultaneous SimpleFlow predictions.

Another key reason for not showing mu ltiple gesture predictions is we strongly suspect that the style of visual design used to show multiple gestures has a very significant impact upon user performance and preferences. The reason for this suspicion is due to research findings in psychology and perception around visual search, i.e. subjects search for a target stimulus (gesture) amongst distractor stimuli (predictions).

Standard Non-Predictive Visual Feedback

The non-predictive style of visual feedback evaluated in this work is Standard Feedback . Standard Feedback is included because it is the typical gesture input mech anism and visual feedback used in many real-world gesture UIs, e.g. StrokeIt [START_REF] Doozan | Strokeit gestures for Windows[END_REF] for Windows, wayV [START_REF] Bennett | wayv: Gestures for Linux[END_REF] for Linux, FireGestures [START_REF] Gomita | Firegestures: Gesture control for Firefox web browser[END_REF] for Firefox.

Standard Feedback does not give predictive visual feedback to users. When ente ring a gesture only the gesture ink is drawn on-screen. The gesture ink shows users the shape of the gesture they have entered, but no feedback about predictions are provided. Standard Feedback does not provide pre-or post-gesture information about whether the gestures are entered correctly or incorrectly.

For our experiment, during the course of Standard Feedback gesture input, gesture prediction does still take place but the predictions are not shown to the participants. This is done to ensure that the algorithm for recognising gestures is the same across all forms of predictive and non-predictive feedback. In this paper, Standard Feedback serves as a control for comparing and evaluating SimpleFlow and SF-Path.

Mike Bennett1,2, Kevin McCarthy2, S ile O'Modhrain3, and Barry S myth2

SimpleFlow

SimpleFlow provides real-time visual feedback during gesture entry and always su ggests a complete gesture. As soon as a user begins entering a gesture they start to receive continuously updated gesture predictions. A gesture prediction is drawn in red underneath the gesture ink (Figure 1). The predicted gesture is automatically scaled so it underlays the gesture ink and visually appears to continue the path of the gesture ink. Often the gesture ink and predicted gesture do not perfectly align on top of each other.

Figure 1 shows a full walk-through of SimpleFlow in action. In Figure 1(a), the user has started to enter their gesture, in this case, a triangle shaped gesture (Figure 4, Gesture 1). SimpleFlow suggests and displays a fully scaled triangle underneath the gesture ink. As the user continues to extend the edge of their input ges ture (as shown in Figure 1(b)) the SimpleFlow feedback scales up to accommodate the change. Figure 1(c) depicts the final stage of entering the triangle. The user draws the base of the triangle gesture, and decides that the full gesture prediction provided by SimpleFlow satisfies their needs. Now they cease their gesture input, safe in the knowledge that the correct complete gesture has been recognised.

SimpleFlow is scale and position invariant, however there is a limitation on the minimum size of the gesture predictions. If there was no minimum size limitation, the predicted gestures could be too small to see. For more details on the gesture prediction algorithm see the Appendix.

Scale Free Dynamic Paths

Scale Free Dynamic Paths (SF-Path) is like SimpleFlow. As with SimpleFlow, SF-Path is scale and position independent, and has an imposed limitation on the min imum size of the gesture predictions. The critical difference between SF-Path and SimpleFlow is how much of the predicted gesture is shown as visual feedback. As the results show, this visually subtle and small difference between SF-Path and Sim-pleFlow has a very significant impact on user performance and preferences.

Figure 2 shows the visual feedback provided while entering the triangle gesture with SF-Path. As with SimpleFlow the gesture ink is black, and the predicted gesture is shown in red. Unlike SimpleFlow, the complete gesture prediction is not shown onscreen by SF-Path. Instead the prediction and gesture ink are merged to form a single combined gesture, so that the gesture prediction looks to be a continuation of the gesture ink.

SF-Path is less visually complex than SimpleFlow, as there are no red lines unde rlying the gesture ink (Figure 2). When the gesture predictions are wrong the visual complexity of SimpleFlow is beneficial. For example, Figure 3 shows a side by side example of the differences between SimpleFlow and SF-Path when a visual prediction is wrong. In this example a user is trying to enter a triangle gesture. The prediction a lgorithm is mistakenly suggesting the gesture C. In Figure 3(a), with SimpleFlow, it is clear that the prediction algorithm is mistakenly suggesting the C gesture. Unfort unately, with SF-Path (Figure 3(b)) it is not at all clear which predicted gesture is merged with the gesture ink to form the combined gesture.

S impleFlow: Enhancing Gestural Interaction With Gesture Prediction, Abbreviation And

Autocompletion 7

SF-Path resembles the visual feedback provided by OctoPocus [START_REF] Bau | Octopocus: A dynamic guide for learning gesture-based command sets[END_REF] -if OctoPocus is altered to show one gesture suggestion at a time, its gesture suggestions are made scale and aspect invariant, and gesture abbreviation is allowed.

Experiment

Our experiment tests whether there are significant user performance and preference differences between predictive gestures and non-predictive gestures. It also establis hes whether there are significant differences between the two styles of predictive visual autocompletion (SF-Path and SimpleFlow).

Hypothesis

For each of these hypotheses we are interested in understanding whether they do or do not hold true between the three styles of visual feedback and autocompletion, i. For H1 Shorten we expect that the continuous predictive visual feedback provided during the course of gesture entry will enable users to significantly shorten the gestures they enter. If true, users will enter abbreviated short partial gestures rather than whole gestures. Of particular interest is how much do they shorten the gestures by, a small amount or large amount?

Accuracy is an important feature of gesture input. Providing effective real-time gesture feedback should improve users' ability to enter gestures correctly. Whether SimpleFlow and SF-Path are effective forms of visual feedback is established with the H2 hypothesis.

The speed with which gestures are entered is important. If gesture entry takes too long it could be distracting and interfere with workflows. With the H3 hypothesis we expect a speed / accuracy tradeoff, where speed is how long it takes to enter a gesture. Predictive gesture entry may be slower because users spend time evaluating and adapting their gesture input based on the real-time feedback.

Hypothesis 4 establishes whether predictive visual autocompletion requires more cognitive resources than no predictive visual feedback. Ideally, well-designed forms of visual feedback should not increase cognitive load during gesture entry.

Mike Bennett1,2, Kevin McCarthy2, S ile O'Modhrain3, and Barry S myth2

Design

The experiment task was to input a series of gestures correctly. A gesture was deemed correctly entered when the gesture inputted by the participant matched the target gesture they had been instructed to enter. Whether a gesture matched was evaluated by the gesture recognition algorithm.

A within-subjects experiment design was used. The experiment was 3x1, where the independent variable was the style of visual gesture autocompletion and the depen dent variable was gesture input accuracy. The three levels of the independent variable were Standard Feedback, SF-Path and SimpleFlow. While the two levels of the dependent variable were Correct or Incorrect, corresponding to whether the entered gesture correctly or incorrectly matched the target gesture.

All participants were presented with three randomly ordered blocks of randomly ordered gestures. Each block was presented once and corresponded to a level of the independent variable, i.e. Standard Feedback, SF-Path, or SimpleFlow. Within each block each gesture was repeated three times, distributed randomly wit hin a block. At the start of each block participants practiced inputting three gestures. The visual autocompletion drawn during practice gestures was based on the level of the independent variable.

Participants completed the experiment in a single session. Each participant co mpleted a short post-hoc questionnaire, where they provided their rank preferences for the visual autocompletion styles.

Participants

Eighteen volunteer participants took part, three of which did the pilot experiments and the remaining fifteen completed the main experiment. Of those fifteen 10 were male and 5 female. Participants' average age was 26.5 years, with a standard deviation of 3.8 years. All participants naturally used their right hand to control the mouse. The fifteen participants entered 2295 gestures, of which 135 were practice gestures.

Materials

Figure 4 shows the sixteen gestures used for the experiment. The ges tures are from the work on the $1 Recognizer [START_REF] Wobbrock | Gestures without libraries, toolkits or training: A $1 recognizer for user interface prototypes[END_REF]. We picked those gestures as they were independently created and have been used in other gesture experiments [START_REF] Wobbrock | Gestures without libraries, toolkits or training: A $1 recognizer for user interface prototypes[END_REF]. Using an independently created set of gestures helps avoid introducing an experiment bias, as the experiment did not run with a set of gestures specifically designed for the experiment task.

The gesture recognition algorithm was kept the same between the three levels. What differed between the levels was whether visual feedback was shown, and what kind of visual feedback it was. Details about the gesture recognition algorithm are provided in the Appendix.

Procedure

The experiment took place in a private room, free from external stimuli. A PC ru nning Windows XP was used to run the experiment, and the same mouse was used by all participants. The PC monitor was equipped with a Tobii T60 eyetracker, which S impleFlow: Enhancing Gestural Interaction With Gesture Prediction, Abbreviation And Autocompletion 9

was used to capture pupil dilation at 60Hz. The eyetracker data was used to calculate cognitive load (Section 5.4). Pixels on the monitor were square.

Participants were first calibrated on the Tobii eye tracker, with Tobii's standard calibration and eye tracking software. Then each participant was given time to practice inputting gestures using the mouse, during which no predictions were displayed. The participants then practiced the stimulus-response gesture input task three times, with gestures from Figure 4. At the start of each block participants also practiced inputting gestures, during which they saw the visual autocompletion sp ecific to the block.

The gesture input stimulus-response task consisted of presenting a target gesture to the participants, which was displayed centered on-screen for two seconds. During the two seconds the mouse pointer was not shown on -screen. Text above the target gesture instructed participants to look at the gesture. After the two seconds expired the screen was blanked, a text message appeared informing participants to draw the ge sture, and the mouse pointer appeared centered on-screen. The mouse pointer was centered on-screen to prevent a carryover effect occurring between tasks, which could arise if the mouse pointer location was carried between gesture tasks.

Participants then inputted a gesture by pressing and holding the left mouse button down and moving the mouse. When the participant released the mouse button, the gesture was considered complete.

Statistical Techniques

To analyse hypothesis H1 and H3 we use Balanced Repeated Measures Within -Subjects ANOVAs with Bonferroni pairwise comparisons. For H4 we first apply a standard log transform to the results, as tests of normality indicated the results had a non-normal distribution until transformed. Then we apply the same analysis tec hniques as for H1 and H3.

For hypothesis H2 we use Repeated Measures Logistic Regression (RMLR) with Bonferroni pairwise comparisons. RMLR is used because gesture accuracy is a cat egorical binary variable, i.e. gestures are either entered Correctly or Incorrectly. Fig. 4. Unistroke gestures used in the experiment, from [START_REF] Wobbrock | Gestures without libraries, toolkits or training: A $1 recognizer for user interface prototypes[END_REF]. Black points are the starting point for drawing the gestures.

Mike Bennett1,2, Kevin McCarthy2, S ile O'Modhrain3, and Barry S myth2

A Bonferroni test compares each level with every other level, and establishes whether performance between the levels is significantly different. Balancing is performed by selecting a subset of results randomly distributed in the results. For exa mple, if there are 600 results for Standard Feedback (Standard), 550 for SF-Path and 500 for SimpleFlow, then 500 results are randomly selected from each of the levels and pairwise comparisons are performed on those results.

Rather than reporting every p value, we report the meaningful p values, i.e. the highest significant p values out of each three way pairwis e comparison. There are nine p values generated per set of results (Standard vs SF-Path, Standard vs Sim-pleFlow, SF-Path vs SimpleFlow) * (All, Correct, Incorrect gestures).

Accepting Gesture Predictions

For the blocks where participants saw predictive visual autocompletion (SF-Path and SimpleFlow), they were informed they could accept a gesture prediction without ha ving to enter the full gesture. Accepting a gesture prediction was achieved by stopping gesture entry. Participants stopped gesture entry by releasing the mouse button. For example, in Figure 1 a participant could stop entering a gesture at time point (a) if they wanted to accept the prediction and input a triangle gesture.

Results

Overall, users of SimpleFlow automatically shorten gestures by 41%, while simultaneously improving the accuracy (+13%, from 68% to 81%) and speed (+10%) of gesture input -and this is achieved with no significant increase in cognitive load. Results from the post-hoc questionnaire indicate that participants prefer SimpleFlow over SF-Path and Standard Feedback.

A total of 135 training gestures and 21 error gestures were removed from the results, leaving 2139 gestures available for analysis, with 714 gestures in Standard Feedback, 713 in SF-Path and 712 in SimpleFlow conditions. Gestures were classified as errors where participants accidentally clicked the mouse button and did not move the mouse.

Hypothesis: H1 Shorten

Users can and do take advantage of the gesture predictions. Participants enter 39%+ shorter gestures with SimpleFlow and SF-Path, hypothesis H1 is true. SimpleFlow and SF-Path successfully enable users to enter abbreviated short partial gestures rather than whole gestures. This implies that the predictive gesture alg orithm makes accurate predictions early and continuously during the course of gesture entry.

Abbreviated Gestures

To establish whether participants abbreviate gestures, we analyse the pixel path length of entered gestures. Gesture path length is a measure of how many pixels are travelled when entering a gesture. Gesture path length is significantly (p < 0.037) shorter for All, Correct and Incorrect SF-Path (39%, All) and SimpleFlow (41%, All) gestures, compared to No Feedback (Figure 5). No significant difference exists between SF-Path and SimpleFlow. is true, based on the gesture pixel path length.

A shorter gesture pixel path length does not definitively establish that participants abbreviate gestures. Shorter path lengths could mean participants enter smaller ge stures. To rule out this possibility, we measure and analyse the entered path length d ivided by the predicted path length. We refer to this measure as the Partial Gesture Ratio (PGR). The lower the PGR the better. A PGR < 1 means the entered gesture is shorter than the predicted gesture, PGR > 1 means the entered gesture is longer than the predicted gesture, and PGR = 1 means the gestures are the same length.

Based on the PGR we find that All, Correct and Incorrect gestures entered with SF-Path and SimpleFlow are significantly (p < 0.002) shorter than Standard Feedback (Table 1). This further confirms that participants do abbreviate gestures.

Early And Continuous Predictions

Do participants enter partial gestures because the prediction algorithm generates acc urate predictions early and continuously during gesture entry? We analyse whether the gesture predictions stabilise early and stop changing. Early stabilisation is measured by dividing the time when gesture predictions stop changing by the time taken to input the gesture.

Predictions do stabilise early and stop changing during gesture entry (Figure 6). For All gestures Standard Feedback is stable for 39% of gesture input time, while SF-Path and SimpleFlow are stable for 47% of input time. All and Incorrect gestures are stable significantly (p < 0.001) longer for SF-Path and SimpleFlow. There are no significant differences for Correct gestures.

Number Of Predictions

How often the predictions change before stabilisation could effect whether users enter partial gestures. Ideally a prediction does not change too often, as too many changes

Hypothesis: H2 Accuracy

By more than 10% both SimpleFlow and SF-Path significantly (p < 0.003) improve gesture entry accuracy, compared to Standard Feedback. Hypothesis H2 is true, predictive gestures with visual autocompletion does improve the accuracy with which users enter gestures.

As shown in Figure 7, for Standard Feedback participants correctly entered gestures 68.1% (SD 15.8%) of the time. The gesture accuracy increased to 78.6% (SD 10.9%) and 81% (SD 12.2%) for SF-Path and SimpleFlow respectively. Performance is not significantly (p < 0.05) different between SF-Path and SimpleFlow. In Figure 7 the dot is the mean accuracy, the error bars are the 95% Wald confidence interval and the dashed lines the standard deviation.

Hypothesis: H3 Speed

Unexpectedly, SimpleFlow is more than quarter of a second faster (+10%, p < 0.012) for entering gestures than Standard Feedback and SF-Path. Entering gestures with SF-Path is as fast as with Standard Feedback (Figure 8). Hypothesis 3 is rejected, as vis ual autocompletion does not slow gesture input.

Rate Of Gesture Input

Is SimpleFlow the fastest input style because participants' rate of input is higher? When entering gestures how fast do users move the on -screen mouse pointer? Rate of input is calculated as the average number of pixels travelled by the mouse pointer per millisecond during gesture input.

Standard Feedback leads to a significantly (p < 0.0001) higher rate of gesture input than both SF-Path and SimpleFlow; for All, Correct and Incorrect gestures (Figure 9). There are no significant differences between SF-Path and SimpleFlow.

This confirms that the speed improvement for SimpleFlow is not due to an improved rate of input. The speed improvement occurs because participants ent er abbreviated gestures, which means it takes less time to enter the partial gestures.

Hypothesis: H4 Cognitive Load

Does the visual autocompletion place higher cognitive demands on users, than no visual feedback? Or does the visual feedback even decrease the cognitive load? Surprisingly H4 is rejected, SF-Path and SimpleFlow do not lead to increased cognitive load. There are no significant differences (p < 0.05) in cognitive load between Standard Feedback, SF-Path and SimpleFlow (Figure 10). This result applies to All, Correct and Incorrect gestures. We measured cognitive load by measuring Task-Evoked Pupillary Response (TEPR), which is known to be a reliable indicator of cognitive load [START_REF] Beatty | The Pupillary System, chapter 6[END_REF][START_REF] Iqbal | Task-evoked pupillary response to mental workload in Human-Computer Interaction[END_REF][START_REF] Klingner | M easuring the task-evoked pupillary response with a remote eye tracker[END_REF]. Task-Evoked Pupillary Response is a measure of how much the pupil dilates over time as a function of task hardness. The larger the change in pupil dilation the larger the cogn itive load.

As recommended in the literature we removed blink artifacts, applied a low-pass filter to the dilation measures, used a 500ms baseline and used TEPR Peak Dilation to calculate cognitive load [START_REF] Beatty | The Pupillary System, chapter 6[END_REF][START_REF] Klingner | M easuring the task-evoked pupillary response with a remote eye tracker[END_REF]. The baseline was captured each time participants entered a gesture, right before they began entering the gesture. Measuring TEPR with a non-contact eyetracker enables us to measure real-time non-subjective quantifications of cognitive load, as an alternative to subjective work load assessments such as NASA TLX [START_REF] Hart | Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research[END_REF].

User Preferences

Participants preferred SimpleFlow 2.2 times more often than SF-Path, and 2.75 times more often than Standard Feedback (Figure 11). In the post-hoc questionnaire participants ranked Standard Feedback, SF-Path and SimpleFlow on a scale from 1 to 3, where 1 is most preferred and 3 least preferred. They could assign equal ranks.

Figure 11 shows the number times each rank was assigned to Standard Feedback, SF-Path and SimpleFlow. Participants ranked SimpleFlow 1st eleven times, SF-Path 1st five times, and Standard Feedback 1st four times. SF-Path is most often ranked 2nd, and Standard Feedback is most often ranked 3rd.

Discussion & Limitations

Our design aim for SimpleFlow, was to keep the complexity of the visual prediction feedback as simple as possible, while improving user performance. A related goal was to enhance gestural interaction without significantly changing it; in an effort to avoid requiring users to spend considerable amounts of time skilling up on a new interaction technique. Two further key aims were to enable users to take advantage of the gesture predictions without forcing them to use the predictions, and enable users to enter shorter gestures, rather than having to input full gestures. Looking at the results we find that SimpleFlow performs best out of all three forms of visual while also achieving our goals and aims. During the experiment users had few practice oppo rtunities with the predictive gestures, yet they performed significantly better with predictive gestures versus non-predictive, while also preferring the predictions over no predictions.

Of particular interest we found that SimpleFlow is faster and more preferred to SF-Path, which is the same style of visual feedback as OctoPocus (when one gesture su ggestion is shown at a time and the suggestions are scale and aspect invariant). This is interesting because the visual difference between SimpleFlow's visual feedback and SF-Path's visual feedback is very small, even though it has a critical impact. Sim-pleFlow shows a whole gesture suggestion in conjunction with the partial gesture a user has entered, while SF-Path completes the gesture for a user (Figure 3). When we followed up with users and informally asked them why they preferred SimpleFlow over SF-Path, the general consensus was that showing whole gestures clearly tells the users exactly which predicted gesture is getting matched with their partial gesture. While when partial gestures are completed in SF-Path, it is not always clear to the user why a gesture is completed the way it is, especially when the gesture prediction is wrong.

An interesting limitation of this work, which strongly suggests worthwhile future directions, is that we treat the users' performance and the performance of the ge sture prediction algorithm as a combined system. This leaves open questions about the relationship between a user's performance, and what would happen if a better or worse gesture prediction algorithm is used? For example, if a perfect gesture recognition a lgorithm is used would we find that users enter even shorter gestures? If so, what is the absolute lowest bound on how much users will abbreviate gestures by? Does that interact with the number of available gestures? Related questions include what, if any, are the effects of other performance characteristics of the gesture prediction algorithm, e.g. stability of predictions. Ultimately, what are the desirable characteristics of gesture prediction algorithms? Another interesting question arises around the set of gestures we ran the experiment with, as the range of gesture shapes could impact on user performance. For example, in Figure 4 There are many other possibilities for future research, including extending Sim-pleFlow to handle multi-stroke gestures, creating UI techniques that enable users to interactively refuse and cancel a gesture suggestion, establishing what styles of preand post-gesture feedback is beneficial with visual predictions, and enabling users to quickly select from a set of gesture suggestions without having to complete the ge stures. Finally, understanding more about why and how users decide to shorten predictive gestures would be very useful. Understanding this would help us further enhance and influence gesture abbreviation, i.e. enable users to enter even shorter gestures more quickly.

Conclusions

We found that users of predictive gestures with SimpleFlow and SF-Path visual feedback will significantly shorten gestures (like predictive text entry systems), with no significant increase in cognitive load. Further, SimpleFlow successfully enhances u sers' gesture entry, both speeding it up and improving the accuracy; along with users preferring SimpleFlow the most. The visual feedback provided by SimpleFlow is visually simple and minimal, and could easily be added to existing gesture entry systems without requiring significant changes to them.

Fig. 1 .

 1 Fig. 1. Example of SimpleFlow. Shown is the visual feedback displayed when entering a triangle gesture. Star is starting point of gesture entry. Black line is the gesture entered so far by the user -(a) is time point 1, (b) time point 2, (c) time point 3. Red line is the real-time predicted gesture.

Fig. 2 .

 2 Fig. 2. Example of Scale Free Dynamic Paths shown when entering a triangle gesture. Star, black line & red line have the same meaning as in Figure 1.

Fig. 3 .

 3 Fig. 3. Example of erroneous gesture predictions that could be shown when entering a triangle gesture. (a) SimpleFlow, (b) SF-Path.

  e. Standard Feedback, SF-Path and SimpleFlow. For example, are SF-Path gestures faster than SimpleFlow gestures? We hypothesise that:  H1 Shorten: Predictive gestures enable users to reduce gestures to a shorter form, by entering abbreviated gestures rather than full gestures. Like predictive text entry systems.  H2 Accuracy: Predictive gestures with visual autocompletion improves the accuracy with which users enter gestures.  H3 Speed: Inputting predictive gestures with visual autocompletion is slo wer.  H4 Cognitive Load: Cognitive load is higher for predictive gestures. Visual autocompletion places higher cognitive demands on users during gesture entry.

Fig. 5 .Fig. 6 .

 56 Fig. 5. Pixel path length of entered gesture. Error bars are 95% Wald confidence interval, dashed lines standard deviation Fig. 6. How long was the accepted gesture stable on screen, as percentage of gesture input time.

S 13 Fig. 7 .

 137 Fig. 7. Predictive gestures with visual autocompletion improves performance when entering gestures.

Fig. 8 .

 8 Fig. 8. Length of time taken to input gestures varies based on visual autocompletion.

Mike Bennett1, 2 ,Fig. 9 .Fig. 10 .

 2910 Fig. 9. Rate of input. Fig. 10. TEPR Peak Dilation measures of Cognitive Load.

Fig. 11 .

 11 Fig. 11. User preferences as indicated in post-study questionnaire.

  gesture 2, 11 and 12 all share a common Mike Bennett1,2, Kevin McCarthy2, S ile O'Modhrain3, and Barry S myth2 initial stroke (downward diagonal left stroke) -what would happen if a set of gestures is used that is optimised to minimize the shared starting strokes?

Table 1 .

 1 Path length of entered gestures, as PGR.

		All	Correct	Incorrect
	Standard	1.014 (0.288)	0.994 (0.057)	1.125 (0.535)
	SF-Path	0.733 (0.395)	0.668 (0.314)	0.901 (0.542)
	SimpleFlow	0.697 (0.323)	0.685 (0.297)	0.791 (0.393)

Table 2 .

 2 Number of times gesture prediction changed. Path and SimpleFlow generate 5.84 and 5.48 prediction changes for All gestures (5.28 and 5.13 for Correct), which are significantly less gesture prediction changes, for All (p < 0.003) and Correct (p < 0.017) gestures. No significant differences occur for Incorrect gestures. Shown in Table 2 are the mean number of times the gesture predictions changed.

		All	Correct	Incorrect
	Standard	6.51 (3.73)	6.01 (3.69)	7.74 (3.50)
	SF-Path	5.84 (4.30)	5.28 (4.02)	7.82 (4.93)
	SimpleFlow	5.48 (3.88)	5.13 (3.71)	6.94 (3.87)
	could make it harder for a user to judge which prediction is emerging as the winning
	prediction.
	SF-

For a video demonstration of the visual autocompletion styles please view the video accompanying this paper.

Appendix: Gesture Prediction Algorithm

For experimental reproducibility this appendix outlines the gesture prediction algorithm (Figure 12), which is composed of a gesture concatenation algorithm and a gesture recognition algorithm. When a user enters part of a gesture, the gesture concatenation algorithm generates a

Mike Bennett1,2, Kevin McCarthy2, S ile O'Modhrain3, and Barry S myth2

complete gesture based on the partially entered gesture. Then the complete gesture is sent to a gesture recognition algorithm, to check whether it matches any of the trained gestures. The best matching gesture is used as the gesture prediction.

Gesture Concatenation Algorithm

The gesture concatenation algorithm is easy to implement, though the computational efficiency of it is open to improvement. The algorithm assumes that gestures start from the same points, like the $1 Recognizer [START_REF] Wobbrock | Gestures without libraries, toolkits or training: A $1 recognizer for user interface prototypes[END_REF]. Unlike OctoPocus [START_REF] Bau | Octopocus: A dynamic guide for learning gesture-based command sets[END_REF] and ShadowGuides [START_REF] Freeman | Shadowguides:Visualizations for in-situ learning of multi-touch and whole-hand gestures[END_REF] however, the algorithm is scale independent. Like $1 Recognizer and Oc-toPocus, and unlike ShadowGuides, our algorithm is for single stroke continuous gestures. The algorithm also handles different gesture aspect ratios, like Protractor [START_REF] Li | Protractor: A fast and accurate gesture recognizer[END_REF], but like OctoPocus and ShadowGuides it is not rotational invariant to gesture orient ations. Below are the seven steps in the algorithm (see Figure 12): 1. During gesture entry, continuously capture the path of the incoming gesture, generating a partial gesture P (Figure 12(a)). 2. M easure the width w, height h and calculate the length l of the partial gesture P (Figure 12(b)). 3. Scale the trained gestures (Figure 12(c)) so they share the same width w and height h as the partial gesture P. We refer to this set of scaled gestures as SC (Figure 12(d)). 4. For each scaled gesture in SC (Figure 12(d)), remove a path length l from the start of each gesture. This generates a set of cropped gestures CG (Figure 12(e)). 5. M erge the partial gesture P with every cropped gesture in CG. This generates a new set of gestures PG, as shown in Figure 12(f). Each gesture in PG is a gesture prediction. The merging should be done such that the end of P is merged with the start of each cropped gesture in CG. 6. If the gesture recognition algorithm is scale dependent, then the gestures in PG may need to be rescaled to match the scale of the gestures used to train the gesture recognition algorithm. 7. To find the winning gesture prediction send each gesture prediction in PG to the gesture recognition algorithm. The best scoring gesture in PG is the gesture prediction.

We imposed a constraint on the above algorithm, such that the predicted gesture cannot be scaled below a specific size. Without that constraint gestures could be scaled to only a few pixels wide or high, which would make the gestures visually indistinguishable.

Gesture Recognition Algorithm

The gesture recognition algorithm used is a scale and aspect invariant template matching algorithm. The code for which is based on wayV's gesture recognition algorithm [START_REF] Bennett | wayv: Gestures for Linux[END_REF]. wayV's algorithm accounts for bounding box issues that can arise where the input starts off purely vertical or horizontal.