Thammathip Piumsomboon
email: thammathip.piumsomboon@pg.canterbury.ac.nz

Adrian Clark
email: adrian.clark@canterbury.ac.nz

Mark Billinghurst
email: mark.billinghurst@canterbury.ac.nz

Andy Cockburn

User-Defined Gestures for Augmented Reality

Keywords: Augmented reality, gestures, guessability

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

By overlaying virtual content onto the real world, Augmented Reality (AR) allows users to perform tasks in the real and virtual environment at the same time [START_REF] Azuma | A Survey of Augmented Reality[END_REF]. Natural hand gestures provide an intuitive interaction method which bridges both worlds.

While prior research has demonstrated the use of hand gesture input in AR, there is no consensus on how this combination of technologies can best serve users. In studies involving multimodal AR interfaces, hand gestures were primarily implemented as an add-on to speech input [START_REF] Heidemann | Multimodal interaction in an augmented reality scenario[END_REF] [START_REF] Kolsch | Multimodal interaction with a wearable augmented reality system[END_REF]. In cases of unimodal gesture interfaces, only a limited number of gestures have been used and the gestures were designed by researchers for optimal recognition rather than for naturalness, meaning that they were often arbitrary and unintuitive [START_REF] Kaiser | Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality[END_REF] [5] [START_REF] Fernandes | Bare hand interaction in tabletop augmented reality[END_REF]. Recent research has integrated hand tracking with physics engines to provide realistic interaction with virtual content [7] [8], but this provides limited support for gesture recognition and does not take into account the wide range of expressive hand gestures that could potentially be used for input commands.

To develop truly natural gesture based interfaces for AR applications, there are a number of unanswered questions that must be addressed. For example, for a given task is there a suitable and easy to perform gesture? Is there a common set of gestures among users that would eliminate the need for arbitrary mapping of commands by designers? Is there a taxonomy that can be used to classify gestures in AR? Similar shortcomings were encountered in the fields of surface computing and motion gestures, where Wobbrock et al. [START_REF] Wobbrock | User-defined gestures for surface computing[END_REF] and Ruiz et al. [START_REF] Ruiz | User-defined motion gestures for mobile interaction[END_REF] addressed absences of design insight by conducting guessability studies [START_REF] Wobbrock | Maximizing the guessability of symbolic input[END_REF].

In this study, we focus explicitly on hand gestures for unimodal input in AR. We follow Wobbrock's approach and employ a guessability method, first showing a 3D animation of the task and then asking participants for their preferred gesture to perform the task. Users were also asked to subjectively rate their chosen gestures, based on the perceived "goodness" and ease to perform. By analyzing the results of this study, we were able to create a comprehensive set of user-defined gestures for a range of tasks performed in AR.

This work makes a number of contributions; [START_REF] Azuma | A Survey of Augmented Reality[END_REF] The first set of user-defined gestures captured from an AR interface, (2) Classification of these gestures based on a gesture taxonomy for AR which was extended from Wobbrock's surface gesture taxonomy [START_REF] Wobbrock | User-defined gestures for surface computing[END_REF], (3) Agreement scores of gestures for selected tasks and subjective rating of the gestures, (4) Qualitative findings from the design process, and (5) Discussion of the implications of this work for AR, gesture interfaces, and gesture recognition.

Related Work

The topic of hand gesture classification as based on human discourse was excellently covered by the work of Wobbrock et al. [START_REF] Wobbrock | User-defined gestures for surface computing[END_REF]. As our work extends this approach to gesture interaction in AR, we focus on related work in bare hand and glove-based unimodal hand gestures interfaces, multimodal interfaces coupled with speech, and recent advancements in AR relevant to interaction. In addition, we briefly discuss previous research that utilized elicitation techniques.

Hand Gesture Interfaces in AR

Lee et al. [START_REF] Lee | Hand gesture-based tangible interactions for manipulating virtual objects in a mixed reality environment[END_REF] designed gloves with conductive fabric on the fingertips and the palm for gesture recognition and vibration motors for haptic feedback. The gloves were tracked using markers placed around the wrist area. A small set of gestures were used to allow selection, gripping, cutting and copying actions. Lee and Hollerer [START_REF] Lee | Handy AR: Markerless inspection of augmented reality objects using fingertip tracking[END_REF] created Handy AR, a system capable of bare hand interaction using a standard web camera. The supported gestures were limited to an opened/closed hand for object selection and hand rotation for object inspection. Their follow up work allowed objects to be relocated using markerless tracking [START_REF] Lee | Multithreaded Hybrid Feature Tracking for Markerless Augmented Reality[END_REF].

Fernandes and Fernandez [START_REF] Fernandes | Bare hand interaction in tabletop augmented reality[END_REF] trained statistical models with hand images to allow bare hand detection. Virtual objects could be translated using the hand in a palm upwards orientation, while rotation and scaling along the marker plane was achieved using two handed pointing.

The main shortcoming of all these interfaces was that they only recognize a small number of gestures, and this gesture set was designed by the researchers for easy recognition. No support was provided for users to define their own gestures which were more comfortable or had contextual meaning.

Hand Gesture and Speech Interfaces in AR

SenseShapes [START_REF] Olwal | SenseShapes: using statistical geometry for object selection in a multimodal augmented reality[END_REF] aimed to find spatial correlation between gestures and deictic terms such as "that", "here", and "there" in an object selection task. The user's hands were tracked using data gloves, and object selection was facilitated by a virtual cone projected out from the users' fingers. The region of interest was estimated based on speech, gaze projection and the pointing projection. Heidemann et al. [START_REF] Heidemann | Multimodal interaction in an augmented reality scenario[END_REF] demonstrated an AR interface which identified objects on a tabletop. Skin color segmentation was used to identify the user's index finger, allowing users to select virtual objects by pointing and make menu selections. Speech could also be used to issue information queries and interact with the 2D menu.

Kolsch et al. [START_REF] Kolsch | Multimodal interaction with a wearable augmented reality system[END_REF] created a mobile AR system that supported interaction by hand gesture, speech, trackball and head pose. Gesture recognition was implemented using HandVu, a computer vision-based gesture recognition library. They categorized tasks by dimensionality. For example taking a snapshot was defined as 0D, adjusting the focus region depth was 1D, using a pencil tool was 2D, and orienting virtual objects was 3D. Some actions such as relocating/resizing/orienting could be performed multimodally by speech, gesture or trackball, while other actions such as take/save/discard snapshot could only be performed by speech.

The work most closely related to ours was that of, Lee [START_REF] Lee | Multimodal Speech-Gesture Interaction with 3D Objects in Augmented Reality Environments[END_REF], who conducted a Wizard of Oz study of an AR multimodal interface to measure the types of gestures people would like to use in a virtual object manipulation task. In this study pointing, translation and rotation gestures were captured. She later developed a multimodal gesture and speech interface for a design related task, however speech was used as a primary input as in typical multimodal systems therefore gestures were only mapped to limited number of spatial related tasks for example pointing, grabbing and moving.

Recent Advancements in AR Technology

Advancements of markerless tracking algorithms and consumer hardware have enabled greater possibilities for gesture based AR interfaces. Modern vision-based tracking algorithms can robustly register the environment without markers, allowing for higher mobility [START_REF] Newcombe | KinectFusion: Real-time dense surface mapping and tracking[END_REF]. Furthermore, an introduction of consumer depth sensors such as the Microsoft Kinect has made real-time 3D processing accessible and have introduced a new interaction paradigm in AR through real-time physically-based natural interaction [START_REF] Hilliges | HoloDesk: direct 3d interactions with a situated see-through display[END_REF][START_REF] Benko | MirageTable: freehand interaction on a projected augmented reality tabletop[END_REF].

Previous Elicitation Studies

Wobbrock et al. describe prior studies involving elicitation of input from users [START_REF] Wobbrock | User-defined gestures for surface computing[END_REF]. The technique is common in participatory design [START_REF] Schuler | Participatory Design: Principles and Practices[END_REF] and has been applied in a variety of research areas such as unistroke gestures [START_REF] Wobbrock | Maximizing the guessability of symbolic input[END_REF], surface computing [START_REF] Wobbrock | User-defined gestures for surface computing[END_REF] and motion gesture for mobile interaction [START_REF] Ruiz | User-defined motion gestures for mobile interaction[END_REF]. In AR, a Wizard of Oz study [START_REF] Lee | Multimodal Speech-Gesture Interaction with 3D Objects in Augmented Reality Environments[END_REF] for gestures and speech was conducted and aimed to capture the type of speech and gesture input that users would like to use in an object manipulation task. It was found that the majority of gestures used hand pointing due to reliance on speech for command inputs. In this research, our focus is to explore the potential of hand gestures as the unimodal input.

Developing a User-defined Gesture Set

To elicit user-defined gestures, we first presented the effect of the task being carried out by showing a 3D animation in AR, and then asked the participants to describe the gestures they would use. Participants designed and performed gestures for forty tasks across six categories, which included gestures for three types of menu. Participants were asked to follow a think-aloud protocol while designing the gestures, and also to rate the gestures for goodness and ease to perform. They were asked to ignore the issue of recognition difficulty to allow freedom during the design process, and to allow us to observe their unrestricted behavior. At the end of the experiment, brief interviews were conducted and preferences of the three types of proposed gesture menus were collected.

Task Selections

In order for the gesture set to be applicable across a broad range of AR applications [START_REF] Van Krevelen | A Survey of Augmented Reality Technologies, Applications and Limitations[END_REF], we surveyed common operations in previous research e.g. [START_REF] Kolsch | Multimodal interaction with a wearable augmented reality system[END_REF][START_REF] Lee | Handy AR: Markerless inspection of augmented reality objects using fingertip tracking[END_REF][START_REF] Fernandes | Bare hand interaction in tabletop augmented reality[END_REF][START_REF] Lee | Hand gesture-based tangible interactions for manipulating virtual objects in a mixed reality environment[END_REF][START_REF] Broll | ARTHUR: A Collaborative Augmented Environment for Architectural Design and Urban Planning[END_REF], which resulted in forty tasks that included three types of gesture menu, which are horizontal [START_REF] Lee | Occlusion based interaction methods for tangible augmented reality environments[END_REF], vertical [START_REF] Broll | ARTHUR: A Collaborative Augmented Environment for Architectural Design and Urban Planning[END_REF], and object-centric that we proposed. These tasks were then grouped into six categories based on context, such that identical gestures could be used across these categories, as shown in Table 1.

Participants

Twenty participants were recruited for the study, comprising of twelve males and eight females, ranging in age from 18 to 38 with mean of 26 (σ = 5.23). The participants which were selected had minimal knowledge of AR to avoid the influence of previous experience with gestures interaction. Nineteen of the participants were right handed, and one was left handed. All participants used PCs regularly, with an average daily usage of 7.25 hours (σ = 4.0). Fifteen owned touchscreen devices, with an average daily usage of 3.6 hours (σ = 4.17). Eleven had experience with gesture-in-the-air interfaces such as those used by the Nintendo Wii or Microsoft Kinect gaming devices.

3.3

Apparatus

The experimental interaction space, shown in Figure 1 (Left), was the area on and above a 120 x 80cm table. Each participant was seated in front of the table, and a Sony HMZ-T1 head mounted display (HMD) at 1280 x 720 resolutions was used as the display device. A high definition (HD) Logitech c525 web camera was mounted on the front of the HMZ-T1 as a viewing camera, providing a video stream at the display resolution. This HMD and camera combination offered a wide field of view, with a 16:9 aspect ratio, providing a good view of the interaction space and complete sight of both hands while gesturing. An Asus Xtion Pro Live depth sensor was placed 100 cm above the tabletop facing down onto the surface to provide reconstruction and occlusion between the user's hands and virtual content. An RGB camera was placed in front of and facing the user to record the users' gestures. A PC was used for the AR simulation and to record the video and audio stream from the user's perspective. A planar image marker was placed in the center of the table, and the OPIRA natural feature registration library [START_REF] Clark | Optical-Flow Perspective Invariant Registration[END_REF] was used for registration and tracking of this marker. The 3D graphics, animation and occlusion were handled as described by Piumsomboon et al. [START_REF] Piumsomboon | Poster: Physically-based natural hand and tangible AR interaction for face-to-face collaboration on a tabletop[END_REF].

Procedure

After an introduction to AR and description of how to operate the interface, the researcher described the experiment in detail and showed the list of tasks to the participant. The forty tasks were divided into six categories, as shown in Table 1, and the participant was told they could choose to carry out the categories in any order, providing that there was no conflict between gestures within the same category. For each task, a 3D animation showing the effect of the task was displayed, for example, in the "Move -long distance" task, participants would see a virtual toy block move across the table. Within the same task category, the participant could view each task as many times as she/he needed. Once the participant understood the function of the task, she/he was asked to design the gesture they felt best suited the task in a think-aloud manner. Participants were free to perform one or two-handed gestures as they saw fit for the task (See Figure 1, Right).

Once the participant had designed a consistent set of gestures for all tasks within the same category, they were asked to perform each gesture three times. After performing each gesture, they were asked to rate the gesture on a 7-point Likert scale in term of goodness and ease of use. At the end of the experiment, a final interview was conducted, where participants were asked to rank the three types of menu presented (horizontal, vertical, and object-centric as shown in Figure 5) in terms of preference and the justification for their ranking. Each session took approximately one to one and a half hours to complete.

Result

A total of 800 gestures were generated from the 20 participants performing the 40 tasks. The data collected for each user included video and audio recorded from the camera facing towards the user and the user's viewpoint camera, the user's subjective rating for each gesture, and transcripts taken from the think-aloud protocol and interview.

4.1

Taxonomy of Gestures in AR

We adapted Wobbrock's surface taxonomy [START_REF] Wobbrock | User-defined gestures for surface computing[END_REF] to better cover the AR gesture design space by taking their four-dimensional taxonomy, (form, nature, binding, and flow) and extending it with two more dimensions; symmetry and locale. Each dimension is comprised of multiple categories, as shown in Table 2.

The scope of the form dimension was kept unimanual, and in the case of a twohanded gesture, applied separately to each hand. In Wobbrock's original taxonomy, form contained six categories including one-point touch and one-point path, however, these two categories were discarded as they were not relevant to AR gestures that occur in three dimensional space.

The nature dimension was divided into symbolic, physical, metaphorical and abstract categories. Examples of symbolic gestures are thumbs-up and thumbs-down for accept and reject. Physical gestures were classified as those that would act physically on the virtual object as if it was a real object for instance grabbing a virtual block and relocating it for a move task. Metaphorical gestures express actions through existing metaphor e.g. pointing an index finger forward and spinning it clockwise to indicate play or increase speed as if one was playing a roll film. Any arbitrary gestures were considered abstract, such as a double-tap on the surface to deselect all objects.

The binding dimension considered relative location where gestures were performed. The object-centric category covered transform tasks such as move, rotate, and scale, as these are defined with respect to the objects being manipulated. Opening and closing horizontal or vertical menus were classified in the world-dependent category as they are located relative to the physical workspace. Gestures in the Worldindependent category could be performed anywhere, regardless of the relative position to the world, such as an open hand facing away from one's body to indicate stop during a simulation. Gestures performed across multiple spaces, such as insert where selection is object-centric and placement is world-dependent, fell into the mixed dependencies category.

In the flow dimension, gestures were categorized as discrete when the action is taken only when the gesture is completed, for example an index finger must be spun clockwise in a full circle to perform the play command. The gestures were considered continuous if the simulation must respond during the operation, such as manipulating an object using the transform gestures.

The first additional dimension we developed, symmetry, allowed classification of gestures depending on whether they were one handed (unimanual) or two handed (bimanual). The unimanual category was further split into dominant and nondominant, as some participants preferred to use their nondominant hand to perform gestures that required little or no movement, leaving their dominant hand for gestures requiring finer motor control. An example of this would be to use the dominant hand to execute a selection, and then use the non-dominant hand to perform a scissor pose for a cut operation. The bimanual category also subdivided, symmetric gestures representing two-handed gestures where both hands executed the same form, such as scaling, where both hands perform a pinch moving toward or away from each other. Two handed gestures, where the forms of the hands are different, fall into the asymmetric bimanual category. An example of this is the copy (1) gesture where one hand is used to select the target object (static pose) while the other hand drags the copy into place (static pose and path).

The other dimension we introduce is locale. If a gesture required physical contact with the real surface, they are considered on-the-surface as opposed to in-the-air. Gestures that require both are considered mixed locales. For example, an index finger tapped on-the-surface at a virtual button projected on the tabletop to perform horizontal menu selection task, as opposed to an index finger pushed in-the-air at a floating button to execute vertical menu selection. An example of a mixed locales gesture is, the box selection [START_REF] Azuma | A Survey of Augmented Reality[END_REF], where one hand indicated the area of the bottom surface of the box by dragging an index finger diagonally along the table's surface, while another hand lifted off the surface into the air to indicate the height of the box (See Figure 5).

Findings from Classification

Classification was performed on the 800 gestures as shown in Figure . 2. Within the six dimensional taxonomy, the most common characteristics of gestures were static pose and path, physical, object-centric, discrete, dominant unimanual, and in-the-air. Within the form dimension, there was a slightly higher number of static poses (3%) performed with a non-dominant hand and lower for static poses with path gestures (2.5%) over a dominant hand. This slight discrepancy was contributed by some participants preferring to use their dominant hand for gestures with movement while using a non-dominant for a static pose.

In the nature dimension, overall the gestures were dominantly physical (39%) and metaphorical (34.5%). The gestures chosen to perform transform, selection, and menu tasks were predominantly physical, with the percentage of 76.1%, 50%, and 57.8% respectively. The browsing and editing task gestures were mainly metaphorical (100% and 40.9% respectively), while the simulation task, gestures were split across symbolic (37%), metaphorical (34%), and abstract (29%) categories. For the binding dimension, the majority of gestures for the transform and selection tasks were objectcentric (100% and 75% respectively). Simulation (93%) and browsing (100%) task gestures were mainly world-independent (93% and 100%), while editing tasks gestures were world-independent (39.5%) and object-centric (32.3%). Menu tasks gestures were object-centric (50%) and world-dependent (45.6%).

For the remaining dimensions including flow, symmetry, and locale, the gestures chosen across all tasks were primarily discrete (77.5%), dominant unimanual (67.8%) and in-the-air (78%).

A User-defined Gesture Set

As defined in prior work by Wobbrock et al. [START_REF] Wobbrock | User-defined gestures for surface computing[END_REF] and Ruiz et al. [START_REF] Ruiz | User-defined motion gestures for mobile interaction[END_REF], the user defined gesture set, known as the "consensus set", is constructed based on the largest groups of identical gestures that are performed for the given task. In our study, each gesture valued at one point; therefore there were 20 points within each task and a total of 800 points for all tasks. We found that participants used minor variations of similar hand poses, for example a swiping gesture with the index finger or the same swipe with the index and middle fingers, and therefore chose to loosen the constraints from "gestures must be identical within each group" to "gestures must be similar within each group". We defined "similar gestures" as static pose and path gestures that were identical or having consistent directionality although the gesture had been performed with different static hand poses.

We had classified the major variants of observed hand poses into 11 poses with the codes, H01 to H11, as illustrated in Figure 4. For tasks where these variants existed, the variant poses could be used interchangeably, as indicated by the description under each user-defined gesture's illustration (Figure 5).

Exercising the "similar gesture" constraint, we were able to reduce the original 800 gestures into 320 unique gestures. The top 44 highly scored gestures were selected to make the consensus set, while the remaining 276 lowest scored gestures were discarded, defined by Wobbrock et al. [START_REF] Wobbrock | User-defined gestures for surface computing[END_REF] as the discarded set. The selected gestures of the consensus set represented 495 (61.89%) of the 800 recorded gestures (495 of 800 points). The consensus set of gestures comprised the overall task gestures in the following percentage transform (19.38%), menu (17.75%), editing (11.75%), browsing (5.00%), selection (4.63%), and simulation (3.38%), which sum up to 61.89%.

Level of Agreement.

To compute the degree of consensus among the designed gestures, an agreement score A was calculated using Equation 1[11]:

(1) where P t is the total number of gestures within the task, t, P s is a subset of P t containing similar gestures, and the range of A is [0, 1].

Consider the rotate-pitch (y-axis) task that contained five gestures with scores of 8, 6, 4, 1, and 1 points. The calculation for A pitch is as follows:

(2)

The agreement scores for all forty tasks are shown in Figure 3. While there is low agreement in the gestures set for tasks such as all select, undo, redo and play, there were notable groups of gestures that stood out with higher scores. User-defined Gesture Set and Its Characteristics. As mentioned in Section 3.4, we allowed users to assign the same gesture to different tasks as long as the tasks were not in the same category. In addition to this, there were some tasks where there were two or more gestures commonly assigned by the participants. This non one-to-one mapping resulted in a consensus set of 44 gestures for a total of 40 tasks, which resulted in improved guessability [START_REF] Wobbrock | Maximizing the guessability of symbolic input[END_REF].

When mapping multiple gestures to a single task, there was one task which had three gestures assigned to it (uniform-scaling), seven tasks had two gestures (x, y, z scaling, box select, stop, delete, and copy), and 23 tasks only had gesture. On the contrary, for the nine remaining tasks, two gestures were assigned to four tasks (short, long move, insert, and paste), one gesture assigned to three tasks (play, increase speed, and redo), and one gesture assigned to two tasks (decrease speed and undo).

When creating the consensus set, we only found one conflict between gestures within the same category. This was between the pause and stop tasks, where the gesture of an open-hand facing away from the body was proposed for both with scores of 4 and 7 points respectively. To resolve this, we simply assigned the gesture to the task with the higher score, in this case stop.

Play and increase speed as well as insert and paste were the exceptions where a single gesture was assigned to two tasks within the same category with no conflict. For play and increase speed, the participants intention was to use the number of spin cycles of the index finger to indicate the speed of the simulation i.e. a single clockwise spin to indicate play, two clockwise spin to indicate twice the speed and three spins for quadruple speed. For insert and paste, the participants felt the two tasks served a similar purpose; insert allowed a user to select the object from menu and placed it in the scene, whereas paste allowed a user to place an object from the clipboard into the scene. In the follow up interviews, participants suggested a simple resolution to this would be to provide unique selection spaces for the insert menu and paste clipboard.

With the minor ambiguities resolved, we were able to construct a consistent set of user-defined gestures which contained 44 gestures, where 34 gestures were unimanual and 10 were bimanual. The complete gesture set is illustrated in Figure 5.

The Subjective Rating on Goodness and Ease. After the participants had finished designing gestures for a task category, they were asked to subjectively rate their gestures for goodness and ease to perform on a 7-point Likert scale. By comparing these subjective ratings between the consensus set (user-defined set) and the discarded set, we found that the average score for gestures that users believed were a good match for the tasks was 6.02 (σ = 1.00) for the consensus set and 5.50 (σ = 1.22) for the discarded set, and the average score for the ease of performance was 6.17 (σ = 1.03) for the consensus set and 5.83 (σ =1.21) for the discarded set. The consensus set was rated significantly higher than the discarded set for both goodness (F 1, 798 = 43.896, p<.0001) and ease of performance (F 1, 798 =18.132, p<.0001). Hence, we could conclude that, on average, gestures in the user-defined set were better than those in the discarded set in terms of goodness and ease of performance.

4.4

Findings from the Design Process

Participants were asked to think-aloud when designing their gestures, and a follow-up interview was conducted after the experiment was complete. Analysis of the resulting empirical data showed recurring thought processes. We present seven motifs which describe the mutual design patterns encountered in designing gestures for AR, which we describe as reversible and reusable, size does matter, influence from existing UI, the obvious and the obscure, feedback backfired, menu for AR, axes and boxes, and variation of hand poses.

Reversible and Reusable. The consensus set included reversible and reusable gestures. We defined reversible gestures as those when performed in an opposite direction yielded opposite effects e.g. rotation, scaling, increase/decrease speed etc. We defined reusable gestures as those which were used commonly for tasks which were different, but participants felt had common attributes e.g. increase speed/ redo, decrease speed/undo and insert/paste. In the experiment there were several dichotomous tasks that are separate tasks which perform the exact opposite operation. Participants used reversible gestures for both tasks where the opposite effect was presented in the single animation, such as rotation and scaling, as well as tasks where the opposite effects were shown separately, such as increase/decrease speed, previous/next, undo/redo, group/ungroup, and open/close menus. All two-handed dichotomous tasks were symmetric bimanual with the gestures performed on both hands being the same form.

Size Does Matter. We found that the virtual object's size influenced the design decision of some participants, especially with regards to the number of hands that they would use to manipulate the object for example the majority of gestures performed for scale tasks were bimanual. This was due to scaling involving shrinking and enlarging the target object within and beyond the palm size. Some comments are as follows:

"

Instinctively, I would use two hands to adapt to the size of the model but it's cool if I can use just the two fingers (one-handed) for something as large." -P04

"Depending on the size of the piece, I can use two hands when it's big but in the case of small piece, it's enough to use the two fingers (thumb and index)." -P12 Fig. 4. Variants of hand poses observed among gestures where the codes, H01-H11, were assigned for ease of reference.

Influence from Existing UI. When participants found it difficult to come up with a gesture for a particular task, they would often resort to using metaphors from familiar UI. For example when designing a gesture for the delete task, several participants imagined having a recycle bin that they could move the target object to. For other arbitrary tasks, users would often resort to double-tapping. Some examples of how participants explained these actions were: "I would select and double-click… I'm thinking too much like Microsoft. It's just the thing that I'm used to." -P10

"The way I do it on my phone is that I would scale like this and then tap it once." -P14

The Obvious and the Obscure. Gesturing in 3D space allows for higher expressiveness, which in turn led to use of commonly understood gestures in the real-world. For example, there was a high level of agreement on the symbolic gestures thumbs up/down for accept/reject with scores of 9 and 10 respectively (out of 20). This was also the case for metaphoric gestures such as a scissor gesture for the cut task with the score of 7. User's liked the idea of using these gestures from the real world, resulting in higher than average goodness and ease scores, with averages of 6.87/6.75 (σ =.35/.71) for thumbs up, 6.5/6.5(σ =.71/.85) for thumbs down and 6.5/6.67(σ =.84/.82) for scissor pose.

The majority of participants found it challenging to come up with metaphors to design gestures for 3D tasks that they referred to as "abstract", such as box selection. In this task, users' had to design a gesture to define the width, depth and height of a 3D bounding box around target objects for selection. There was little agreement upon a common gesture, with a low agreement score of 0.095. In cases where the agreement score is below 0.1, we recommend further rigorous studies and usability tests to select the best gesture for the task. One participant expressed an opinion which was shared by many others:

"I don't think that it's unsuitable (the proposed gesture) but it's just very arbitrary and there is not a lot of intrinsic logic to it. If somebody told me that this is how to do it then I would figure it out but it's not obvious. It's just an arbitrary way of selecting a 3D area." -P11 Feedback Backfired. Our experimental design included the use of a 3D camera to support hand occlusion, which gave users some concept of the relative position between the virtual contents and their hands, however some participants found it to be obtrusive. We present ideas on how to improve this component of the experience in Section 5.1. One example of this criticism was as follows:

"Your hand gets in the way of the object so it can be hard to see how you're scaling it." -P11 Menus for AR. There was no significant difference in menu ranking. Some participants favored the horizontal menu because it was simple, familiar, easy to use/understand, supported on-the-surface gestures for touch sensing and did not inter-fere with virtual content. Others disliked the horizontal menu and felt it did not take advantage of 3D space with some options being further away and hard to reach.

The majority of participants found the vertical menu novel, and some found it to be appealing, easy to understand and that it made a good use of space with the distance to all options was evenly distributed. However, some found it harder to operate as they needed to lift their hands higher for options at the top if the buttons were arranged vertically.

Finally, some participants liked the object-centric menu because it was unique and object-specific so they knew exactly which object they were dealing with. However, some participants thought that it was unnatural and harder to operate in a crowded workspace. Furthermore, the open/close gestures for the object-centric menu were not as obvious, as indicated by the low agreement score of 0.11, as opposed to horizontal and vertical that scored 0.905.

Axes and Boxes.

The rotation and scaling tasks, allowed for three possible coordinate systems, local, global, and user-centric, which corresponded to the object-centric, world-dependent, and world-independent categories in the binding dimension. In practice, we found that the transformations were mostly object-centric; the participant would perform gestures based on the direction of the transformation presented on the object. This was expected because people would naturally perform these tasks physically and adapted their bodies and gestures to suit the operation.

To perform a rotation, participants would grasp the object with at least two contact points and would move their hand or turn their wrist accordingly. For scaling on 3 axes, participants would grasp or use open-hands to align with the sides of object and increased or decreased the distance between them to enlarge or shrink in the same direction as the transformation. Uniform scaling was less obvious, for example some participants preferred using open hands moving along a single axis in front of them, as shown in Figure 5 uniform scale [START_REF] Azuma | A Survey of Augmented Reality[END_REF]. Others preferred grasping the objects' opposing diagonal corners and moving along a diagonal line across the local plane parallel to the table surface as shown in Figure 5 uniform scale [START_REF] Heidemann | Multimodal interaction in an augmented reality scenario[END_REF]. Some user's expressed concern about how to perform the task for a round object, and suggested that bounding volumes must be provided for these models for manipulation.

Variation of Hand Poses.

Variants of a single hand pose were often used across multiple participants, and sometimes even by a single participant. We clustered common hand poses into eleven poses, as shown in Figure 4. Multiple hand poses can be used interchangeably for each gesture in a given task.

Discussion

In this section, we discuss the implications of our results for the fields of AR, gesture interfaces, and gesture recognition.

5.1

Implications for Augmented Reality

While our experiment was conducted in a tabletop AR setting, the majority of the user-defined gestures are equally suitable to be performed in the air. Only four gestures were on-the-surface, select all and open/close/select horizontal menu, with three mixed locale, box select [START_REF] Azuma | A Survey of Augmented Reality[END_REF] and insert/paste [START_REF] Heidemann | Multimodal interaction in an augmented reality scenario[END_REF]. This opens up our gesture set to other AR configurations, including wearable interfaces.

For our experiment, we implemented hand occlusion to give better understanding of the relative positions of the users' hands and virtual content. However we found that this could hinder user experience when virtual objects are smaller than the user's hand, occluding the object completely. We recommend that the hands should be treated as translucent rather than opaque, or occluded objects are rendered as outlines to provide some visual feedback of the objects' location.

As discussed in axes and boxes motif, a clear indicator of axes and bounding boxes should be provided during object manipulation tasks. Due to an absence of haptic feedback, visual feedback should be provided to inform users of the contact points between hands and objects.

Implications for Gesture Interfaces

We found most of the gestures elicited were physical (39%). Wobbrock et al. reached a similar outcome for surface gestures and suggested using a physics engine for handling these gestures. This approach was implemented by Hilliges et al. [START_REF] Hilliges | HoloDesk: direct 3d interactions with a situated see-through display[END_REF] and Benko et al. [START_REF] Benko | MirageTable: freehand interaction on a projected augmented reality tabletop[END_REF], who introduced "physically-based interaction", however only basic manipulations were demonstrated, with limited precision and control over the virtual contents. We believe that better control can be achieved by manipulation of the dynamical constraints imposed by the engine. Many gestures can make use of the collision detection component without the dynamics for tasks such as object selection, scaling etc.

In the size does matter motif we described how object size influences the number of hands used for manipulation. Since the resulting user-defined gesture set contains both one-handed and two-handed gestures for tasks such as scaling, we suggest taking an advantage of this fact to provide different levels of control. For example, in scaling tasks, by combining a snap-to feature for different granularities, unimanual scaling could offer snap-to in millimeter steps and bimanual in centimeter steps, as users tend to use one hand for an object smaller than their palm's size and two when it is larger.

As mentioned in the obvious and the obscure motif, care must be taken when choosing gestures for tasks with low agreement scores. We recommend follow up studies to determine usability by comparing these gestures, designer-refined gestures, menu options and even alternative modalities in case of multimodal interface.

Implications for Gesture Recognition

High degree of freedom hand pose recognition is achievable, however it is computationally expensive. In the variation of hand poses motif, we found a limited number of common poses (Figure 4), reducing the search space. Furthermore, the majority of the resulting gestures were static pose and path, which are simpler to recognize than dynamic pose and path gestures.

Conclusion and Future Work

We have presented an experiment and the results of a guessability study for natural hand gestures in AR. Using the agreement found among the elicited gestures, 44 userdefined gestures were selected as a "consensus set". Although gestures were found for all 40 tasks, agreement scores varied, suggesting that some gestures are more universally accepted than others. We are conducting a further study to validate our gestures, where a different group of participants will be shown the elicited gestures from both consensus and discarded sets and their preferences determined for each task to confirm our result.

Fig. 1 .

 1 Fig. 1. (Left) A participant performs a gesture in front of the image marker. (Right) The participant sees an AR animation of a shrinking car, and performs their gesture for a uniform scale task

Fig. 2 .

 2 Fig. 2. The proportion of gestures in each category in the six dimensional taxonomy. Form has been calculated for each hand separately.

Fig. 3 .

 3 Fig. 3. Agreement scores for forty tasks in descending order (bars) and ratio of two-handed gestures elicited in each task (line).

Table 1 .

 1 The list of forty tasks in six categories.

	Category	Tasks	Category		Tasks
	Transforms Move 1. Short distance	Editing		21. Insert
		2. Long distance			22. Delete
		3. Roll (X-axis)			23. Undo
	Rotate 4. Pitch (Y-axis)			24. Redo
		5. Yaw (Z-axis)			25. Group
		6. Uniform scale			26.Ungroup
	Scale	7. X-axis			27. Accept
		8. Y-axis			28. Reject
		9. Z-axis			29. Copy
	Simulation	10. Play/Resume			30. Cut
		11. Pause			31. Paste
		12. Stop/Reset	Menu	Horizontal	32. Open
		13.Increase speed		(HM)	33. Close
		14.Decrease speed			34. Select
	Browsing	15. Previous		Vertical (VM) 35. Open
		16. Next			36. Close
	Selection	17.Single selection			37. Select
		18.Multiple selection		Object-centric	38. Open
		19.Box selection		(OM)	39. Close
		20.All selection			40. Select

Table 2 .

 2 Taxonomy of gestures in AR extended from taxonomy of surface gestures.

		Taxonomy of Gestures in AR
	Form	static pose	Hand pose is held in one location.
		dynamic pose	Hand pose changes in one location.
		static pose and path	Hand pose is held as hand relocates.
		dynamic pose and path	Hand pose changes as hand relocates.
	Nature	Symbolic	Gesture visually depicts a symbol.
		physical	Gesture acts physically on objects.
		metaphorical	Gesture is metaphorical.
		abstract	Gesture mapping is arbitrary.
	Binding	object-centric	Gesturing space is relative to the object.
		world-dependent	Gesturing space is relative to the physical world.
		world-independent	Gesture anywhere regardless of position in the world.
		mixed dependencies	Gesture involves multiple spaces.
	Flow	Discrete	Response occurs after the gesture completion.
		continuous	Response occurs during the gesture.
	Symmetry dominant unimanual	Gesture performed by dominant hand.
		nondominant unimanual	Gesture performed by nondominant hand.
		symmetric bimanual	Gesture using both hands with the same form.
		asymmetric bimanual	Gesture using both hands with different form.
	Locale	on-the-surface	Gesture involves a contact with real physical surface.
		in-the-air	Gesture occurs in the air with no physical contact.
		mixed locales	Gesture involves both locales.

[H08, [START_REF] Ruiz | User-defined motion gestures for mobile interaction[END_REF][START_REF] Wobbrock | Maximizing the guessability of symbolic input[END_REF] Next: Swipe right to left. [H08, [START_REF] Ruiz | User-defined motion gestures for mobile interaction[END_REF][START_REF] Wobbrock | Maximizing the guessability of symbolic input[END_REF] Move, insert, paste (1): Select target from menu/clipboard, move it to a location to place.


Paul Schmieder

John Hosking
email: john.hosking@anu.edu.au

Andrew Luxton-Reilly
email: andrew@cs.auckland.ac.nz

Beryl Plimmer

Thumbs Up: 3D Gesture Input on Mobile Phones Using the Front Facing Camera

Keywords: Motion gestures, mobile interaction, image recognition

We use the front facing camera in a smart phone to capture gesture input. Thumb gestures performed above the camera are recognized and used to invoke commands. In contrast to other input modalities the camera requires no device movements and no valuable screen space is used. To be viable, this type of interaction requires gestures which are comfortable and memorable for the user and real-time accurate recognition of those gestures. Given the performance constraints of phones and their cameras we needed to determine whether accurate and reliable recognition is possible and identify types of gestures that are recognizable and user appropriate. As a proof of concept, we conducted a user study testing three gestures for performance and user satisfaction. The results demonstrate that the 3D gestural input is successful and we provide detailed insights into successful recognition strategies for this novel interaction modality.

Introduction

Typical mobile devices have limited input modalities available: touch screen, two or three buttons, sensors, cameras and microphones. Gesture input has the advantage of supporting continuous input thus reducing the demand for input actionsone successful example of gesture input is the swipe keyboard [START_REF] Kushler | System and method for continuous stroke word-based text input[END_REF]. The disadvantage of most continuous input modalities is that they use the touch screen which obstructs the information on the display, or the gyroscope or accelerometer that require distracting device movements [START_REF] Holz | Understanding Touch[END_REF]. Recent smart phones include front facing cameras. Their location means that such cameras are easily accessible away from the screen while their field of view is visible to the user and moving a thumb in this field does not require distracting device movements (Fig. 1). In this paper we explore how 3D gestures can be used to provide continuous input captured by the front facing camera. Using gesture input is common for a variety of devices and situations. A main difference is the number of dimensions gestures are performed in. For mobile devices, 2D Fig. 1. A smartphone with a front facing camera and a thumb performing a gesture above it on the right. gestures are common on the touch screen [START_REF] Bragdon | Hinckley Experimental analysis of touch-screen gesture designs in mobile environments[END_REF], including pinch and zoom, and swipe. 2D touch gestures can also be combined with 3D motion gestures [START_REF] Hinckley | Sensor synaesthesia: touch in motion, and motion in touch[END_REF]. These use a variety of sensors: gyroscope [START_REF] Hinckley | Sensor synaesthesia: touch in motion, and motion in touch[END_REF], accelerometer [START_REF] Iwasaki | Expressive typing: a new way to sense typing pressure and its applications[END_REF] and cameras [START_REF] Wigdor | Lucid touch: a see-through mobile device[END_REF].

A continuous input mechanism is desirable, as it provides affordances suited for controlling continuous variables, such as volume. Ideally, a continuous input mechanism is application independent allowing fast and precise manipulation in both directions (e.g. volume up and down, fast for big changes or slow for precise adjustments) such as with the scroll wheels on MP3-Players like Apple's IPod Classic or Microsoft's Zune. However, to adjust volume most portable devices, including smartphones, use buttons. Alternatively continuous input can be captured by the touch screen but this means displaying an appropriate interface element occluding current information. Sensors in smart phones are also able to provide continuous input but usually require distracting and disruptive device movements. For example Hinckley and Song's techniques [START_REF] Hinckley | Sensor synaesthesia: touch in motion, and motion in touch[END_REF] combine motion and touch using the gyroscope and accelerometer most of which required motions disruptive to user tasks, such as tilting the device away or shaking it.

Gesture recognition is typically a computationally intensive process. Performing such recognition on mobile devices with limited processing power is challenging, particularly when it involves image capture. Wang et al [START_REF] Wang | Camera phone based motion sensing: interaction techniques, applications and performance study[END_REF] used a Motorola v710 to capture gesture input and reported a maximum rate of 15.2 frames per second (fps), but when image processing algorithms were applied to the input, the capture rate dropped to 12fps. Gu et al. [START_REF] Gu | Developing Mobile Phone AR Applications using J2ME. in Image and Vision Computing New Zealand[END_REF] showed that many image processing algorithms reduced frame rates below one when implemented on a variety of different phones. To solve this problem, we developed novel recognition algorithms to reliably recognize the gestures in real-time on mobile devices with limited processing power.

Cameras in mobile devices have been widely used in research systems. This has focused on cameras on the back of the device tracking the devices' position [START_REF] Boring | Touch projector: mobile interaction through video[END_REF] or hand movements [START_REF] Wang | Camera phone based motion sensing: interaction techniques, applications and performance study[END_REF][START_REF] Wigdor | Lucid touch: a see-through mobile device[END_REF]. Kato [START_REF] Kato | A camera-based tangible controller for cellular phones[END_REF] attached a spring with a marker at its end above the back facing camera, allowing continuous input by tracking the marker in three dimensions. However, due to the camera position and the attached spring, operating the marker was rather constrained. Huerst et al. [START_REF] Hürst | Gesture-based interaction via finger tracking for mobile augmented reality[END_REF] tracked fingers with the back facing camera using markers attached to the tips to capture gestures.

Here we investigate the use of a front facing camera on a mobile device for continuous gesture based input, a combination we believe is novel. Using a front facing camera has multiple advantages: first, the required hardware, camera and finger are already present. Second, the input is independent of application (except for those directly using the front facing camera) and therefore always accessible. Third, no distracting device movements are required. Fourth, by offering an interaction space away from the display, screen occlusion is reduced. Fifth, the camera can be combined with other sensors to enrich interaction.

There are three components required to support this interaction modality: the gesture capture, the gesture recognition, and the gesture design: all of which contribute to the user experience. To test the feasibility of using three-dimensional gestures for continuous input, we defined three gestures and developed appropriate recognizers with varying demands on usability and hardware. In the second phase we evaluated the gestures on a smart phone in different scenarios. Besides the gesture recognizer performance we were interested in the demands on participants in terms of interaction space and the comfort of the thumb movements.

The results yield three contributions. First, continuous input can be successfully captured by front facing phone cameras. Second, current smart phones can accurately recognize 3D gestures in real-time. Third, our evaluation provides insights into the design implications of our gestures on usability including the definition of the interaction space above front facing cameras where gestures can be comfortably performed.

Related Work

Our work builds on related research involving interaction techniques for mobile devices; camera based phone applications; gesture input; and computer vision input.

Interaction Techniques for Mobile Devices

Some research focuses on facilitating mode manipulations based on the input channels available in mobile devices. For example Hinckley and Song [START_REF] Hinckley | Sensor synaesthesia: touch in motion, and motion in touch[END_REF] explored gestures combining touch and motion using gyroscope, accelerometer and touch screen. They propose motions for continuous input such as tilt-to-zoom and discrete input such as hard-tap. Research is not limited to the sensor abilities available in current devices. Brewster and Hughes [START_REF] Brewster | Pressure-Based Text Entry for Mobile Devices[END_REF] investigated the use of pressure on touchscreens to differentiate between small and capital letters entered on the phone's keyboard.

Camera Based Interaction on Mobile Devices

Camera based techniques on mobile devices can be differentiated by camera location: integrated in the device or external. Vardy et al. [START_REF] Vardy | The WristCam as input device[END_REF] used a camera attached to the wrist to capture finger movements feeding the recognized gestures to a computer for interaction. More common is to use device integrated cameras which are predominantly on the rear. TinyMotion [START_REF] Wang | Camera phone based motion sensing: interaction techniques, applications and performance study[END_REF] uses a rear facing camera to measure cell phone movements. It computes phone tilts to control games such as Tetris and text input. TinyMotion requires a button press combined with a phone tilt to input a character [START_REF] Wang | Camera phone based motion sensing: interaction techniques, applications and performance study[END_REF]. Another application is a see-through tool for applications such as maps [START_REF] Bier | Toolglass and magic lenses: the see-through interface[END_REF].The device's position over a surface is recognized and augmented with information.

Kato and Kato [START_REF] Kato | A camera-based tangible controller for cellular phones[END_REF]developed a tangible controller for continuous analog input. They attached a spring on top of the rear-facing camera with a marker attached to its end. The user provides input by moving the marker in any direction.

Gesture Input

Gestures, including touch and motion gestures, are commonly used for mobile device interaction. Bragdon et al. [START_REF] Bragdon | Hinckley Experimental analysis of touch-screen gesture designs in mobile environments[END_REF] investigated the impact of distractions on touch interaction. Comparing gestures and their soft button equivalent, they found gestures are less impacted by distractions and perform as well as soft buttons without distractions.

Aiming to build a gesture set for smartphones Ruiz and Lank [START_REF] Ruiz | User-defined motion gestures for mobile interaction[END_REF] conducted a study in which end-users proposed gestures for given scenarios. They found consensus among study participants regarding movement parameters and gesture-tocommand mappings. The resulting taxonomy differentiates between how a gesture is mapped to a command and a gesture's physical characteristics. Using thumb input on touch screens is widely researched. For example, Karlson et al. looked at the interaction between hands and mobile devices including user preferences and mechanical limitations [START_REF] Karlson | Understanding one handed use of mobile devices. Handbook of research on user interface design and evaluation for mobile technology[END_REF].

Computer Vision Based Input

Computer vision systems are widely used to collect visual information for gesture based input [START_REF] Freeman | Computer vision for interactive computer graphics[END_REF]. The choice of technique depends on factors such as desired information, environmental circumstances and available computational power. Commonly sought information is recognition of objects and/or gestures [START_REF] Mitra | Gesture Recognition: A Survey. Systems, Man, and Cybernetics, Part C: Applications and Reviews[END_REF] and object tracking [START_REF] Yilmaz | Object tracking: A survey[END_REF]. To gain this information environmental factors such as lighting, motion speed, blur and occlusions as well as their speed of change must be considered [START_REF] Moeslund | A Survey of Computer Vision-Based Human Motion Capture[END_REF].

Tracking a large fast moving object dominating a scene is challenging. Fast optical flow algorithms and graph cut approaches are viable but computationally expensive [START_REF] Barron | Performance of optical flow techniques[END_REF]. With limited compute power, temporal difference images have been used successfully to detect coarse-scale motion [START_REF] Freeman | Computer vision for interactive computer graphics[END_REF]. To further deal with changing lighting and blur, other properties of the tracked object such as object shape and color [START_REF] Yilmaz | Contour-based object tracking with occlusion handling in video acquired using mobile cameras[END_REF].

Gestures

The novelty of using front facing cameras for gesture input on a mobile device means the feasibility of such a system is unknown. To address this, we conducted a user study testing three gestures of varying demands on hardware performance and user input capability. This study addresses three questions: 1) are current mobile devices powerful enough for camera based gesture recognition in real-time? 2) are 3D gestures performed above the camera an easy and precise way to capture input? 3) what is the interaction space above the camera where gestures can be comfortably per-

formed? To answer these, gestures of differing complexity were tested for recognition accuracy and usability in different scenarios. For the development and testing we exclusively use a Samsung Omnia W, featuring a single core 1.4 GHz Scorpion processor and Windows Phone 7. As results will show, camera based gestural input can be successfully used in mobile devices for enriched natural interaction.

Design

We designed three gestures and an accompanying recognition algorithm to accurately identify each gesture. To allow for continuous input, each gesture can be performed such that it differentiates between two motion directions: for example to increase and decrease the volume, to zoom in and out of an image or to scroll a list up and down.

As the motion direction depends on the phone's orientation, the orientation was used by the recognizers to decide what motion direction maps in to what input direction. Additionally, each gesture has different levels of acceleration: for example, one level may increase the volume by one each iteration, another by three and another by five. Each gesture has a neutral state where no changes occur. Before the gestures are recognized each captured image is preprocessed. The first step scales the image to 40 x 30. We tested different resolutions for the gesture recognizers and found 40 x 30 to be the best tradeoff between performance and accuracy. Additionally, the image is converted from 32 bit RGB to 8 bit grayscale.

The thresholds used for the algorithms were obtained by testing the algorithms under different lighting conditions: bright: outside during a sunny day; neutral daylight: in the office with open blinds; artificial light: in the office with blinds down and light on, half dark: in the office with blinds down. Completely dark was not trialed as the algorithms require some light to track the thumb. Tilt Gesture. The thumb is lying on the camera and its tilt determines the input (Fig. 2). The tilt direction and intensity determines the direction and acceleration of change in input. Tilting the thumb to the right maps to an increase in input and tilting to the left a decrease. The higher the tilt is the higher the level of acceleration. If no tilt is applied to the thumb covering the camera no changes in input are invoked.

The recognition algorithm analyzes the light captured by the camera to calculate the input. The first step is to split the captured image in half depending on the phone orientation: the split divides the image so that each half is covered by a half of the thumb. Second, the average brightness value is computed over the pixels in each half. The image half with the lower brightness is the side the thumb is tilted towards. Similar low brightness values mean that the thumb evenly covers both image halves so no input changes are triggered. Finally, based on the brightness values and thus the amount of tilt, a step function is used to determine the level of input acceleration: the brighter the image half the higher the acceleration (Fig. 2).

A brightness threshold determines if the tilt gesture is currently being applied. The gesture recognizer is only used if the average brightness over the complete image is below a predefined threshold. This threshold is set to 55% of an image brightness value taken when nothing covers the camera (e.g. when the application is started). The chosen threshold provides the best offset between brightness ranges to determine the level of acceleration and activate/deactivate the gesture recognizer.

Distance Gesture. The distance between thumb and front facing camera is used to adjust the input (Fig. 3). A predefined distance marks a neutral point where no input manipulation happens. A smaller thumb-camera distance triggers a decreased input and a larger one an increase. The exact distance determines the level of acceleration. Input is based on the thumb's diameter in the captured image: the larger the diameter the closer the thumb. To calculate the distance, the thumb's position in the captured image has to be determined. Then, brightness values are computed over the image. Finally, the input value is determined using the thumb's distance and brightness.

To detect the thumb in the image a Sobel edge detector with a 3x3 kernel is applied. The resulting Sobel image (Fig. 3) contains objects defined by their borders. To identify the thumb three assumptions are made: 1) the thumb partly covers the center row of the Sobel image; 2) in the center row the thumb is the biggest object; 3) the thumb forms an uninterrupted blob starting at the image. Hence the recognition algorithm analyzes the center row of the image looking for the biggest group of dark pixels bounded by white pixels. If the biggest section is above half the image's width, the algorithm further tests whether the section is part of a blob connected to at least one image edge. If no object satisfies all three the criteria the algorithm discards the image and no input is invoked assuming the gesture is not currently being performed. The size of the range between minimum and maximum thumb distance is divided into smaller spans, each accommodating a different input value. A span has to be sufficiently large to be easily found by the user in terms of distance of his/her thumb to the camera. For example, in a 40 pixel wide image the total range is 20 as the thumb must be at least half the image width wide. This leaves 20 pixels to be split into smaller spans. We found spans of one-eighth of the image width to work best. Brightness values determine if a thumb is covering the camera completely. This allows further differentiations when the thumb covers the captured image width completely but does not yet physically cover the camera. Circular Gesture. A circular motion is performed close above the camera to adjust the input (Fig. 4). Depending on the motion direction (clockwise or counterclockwise) input is either increased or decreased. The speed of the gesture determines the level of acceleration.

The circle gesture recognizer first detects the thumb. Afterwards the thumb's center of mass is used to calculate the motion direction. Motion speed is based on the overlap of thumbs between two consecutive images.

To detect the thumb a flood fill algorithm is used based on the captured image's greyscale values. The thumb appears darker than the rest due to the movement close to the camera: cameras found in smart phones are not fast enough to adjust their blending settings to evenly illuminate the image. As the thumb changes illumination conditions quickly, the camera makes the background behind the thumb appear much brighter than normal. The flood fill algorithm finds the biggest connected thumb blob composed of pixels with an 8-bit greyscale value below 90. If the biggest blob covers more than 21% of the captured image it is assumed to be the thumb. Otherwise the thumb is assumed not to be in the image. If no thumb is detected in more than four successive images further computational steps are stopped and only resumed when a thumb is found in at least four successive images.

Motion direction is calculated using the cross products of three successive images. The weighted average location for each thumb blob is calculated to determine its center of mass (Fig. 4): the center of mass's x-coordinate is the average value over all the blob pixels x values. The same applies to the y-coordinate. Afterwards, the cross products between the four most recent thumb centers of mass are computed. If the sum of the three resulting vector magnitudes is positive the gesture motion is clockwise and otherwise counter clockwise. Motion speed depends on movement and on the number of intersection points between two successive thumb blobs. To detect a stationary thumb the image is divided into four even quadrants. If the thumb remains in the same quadrant over four successive images, the motion counts as stopped. If not, the thumb is moving and the overlap of two thumbs in successive images is computed. A step function based on the overlap percentages is used to define thresholds for the different levels of acceleration depending on the number of acceleration levels. For example, given two acceleration levels, our empirical tests showed a threshold of 42% to be optimal: a higher overlap than 42% means a slower acceleration and a smaller overlap a higher acceleration. Using quadrants to detect a stationary thumb rather than overlaps is better as it compensates for small unintentional thumb movements.

Performance & Robustness

The computation of a new input value consists of multiple steps of which capturing the image is the most time costly (Table 1). If no other computations are performed the Samsung Omnia W captures up to 16 images per second. In the following we comment on the robustness and accuracy of the recognizers in general and under different lighting conditions (see section 3.1). The tilt and circle gestures are both unaffected by different lighting conditions and fast lighting changes. Only when it was completely dark did both recognizers stopped working. The main reason for the robustness is that in both cases the thumb dominates the captured images making the background appears bright and unfocused (Fig. 4).

The distance gesture worked perfectly under the tested lighting conditions except in the half dark condition. In this condition if there were multiple dark objects in the image background they were occasionally merged with the thumb: because of the insufficient lighting the Sobel operator did not detect all the edges. Fast lighting changes did not affect the algorithm.

Evaluation

We conducted an evaluation study to explore the interaction with a smartphone using continuous gestures captured by a smartphone's front facing camera. This study focuses on the requirements of suitable 3D gestures and the space they are performed in with the aim to inform further research regarding the basic requirements of this novel and yet unexplored interaction. Four pilot participants were used to determine algorithm parameters such as the number of acceleration levels and the rate at which changes were applied. Only one of the recognizers was activated at each point in time requiring a button press to change between them. We were specifically interested in the following questions:  What defines the interaction space in which gestures can be comfortably performed above the camera?  Are the evaluated gestures easy and intuitive and are their associated apps an appropriate match?  Is the phone's hardware sufficiently powerful to guarantee real-time performance?  Are the gestures recognized accurately in terms of direction and acceleration changes and reactiveness?

Participants

Participants had to have a smartphone as their mobile device to guarantee device familiarity. We recruited 24 participants, 8 females and 16 males, ranging in age from 19 to 41 (mean: 25.63 years, SD = 5.17). Five were left-handed. There was no compensation for participation. Since accurate tracking objects may depend on the visual properties of the tracked object, we wanted to ensure a variety of different thumb size, shape and color were represented in the evaluation. To test the robustness of our algorithms we intentionally recruited participants from different ethnic groups (10 Caucasians, 6 Indians, 5 Asians and 3 from the Middle East) and of different height as an approximation of thumb size (mean: 171cm, SD = 8.62).

Apparatus

To test the gestures, participants used a Samsung Omnia W I8350 featuring a single core 1.4 GHz Scorpion processor and Windows Phone 7. The front facing camera was located on the shorter side's corner (Fig. 1). The capture rate of the front facing camera is on average between 8-10 images per second on the Windows Phone The gesture recognizers were integrated in a program running on the phone. Each recognizer was configured to support two directions and to differentiate between two acceleration levels: one and three. We decided to use two acceleration levels based on the results from the four pilot study participants. While the tilt and circle gestures would have worked intuitively enough with three acceleration levels, the distance gesture did not. When the thumb was moved with the lower acceleration changes were applied every 500ms to allow for accurate input. This low change rate was chosen as the pilot study participants frequently overshot the target with a faster change rate. For the faster acceleration level changes were applied after each image was analyzed and the new value computed (i.e. on average every 75msecs). Sessions were recorded using a video camera mounted above the participants.

Tasks

We designed three tasks each dedicated to a particular gesture and one for familiarization across all three gestures. Each task required participants to use all facets of the gesture so that the invoked input direction and level of acceleration were frequently changed. The choice of task for a particular gesture was based on similar mappings found in current environments as explained in the following. The order of gestures was varied between the participants forming six conditions.

Contact List -Circle Gesture. To scroll a list using a circular motion is familiar practice. MP3 players such as Apple's iPod Classic use a circular motion to browse their content. Scrolling a list requires participants to bridge bigger distances faster as well as precise manipulations to jump between neighboring list entries. The scrolling mini app displayed a list of 78 contacts to simulate a standard smartphone Contact list (Fig. 5a). To scroll down/up, participants performed a clockwise/anti-clockwise motion.

If the acceleration level was 1 the selector, indicated by a blue color, jumped from the currently selected contact to the next neighbor. An acceleration level of three jumped three contacts at once.

In the top right corner a name was displayed which was also in the contact list. Participants were asked to scroll to that name in the list as fast as possible. The correct name had to be selected for two seconds requiring participants to provide neutral input. After two seconds the name in the top right corner was replaced by another name from the contact list. In total, participants had to scroll to 10 different names. The choice of names made participants perform precise selections (e.g. to go from one name to a neighbor) and to bridge larger gaps (e.g. Paul to Billie).

Zooming -Distance Gesture. Participants controlled the size of a green square on the screen (Fig. 5b). To zoom in, participants moved their thumb close to the camera and to zoom out the thumb was moved further away. This gesture is familiar for zooming as going closer to an object makes it naturally appear bigger and going further away makes it smaller.

An acceleration level of one increased or decreased the square's size by 10 pixels and a level three by 30 pixels. The square can have any width between 0 and 300 pixels resulting in 30 distinct zoom-able sizes.

The participants' aim was to match their square's size with a red frame shown on the display. To indicate equal sizes, the red frame turned white. Participants had to keep the size constant for two seconds, requiring them to adjust the thumb camera position to neutral, before the frame changed size and turned red again. In total the red frame changed size 10 times. BreakOut -Tilting Gesture. Participants controlled the paddle in BreakOut (Fig. 5c) to bounce the ball upwards. The aim was to destroy all ten bricks lined in the top of the game area by hitting each with the ball. After a brick was hit, the ball was deflected down towards the paddle. The angle of deflection when the ball hit the paddle depended on where the ball hit the paddle. To destroy the bricks the user needed both small and large paddle position adjustments forcing them to use all facets of the tilt gesture. The game area was 300 pixels wide. An acceleration level of one meant a paddle move of 4 pixels while a level of three meant a 12 pixel move. In total the paddle had 75 distinct positions. Each destroyed brick scored 10 points making 100 points the highest score. The game was also used in previous research to study camera based input techniques [START_REF] Hansen | Use Your Head: Exploring Face Tracking for Mobile Interaction[END_REF].

Procedure

Participants were seated at a table with the smartphone lying immediately in front of them. They were given a tutorial guiding them through the study. The participants' first task was to complete the first section of the questionnaire.

To familiarize themselves with the Windows Phone participants were asked to pick it up and the facilitator explained the main elements on the screen. Once familiar, the participants started the test application. During this the phone was held in portrait orientation before changing to landscape to interact with the test application.

At the beginning of each gesture the facilitator explained the gesture to the participant. Afterwards the participant was told to navigate to the volume mini app and trial the gesture. Then the participant completed both a gesture specific task and the questionnaire for this gesture. This was repeated for each gesture.

After all tasks were finished the final section of the questionnaire was completed. The experiment concluded with a discussion of other possible gestures allowing continuous input. It took the participants on average 29 minutes to complete the study.

Results

The collected data includes the questionnaires, session video recordings and transcripts of the final discussion. The questionnaire comprised three sections. The first gauged participants' familiarity with gesture input. The second asked participants to rate the gestures regarding their ease and recognition accuracy using a five point Likert scale. The last contained a ranking of the gestures as well as questions gauging the interaction space above the camera including camera location.

Interaction

All 24 participants held the smartphone in landscape orientation with the camera on the right. Regardless of their handedness all participants chose to use their right thumb to perform the gestures. The matches between gesture and associated mini apps were never rated as negative thus confirming our design choices (Fig. 6).

Ideal Camera Position.

The camera should ideally be centered on the horizontal axis in landscape mode on the phone's right side. When asked in the questionnaire to indicate the optimal camera position on the phone all participants chose the right side of the screen. Six participants chose the top corner, eight the center and nine the lower corner. Ten participants noted that the camera on the horizontal axis should be centered between screen and phone edge. The camera on the test phone was closer to the edge than to the screen which sometimes created problems with the tilt and circle gesture: a tilt to the right (of a big thumb) meant a tilt slightly over the phone's edge. While performing the circle gesture some participants drifted away from the camera's center. They stated that they used the space between screen and phone edge as a frame of reference with their circle motion touching both sides.

Keeping the Thumb in the Camera's Viewing Field. Each gesture used a different interaction space above the camera with different demands on the user. The tilt gesture was the easiest one to keep in the camera's viewing field as it essentially was performed as a two dimensional gesture with the thumb lying flat on the camera (Fig. 6). It received ratings above neutral with 83% of the participants strongly agreeing to the ease and 17% agreeing. In contrast, the ease of keeping the distance and circle gestures in the camera' viewing field was rated lower.

The dominant usability problem with the distance gesture was to keep it in the viewing field. To measure the distance between camera and phone, the recognition algorithm calculates the thumb's diameter. If the thumb is close but not centered above the camera, it is only partially visible thus its measured diameter is smaller than it actually is so the calculated distance is incorrect. As the thumb moves away from the camera it does so in an arc. Its tip moves towards the bottom of the phone as it moves up thus leaving the camera's viewing field on the lower edge. However, this usually occurs after the thumb position starts feeling uncomfortable due to the high finger stretch. In the questionnaire we asked participants what the maximum distance between thumb and camera is before holding the thumb becomes uncomfortable. Measured with a ruler, the average distance stated was 4.75 cm (SD = 1.12cm, Min = 2 cm, Max = 6 cm). The majority of the participants stated that it becomes uncomfortable just before thumb and thumb socket form a straight line. For correct recognition, the circle gesture had to be performed at a certain height above the camera and the motion had to circle the camera. The gesture did not require participants to draw a perfectly round circle nor one perfectly centered above the camera. However, moving outside the camera's viewing field, as some participants did, made it impossible to track the thumb resulting in incorrect recognition. On occasion the thumb was too close to the camera so the captured image was almost totally dark leading to incorrect recognition. Participants on average found it harder to keep the thumb in the right area for the circle compared to the distance gesture with the distance gesture having 17% rating it below average and the circle gesture 29%. This was also reflected in the questionnaire where eight participants said it was difficult to keep the circular movement above the camera. There were no complaints about the distance gesture.

In summary, the tilt gesture is the easiest with only positive ratings. The distance gesture is the second highest rated followed by the circle.

Ease of Gesture Motions. The thumb movements for the distance and circle gestures were perceived as physically uncomfortable at times. Having to hold the thumb still at changing positions over an extended duration (Zooming task: Ave = 141 sec, SD = 36) was perceived as uncomfortable by 12.5% of the participants. For the circle gesture, 29% of the participants experienced discomfort when performing the gesture: three stated their thumb movement was restricted as they were holding the phone with the same hand. This was particularly evident when the thumb tip moved close to the thumb base forming a right angle between the thumb's first and second phalanges.

Phone Performance and Recognition Accuracy

All gesture recognizers and mini applications ran in real-time on the phone. In the questionnaire participants rated the overall accuracy of the gesture recognizer as well as the recognizers' reactiveness and the ease and correctness of changing the acceleration level and direction (Fig. 6).

The gestures and their associated mini applications ran smoothly apart from several occurrences when the camera froze for two seconds during interaction. This occurred randomly for different recognizers and appeared to be an artifact of the phone's API to the camera buffer. We speculate that this is unlikely to be an issue with a multicore processor phone where camera and interface run on different threads.

The recognizers' reactiveness was rated neutral and above for the tilt gesture and neutral and above by 21 participants for the distance gesture and by 23 for the circle gesture (Fig. 6). To allow for precise adjustments changes were applied every 500ms, which some participants perceived as too slow. For the distance gesture, changes were invoked instantly when the thumb distance entered a new range. For example, to zoom in with an acceleration level of one the thumb had to be roughly (depending on thumb size) between 3 and 4 cm away. Any other distance range was mapped to another input. If the thumb position was changed but not enough to change ranges, participants perceived this as slow to react. For the circle gesture the thumb had to rotate half a circle to trigger any change in input. A lower threshold meant that sometimes unintended changes were invoked as the circular motion was not performed.

Changing the level of acceleration was rated below neutral for the distance and circle gesture by two and five participants respectively (Fig. 6). For the distance gesture, to change the level of acceleration, the finger has to be in the distance range associated with the desired acceleration. To find the correct range one either recalls it from practice/experience or moves towards the desired range until the input value changes to the desired outcome. As participants had little training not all remembered the positions perfectly so had to use the second strategy which some perceived as suboptimal. For the circle gestures, correctly recognizing the acceleration required the circle to be performed above the center of the camera. If this condition was violated the recognizer could not accurately differentiate between the acceleration levels of one and three. Due to the problems of keeping the thumb in the required space (Fig. 6), the acceleration levels were occasionally recognized incorrectly.

Changing the input direction was always rated as neutral or higher for all gestures (Fig. 6). Differentiating between directions was easier than changing the acceleration levels because the motions were more different thus easier to remember and recognize. For example, for the circle gesture one had to change the motion direction regardless of acceleration. Additionally, to accurately recognize the correct direction it was sufficient for the camera to capture part of the circular motion thus not requiring a centered motion above the camera.

To sum-up, well designed gesture recognition algorithms can run in real-time on current generation smartphones. The gestures allow for accurate adjustments of acceleration and directions if the gesture requirements such as motion path and accuracy are satisfied.

5.3

Summary and New Gestures

Both, natural gestures and accurate recognizers are required for satisfactory interaction. If a gesture is too complicated to be intuitively performed, a recognizer may be unable to deal with the inaccurately performed motion. The three tested gestures varied in terms of motion complexity; this was reflected by how much participants enjoyed each gesture (Fig. 6) and the final gesture ranking. The tilt was the easiest to perform and was ranked first by 21 of 24 participants and second by the remaining participants. The distance gesture was more difficult to perform mainly because the desired camera-thumb distances were hard to remember with the little practice. Thus, the distance gesture was ranked second by over half the participants (1 st : 2, 2 nd : 13, 3 rd : 9 participants). The circle was the most difficult of the three, as it required the participants to first, perform the motion at a certain distance above the camera while, second, centering the motion. The gesture accurately recognized only if both criteria were satisfied. 15 participants ranked the circle gesture third, 8 second and 1 first.

When asked for other gestures, participants proposed one of three gestures. The first, proposed by four participants, is a circular motion but on the z-axis above the camera. Thus staying inside the camera's viewing field would become easier. Circle direction and speed determine acceleration level and direction. The second and third are variants of swipe motions. The second, proposed by four, consists of a horizontal swipe motion with the swipe direction determining the input direction. Once the swipe in a direction reached the outer border the finger is either lifted high above the camera or besides it to return to the other side and continue the gesture. The third proposed by seven has an additional horizontal swipe that makes the extra motion to return to the start position redundant.

Discussion

The aim of the presented work was to investigate the potential of a yet untested interaction to capture continuous input with the focus on performance and 3D interaction. Due to the new challenges we felt that such an evaluation was required before other evaluations such as comparing the proposed techniques with state-of-the-art interaction methods (e.g. touch screen based input) were conducted. Additionally, with the knowledge gained a more detailed study looking at creating user defined gesture set is possible similar to [START_REF] Ruiz | User-defined motion gestures for mobile interaction[END_REF].

Capturing and Recognizing Gestures

Gesture recognition using front facing cameras has to run in real-time despite a mobile devices' limited hardware performance. To track objects like a thumb a number of tracking approaches may be unsuitable due to high performance requirements. We found the following methods provide sufficient information while allowing for real time interaction: working with the image's brightness, edge detection and differential images; the last is unsuitable if fast movements are involved due to fast lighting changes.

A captured image's brightness (or the excess of it) can be used to track moving objects near the camera. If a tracked object moves fast and covers at least quarter of the image, the background is overexposed making it too bright with almost no contours. Using a threshold to extract the close darker object from the background is easy and has low performance cost as shown by the circle recognizer.

To track an object at any distance that is not moving, fast edge detectors can be used. To identify the object based on its edges additional information is required such as its shape, origin and/or size. The last two were successfully used for the distance gesture where the finger always originates from at least one corner and covers at least a certain proportion of the image.

Capturing the gestures with the camera using these algorithms has some disadvantages. Constant capturing of images consumes more power draining the battery faster. While it does not matter whether the light is natural or artificial, sufficient lighting is still required: our tests showed that except for the Sobel operator, the algorithms worked in all conditions except the completely dark one.

Designing Gestures

Designing successful motion gestures means creating intuitive gestures without constraints. As the camera has a limited field of view gestures have to be fitted to this area: gestures close to the camera have to be directly above it while those further away can be further away from the camera's center. However, gestures cannot be too far away as this becomes uncomfortable to the point where holding the device with both hands becomes impossible. Each state in a gesture motion must be easily identifiable and performable. If it is unclear what motion to perform to effect change or if the motion requires physically uncomfortable moves, the gesture may be unsuitable for continuous input. The distance gesture showed that having no explicit knowledge of the required motion to change to a given acceleration level may dissatisfy users.

The number of acceleration levels depends on the differentiability of the gesture motions. For example, with the distance gesture there was a total range of 6 cm available, which had to be divided in sub-ranges each dedicated to a unique input value. Trying to divide this range into more than five ranges (two directions with two acceleration levels each plus a neutral range) would be unrealistic as users' ability to discriminate those ranges is too difficult as shown in the evaluation. User feedback is important, a change in gesture must immediately cause a change of input while providing sufficient control for small changes.

Designing the Device

The camera position preference, regardless of the users' handedness, is on the right side of the phone. When asked for their preferred camera position, the participants' answers did not show any strong preference between the vertical position (top, center or bottom). However, there was a preference for the camera to be centered on the phone border.

Regardless of the camera position the gestures did not impair screen visibility. When asked, participants stated that they were able to see the entire screen during the interaction. An analysis of the video recording showed that items of importance such as the next name for the scrolling tasks situated on the screen's right border next to the camera were no covered by any gesture.

Conclusion & Future Work

We developed, implemented and evaluated three gestures and associated recognizers to allow for continuous input on portable devices. To capture input the front facing camera is used as its location offers an easily reachable input channel away from the screen. Using camera-based gestures avoids the need to move the device, which would potentially interrupt the interaction, and does not occupy valuable screen space.

Our work shows how 3D gestures are successfully captured and recognized. Different low-cost recognition strategies are implemented and trialed. The evaluation demonstrates the robustness of the recognition algorithms under different conditions; e.g. different lighting conditions and different colors and sizes of the tracked thumbs. The evaluation also provides insights into, first, the design of gestures and their requirements and, second, the interaction space above front facing cameras where gestures are comfortably performed.

Motivated by the results of our study we are exploring further gestures for discrete and continuous input. We are also looking at combining camera-based gestures with other input modalities such as data captured by gyroscope and accelerometer. Motivated by [START_REF] Karlson | Understanding one handed use of mobile devices. Handbook of research on user interface design and evaluation for mobile technology[END_REF], we are also exploring making camera based interaction accessible for one handed interaction.

Fig. 2 .

 2 Fig. 2. The tilt gesture: Left: Tilting the finger to either side determines the direction of input and the amount of tilt determines the level of acceleration. Right: The captured images used to compute the tilt. (top) A strong tilt to the left and (bottom) a slight tilt to the right.

Fig. 3 .

 3 Fig. 3. The distance gesture: Left: the input depends on the distance between camera and thumb. A predefined distance is the neutral input where no changes are invoked. Right: The images after the Sobel operator has been applied with the thumb further away (top) and close (bottom) to the camera.

Fig. 4 .

 4 Fig. 4. The circular gesture: Left: The direction of the circular motion determines the input direction and the speed of the motion determines the level of acceleration. Holding the finger at a position invokes no change (i.e. neutral input). Right: The images with the thumb in black after the flood fill algorithm. The white dot in the black blob indicates the center of mass.

Fig. 5 .

 5 Fig. 5. The test program showing the mini apps: (a) the contact app, (b) the zooming app and (c) the Break-Out app. In the top left side an indicator is shown displaying the currently captured input ((a) neutral, (b) fast-left, (c) slow-right).

Fig. 6 .

 6 Fig. 6. The questionnaire ratings for the three gestures on a 5 point Likert scale centered around neutral. The ratings are: 1= strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree.

Table 1 .

 1 Performance measurements in milliseconds using a Samsung Omnia W phone.

		Tilt Gesture	Distance Gesture	Circular Gesture
	Pre-process Image		image captured (66.75msec)
			scaled & converted to greyscale (1.9msec)
	Process Image	-	4.93 (Sobel)
	Detect Thumb	-	1.12	2.88 (Flood Fill &
				mass center)
	Compute Direction	0.97	0.51	1.76
	& Acceleration