Ahmed Kharrufa
email: ahmed.kharrufa@ncl.ac.uk

James Nicholson
email: james.nicholson@ncl.ac.uk

Paul Dunphy
email: paul.dunphy@ncl.ac.uk

Steve Hodges
email: steve.hodges@microsoft.com

Pam Briggs
email: p.briggs@northumbria.ac.uk

Patrick Olivier
email: patrick.olivier@ncl.ac.uk

Using IMUs to Identify Supervisors on Touch Devices

Keywords: IM U, association, authentication, touch interaction, UI design

de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Touch-based computing devices, and in particular a variety of tablet form factors, are becoming prevalent. Initially used as personal devices, they are increasingly being used in work and public settings such as schools, shops, museums and exhibition spaces. The focus in this paper will be on school settings, as a principal example, considering that tablet-based classroom applications are growing rapidly [START_REF]Tablets and Apps in Schools 2013[END_REF] and are seen by many as one of the key classroom innovations of the 21st century. However, the same approach applies to the other settings and for other touch devices such as interactive boards, tabletops, laptops and other devices with interactive screens.

In order to leverage the full benefits that tablets can offer in a classroom setting, a teacher or classroom assistant will often need to override the settings on a student's device and initiate machine-level or even classroom-level effects such as projecting the work of one student onto the class whiteboard or gaining temporary Internet a ccess [START_REF] Alagha | Towards a teacher-centric approach for multitouch surfaces in classrooms[END_REF], [START_REF] Do-Lenh | Supporting Reflection and Classroom Orchestration with Tangible Tabletops[END_REF], [START_REF] Kharrufa | Tables in the wild: Lessons Learned from a Large-Scale M ulti-Tabletop Deployment[END_REF], [START_REF] Kharrufa | Extending tabletop application design to the classroom[END_REF], [START_REF] Kay | M TClassroom and M TDashboard: Supporting Analysis of Teacher Attention in an Orchestrated M ulti-tabletop Classroom[END_REF]. Imagine the following scenario: a student has shown an interesting approach to solving a problem which the teacher wants to share with other students. The teacher initiates supervisor access on that student's tablet, then freezes all the other devices in the classroom and projects to the classroom display for a class discussion. Moving through the class, the teacher and one or more teaching assistants can then authenticate other students ' devices to pull content from the board, or to sanction sharing of content between students.

A key requirement to support scenarios like these is the ability to identify and authenticate interactions by supervisory users such as teachers, and to differentiate these from regular users, in this case the students. Here we encounter a known problemsecure authentication is difficult to achieve with any tablet or surface interface, given the ease with which casual observers can engage in "shoulder surfing". Most co mmonly, user authentication is based on "something you know" that is meant to be held secretsuch as a password or PIN. However, this passcode secret is overly exposed when using touch screen input [START_REF] Kim | M ultitouch authentication on tabletops[END_REF] and can be easily compromised in a public setting such as a classroom [START_REF] Kim | M ultitouch authentication on tabletops[END_REF] where authentication is typically in full view of a number of observers. This inevitably leads to uncontrolled student access to unauthorized applications.

Controlling student access to privileged functions is a very real problem. For e xample, in 2013 Apple was awarded a $50 million contract from the Los Angeles School Board of Education, to roll iPads out into public schools across the stateintended to be the first of many such large scale education initiatives. However a year later, the scheme faltered in large part because of identity management and authentication concerns raised when the students found ways to access unauthorized content and applications, leading to significant problems with classroom discipline and ult imately challenging broadband capacity [START_REF] Leonard | The iPad goes to school[END_REF], [START_REF] Needle | school district puts the brakes on massive iPad deployment[END_REF], [START_REF] Pierra | Tablets-in-school galore causes a bandwidth war in Southern California[END_REF].

New authentication and access management solutions in the form of biometrics and near field communication (NFC) may help, but these are only available in some of the latest tablets and there is a pressing need for effective, usable access control for existing tablets including the estimated 10 million iPads already in the classroom in the US alone [START_REF] Leonard | The iPad goes to school[END_REF].

In this paper, we offer a novel and elegant authentication so lution called SwipeID that simplifies supervisory access to tablets with no NFC or biometric capability. Importantly, it requires no hardware augmentation of the touch device itself. Instead, Fig. 1. SwipeID. Associating touch data with IM U data using a challenge-response protocol.

our solution comprises a wrist mounted inertial measurement unit (IMU) worn only by the supervisor(s), and a challenge-response style interaction protocol involving a very simple set of movements. This setup is depicted in Figure 1. The aim is to provide a practical alternative when other solutions such as NFC and fingerprint readers are not possible (e.g. for iPads and interactive boards in schools , and for large/fixed touch displays in public settings that do not normally have NFC, fingerprint readers, or cameras). Unlike other options, ours allows for identification at the point of touchespecially useful for large displays like interactive boards. We precede a more detailed description and evaluation of our solution with an overview of related work in the areas of touch and sensor interactions and their deployment in a classroom setting.

Our contribution is as follows: We present a novel system that allows simple, point of contact authentication for any touch screen device. We show how such a system can be used to solve known access management problems when using ta blets and other touch screens in the classroom as an example. We demonstrate the efficacy of our system in user studies with four exemplar user interface controls.

RELATED WORK

In this section we cover previous work which tackles the challenge of identifying users of touch devices. We also briefly review literature relating to the use of mov ement correlations for building trust, a very relevant topic. Finally we consider prev ious approaches to the specific challenge of teacher orchestration in the classroom, a key application area which inspired this work.

Identifying Touches

Much work that seeks to identify users on touch devices has focused on the tabletop context. In DiamondTouch [START_REF] Dietz | DiamondTouch: A M ulti-User Touch Technology[END_REF] sensors are embedded in the chairs of users. In conjunction with a transmitter built into the display it is possible to identify the owner of each touch event. Roth et al. [START_REF] Roth | The IR ring: authenticating users' touches on a multi-touch display[END_REF] proposed the IR Ring; an infra-red (IR) augmented finger-worn ring that transmits IR pulses detectable by cameras embedded into a tabletop. Particular patterns of pulses are unique to particular users, making them suitable for identifying touches across sessions and across different tabletops. A similar approach using a wristband was proposed by Meyer and Schmidt [START_REF] Schmidt | IdWristbands: IR-based user identification on multi-touch surfaces[END_REF]. However, both technologies are only suitable for IR-sensitive optical touch detection systems. Holz and Baudisch [START_REF] Holz | Fiberio: A touchscreen that senses fingerprints[END_REF] designed a special touch screen to support biometric user authentication based on fingerprint recognition carried out dynamically during each touch interaction. This provides natural per-touch user identification but relies on high resolution, high speed scanning and processing hardware. Harrison et al. [START_REF] Harrison | Capacitive fingerprinting: exploring user differentiation by sensing electrical properties of the human body[END_REF] introduced an approach for user identification that relies on sensing electrical properties of the human body when using capacitive touch devices. However, while promising, the experimental results showed that the variability of these electrical properties due to biological and environmental factors can be larger than the variability of such properties between users. Mock et al. [START_REF] Edelmann | User identification using raw sensor data from typing on interactive displays[END_REF], on the other hand, explored using raw sensor data from typing on an interactive display for user identification. The system was based on optical touch sensing rather than the more common capacitive touch displays.

HandsDown [START_REF] Schmidt | HandsDown: hand-contour-based user identification for interactive surfaces[END_REF] and MTi [START_REF] Blažica | M Ti: A method for user identification for multitouch displays[END_REF] are two approaches that rely on handprints for identification against a database of users' hands characteristics rather than fingerprints. Unlike HandsDown, MTi is not limited to camera-based touch sensing. However, for both the touch surface needs to be big enough to accommodate an outstretched hand. Other approaches (e.g. [START_REF] Collins | Who did what? Who said that? Collaid: an environment for capturing traces of collaborative learning at the tabletop[END_REF], [START_REF] Ramakers | Carpus: a nonintrusive user identification technique for interactive surfaces[END_REF]) use an overhead camera for tracking and identifying users. These are most suitable for large fixed displays, in particular tabletops.

Inferring relationships by sensor correlation

Researchers have developed a number of techniques to infer the physical relationship between devices being used in conjunction with each other. The use of accelerometers for making associations between multiple devices has been a subject of many investigations [START_REF] Fujinami | A study on a correlation coefficient to associate an object with its user[END_REF], [START_REF] Holmquist | Smart-its friends: A technique for users to easily establish connections between smart art efacts[END_REF], [START_REF] Patel | A gesture-based authentication scheme for untrusted public terminals[END_REF]. Fujinami and Pirttikangas [START_REF] Fujinami | A study on a correlation coefficient to associate an object with its user[END_REF] used the correlation of accelerometer signals in wrist-worn devices and devices embedded in objects to reason about the identity of an object's user. In Smart-Its [START_REF] Holmquist | Smart-its friends: A technique for users to easily establish connections between smart art efacts[END_REF] two objects held together and shaken were associated based on a correlation threshold. Similarly, Patel et al. [START_REF] Patel | A gesture-based authentication scheme for untrusted public terminals[END_REF] proposed the use of a shake/pause gesture sequence to pair a mobile device with a public terminal. If the mobile device produces the same shake/pause sequence as that displayed on the terminal, then it is assumed that this is the correct device with whic h to establish an association. For devices that have already been paired, Chen et al. [START_REF] Chen | Duet: exploring joint interactions on a smart phone and a smart watch[END_REF] explored the design space of joint interactions between a smart watch and a smart phone. One of the proposed interactions was to use the accelerometer data from the smart watch to augment the interactions with the phone. Shrirang et al. [START_REF] Shrirang M Are | ZEBRA: Zero-Effort Bilateral Recurring Authentication[END_REF] relied on the correlation between input from a wrist-worn accelerometer and from the keyboard/mouse of a computer terminal to confirm the continuous presence of a user. The user is logged-out in the absence of such correlation. PhoneTouch [START_REF] Schmidt | PhoneTouch: a technique for direct phone interaction on surfaces[END_REF] used server-side correlation-in-time to allow the use of phones to select targets on an interactive surface using direct touch. One application of associating a detected touch with the phone that caused it, is to use the phones for user identification. However, the need to rely on computer vision to detect phone touches and to distinguish them from finger touches limits its use to vision based touch screen s. Moreover, since the system is not specifically designed for user identification, association fails when the system detects more than one touch within the same recognition time frame making it very susceptible to attacks.

Classroom orchestration

With the emergence of affordable tablet and tabletop computers, there has been an increased interest in deploying large numbers of single-and multi-user devices in classroom environments [START_REF] Alagha | Towards a teacher-centric approach for multitouch surfaces in classrooms[END_REF], [START_REF] Do-Lenh | Supporting Reflection and Classroom Orchestration with Tangible Tabletops[END_REF], [START_REF] Kharrufa | Tables in the wild: Lessons Learned from a Large-Scale M ulti-Tabletop Deployment[END_REF], [START_REF] Kharrufa | Extending tabletop application design to the classroom[END_REF], [START_REF] Kreitmayer | UniPad: orchestrating collaborative activities through shared tablets and an integrated wall display[END_REF], [START_REF] Kay | M TClassroom and M TDashboard: Supporting Analysis of Teacher Attention in an Orchestrated M ulti-tabletop Classroom[END_REF], [START_REF] Higgins | Interactive Surfaces and Spaces: A Learning Sciences Agenda[END_REF]. The support of teacher orchestration of the classroom has repeatedly been identified as a key challenge [START_REF] Alagha | Towards a teacher-centric approach for multitouch surfaces in classrooms[END_REF], [START_REF] Kharrufa | Tables in the wild: Lessons Learned from a Large-Scale M ulti-Tabletop Deployment[END_REF], [START_REF] Kay | M TClassroom and M TDashboard: Supporting Analysis of Teacher Attention in an Orchestrated M ulti-tabletop Classroom[END_REF]. Teachers are often provided with remote monitoring and control tools and the ability to project the content of one of the devices to a large classroom display.

This approach may be facilitated by providing the teacher with a dedicated device [START_REF] Alagha | Towards a teacher-centric approach for multitouch surfaces in classrooms[END_REF], [START_REF] Kreitmayer | UniPad: orchestrating collaborative activities through shared tablets and an integrated wall display[END_REF], [START_REF] Kay | M TClassroom and M TDashboard: Supporting Analysis of Teacher Attention in an Orchestrated M ulti-tabletop Classroom[END_REF], [START_REF] Higgins | Interactive Surfaces and Spaces: A Learning Sciences Agenda[END_REF]. However, confining acts of orchestration to a single, static device [START_REF] Alagha | Towards a teacher-centric approach for multitouch surfaces in classrooms[END_REF] fails to recognize the real nature of teaching a class, which involves dynamic engagements with the whole group, sub-groups and individuals [START_REF] Reid | Spatial Involvement and Teacher-Pupil Interaction Patterns in School Biology Laboratories[END_REF], [START_REF] Smith | Nonverbal communication in teaching[END_REF]. Alternative proposals include the provision of orchestration functionalities on a teacher's hand -held device such as a tablet [START_REF] Kay | M TClassroom and M TDashboard: Supporting Analysis of Teacher Attention in an Orchestrated M ulti-tabletop Classroom[END_REF], [START_REF] Higgins | Interactive Surfaces and Spaces: A Learning Sciences Agenda[END_REF]. While this improvement was reported to be useful by one teacher [START_REF] Higgins | Interactive Surfaces and Spaces: A Learning Sciences Agenda[END_REF], that same teacher described how having to hold and interact with such a device limited their ability to work with the students directly on their tables. The realities of classroom environments result in a wide range of situations where holding a tablet will restrict the quality of teachers' interaction with students. In recognition of such restrictions and the need for teachers to interact directly with the students' devices, the TinkerLamp project [START_REF] Do-Lenh | Supporting Reflection and Classroom Orchestration with Tangible Tabletops[END_REF] used a 'TinkerKey' tangible object with specific visual markers on it. This was automatically identified by the students' tabletops, to allow teachers to issue special commands. However, TinkerKey relies on optical multitouch sensing and is therefore incompatible with capacitive tablets

SwipeID

SwipeID is a system we have developed to support supervisors, such as teachers, as they interact with one or more touch devices. We leverage a wrist-worn IMU connected through Bluetooth to a nearby SwipeID server to identify the teacher's intera ctions (The server can be any available machine that can con nect to both the IMU and the other devices and that can perform simple correlation calculations). A key assumption is that only the supervisors in a particular context are wearing IMUs on their wrist. Each of the touch-based client devices runs the SwipeID client service. The client software allows access to commands and tools that can only be successfully activated by a user wearing a pre-configured IMU. When a touchscreen user command that requires privileged access is executed, a challenge-response protocol is initiated on-screen, requiring the user to perform a particular sequence of gestures. While interacting with this control, all associated touch data is sent over the network to the server. The server, which is continuously reading IMU sensor data from connected wrist devices, compares this sensor data with the touch data transmitted by the client device and then calculates the correlation between them. If the correlation value is above a predefined threshold, a go -ahead is sent back to the client device. If the correlation is low, a 'reject' is sent back to the client. The client device only communicates with the server when privileged access is requested ensuring that the network is not overloaded during normal usage.

SwipeID has the following properties:

 It allows an arbitrary number of people privileged control of any number of touchbased devices.  It requires no special hardware on the client devices.  It removes the need to try and keep a password secret and the need to remember passwords.

 It requires a small and relatively low-cost wrist-worn IMU such as a smart watch for each authorized person, along with a networked machine acting as a server.

Correlating touch and sensor data

During the challenge-response phase, the magnitude of the acceleration of each touch stroke is derived from both IMU and touch data. In the case of the IMU, the data from the on-board accelerometer, gyroscope and magnetometer must be integrated to d erive the linear acceleration of the device independent of any confounding movement such as rotation. By using the magnitude of the linear acceleration there is no need to consider the relative orientation of the IMU and touch sensor which could vary with time and depending on how the device is worn. The two data streams are then matched using a correlation-in-time function [START_REF] Fujinami | A study on a correlation coefficient to associate an object with its user[END_REF] (Equation 1) and the resulting probability is thresholded. The touch and IMU data are collected at the same rate , 33Hz. The data is resampled to account for the intermittent sampling latencies inherent in the devices used. The X and Y touch data is differentiated twice to calculate acceleration. The data is filtered using a 5-sample moving average before and after each derivative calculation, and the magnitude of the acceleration is calculated for both the touch (d1) and linear acceleration (d2) data. The data correlation was calculated as follows:

A network based challenge-response approach

SwipeID relies on the correlation between the acceleration calculated from the touch data and that measured by the IMU. Accordingly, in theory any user could attempt to trigger a privileged command as authentication can only be carried through performing some gestures aiming to achieve the correlation threshold. This means that without imposing constraints on the type of gestures required for authentication, a user not wearing an IMU could attempt to trigger a privileged command concurrently with a supervisor (who is working on another device), and mirror the movement pattern of the supervisor. If the unauthorized user were to do this well enough, there is a poss ibility that the correlation calculated between their touch data and that of the IMU on the supervisor's wrist exceeds the threshold and might even be higher than that of the supervisor -due to measuring acceleration at the wrist which does not map perfectly to the touch data at the fingertip.

To prevent such attacks, a challenge-response approach is used. The server has a pool of distinct challenges equal to or larger than the number of devices in use at any one time. If two users try to gain privileged access at the same time, they will be a ssigned two different challenges that require different movement patterns . If unauthorized users try to copy the movement pattern of the person wearing the IMU, they will have to deviate from the movement pattern required by their own challenge and thus fail their challenge locally regardless of how well their movement correlates wit h that of the IMU. Accordingly, the only option users have to gain access to privileged commands is to both accurately follow their own challenges be wearing the IMU.

To best design a set of challenge patterns that ensure distinct movement patterns which will not inadvertently cause high correlation between different challenges, we looked at the basic acceleration signals generated from touch data for the most primitive strokes (Table 1). From these graphs we can see that by interleaving sho rt strokes and periods of no movement, it is possible generate a number of distinct patterns. If we represent a movement by M and a pause by P, we can design as many different challenges as desired. We want to keep the sequence short to keep it quick to en ter, but to decrease the correlation between the different patterns we decided to choose challenges that differ in at least two segments. Table 2 lists 12 different challenges which meet this criteria and also have a minimum of three movements and one paus e.

Designing sensor-coupled user controls

To require a user to follow a specific challenge, a custom user interface control must display it and verify the response. We would like these controls to:

1. be user friendly, i.e. feel simple, be quick to use and not too demanding; 2. support the proposed challenge-response protocol; and 3. result in sufficient movement to generate high correlation.

To aim for user friendliness (goal 1), we avoided controls that increase cognitive load, such as requiring users to trace a dynamic path that appears incrementally or changes in real time. We limited the design space to only include controls that simply require users to move along a path or to move to clearly marked targets. With such controls, it is possible to enforce a pattern that results in different levels of mov ement and to verify that the user is following the displayed challenge (goal 2).

To meet goal 3 we considered two options for imposing distinct changes in the speed of movement:

Present a sequence of targets one at a time. The user is required to move to t he next target within a certain time otherwise the challenge fails. Once a target is reached, the subsequent target appears, either immediately or after a certain period, depending on the challenge. This enforces a distinctive movement/no movement pattern. We call this 'discrete' interaction as there are explicit wait periods. Note that the user is not aware of the full challenge pattern beforehand.

Show the full challenge, represented by a certain path to navigate through, from the outset. The required pattern of movements is achieved by switching between a wide straight paths which can be navigated through quickly, and shorter narrow and/or curvy paths that require more careful (and slow) navigation. This idea is derived from Accot and Zhai's work [START_REF] Accot | Beyond Fitts' law: models for trajectory -based HCI tasks[END_REF] that looked at the relationship between movement time and width, and to a lesser extent curvature, of a path to steer through. We refer to this as 'continuous' interaction because no pauses are expected. In this case the users can see the full challenge from the outset. Table 3. Design space for four different IM U-coupled user interface control types. The shapes in the second row are one of 12 different challenge shapes.

Sma l l footpri nt

La rge footpri nt Di s crete Cha l l enge

Conti nuous Chal l enge

It is hypothesized that users associate different levels of ease of use to options (a) and (b) because of the different style of interaction used. In particular, the use of continuous movement and exposure to the full challenge beforehand is assumed to help in perceiving the interaction as one action (or chunk [START_REF] Buxton | Chunking and phrasing and the design of human-computer dialogues[END_REF]) rather than a number of discrete actions. Pauses or very slow movement in a straight wide path area results in failing the challenge. Moreover, as long as the user does not go outside the path they can speed up the response for a continuous challenge by navigating more quickly. This is not the case with the discrete challenge as the user must wait for each target to appear.

We need to ensure sufficient wrist movement to generate meaningful IMU data. We hypothesized that asking users to perform the task as quickly and as accurately as possible would help with this. We also need to avoid finger-only movements. For this reason, we wanted to compare gestures which use predominantly one axis of mov ement with those which require movement in two dimensionswith both cases requiring movements above a certain minimum physical distance. The full design space is summarized in Table 3.

For discrete, small footprint interactions, a horizontal path is presented with the next target to move to indicated as a circular region. The following target either a ppears immediately after reaching this target or after a certain pause, depending on the challenge. Similarly, for the large footprint discrete option, a rectangular shape is displayed with the target appearing at one of the four corners.

For continuous interaction, the control is a path that changes between straight, wide segments and narrow curvy segments that the user needs to navigate (steer) through. The path, which ends with a clearly-marked target, is designed to either stretch mostly along one axis (small footprint) or along both (large footprint).

We envisage privileged commands to be presented as normal buttons that expand once touched. When a small discrete challenge button is touched for example, it expands horizontally to the required width showing the next target . The user must then slide their finger without removing it to the next target and so on. Upon sliding to the last point in the challenge, which is the same as the starting point, the button collapses. The button also collapses immediately if the user fails to follow the challenge.

Figure 2 shows the magnitude of acceleration signal at the server side calculated from touch data and that from the IMU for two different challenges. These signals, A pause-move-move-pause-move-pause challenge (PMMPMP).

A move-pause-move-pause-move-pause challenge (MPMPMP).

4

User study

We conducted a study as a proof of concept for SwipeID and to compare the perfo rmance and user perception of the four different control types: discrete small footprint, discrete large footprint, continuous small footprint, and continuous large footprint. A repeated measures design was chosen to directly compare participants' performance with each control. We recruited 19 participants for the study (mean age = 26 years; all right handed). Inclusion criteria included good (correctable) vision and some experience with touch-based devices such as smartphones or tablets. The touch device used was a 10.1" Windows 8 tablet. A commercial IMU, LPMS-B from Life Performance Research was used in the study and was fixed on the participants' wrists using a sweatband (Figure 3). This device supports on -board integration of accelerometer, gyro and magnetometer data to calculate linear acceleration. The tablet was placed horizontally on a table and the participants interacted with it from a seated position. All the controls had a physical length of 15.9cm and the large foo tprint controls had a height of 12.0cm.

Initially participants were briefed regarding the purpose of the study and were shown the four control types. They were then given two practice challenges for each control before commencing the study proper. The presentation order for the four co ntrols was counterbalanced and the 12 challenges were rando mized for each control (see Table 2 for challenges). Participants were asked to complete each challenge as quickly and accurately as possible. If the participant completed the challenge correc tly they were presented with a green feedback screen, whereas a red screen was presented if the challenge was not completed correctly and they were required to retry the challenge until successful. All touch and IMU data from each trial was automat ically logged for subsequent analysis.

Upon completing the challenges for each of the four controls, participants were presented with a six-item questionnaire exploring their preferences and perceptions regarding the proposed controls. Four of the questions directly asked the participants to rate their experience using each of the four configurations using a scale ranging from Very Good to Bad. The other questions asked participants to write down their preferred configuration and the configuration they would like to avoid, along with explanations for each selection. Participants were fully debriefed upon completing the questionnaire.

Results

We performed data analysis to determine the efficacy of our challenge-response protocol. To do this we calculated (i) the correlation between corresponding t ouch and IMU data and (ii) the similarity between touch data for a challenge performed by the person wearing the IMU and touch data from other challenges. Table 4 shows the average correlation between touch data and IMU data for the different control types as well as the average time to respond to a challenge and the average number of failed attempts to respond to the challenge out of the total of 12 trials. Failure occurs when the user does not follow the challenge accurately, leading to rejecting the response locally without the need to send data to the server. Upon failure, the user had to repeat the challenge. The table shows that the discrete controls resulted in higher average correlation, shorter average time to completion and a lower number of failed attempts on average than the continuous controls. The receiver ope rating characteristic (ROC) curves for the four controls (Figures 4 and5) show that the discrete controls have better performance than the continuous controls. The ROC curves show that for the discrete controls a threshold of 0.6 allows for 86% (large footprint) to 88% (small footprint) true positives and 0.2% (small footprint) to 0.4% (large footprint) false positives. As for the continuous controls, while the curve still shows that the controls perform well in separating true positives from false positives, a threshold of 0.5 is probably the best choice, which results in 59% (small footprint) to 64% (large footprint) true positives but also allows for 5% false positives. The results show that participants were faster and failed less challenges when using the discrete small footprint controls, while the mean correlation was also the highest. The discrete large footprint control ranked second overall in the three categories, with continuous small footprint third. The continuous large footprint control performed the worst of all the conditions.

The results also show that the large footprint with horizontal, vertical and diagonal movements did not clearly result in better performance than the small footprint with mostly horizontal movements. In the discrete controls case the small footprint pe rformed marginally better than the large footprint control in terms of correlation, time, and failed challenges. In the continuous controls case the large footprint control performed slightly better in terms of correlation and errors, but worse with regards to time. For the discrete control case, one reason why the large footprint control performed slightly worse than its small counterpart may have been the result of partici-pants having to sometimes lift their wrist to see if the next target had been obscured by their hands. This could have led to unwarranted wrist movements and thus IMU acceleration that did not correspond to any touch movement. Such a scenario would result in a reduced correlation between the touch and the IMU data. For the continuous interaction-large footprint control, participants were tracing a path gradually thus no such wrist movements were observed. The higher average correlation in the continuous interaction-large footprint control compared to its small counterpart may be due to the fact that the horizontal only movement could be performed with more finger movement and less wrist movement as compared to a gesture performed in both horizontal and vertical dimensions.

Participant Feedback

All the participants were asked to answer a simple questionnaire to provide subjective 'experience ratings' regarding the four control types used in the study. Participants were asked to rate their experience of using each of the four controls on a scale from 1 (bad) to 4 (very good). Additionally, participants were asked to select the control they would prefer to use on a daily basis and also indicate the control they most wanted to avoid. Below we present the findings from the questionnaire. After aggregating the participants' experience rating scores for each of the four techniques, we found that the ratings for three of the controls were very close. The discrete small footprint came top with a rating of 3.3, while the discrete large foo tprint and continuous small footprint controls shared a score of 3.2. The continuous large footprint technique received the lowest rating: 2.8.

Despite the discrete small footprint control receiving the highest experience rating, the most popular control amongst participants (the one most participants selected as their favorite for daily use) was the discrete large footprint technique (see Table 5). This control was perceived to be fast whilst yielding a low number of errors, but participants also noted, as we have observed as well, that some angles could be obscured at times by the placement of the hand (e.g. when target is located in lower-right hand corner) and waiting for the dot to move could become tedious. This may help explain why, despite being the preferred technique by some, it also had a higher avoidance percentage than both the discrete and continuous small footprint controls by others . The general consensus regarding the discrete small footprint control, the tied-second most popular technique, was that this technique was fast, easy and intuitive. With regards to the continuous controls, participants agreed that the small footprint technique was fast, but some argued that it was potentially too much work for ident ification. However, while the small footprint control shared almost the same prefe rence rating as both types of discrete challenge controls, the large footprint version did not obtain any preference votes, and received the majority of avoidance votes . Users mentioned that the technique required too much concentration (i.e., was too much work) and caused too many errors resulting in a 'frustrating' experience. This was a somewhat surprising considering that it had slightly lower failure rate and slightly higher average correlation than the small footprint version, although the average completion time was slightly longer.

The results of our questionnaire show that despite quantitatively being the fastest, leading to fewer errors, and being the highest rated control in terms of experience, the discrete small footprint control was only the second most popular choice amongst participants. However, it is unclear why this was the case as there was no reported negative feedback on this technique. It may be a case of participants' perception of the large footprint being more suitable for the identification task. Clearly, however, participants did not favor the continuous large footprint technique.

Discussion and future work

SwipeID can identify any user wearing the IMU on any touch -based device without the need for special hardware on the touch device, and at the point of contact on the screen. This approach can be a practical solution for devices that do not have a fingerprint or NFC reader and, most immediately, we envision it as a practical solution for supervisor authentication to iPads already in classroom use. Accordingly, we view the decision of using SwipeID as one of practicality rather than based on performance measures alone when compared to NFC, biometric, or password authentication. For example, with devices that do not have NFC or biometric, password is the most likely alternative but, as we have discussed earlier, passwords are not as practical with touch devices in public settings as they are more prone to shoulder surfing especially when it is not possible to use the device in a private environment. Moreover, even with devices that do have NFC or biometric identification, SwipeID could be useful in scenarios where identification is required at a specific point on a large screen where it may be impractical to move to a specific location for biometric or NFC identification. This includes collaborative work on a large classroom whiteboard where passcodebased authentication procedures are even more visible and could be easily compromised, or when collaborating around digital tabletops for example where issues of reach caused by the large surface render other techniques impractical. With SwipeID authentication is done at the point of contact using simple gestures. The combined use of a challenge-response protocol and our special user control ensure that only the user wearing the IMU is able to successfully respond to the challenge. While this approach does have a certain level of technical complexity, from users' perspective it is a simple technique that is unobtrusive and makes only few assumptions on the user's part.

One of the motivations for this work was to improve the ways in which teachers can interact with students and their touch-based devices in a classroom setting. We note that SwipeID offers a number of opportunities in this space, given that it allows for freedom of movement around the class and allows for device or scree n-specific authentication at the point of contact. It also removes the need for special hardware on the students' devices, the use of a secret passcode, or having to issue commands through a separate fixed or handheld computer. As we noted in the introduct ion, tablets are increasingly used in the classroom, but their uptake is currently being limited by known access management concerns. Better tools for user identification are therefore an important educational issue. However, considering that even with dis crete controls, authentication gestures took 4-5 seconds, this may be perceived to be too long in a classroom context. Exploring the use of SwipeID in a real classroom setting and exploring options for reducing the required gesture time for authentication are two important areas for future investigation.

SwipeID can also be used in other scenarios where supervisor access of touchbased interactions needs to be identified. For example, in a retail context a sales assistant may need to configure the information displayed in-store, while customers are only allowed to browse the information. Alternatively, in a museum a staff member might need to dynamically update information displayed on interactive terminals. Moreover, SwipeID lends itself well to devices with large or fixed interactive displays where it is not possible to interact with the display privately without being prone to shoulder surfing.

In some applications, multiple users share touch-based devices or need to interact collaboratively on a large interactive surface. In these cases it can be beneficial to identify each specific individual in a way that is resistant to shoulder-surfingagain, a known problem with traditional authentication on touch screen devices. If each user is able to use a wrist-worn motion sensor then SwipeID can be used to identify them.

Another possibility for future work is to explore the use of SwipeID with mouse interactions for user authentication (rather than just verifying user presence as in Shrirang et al. [START_REF] Shrirang M Are | ZEBRA: Zero-Effort Bilateral Recurring Authentication[END_REF]). This could allow the use of a single system as a universal authentication solution across a full range of devices within a particular contexteducational or otherwise. However we should note that a key challenge with using a mouse is that the user can perform relatively large gestures on the screen without a significant movement of the wrist. The level of correlation between the physical movement of the mouse and the movement of the mouse pointer is also highly dependent on the gain setting of the mouse so this would also need to be factored in.

Our user study aimed to demonstrate the validity of our approach and to evaluate the performance and perception of four proposed controls. It showed that discrete controls performed better than continuous contro ls in all regards. However, we observed that when users failed a challenge and had to repeat it for the continuous co ntrols case, they performed the gesture faster which resulted in better correlation for that challenge. This appears to support the theory that it is possible for users to improve their performance over time with continuous controls, giving them a longer term advantage over the discrete controls. In future work it may be possible to further optimize the continuous controls to improve performance.

A limitation of our study was that it was conducted in a lab environment rather than in one of the contexts within which we claim its utility. Future work needs to investigate its longitudinal use in an 'in-the-wild' environment where supervisors will be interacting with devices of different form factors and from different seating and standing positions. A longitudinal, in the wild evaluation will also help in gaining better understanding of the user's preferences and may explain some of the discrepa ncies between preferences and performances recorded in Tables 4 and5. It can also show whether the measured failure rate can improve with repeated use and whether the level it settles at can cause annoyance to users or not.

SwipeID user identification is reliant on a wrist worn IMU. In other words, the framework identifies the IMU and not the actual user. This means that whoever has the IMU will have privileged access regardless of the actual identity of the user (relying on what the user has rather than what the user knows). This means that the main threat for the system is getting access to the IMU by unauthorized users. This threat is common with other techniques that rely on a hardware key for authentication such as NFCs. Designing the system to overcome this limitation was outside the scope of the work reported here, but one possible solution would be to require a secret passcode to be entered into the wrist-worn IMU prior to use. The user would only be required to enter the passcode when the wrist-band with the IMU is first put on, which will normally just be once and could be done discreetly (thus adding the requirement of something the user knows as well). An alternative to the passcode is to take advantage of the motion sensors and use behavioral biometric to continuously authenticate the person wearing the sensor.

With the increasing popularity of wrist-worn devices incorporating motion sensors, such as smart watches, the need for a dedicated wrist-worn sensor could ultimately be eliminated. While our current work uses linear acceleration data from the IMU which is derived from data collected from an accelerometer, a gyroscope and a magnetometer, future work will investigate the use of accelerometer data only, thus eliminating the need for the gyroscope and the magnetometer, reducing cost and increasing co mpatibility with wrist-worn consumer devices.

Conclusion

In this paper we considered the problem of identifying supervisors across multi-touch devices and gave an illustrative classroom scenario. We proposed SwipeID, a system that works across any touch device and which provides an IMU to supervisors and searches for correlations between the IMU and touch events on known devices to identify the display being used by a supervisor. In our user study we explored IMUcoupled user interface controls to accompany the proposed system and found that requiring the user to perform discrete gestures, on small interface controls enabled us to identify the best correlation between touch and IMU data. However, the time required to complete the authentication process may be perceived to be too long in ce rtain contexts, thus reducing authentication time is identified as an area for further investigation. We propose that SwipeID can also be used across multiple devices, that it has significant classroom potential and that future work can explore its applicability with other input techniques and its relevance to other contexts.

  Stroke typeMagnitude of touch data acceleration vs time Si ngl e, s tra i ght s troke A 2 s troke ges ture wi th a 180 o cha nge i n di recti on A 2 s troke gesture with 90 o cha nge i n di re cti on A 2 s troke gesture with no cha nge i n di re cti on, but a s mall pause between the s trokes A 2 s troke gesture with 180 o cha nge in directi on a nd s mall paus e between the s trokes

Fig. 2 .

 2 Fig. 2. The magnitude of acceleration signal from touch (black) and IM U (red) for two different challenges using the discrete, small footprint control.

Fig. 3 .

 3 Fig. 3. User study. Discrete, small footprint control (left). Continuous, large footprint control (right).

Fig. 4 .Fig. 5 .

 45 Fig. 4. Discrete small footprint and discrete large footprint receiver operating characteristic curves.

Table 2 .

 2 Twelve different challenges that combine movements (M ) and pauses (P). (See Figure2for example shapes of signals associated with these challenges)

	M	P	M	P	M	P
	M	P	M	P	P	M
	M	P	P	M	M	P
	M	P	P	M	P	M
	P	M	M	P	M	P
	P	M	M	P	P	M
	P	M	P	M	M	P
	P	M	P	M	P	M
	M	M	P	P	M	M
	P	M	M	M	M	M
	M	M	M	M	M	P
	M	M	M	M	P	M

Table 1 .

 1 The magnitude of acceleration from touch data for basic gestures.

Table 4 .

 4 M ean (and SD) for correlation, execution time and number of failed challenges across the four different control types proposed.

	Control type	Mea n correl a-ti on	Mea n ti me i n ms ec	Mea n no. of fa i l ed cha l l enges
	Di s crete, s ma l l footpri nt	0.77 (0.08)	4800 (435)	1.58 (1.77)
	Di s crete, l a rge footpri nt	0.71 (0.07)	5280 (362)	1.79 (2.37)
	Conti nuous, small footpri nt	0.49 (0.11)	7030 (1800)	6.47 (5.47)
	Conti nuous, la rge footpri nt	0.53 (0.07)	7130 (1980)	5.53 (3.01)

Table 5 .

 5 Summary of participants' control preference ratings

	Control	Preferred Choice (%)	Avoid (%)
	Discrete Small Footprint	31.6	10.5
	Discrete Large Footprint	36.8	21.1
	Continuous Small Footprint	31.6	10.5
	Continuous Large Footprint	0	57.9

Acknowledgments

This work was supported by the RCUK Digital Economy Programme-SIDE: Social Inclusion through the Digital Economy EP/G066019/ 1


Eduardo Velloso
email: e.velloso@lancaster.ac.uk

Jayson Turner
email: j.turner@lancaster.ac.uk

Jason Alexander
email: j.alexander@lancaster.ac.uk

Andreas Bulling
email: andreas.bulling@acm.org

Hans Gellersen

An Empirical Investigation of Gaze Selection in Mid-Air Gestural 3D Manipulation

Keywords: 3D User Interfaces, Eye Tracking, M id-Air Gestures

In this work, we investigate gaze selection in the context of mid-air hand gestural manipulation of 3D rigid bodies on monoscopic displays. We present the results of a user study with 12 participants in which we compared the performance of Gaze, a Raycasting technique (2D Cursor) and a Virtual Hand technique (3D Cursor) to select objects in two 3D mid-air interaction tasks. Also, we compared selection confirmation times for Gaze selection when selection is followed by manipulation to when it is not. Our results show that gaze selection is faster and more preferred than 2D and 3D mid-air-controlled cursors, and is particularly well suited for tasks in which users constantly switch between several objects during the manipulation. Further, selection confirmation times are longer when selection is followed by manipulation than when it is not.

Introduction

Interaction fidelity-the degree with which the actions used for a task in the UI correspond to the actions used for that task in the real world [START_REF] Bowman | Questioning Naturalism in 3D User Interfaces[END_REF]-is an active topic of research in 3D user interfaces (3DUI). Interfaces based upon free-space spatial input (e.g. mid-air gestures, tilt and turn gestures, magnetic trackers, etc.) offer this fidelity for 3DUI due to their multiple degrees of freedom and high integration of dimensions of control (i.e. many degrees of freedom can be controlled simultaneously with a single movement) [START_REF] Bowman | 3D user interfaces: theory and practice[END_REF] [START_REF] Hinckley | A survey of design issues in spatial input[END_REF]. In particular, recent advances in unobtrusive motion capture (e.g. Kinect, Leap Motion) created a renewed interest in mid -air gestures for 3DUI.

In immersive virtual reality environments and on stereoscopic displays, such interactions allow users to manipulate virtual objects using interaction metaphors that relate more closely to real world interactions, for example, by using an isometric mapping between the virtual and physical spaces, users can reach virtual objects directly where they see them. However, a large number of 3D activities, such as gaming, graphic design and 3D modelling are still mostly conducted on conventional monoscopic desktop displays. This setup creates a discontinuity between the p hysical and the virtual environments, and therefore does not allow users to directly grasp objects in three dimensions. In this desktop context, common mid-air interaction techniques for 3D selection are Raycasting (in which the user's hand controls a 2D point that determines the direction of pointing) and the Virtual Hand (in which the user controls a 3D representation of his hand and makes selections by intersecting it with virtual objects) [START_REF] Bowman | 3D user interfaces: theory and practice[END_REF]. See Argelaguet et al. for a survey of selection techniques for 3D interaction [START_REF] Argelaguet | A survey of 3D object selection techniques for virtual environments[END_REF].

As the eyes provide a natural indication of the focus of the user's interest, eye trackers have been used for pointing in a wide variety of contexts without necessarily requiring a representation on the screen, showing higher speeds than conventional techniques [START_REF] Sibert | Evaluation of eye gaze interaction[END_REF]. Even though gaze pointing for computing input has been investigated since the 80's [START_REF] Ware | An Evaluation of an Eye Tracker As a Device for Computer Input[END_REF], studies on gaze pointing for 3DUI started with work by Koons et al., who built a multimodal interface integrating speech, gaze and hand gestures [START_REF] Koons | Integrating Simultaneous Input from Speech, Gaze, and Hand Gestures[END_REF]. Early work was also conducted by Tanriverdi and Jacob, who found it to be faster than an arm-extension technique with a 6DOF magnetic tracker in a VR environment [START_REF] Tanriverdi | Interacting with eye movements in virtual environments[END_REF]. Cournia et al. found conflicting results that suggest gaze is slower than a handbased Raycasting technique with a wand [START_REF] Cournia | Gaze-vs. hand-based pointing in virtual environments[END_REF]. These works only investigated selection tasks, but in practice, common 3D interaction tasks involve further manipulation steps after selection, such as translation and rotation. Given that gaze alone is impractical for all steps, several works combined gaze with additional modalities, but few explored the context of 3D user interfaces . In particular, when using gaze for selection and mid-air gestures for 3D manipulation, is there a cost in performance in switching modalities?

Even though gaze has been explored in a variety of multimodal configurations [START_REF] Stellmach | Gaze-supported Multimodal Interaction[END_REF], few works explored the combination of gaze and mid-air gestures. Kosunen et al. reported preliminary results of a comparison between eye and mid -air hand pointing on large screen in a 2D task that indicate that pointing with the eyes is 29% faster and 2.5 times more accurate than mid-air pointing [START_REF] Kosunen | Comparing eye and gesture pointing to drag items on large screens[END_REF]. Hales et al. describe a system in which discrete hand gestures issued co mmands to objects in the environment selected by gaze [START_REF] Hales | Interacting with Objects in the Environment by Gaze and Hand Gestures[END_REF]. Pouke et al. investigated the combination of gaze and mid-air gestures, but in the form of a 6DOF sensor device attached to the h and [START_REF] Pouke | Gaze tracking and non -touch gesture based interaction method for mobile 3D virtual spaces[END_REF]. They compared their technique with touch, and found that the touch -based interaction was faster and more accurate.

The conflicting results in the literature highlight the importance of further work investigating gaze selection for 3DUI, particularly considering that technical advances made eye tracking technology significantly more accurate, precise and robust than the devices and techniques used in previous works . In this work, we present an investigation of gaze selection for mid-air hand gestural manipulation of 3D rigid bodies in monoscopic displays. We conducted a study with three tasks. In the first task, we compared three 3D interaction techniques for selectio n and translation: a 2D cursor controlled by the hand based on Raycasting, a 3D cursor controlled by the hand analogous to a Virtual Hand and Gaze combined with mid -air gestures. In the second task, we also compared the same three techniques but in a selection and translation task involving multiple objects. In our pilot studies we found that when participants used the Gaze + Mid-Air Gestures technique, they reached out for objects even though they did not have to. We hypothesised that this action was due to the clutching required for manipulation. To test this hypothesis, users performed a third task, in which we compared the selection time in the case where users were only required t o select an object to the case where they also had to translate the object after selecting it.

Our results show that gaze selection is faster and more preferred than conventional mid-air selection techniques, particularly when users have to switch their focus between different objects. We also discovered a significant difference in the time to pinch after the object was gazed at between selection only tasks and selection followed by translation, indicating that the context of the selection impact the selection confirmation time.

Related Work

Human Prehension

Prehension is formally defined as "the application of functionally effective forces by the hand to an object for a task, given numerous constraints" [START_REF] Mackenzie | The grasping hand[END_REF], or more informally as the act of grasping or seizing. Different authors proposed ways of modelling this process. In Arbib's model, the eyes (perceptual units), arms and hands (motor units) work together, but under distributed control to reach and grasp objects [START_REF] Arbib | Perceptual structures and distributed motor control[END_REF] [START_REF] Mackenzie | The grasping hand[END_REF]. The perceptual schema uses the visual input provided by the eyes to locate the object and recognise its size and orientation. The motor schema can be divided into two stages: reaching (comprised of a quick ballistic movement followed by an adjustment phase to match the object's location) and grasping (including adjusting the finger and rotating the hand to match the object's size and orientation , followed by the actual grasping action). Paillard's model begins with the foveal grasping, in which the head and the eyes position themselves towards to object . Then, according to shape and positional cues, the arms and hands locate and identify the obje ct, in open and closed loops, until finally grasping it, performing mechanical actions and sensory exploration [START_REF] Paillard | Le corps situé et le corps identifié[END_REF] [START_REF] Mackenzie | The grasping hand[END_REF].

In the context of mid-air gestures for 3D user interfaces, reaching is analogous to selection and grasping to the confirmation of the selection. In this work, we investigate how human prehension can be supported in a desktop monoscopic 3D environment. In all conditions we studied, grasping (confirmation) was performed by a pinch gesture, similar to how we would grasp physical objects , but the selection step varied across conditions . The 3D cursor includes a reaching step similar to normal prehension, only offset due to the discontinuity between the virtual and physical worlds. The 2D cursor also contains a reaching step, but only in two dimensions. The Gaze condition only requires foveal grasping, as when the user looks at the object, she only needs to pinch to confirm the selection. However, as we show in the results of task 3, when the user grasps the object for further manipulation, she still reaches out for it.

Mid-Air Interaction for 3D Manipulation

Due to our familiarity in manipulating physical objects with our hands, a considerable effort of the HCI community has been put into developing input devices and interaction techniques that leverage our natural manual dexterity to interact with digital content.

An important interaction paradigm in 3D interaction is isomorphism: a strict, geometrical, one-to-one correspondence between hand motions in the physical and virtual worlds [START_REF] Bowman | 3D user interfaces: theory and practice[END_REF]. Even though isomorphic techniques are shown to be more natural, they suffer from the constraints of the input device (e.g. the tracking range of the device) and of human abilities (e.g. the reach of the arm). When targets are outside the user's arm reach, techniques such as Go-Go [START_REF] Poupyrev | The Go -go Interaction Technique: Non-linear Mapping for Direct Manipulation in VR[END_REF] and HOMER [START_REF] Bowman | An evaluation of techniques for grabbing and manipulating remote objects in immersive virtual environments[END_REF] can be used to extend the length of the virtual arm [START_REF] Argelaguet | A survey of 3D object selection techniques for virtual environments[END_REF]. Other works have explored different modalities for 3D interaction, including feet movements [START_REF] Simeone | Feet Movement in Desktop 3D Interaction[END_REF], tangible interfaces [START_REF] Kitamura | Real-time 3D Interaction with ActiveCube[END_REF], and computer peripherals (e.g. 3D Connexion SpaceNavigator).

Gaze in Multimodal Interactions

Gaze-based interaction is known to suffer from a few challenges [START_REF] Stellmach | Still looking: investigating seamless gaze-supported selection, positioning, and manipulation of distant targets[END_REF]: inaccuracy (due to the jittery nature of eye movements and technological limitations), double -role of visual observation and control, and the Midas Touch problem (the unintentional activation of functionality due to eye tracking being always -on [START_REF] Jacob | What you look at is what you get: eye movement -based interaction techniques[END_REF]). To address these problems gaze is usually combined with other input modalities and devices.

Stellmach et al. investigated combinations of gaze with a wide variety of modalities [START_REF] Stellmach | Gaze-supported Multimodal Interaction[END_REF], including a keyboard [START_REF] Stellmach | Designing gazesupported multimodal interactions for the exploration of large image co llections[END_REF], tilt gestures [START_REF] Stellmach | Designing gazesupported multimodal interactions for the exploration of large image co llections[END_REF][24], a mouse wheel [START_REF] Stellmach | Investigating gaze-supported multimodal pan and zoom[END_REF], touch gestures [START_REF] Stellmach | Investigating gaze-supported multimodal pan and zoom[END_REF][25] [START_REF] Stellmach | Still looking: Investigating seamless gaze-supported selection, positioning, and manipulation of distant targets[END_REF] and foot pedals [START_REF] Göbel | Gazesupported foot interaction in zoomable information spaces[END_REF]. A common interaction paradigm in gaze-based interaction is that of gaze-supported interaction-gaze suggests and the other modality confirms [START_REF] Stellmach | Look & touch: gaze-supported target acquisition[END_REF]. An example of a gaze-supported interaction technique is MAGIC pointing, which warps the mouse cursor to the area around the gaze pointing [START_REF] Zhai | Manual and gaze input cascaded (M AGIC) pointing[END_REF]. Fine positioning and selection confirmation are performed normally with the mouse.

These works have shown that multimodal gaze-based techniques are intuitive and versatile enough to work in a wide variety of contexts, ranging from small mobile devices to large public displays [START_REF] Turner | Eye pull, eye push: Moving objects between large screens and person al devices with gaze and touch[END_REF].

In this work, we have a similar goal to Stellmach and Dachselt, that of seamless selection and positioning [START_REF] Stellmach | Still looking: Investigating seamless gaze-supported selection, positioning, and manipulation of distant targets[END_REF]. Whereas in their work, they achieved this with different touch-based techniques for mobile devices, the context of 3D user interfaces requires extra degrees of freedom that are better suited for mid-air gestures.

Gaze and Mid-Air Gestures

Kosunen et al. reported preliminary results of a comparison between eye and midair hand pointing on large screen in a 2D task that indicates that pointing with the eyes is 29% faster and 2.5 times more accurate than mid-air pointing [START_REF] Kosunen | Comparing eye and gesture pointing to drag items on large screens[END_REF]. The techniques investigated in their paper are analogous to our 2D Cursor and our Gaze technique, as they also used a pinch gesture for selection confirmation. In this paper, we extend their work to 3D manipulation, also comparing them to a 3D cursor. Also, their task involved 2D translation of objects, whereas ours involves 3D translation.

Pouke et al. investigated the combination of gaze and mid-air gestures, but in the form of a 6DOF sensor device attached to the hand [START_REF] Pouke | Gaze tracking and non -touch gesture based interaction method for mobile 3D virtual spaces[END_REF]. Their system supported tilt, grab/switch, shake and throw gestures. They compared their technique with a touchbased one, and found that the touch-based interaction was faster and more accurate, mainly due to accuracy issues with their custom-built eye tracker. We aimed to minimise tracker accuracy problems, by using a commercial eye tracker with a gaze estimation error of 0.4 degrees of visual angle. Our study also differs from theirs in that the mid-air gestures investigated by them were based on a tangible device, rather than hands-only gestures.

Yoo et al.'s system tracked the user's head orientation (as an approximation for the gaze point) and the 3D position of the hands for interacting with large displays [START_REF] Yoo | 3D user interface combining gaze and hand gestures for large-scale display[END_REF]. Bowman et al. investigated pointing in the direction of gaze, but also approximating it to the head orientation [START_REF] Bowman | 3D user interfaces: theory and practice[END_REF]. Such approximations only work when the user is looking straight ahead. Hence they are only suitable for large scale interactions, such as with large displays and fully immersive virtual environments. In a desktop setting, the head orientation is not a good approximation for the point of regard , as the user is constantly facing the same direction.

Cha and Maier proposed a combination of gaze and mid-air gestures for a multidisplay use case [START_REF] Cha | Eye gaze assisted human-computer interaction in a hand gesture controlled multi-display environment[END_REF]. These authors presented architectural implementation details of their system, but did not present any evaluation results or interaction design decisions.

Gaze and 3D User Interfaces

Stellmach and Dachselt proposed two ways in which 3D user interfaces can benefit from eye tracking: first, understanding how users visually perceive 3D scenes can assist the design of new 3DUIs and second, eye trackers can be used for direct control of 3DUIs [START_REF] Stellmach | Looking at 3D User Interfaces[END_REF].

Examples of the first group of applications include studying players' gaze patterns in 3D video games to improve level design and graphics [START_REF] El-Nasr | Visual attention in 3D video games[END_REF], to improve gaze behaviour and body animations of virtual agents [START_REF] Vinayagamoorthy | An eye gaze model for dyadic interaction in an immersive virtual environment: Practice and experience[END_REF] and to enhance the rendering of depth-of-field blur effects [START_REF] Hillaire | Using an eye-tracking system to improve camera motions and depth-of-field blur effects in virtual environments[END_REF].

In the second group are applications controlled directly by gaze. In the past few years, companies such as Tobii and SMI started marketing eye trackers for the wider consumer market aimed primarily at gaming, which stimulated developers to create the first commercial gaze-enabled games [START_REF] Turner | EyePlay: applications for gaze in games[END_REF]. The research community has also demonstrated several examples of interaction techniques and game prototypes in this context (see Sundstedt for an overview [START_REF] Sundstedt | Gazing at games: using eye tracking to control virtual characters[END_REF]).

The popularity of head-mounted displays such as the Oculus Rift created renewed interest in exploring eye tracking within Virtual Reality. Tanriverdi and Jacob compared selection time and the users' ability to recall spatial information in a VR between two techniques: the gaze position and pointing with a ma gnetic tracker [START_REF] Tanriverdi | Interacting with eye movements in virtual environments[END_REF]. They found that the gaze technique was significantly faster, but led to more difficulties in recalling the locations of items they had interacted with. On the other hand, Cournia et al. compared gaze and hand pointing in VR and found hand pointing to perform better [START_REF] Cournia | Gaze-vs. hand-based pointing in virtual environments[END_REF]. Following Cournia et al.'s recommendation, we implemented our 2D cursor using raycasting, as it seemed to outperform arm extension techniques (such as the one used by Tanriverdi and Jacob [START_REF] Tanriverdi | Interacting with eye movements in virtual environments[END_REF]). These works, however, investigate 3D interaction in an immersive VR environment, whereas we use a monoscopic display. Duchowski et al. also investigated eye tracking in virtual reality in applications ranging from monitoring users' gaze for aircraft inspection training [START_REF] Duchowski | Binocular eye tracking in virtual reality for inspection training[END_REF] to providing a visual deictic reference in collaborative environments [START_REF] Duchowski | Visual deictic reference in a collaborative virtual environment[END_REF].

Experimental Setup

We recruited 12 right-handed participants (6M/6F), aged between 20 and 43 years (median=28). Three wore glasses and one wore contact lenses in the study. Figure 1 shows our experimental setup. Participants sat in front of an 18'' laptop running a custom application built in the Unity game engine. Gaze was tracked at 30fps using a Tobii EyeX tracker mounted under the display, with an average gaze estimation accuracy of 0.4 degrees of visual angle. Hands were tracked using an Asus Xtion PRO LIVE sensor, with resolution of 640x480 (30Hz), mounted facing down on a 0.82m×1.0m rig. Pose estimation and gesture recognition were performed using 3Gear Systems' Nimble SDK.

We implemented three interaction techniques for selecting and translating objects in our 3D scene:  Gaze (Gaze-Supported Mid-Air Gestures): the user looks at the object he wishes to select, pinch, move his hand to translate the object, and releases the pinch to disengage from the interaction.  2D Cursor (Raycasting): the user moves his hand on the plane parallel to the screen (up/down and left/right), which moved a cursor on the camera plane of the scene (moving the hand towards and away from the screen had no effect on the cursor). Targets were selected by hovering over them, (similar to a mouse cursor) and pinching. Then, the user moved his hand to translate the object and released the pinch to disengage from the interaction. Note that in this interaction technique, whereas the selection step uses only the XY coordinates of the hand, the translation step uses all three (XYZ).  3D Cursor (Virtual Hand): the user moves his hand around the space above the desk, which moved a sphere cursor in the virtual environment in three dimensions. Because we used an isomorphic mapping between the physical space and the 3D scene, any movement of the hand was directly translated in an equivalent movement of the cursor. To select an object, the user intersects the sphere cursor with the desired object and pinches. The user then moves his hand to translate the object and releases the pinch to disengage from the interaction. Upon arrival, participants completed a consent form and a demographics questionnaire. We calibrated the eye and hand trackers with the manufacturers' default procedures. Participants then performed three 3D interaction tasks, described in the following sections. After all tasks were completed, we conducted an openended interview about their experience in using the interaction techniques.

Task 1: Translating a Single-Object

In Task 1, we compared completion times for two hand-based and one gaze-based selection techniques in a translation task. Participants were presented with a 3D environment containing one blue and one red cube (see Figure 2). The task was to pick up the blue cube with a pinch gesture using each technique to select it, match its position to that of the red cube by moving their right hand whilst pinching, and drop it at the position of the red cube by releasing the pinch. When the blue cube intersected with the red cube, the red cube would turn green, indicating that the object could be released.

Participants performed the tasks in three blocks, each with 18 trials for each technique, in a counter-balanced order, for a total of 3 blocks × 3 techniques × 18 trials = 162 interactions. In each trial, the starting position of the cubes changed, but the distance between them remained constant. In the final block, after completing all trials for each technique, participants completed a questionnaire in which they rated each technique on a 7-point scale with respect to speed, accuracy, ease of learning and use, eye, hand and arm fatigue, intuitiveness, mental and physical effort, comfort, suitability for the task and personal preference. After completing all blocks they ranked the techniques in terms of speed, accuracy, comfort and personal preference. We discarded the first block from further analyses as a practice round.

Results

We compared the mean completion times between each technique across all trials, as well as the times of each step of the task, namely the time to acquire the blue cube (Acquisition), the time to pinch to confirm the selection (Confirmation) and the time to move it to the red cube (Translation). We tested the effects of the technique on the dependent variables using a one-way repeated-measures ANOVA (Greenhouse-Geisser corrected in case Mauchly's test revealed a violation of sphericity) and posthoc pairwise t-tests (Bonferroni corrected).

The mean trial completion time using Gaze (3.76s) was 21.3% shorter than using the 2D Cursor (4.78s) and 37.0% shorter than using the 3D Cursor (5.97s) (see Figure 3). The effect of technique on mean completion time was significant (F 2,22 =24.5, p<.01) with significant differences between all combinations of techniques (p<.05).

The Acquisition Time using Gaze (161ms) was 87.2% shorter than using the 2D Cursor (1.25s), and 91.9% shorter than using the 3D Cursor (1.98s), with a significant effect of the technique (F 2,22 =194.5, p<.01). Post-hoc tests showed significant differences between all combinations of techniques at p<.05. We did not find a significant effect of the technique neither on the confirmation time (F 1.2,13.2 =3.1, p=.07) nor on the translation time (F 2,22 =.12, p=0.88).

In the questionnaires, Gaze received higher scores than the other two techniques along all dimensions, except for eye fatigue, for which it scored the lowest of all three (but the difference was not statistically significant). Eleven participants ranked gaze as their preferred technique overall, with only one user preferring the 2D cursor. Nine users indicated the 3D cursor as the worst technique and three indicated the 2D cursor. A similar pattern was found for Accuracy, Speed and Comfort rankings.

Discussion

The results from Task 1 are in line with Tanriverdi and Jacob [START_REF] Tanriverdi | Interacting with eye movements in virtual environments[END_REF]. Even though their setup was VR-based, it seems that Gaze also outperform other 3D selection techniques in monoscopic displays. Unlike Cournia et al., Gaze also outperformed Raycasting for selection, but as suggested by these authors, Raycasting performed better than Virtual Hand [START_REF] Cournia | Gaze-vs. hand-based pointing in virtual environments[END_REF].

Figure 3 -Mean task 1 completion time split by step

Both Tanriverdi and Jacob and Cournia et al. investigated 3D selection, but not in the context of further manipulation. We also included a translation task to analyse whether the selection technique influenced the completion time of subsequent manipulation tasks (for example, by requiring clutching or adjustment of the hand position after selection). Because we found no significant difference in the confirmation and translation tasks, we cannot affirm that these interaction techniques have any effects on the manipulation task time, even though we observed certain hand clutching in the Gaze and 2D Cursor conditions. As shown in Figure 3, the only significant cause for the difference in the task completion time was in the object acquisition.

Task 2: Sorting Multiple Objects

In Task 1, we showed that the acquisition time using gaze is significantly shorter than using the other techniques. However, Gaze is known to suffer from inaccuracies, due to the jittery nature of eye movement, calibration issues and gaze estimation error. The goals of the second task were twofold: to investigate whether eye tracking inaccuracies would impair object selection in cluttered environments and to investigate how the faster selection times enabled by gaze can speed up tasks in which the user is required to rapidly manipulate different objects in sequence. We hypothesised that, because users do not have to necessarily move their hands to pick up new objects with Gaze, the fact that they could start the manipulation from wherever their hands were would speed up switching between objects.

Participants were presented with the same environment, now containing six black and six white chess pieces (see Figure 4). The right and left walls were coloured in white and black, respectively. Participants were asked to pick up each chess piece and move it to the appropriate wall. When the object collided with the corresponding wall, it disappeared. If the object collided with the wrong wall, it did not disappear, but we incremented an error counter. Each participant performed three trials with 12 pieces, totalling 3 trials × 3 techniques × 12 pieces = 108 interactions. In the last trial, after each technique, they answered the same questionnaire as before. After all trials were completed, they completed the preference ranking questionnaire again. We discarded the first trial of each technique as a practice round.

Results

The mean time to put away each piece with Gaze (2.27s) was 38.4% shorter than with the 2D cursor (3.67s) and 49.4% shorter than the 3D cursor (4.47s) (see Figure 5). We found a significant effect of the technique on Completion Time (F 2,22 =37.7, p<.01) and significant differences between all combinations at p<.05.

The mean rate of incorrectly placed pieces with the 3D Cursor (1.92%) was 71.3% smaller than with the 2D cursor (6.70%) and 82.7% smaller than the Gaze (11.1%). We found a significant effect on Error Rate (F 2,22 =8.19, p<.01). The post-hoc tests showed significant differences only between Gaze and the 3D Cursor (p<.05). No considerable differences were found in the questionnaire responses between the first and second task.

Discussion

The task completion times in Task 2 were significantly shorter than in Task 1. The reason for this is that whereas the selection step required precision, the translation step did not-as soon as the object hit the correct wall, the task was complete. For the hand-based tasks this represented a similar gain in speed (30% for the 2D Cursor and 34% for the 3D Cursor), but a much higher gain in speed for the gaze technique (66%). This shows that, even though it comes at a price of accuracy , Gaze is particularly well suited for tasks in which there is constant switching between different objects being manipulated. Examples of such tasks include organising furniture in architectural applications, playing speed-based 3D puzzle games and switching between different tools in a 3D modelling application.

Task 3: Selection Only vs. Selection and Manipulation

In our pilot studies, we noticed an interesting phenomenon when observing participants using gaze-assisted mid-air gestures. In the Gaze condition, once participants looked at the object, they could pinch from wherever their hands were and start manipulating the object from there. However, users still slightly reached out to the general position of the object, either to open up space for subsequent manipulations or due to a natural tendency to reach out as when handling real objects.

We hypothesized that this "clutching" before the translation would delay the selection confirmation when compared to selecting the object without any subsequent manipulation. To test this, we conducted a third task with the same 3D environment (see Figure 6-A). In one condition, a white cube appeared at random positions, but always at the same Y coordinate and at one of two Z coordinates (one in the foreground and one in the background). To reset the gaze point, between each trial the cube would show up at the centre of the environment. Participants were asked to look at the cube and make a pinch gesture, after which the cube would disappear. To avoid participants predicting the timing of the pinch gestures, we added a random delay uniformly distributed between 500 ms and 1.0 s before each trial. The second condition was similar to the first, but after pinching, the user was asked to drag the white cube to a red cube at centre of the environment (see Figure 6-B). Participants performed three blocks, each containing 20 trials of each task (not counting the gazeresetting steps), for a total of 3 blocks × 2 tasks × 20 trials = 120 interactions.

Results

We compared the time to perform the pinch gesture after having acquired the object with their gaze. The time in the Selection Only condition (557 ms) was 26.5% shorter than in the Selection + Translation (758 ms) (see Figure 7). A Welch's t-test revealed that this difference was significant (t 11 =-2.69, p<.05).

Discussion

Our results show that the time taken to select an object with Gaze is significantly longer when the user plans on manipulating it afterwards. W e offer three possible explanations for this phenomenon. First, when the user must translate the object after picking it, there is an additional planning step in the prehension process, adding some extra time for cognitive processing. Second, it is our natural behaviour to reach out in the general direction of where objects are. Third, with Gaze, even though the object can be selected from wherever the hand is, this initial position must allow enough room for the subsequent manipulation. Therefore, if the user's hand is not in an appropriate position, she must clutch it before picking the object up. From our observations, we believe the third explanation to be the most likely one.

Discussion

The results of Task 1 show that the acquisition time varied significantly between techniques, with Gaze being the fastest, followed by the 2D Cursor. We did not find a significant modality switch latency, as once the object was acquired, participants took approximately the same time to pick it up with a pinch gesture and move it to the target. As the results from Task 2 show, the advantage of Gaze is even stronger in tasks in which multiple objects are manipulated in sequence. This gain in speed comes at the cost of accuracy, particularly in densely populated environment s. Participants' opinions on the techniques also confirmed that Gaze was the most popular technique.

In task 3, we discovered a significant difference in the time to pinch after the object was gazed at between selection only tasks and selection followed b y translation. Although this difference is negligible for practical purposes, it reveals an interesting aspect of human behaviour when interacting using gaze. Even though the system was calibrated so that no clutching was necessary, participants still reached out in the general direction of where the object was positioned before pinching, similarly to how they would do with physical objects. Gaze selection elegantly supports this natural behaviour. This result suggests that gaze selection should be analyse d in the context of the subsequent tasks, and not as an independent phenomenon.

We conducted our experiment in a desktop environment. The presented techniques could, however be extended to standing interaction with large displays and immersive environments. Moreover, in stereoscopic displays, as the hands do not need to intersect the objects, Gaze-selection can be used without breaking the 3D illusion. Another limitation was that we only looked at translation tasks, but the same could be investigated for rotation and scaling.

Gaze-assisted mid-air manipulation allows users to select objects far away and manipulate them comfortably as if they were within reach. This allows users to rest their wrists on the desk, minimising the Gorilla Arm problem. This technique is are also particularly useful for monoscopic displays, where the inherent discontinuity between the virtual and the physical spaces do not allow for direct manipulation and often require an extra step for positioning the cursor on the target. In fact, participants reported not having to think about this step at all and that all they had to do was to think about the object and pinch, allowing for an arguably more immersive experience and an interaction with more fidelity.

Conclusion

In this work we evaluated gaze as a modality for object selection in combination with mid-air hand gestures for manipulation in 3DUI. Whereas previous work has found conflicting results on the performance of gaze for 3D interaction, we found that gaze outperforms other mid-air selection techniques and supports users' natural behaviours when reaching out for objects. Our findings suggest that gaze is a promising modality for 3D interaction and that it deserves further exploration in a wider variety of contexts. In particular, in future work we would like to explore how gaze can modulate the mapping between the physical and virtual environments, making it easier to reach distant objects, for example. Another avenue for investigation is how gaze can be incorporated into existing 3D applications.

Figure 1 -

 1 Figure 1 -Gaze selection for 3DUI: The user selects the object by looking at it (A), pinches (B), and moves her hand in free-space (C) to manipulate it (D).

Figure 2 -

 2 Figure 2 -Task 1: Users picked up the blue cube using each of the three techniques to select it and pinching to confirm the selection. They then moved this cube until it touched the red cube, which, in turn, would change its colour to green. The trial was concluded by releasing the pinch.

Figure 4 -

 4 Figure 4 -Task 2: Users picked up each chess piece and moved it to the appropriate side of the virtual environment.

Figure 5 -

 5 Figure 5-Mean trial completion times in task 2. Gaze was significantly faster than the other two techniques.

Figure 6 -

 6 Figure 6 -Task 3: We compared the selection time between only selecting the object to the selection time with subsequent manipulation.

Figure 7 -

 7 Figure 7 -Selection times in task 3. Users took longer to select the object when they were going to manipulate it afterwards.