Kaisa Väänänen-Vainio-Mattila
email: kaisa.vaananen-vainio-mattila@tut.fi

Thomas Olsson
email: thomas.olsson@tut.fi

Jonna Häkkilä
email: jonna.hakkila@gmail.com

Towards Deeper Understanding of User Experience with Ubiquitous Computing Systems: Systematic Literature Review and Design Framework

Keywords: User experience, ubiquitous computing, literature review, UX design and evaluation framework

Over the past decades, a plethora of innovative ubiquitous computing (ubicomp) systems have been constructed. The acceptance of the systems, however, depends on how users experience them in real contexts. While many of the ubicomp research projects include some form of user study, there is no overview of how user experience (UX) is approached in ubicomp research. To this end, we conducted a systematic literature review of ubicomp UX studies. Our findings reveal that users' experiences with ubicomp systems have often been investigated in rather lightweight ways, for example by addressing basic usability issues, collecting ratings by simple, predetermined scales, or producing descriptions of general experiences such as fun and trust. Based on the findings we argue that a deeper and more fine-grained understanding of user experience would help developing more successful ubicomp systems. We propose a ubicomp UX framework that can help design and evaluate ubicomp systems with a desirable set of target experiences.

Introduction

The core principles of ubiquitous computing visions include the omnipresence of computing devices integrated to our everyday environments, smooth interoperability between distributed platforms, and hiding computing in the periphery of human attention. [START_REF] Weiser | The Computer for the 21st Century[END_REF] Since the early days of Mark Weiser's vision, ubiquitous computing, or ubicomp, has become one of the dominating technology trends. With increasing numbers of sensor-enhanced everyday objects and infrastructures, such as smart home controls, activity tracking applications and context-sensitive mobile devices, ubicomp systems have already gained initial foothold in our environments. As a field, ubicomp research is pushing further to expand the frontiers of current solutions, and seeking ways to fulfill the promise of its visions to the users [START_REF] Abowd | The Human Experience[END_REF]. The original vision has been develo-ped further, for example to address ubicomp which can actively engage the users in everyday interactions [START_REF] Rogers | Moving on from Weiser's Vision of Calm Computing: Engaging UbiComp Experiences[END_REF].

The promise of ubicomp includes a strong emphasis on end-user's perceptions and experiences. In order to hide the technology and make its use fluent and eventually transparent, incorporating the end-user's perspective is essential in the design of the devices, applications and services [START_REF] Kuniavsky | Smart Things: Ubiquitous Computing User Experience Design[END_REF]. Including user experience in the design goals as well as evaluation criteria becomes essential especially when taking steps from early proof-of-concept level technical demos towards more mature prototypes and commercial products. The commercial success is in the end determined by the suitability in authentic usage situations "in the wild".

User experience (UX) as a field seeks to offer a systematic approach to design and analysis of the user's holistic experiences with the technology. As a quality attribute that is an increasingly important success factor of any interactive technology, UX has become one of the major interest areas in the field of human-computer interaction (HCI) since the early 2000's. On a general level, UX refers to users' perceptions and responses that arise in the use of an interactive system [START_REF]Ergonomics of human system interaction-Part 210: Human-centred design for interactive systems[END_REF]. Going beyond this definition, UX covers a broad set of users' experiences based on the instrumental (pragmatic) and non-instrumental (hedonic) system qualities [START_REF] Hassenzahl | User experience -a research agenda[END_REF]. Instrumental qualities cover traditional viewpoints like usability and efficiency of the system but also other experiential aspects, such as supporting sense of achievement, flow and self-esteem. The hedonic aspects enable experiences related to pleasure, stimulation, social connectedness, inspiration and self-expression, for example. Many of such aspects have been acknowledged in the UX literature only recently as a result of elaborate analysis of the concept of user experience.

Understanding subjective and emotional experiences will help set meaningful and explicit targets for system design [START_REF] Hassenzahl | Experience Design, Technology for All the Right Reasons[END_REF]. UX studies thus need to go beyond traditional usability tests and field studies where the focus is on the efficiency and ease of use of the system interaction. To provide detailed guidance for the design of experiential aspects of especially such novel and versatile technology as ubicomp, UX studies need to take into account a broad spectrum of human experiences.

Coming from different research traditions, ubicomp has roots in the engineering fields and computer science whereas UX is strongly connected with HCI, psychology and design. These two fields have not yet been fully entwined. Ubicomp is addressing paradigmatic changes on how technology is interacted with and what are the potential technology-mediated services in different surroundings. Experiential design thinking has become a prominent trend in HCI and we believe that the UX approach can bring opportunities for enticing ubicomp experiences. Deep, detailed understanding of experiences that are desirable and opportune in ubicomp is necessary to guide the design of successful ubicomp systems.

In this paper, we seek to provide understanding of the current study practices and types of UX findings in the area of ubiquitous computing research. Such overview has so far been missing. We do this by a systematic literature review and the following analysis of current state of UX research in ubicomp. Specifically, the main goals of our study were:

 to understand what kind of UX studies have been conducted in ubicomp research, and  to understand what kind of user experience findings have been gained in the empirical UX studies of ubicomp systems.

The main contribution of this paper is the overview of the status of empirical UX research in ubicomp. Based on the findings, we argue that understanding different user experience types in more depth can give a basis to future designs of ubicomp systems. We also propose a ubicomp UX framework which can help design and evaluate successful ubicomp systems with desirable target experiences.

Background and Positioning of this Study

In order to understand the roots of ubicomp research, we first take a glance to the early work conducted in the field. From the early days of ubicomp, there has been an emphasis on building interactive systems and integrating existing technologies, such as sensors and location tracking, to application and device concepts that can be trialed out. Especially, the seminal work conducted in Xerox PARC in the dawn of 1990's illustrates the research approach and the motivation of bringing the ubicomp philosophy, as verbalized by Weiser [START_REF] Weiser | The Computer for the 21st Century[END_REF] alive. As an example, Active Badge project [START_REF] Want | The active badge location system[END_REF] demonstrated and trialed in the wild the device and service for locating people in the large office and routing the telephone calls to correct locations.

Sensor data fusion and the idea of context-aware devices became integral part of ubicomp. Probably the most used definition related to context-awareness dates back to 2000 when Dey & Abowd define context as "any relevant information used to characterize an entity" [START_REF] Coulton | Harnessing player creativity to broaden the appeal of location based games[END_REF]. Other landmark early works from ubicomp include mobile technology focused TEA project, which investigated technology enabled awareness through context recognition and device integrated sensor modules using e.g. mobile phones as platforms [START_REF] Schmidt | There is more to context than location[END_REF][START_REF] Schmidt | Advanced Interaction in Context. Handheld and ubiquitous computing[END_REF]. Location-awareness became already early one of the key attributes when demonstrating ubiquitous technologies (e.g. Cybreminder reminder system [START_REF] Dey | CybreMinder: A Context-Aware System for Supporting Reminders[END_REF] and the first location-aware tourist guide publicly available for city visitors, GUIDE [START_REF] Cheverst | Experiences of developing and deploying a context-aware tourist guide: the GUIDE project[END_REF]). Also new output technologies such as peripheral and public displays started to emerge (e.g. [START_REF] Mccarthy | UniCast, OutCast & GroupCast: Three Steps Toward Ubiquitous, Peripheral Displays[END_REF]). While these works already included the aspect of taking the technology out from the laboratory conditions and to the field, it is evident that the development of prototype devices and applications was conducted very much from the technical viewpoint. The central outcome of the field studies was very much to verify that the technology concept actually worked in the real world settings, rather than that it was valid for the actual end users and targeted contexts of use.

Over the years, the trend to demonstrate more complex systems and multitude of ubicomp technologies became evident. Living lab type environments such as Georgia Tech Aware Home [START_REF] Kientz | The Georgia Tech Aware Home[END_REF] gave opportunities to evaluate the research concept in an instrumented environment but yet with authentic, real world users. Moreover, demo environments focused on a specific topic, such as shopping [START_REF] Krüger | Innovative Retail Laboratory -Investigating Future Shopping Technologies[END_REF], mixing the controlled and in-the-wild research setting were set up. During past few years, sensor instru-mentation in the form of smart phones has enabled data collection of numerous everyday activities even in the global scale [START_REF] Böhmer | Falling asleep with Angry Birds, Facebook and Kindle: a large scale study on mobile application usage[END_REF].

In our research, we are especially interested in detailed understanding of user experience (UX), and how it can be applied in ubicomp system development. As was stated above, UX goes beyond the traditional instrumental aspects of conventional usability [START_REF]Ergonomics of human system interaction-Part 210: Human-centred design for interactive systems[END_REF] and extends the focus towards hedonic and emotional aspects of interaction with the product or service [START_REF] Hassenzahl | User experience -a research agenda[END_REF]. Due to the maturing ubicomp technologies, more high fidelity prototypes are developed and employed in real world use. While this is happening, it becomes more relevant to investigate the user perceptions in a holistic manner in order to find solutions how to introduce the technology for larger audiences and in the real world use.

Earlier literature reviews and surveys in the area of ubicomp have mostly considered the topic from the technology point of view. These literature reviews include, for instance, surveys focusing on Internet of Things [START_REF] Atzori | The Internet of Things: A survey[END_REF] or context-aware smart homes [START_REF] Meyer | A survey of research on context-aware homes[END_REF]. HCI-oriented surveys have been conducted on research methods in mobile HCI [START_REF] Kjeldskov | A longitudinal review of Mobile HCI research methods[END_REF], virtual environments [START_REF] Bowman | A Survey of Usability Evaluation in Virtual Environments: Classification and Comparison of Methods[END_REF], augmented reality (AR) [START_REF] Swan | Survey of user-based experimentation in augmented reality[END_REF][START_REF] Dünser | A survey of evaluation techniques used in augmented reality studies[END_REF] and UX study practices in general [START_REF] Bargas-Avila | Old Wine in New Bottles or Novel Challenges? A Critical Analysis of Empirical Studies of User Experience[END_REF]. For example, Swan et al. [START_REF] Swan | Survey of user-based experimentation in augmented reality[END_REF] show that user experiments have been conducted only in 8% of AR studies. Regarding ubicomp, so far no comprehensive review investigating the UX research in the field has been conducted.

Review of Literature of Empirical Ubicomp UX Studies

Our review process was based on iterative evaluation, filtering and analysis of literature. We started by defining and selecting appropriate sources and keywords for the survey. This was followed by a multi-phased iterative filtering of the collection of articles to meet our relevance requirements, and finally a systematic analysis of the UX related findings reported in the selected publications.

Database and Keyword Selection

We focused the search to six major publishers, digital libraries or meta search engines in the field: Scopus, Springer Link, IEEE Xplore, Science Direct, ACM, and ProQuest. These were chosen because they cover the majority of the publications in the fields of ubicomp and HCI. The query was conducted on August 21st 2014. The initial search from each source was executed with a combination of three types of keywords, one focusing on the ubicomp system type, the second on the empirical user study and the third on the UX related terms (see Table 1). Regarding the technology, we aimed at covering keywords that represent the broad field of ubicomp. We included generic ubiquity-related terms (e.g. pervasive, ambient, everyware), recent related trends that are considered to belong to ubicomp (e.g. internet of things), as well as technologies related to interaction, such as tangibility. We focused especially on emerging ubicomp technologies, i.e. computing and interaction technologies that are relatively novel and have not yet produced well-established business, and thus would benefit from user-centric research. Consequently, we defined that pure mobile applications would not be included in the review. Only mobile systems with further ubicomp aspects, such as location sensing or other novelties were agreed to be acceptable to the sample. With the second keyword type (empirical) we wanted to ensure that there would be some actual user experiences found in the ubicomp study. The focus in the UX keywords was in experiences, emotions, user perceptions and value, to cover the subjective aspects of UX and to exclude papers which focused on usability or practices of use. Narrowing down the scope already at this phase was done to avoid excessive noise in the resulted data (i.e. false positives, such as papers about ubicomp demonstrators without any experiential findings). The initial query resulted in 1016 publications that we selected for further investigation (see Table 2 for the distribution according to the source). Instead of accepting the approximately 8500 results based on the entire publication data (including full text) we included only the search results based on the title, abstract, keywords and other metadata. This was to avoid investigation of a vast body of probably mostly irrelevant publications. We assumed that for the types of research papers we were looking for, the defining keywords would be mentioned in the abstracts, keywords or metadata. We tested this assumption by going through the Springer Link results where the search results could only be based on all article data. Only 19 of the 640 search results were relevant, so we assumed that the situation would be similar also for the other sources and continued with the narrower set of papers.

Iterative Analysis and Selection of Relevant Papers

Selection Criteria. We assessed the relevance of each paper in several phases and with three main criteria. First, the publication had to fit in the technological scope of ubicomp (e.g., not merely a mobile application or web service). Second, the research approach had to be based on an empirical study, rather than, e.g., pure literature review or a vision paper. Most of the exclusions took place based on these two aspects. Third, the UX focus determined the final relevance, based on how detailed and analytical the user study findings were.

The Process of Iterative Analysis-Based Filtering. The first analysis phase was conducted to filter out publications that were obviously irrelevant or incomplete (missing authors, publication names or abstracts; papers from completely different disciplines, abstracts of journal and book chapters, dictionary entries). In this phase, 156 publications were excluded, remaining 860. Furthermore, majority of the duplicates were eliminated (due to the slight overlap between what sources the different search engines use). This phase was executed by one researcher.

In the second phase, a relevance inspection based on the title and abstract was conducted in parallel by three researchers. Three levels of relevance (no relevance, maybe relevant, relevant) were considered based on the abovementioned three aspects of search. This resulted in 31 relevant and 158 maybe relevant publications. The relevance requirements that excluded most of the publications in this phase were the lack of UX viewpoint or a missing empirical user study. In the third phase, we performed a cross-analysis by three researchers of eight randomly selected relevant publications by reading the publications thoroughly and inspecting the user study methodology and reported results. The aim was to refine the analysis scheme for the rest of the publications. This helped create a commonly agreed view and later systematically analyze the entire variety of relevant aspects. In the fourth phase we performed an analysis of the relevant and maybe relevant papers (189) based on the full text. This included a systematic inspection of the key criteria in each paper to determine the final relevance.

Finally, the analysis resulted in 75 relevant publications as our data set. This means that originally, based on the abstract, there were 114 publications that were thought as relevant but after reading the full text were found irrelevant. Again, the most important reasons which led to filtering out papers at this phase were the lack of empirical user study that focused on any experiential aspects and some of the systems being mere mobile applications without any aspects that could be considered as ubicomp. In addition, a few duplicates and five papers that we could not access were excluded.

Content Analysis of the Relevant Papers. After gaining the set of 75 relevant papers, three researchers analyzed the contents bottom up from the full text, extracting items that were sought for in the research questions relating to the user study approaches and types of UX findings. Unclear items were discussed and resolved for the final classification of the found items.

Results

We first present an overview of the resulting 75 papers and their publication forums.

A majority of the selected papers are conference papers from established fora such as UbiComp, CHI, AmI, MobileHCI, Pervasive and MUM. It is noteworthy that the set of papers represent almost 20 further conferences, relating to, e.g., entertainment technologies, assistive technologies, interactive tabletops, persuasive systems, mixed reality, and children and technology. This implies that ubicomp is indeed a broad area with a variety of subtopics and related themes, and that UX studies of ubicomp are published in both ubicomp-specific technical forums and HCI-centered forums. There are altogether 13 journal publications, seven of them from Personal and Ubiquitous Computing. Furthermore, the resulting set of publications includes six book chapters, all in books with a theme closely related to Ubicomp. Figure 1 summarizes the publication years of the papers in our final data set. We can see that the majority of relevant user studies have emerged in the field rather recently. All the papers in the corpus, including the main categorizations, are listed at https://sites.google.com/site/ubiuxcorpus/. The majority of the systems in the relevant papers are for everyday pragmatic [START_REF] Ervasti | Experiences from NFC supported school attendance supervision for children[END_REF] and leisurely [START_REF] Chi | Enabling Calorie-Aware Cooking in a Smart Kitchen[END_REF] tasks, and for communication [START_REF] Costanza | Understanding domestic energy consumption through interactive visualisation: A field study[END_REF]. Other intended task types include various cognitive tasks, such as learning and searching [START_REF] Chi | Enabling Calorie-Aware Cooking in a Smart Kitchen[END_REF], transport and navigation [START_REF] Cheverst | Experiences of developing and deploying a context-aware tourist guide: the GUIDE project[END_REF], monitoring and self-reflection (9) and sports and exercise [START_REF] Bargas-Avila | Old Wine in New Bottles or Novel Challenges? A Critical Analysis of Empirical Studies of User Experience[END_REF]. The types of ubicomp technologies used are location or proximity sensing [START_REF] Dey | CybreMinder: A Context-Aware System for Supporting Reminders[END_REF], smart objects [START_REF] Costanza | Understanding domestic energy consumption through interactive visualisation: A field study[END_REF], smart environment [START_REF] Chi | Enabling Calorie-Aware Cooking in a Smart Kitchen[END_REF], public displays and projectors [START_REF] Chi | Enabling Calorie-Aware Cooking in a Smart Kitchen[END_REF], context awareness and activity tracking [START_REF] Cheverst | Experiences of developing and deploying a context-aware tourist guide: the GUIDE project[END_REF], wearable technology [START_REF] Böhmer | Falling asleep with Angry Birds, Facebook and Kindle: a large scale study on mobile application usage[END_REF], augmented/mixed reality [START_REF] Böhmer | Falling asleep with Angry Birds, Facebook and Kindle: a large scale study on mobile application usage[END_REF], interactive tags and near-field communication (3), cameras and object recognition (3), proactive assistants or agents (1) and novel interaction techniques (1).

In the following subsections, we present the findings from the literature review to the two key areas: 1) empirical user study approaches which have been used with ubicomp systems and 2) types of user experience findings that have been presented in the relevant literature.

Ubicomp User Experience Study Approaches

Most of the papers (47) included field studies in real contexts of use. 20 of the studies were laboratory studies. In eight of the studies, a laboratory was constructed to simulate the real context, for example a smart home. Most studies represented primarily qualitative research [START_REF] Lucero | The PLEX Cards and its Techniques as Source of Inspiration When Designing for Playfulness[END_REF], whereas mainly quantitative approach was used in 16 papers. Many papers included both qualitative and quantitative approaches [START_REF] Ervasti | Experiences from NFC supported school attendance supervision for children[END_REF]. Table 3 presents key aspects of the user study approaches and methodology of the empirical user studies in the surveyed set of papers. In some cases one paper fits in several categories, increasing the total number of the papers in the table (e.g. multiple methods used). Additionally, not all papers have explicitly reported all the inspected matters (marked as "N/A").

N/A 5 N/A 20 N/A 9
As Table 3 shows, the data gathering methods represent traditional user study methods, based largely on interviews, questionnaires, system logging and observations. Methods suitable for long-term field studies, such as diaries and experience sampling are rare. The system logging some of the users' activities was well utilized in this set of studies.

A majority of the papers reported studies with fewer than 30 participants (23 papers with 1-12 participants and 29 papers with 13-30 participants), which is a typical number in user studies conducted in HCI. Only 15 papers had a sample size of 30-100. The seven studies employing over 100 participants were largely using questionnaires to gather UX data (e.g. [START_REF] Meschtscherjakov | AmIQuin -An Ambient mannequin for the shopping environment[END_REF][START_REF] Rothensee | User Acceptance of the Intelligent Fridge: Empirical Results from a Simulation[END_REF]).

Only 14 of the papers reported a study where we interpreted the participants to represent the system's intended target users. Mostly, various kinds of "convenience samples" were used (e.g. students or colleagues) or the user representatives were not reported at all. This may naturally has an effect on the validity of the experiences that the users have with the system, as opposed to studying UX with the real users. As Table 3 shows, in this set of studies long periods (over a week) were quite common. Slight majority, however, is based on short-term studies that allow only the first-time experiences to be studied. When trying to understand the experiences beyond firsttime impressions, longer periods of use increase the validity of the UX results.

Types of UX Findings

In the analysis of the papers we looked for user study results that related to experiential aspects of using novel ubicomp systems. We grouped the UX findings bottom-up into six UX finding types. Table 4 shows the distribution of the papers in these categories. Many studies included aspects of several UX finding types, and our categorization was based on the central result type elicited from each paper (i.e., each paper is only counted once in this grouping). The types of UX findings shown in Table 4 are described below in association with example studies for each type.

Usability Findings. Even though our filtering process aimed at excluding usability, some papers addressed usability issues labeled under the term of user experience. In this category, the user study findings focused on the traditional usability issues like efficiency and ease of use on the user interface level.

Harrison et al. [START_REF] Harrison | Using Multi-modal Sensing for Human Activity Modeling in the Real World[END_REF] studied a system called UbiFit Garden which automatically infers and communicates information about particular types of user's physical activities, and reflects this on the mobile user interface. A two-phase study was conducted with altogether 40 participants in field conditions. The results reveal issues related to the system's form factor and design, power consumption, connectivity, accuracy, generalizability, and adaptability.

Koskela et al. [START_REF] Koskela | User Experience in Added Value Location-Based Mobile Music Service[END_REF] studied a mobile, location-based music recommendation service in the restaurant context with 53 participants. The study assessed the technical feasibility of the system and general understanding of central usability factors of the service, including ease of use, effectiveness and speed of using the service user interface.

User Acceptance Findings. The papers that fell in this category contained views to the user acceptance. In some cases, Technology Acceptance Model (TAM, [START_REF] Davis | Perceived usefulness, perceived ease of use, and user acceptance of information technology[END_REF]) was used as the reference model for the analysis and to construct Likert statements; in others the issue of acceptance was addressed more as a general term to describe how users perceive the system.

Arning et al. [START_REF] Arning | Insights into user experiences and acceptance of mobile indoor navigation devices[END_REF] studied a location-based indoor navigation system that uses pico projectors to display additional navigation information into the surroundings. In their laboratory study of 24 participants the pico projector based solution was compared to the mobile screen. The main findings related to UX are user acceptance factors such as visibility, trust, and disorientation. A major outcome of the study is a structural model of user acceptance of the compared ubiquitous display solutions.

In the study of Portet et al. [START_REF] Portet | Design and evaluation of a smart home voice interface for the elderly: acceptability and objection aspects[END_REF], a smart home environment was developed and studied in a simulated home laboratory with 8 elderly people and their family members and caretakers. The specific features studied were the voice command UI, alert messages and a shared calendar. The results show that the voice UI was acceptable to the elderly, but they were worried about their privacy and potential decrease in activity caused by the automatisation of home.

Scaled Ratings on Predetermined UX Aspects. This subset of papers used one or more scaled questions to ask the participants to rate their subjective experiences of the evaluated ubicomp system. The scales were based on predetermined experiential aspects. Ratings provide summative information about the perceived system quality but do not allow analysis or understanding of the assumed or emerging experiences.

Ecker et al. [START_REF] Ecker | EcoChallenge: a race for efficiency[END_REF] developed and studied a community-and location-based persuasive game for the car context with the goal to motivate and support behavioral change towards a fuel-saving driving style. In a field study with 37 people, the results regarding driving style showed the effectiveness of the approach. In the final questionnaire, a question on "joy of use" got the highest average score of 4.5 on the Likert scale from 1 to 5. The paper concludes that the users confirmed a very positive experience with the system.

Ankolekar et al. [START_REF] Ankolekar | Play it by ear: a case for serendipitous discovery of places with musico-ns[END_REF] studied the performance and emotional engagement of different types of audio-based cues for directing users' attention to specific points of interest (POI) in a city. In the field study evaluation, effectiveness and affect with five types of cues were studied amongst the 15 participants. Experience sampling was used with a scale consisting of six pairs of words related to 3 dimensions of emotional response, namely pleasure, arousal and autonomy: Annoyed -Pleased; Bored -Engaged; Calm -Excited; Relaxed -Stimulated; Influenced -Influential; Guided -Autonomous. The comparison results show that musicons and mixed-modality cues create more pleasant and engaging user experience than other forms of cues.

Individual User Comments or Observations. In many studies, understanding UX itself is not the main aim of the user study but the focus is on assessing more specific aspects of the studied system. Thus, even if the paper addresses user experience, the results might contain individual findings or sporadic mentions related to how the users experienced or perceived the system.

Hoffman et al. [START_REF] Hoffman | In-car game design for children: child vs. parent perspective[END_REF] developed an in-car game that combines location-based information, AR and virtual characters. They studied how children's and parents' experiences could be supported by this game in a field study of six families (27 persons). One of the user comments with regards to UX was that capturing the game characters was fun, especially for the children. The paper concludes with key challenges for incar game design for children.

In the first user study with SenseCam by Hodges et al. [START_REF] Hodges | SenseCam: A retrospective memory aid[END_REF], a single participant suffering from amnesia was using the device in her real life, and it was compared to using a diary as a memory aid. The main result of this study is that the participant's ability to remember things increased. With regards to UX, the researchers observed reduced anxiety and increased ability to relax when using SenseCam.

Overall Summaries about UX. In contrast to the previous UX finding type, the studies belonging to this category summarize the study findings by describing certain types of prevalent experiences users have with the system. Still, these experiences are not necessarily in the core of the presentation of the findings but complement the more pragmatic and technical findings.

In their study of an interactive theatre experience for the blind and sighted, van der Linden et al. [START_REF] Van Der Linden | Haptic reassurance in the pitch black for an immersive theatre experience[END_REF] had 96 participants explore the theatre space blindfolded, enhanced with a haptic mobile device. In regard to UX, they conclude that the participants' responses to the overall immersive experience were very positive, for example related to surprises while interacting in the space. The paper presents findings also related to the haptic device and arm gestures used while moving around in the space.

Holmquist et al. [START_REF] Holmquist | Supporting group collaboration with interpersonal awareness devices[END_REF] studied Hummingbird, a location-aware system to support awareness and collaboration between people who are in the physical vicinity of each other. Based on their field studies in the office, rock festival and conference, the conclusion was that users did not find Hummingbird immediately as compelling to use in the familiar setting as in the unfamiliar settings. The summarized user experiences include the feelings of connection and comfort from others being around.

Description of Several UX Types. Papers in this category address several specific user experience types systematically or in detail. In the reviewed papers none of the studies provided a very broad set of such experience type descriptions but the level of detail in the description of the selected ones is much higher than in the categories discussed above.

In their study of a persuasive, sensor-based bin can, Comber and Thieme [START_REF] Comber | Designing beyond habit: opening space for improved recycling and food waste behaviors through processes of persuasion, social influence and aversive affect[END_REF] had 22 participants use the system for five weeks. Their findings point out several experience related areas, including self-consciousness by awareness raising and aversive feelings of guilt or shame based on the social influence arising from the system use.

Persson et al. [START_REF] Persson | DigiDress: A Field Trial of an Expressive Social Proximity Application[END_REF] studied DigiDress, a mobile system for proximity-based social interaction in a long-term (avg. 25 days) field trial with 619 participants in the corporate office environment. The lookaround feature to identify nearby users was the most valued aspect of the system. The found experiences included fun, liveliness and social play, which all relate to the increased social awareness and interaction.

Summary. Overall, the papers found relevant in our review contained less detailed UX findings than what we originally expected. Only 4 of the 75 papers were at a level of description of the subjective user experiences that foster deep understanding of how the ubicomp systems are experienced. Most user studies in this sample are limited to more pragmatic or "lighter" forms of UX results. Such information can help assessing important aspects like the specific user interface design or measuring acceptance but they may not help in concepting and designing desirable ubicomp systems. To formulate meaningful experiential design targets for ubicomp, we argue that there is a need for more thorough and fine-grained understanding of the types of experiences and how different technology features and design solutions can enable them.

Discussion

The presented systematic literature review aimed at revealing the status of what kind of empirical UX studies have been conducted within ubicomp research, as well as the kinds of user experience findings these studies repor. Based on our findings it can be concluded that the subjective user experiences have been investigated in rather lightweight ways in the vast majority of ubicomp research projects. UX is an evolving concept and it has often been seen to relate to any form of user data or feedback. While general understanding of user responses and practices can be useful, we argue there is a need for more detailed understanding of UX. Thus, we have investigated UX from a viewpoint that addresses specific user experiences that go beyond the traditional pragmatic qualities of the system such as usability or usage patterns.

With regard to the user study approaches and methods, field studies have clearly been the most often used approach. Qualitative data gathering, especially interviews, has been a prominent approach in the ubicomp UX studies. Still, many studies have employed simple scales for summative evaluation based on narrow sets of experiential aspects, such as fun or sense of privacy. Furthermore, using convenience samples (often students or any accessible people) has been quite common. We propose that qualitative, open-ended methods are applied to gain understanding of the experiences of the developed ubicomp systems. The aim should be to understand the reasons for both positive and negative experiences, and ways to enable the desirable ones. These user studies should ideally be conducted with real target users, in the real contexts of use and in long-term use. These approaches will increase the ecological validity of the UX findings, and will enable transfer of the findings to the design of further, similar ubicomp systems.

The focus in this study was on the experiential aspects beyond basic usability factors, including the subjective and hedonic aspects, such as pleasure, self-expression, discovery, social connectedness, empowerment, awareness and engagement. User experiences like this have been envisioned to be probable results of the use of ubicomp systems by early visionary papers like [START_REF] Rogers | Moving on from Weiser's Vision of Calm Computing: Engaging UbiComp Experiences[END_REF][START_REF] Weiser | The Computer for the 21st Century[END_REF]. We were surprised to find very little systematic, in-depth analyses of such experiences. In many studies there are interesting findings about usage practices and design choice preferences, and these findings can be used for gaining insights of system use and needs for redesign. Still, such findings do not reveal insights to actual subjective experiences. Instead, the majority of the UX findings mention only individual, general experiencessuch as fun or trustor overall summary statements about user judgements of the system. In addition, even though we excluded mere usability studies from our selection criteria, some studies addressed usability issues, termed as UX. This illustrates the fact that the term UX is still used very loosely and often seems to refer to any form of user feedback about the system. Many systems in ubicomp are still in rather immature stage and the user evaluations bring up technical challenges, leading to unavoidable usability problemseven when the intended focus of the study would initially have been in UX. Having said that, even many of the qualitative studies seem not to have aimed at digging deep into the detailed experiences with the systems.

Then why should we gain more in-depth UX understanding for ubicomp systems? Ubicomp is maturing as a field of technology, and products and services are starting to enter the market. Human-centered design and the resulting pleasurable user experiences are becoming important competitive factors in the services offered by ubicomp systems. Designers of these systems and services can benefit from insights of the kinds of target experiences that the systems should support. The paradigmatic changes that ubicomp has to do with (e.g., implicit interaction, context awareness, proactivity and engagement [START_REF] Abowd | The Human Experience[END_REF][START_REF] Rogers | Moving on from Weiser's Vision of Calm Computing: Engaging UbiComp Experiences[END_REF]) could allow types of user experiences that cannot be reached with other types of systems. This experience design potential deserves to be explored.

It is interesting to compare the number of papers that were relevant (75) to our review criteria (ubicomp technology + empirical user study + user experience) with all surveyed ubicomp literature. The fact that so little ubicomp literature deals with subjective, detailed aspects of UX may be because ubicomp comes largely from groups with strong technology and engineering backgrounds. Only recently have ubicomp systems started to reach such technical maturity level that it is feasible to evaluate the systems in real contexts of use, with real users. We hope that in the coming years we will see an increasing number of studies focusing on detailed understanding of a variety of UX aspects. All in all, a continuing dialogue between HCI/UX and ubicomp communities is needed to reach the ubicomp visions in the most desirable and appropriate way. For example, the HCI/UX community should provide methodologies, such as advanced simulations, to better suit the development challenges in ubicomp.

Regarding the validity of this research, literature reviews face inherent challenges regarding the coverage. Because of the keyword and database selection in this broad field, some relevant papers have probably been missed. For example, papers discussing phenomena related to UX but not with that specific term may have been left out (e.g. related to affective interaction or novel, embedded systems). Furthermore, we focused on ACM, IEEE, etc. because of their technical ubicomp relevance, and thus it is possible that we left out some relevant social science research focusing on ubicomp systems. After conducting the systematic literature review, we found some further ubicomp studies (such as [START_REF] Brown | Locating family values: A field trial of the Whereabouts Clock[END_REF][START_REF] Bentley | Promoting intergenerational communication through location-based asynchronous video communication[END_REF][START_REF] Håkansson | Facilitating Mobile Music Sharing and Social Interaction with Push!Music[END_REF]) which would have been relevant but were not found by the review because the keywords were not present in the title or abstract. thus acknowledge that, due to limitations of keyword-based systematic searches, our literature sample omits some relevant papers in the field.

Despite the inevitable coverage challenges, we believe that the identified issues on the user study and UX finding types are sufficiently well covered to form the "big picture" of UX in research of ubicomp systems. The findings indicate trends and gaps in the research approaches that we believe that both the ubicomp and UX fields can agree with, and hence the results point out important steps towards the next, more human-centered and successful era of ubicomp research and product development.

Towards a Framework for Design and Evaluation of UX in Ubicomp Systems

The findings from our literature survey revealed a need for detailed understanding of user experiences when designing ubicomp systems. To theorize our findings, in Figure 2 we propose an initial framework for how ubicomp design and evaluation could be conducted with a specific emphasis on user experience. It describes how specific experiences could serve as starting points and be matched with ubicomp technologies to drive design and evaluation of successful, human-centered systems.

An experience category, such as relatedness, can be manifested by specific experiences (X) such as social connectedness, intimacy and nurture. Experience categories can set design goals or targets for the design and evaluation of ubicomp system.

A designable feature, such as proximity view or camera surveillance, is a type of functionality which is enabled or implemented by the selected ubicomp technologies. Designable features should support the design goals set by the targeted experience types. The features then form the basis for the ubicomp system design.

The following examples from our data set illustrate the elements of the framework. In the study by Chi et al. [START_REF] Chi | Enabling Calorie-Aware Cooking in a Smart Kitchen[END_REF], a cooking assistant system was built with the aim to increase the experience of self-awareness of the user's processes and habits. The developed system supported this by instant feedback of own actions, enabled by an awareness display in the cooking space. Coulton et al. [START_REF] Coulton | Harnessing player creativity to broaden the appeal of location based games[END_REF] studied a mobile outdoors game with kids and their parents. The resulting experiences were mainly about stimulation, and they included fun, discovery and creativity. The central feature of the system was a competition to create "monsters", and this feature was enabled by mobile augmented reality. We envision that this framework can be used and developed further in the following ways.

Experience-driven design. In the beginning of the design process, the designer of a ubicomp system can define target user experiences based on user needs studies or experience frameworks in literature. One or several related experiences are set as design goals. The designer then chooses and designs the features and enabling technologies to fit those targets. This approach is along the lines of supporting users' feelings as in Kansei engineering [START_REF] Nagamachi | Kansei engineering as a powerful consumer-oriented technology for product development[END_REF] as well as in more recent proposals of experiencedriven design [START_REF] Hassenzahl | Experience Design, Technology for All the Right Reasons[END_REF][START_REF] Lucero | The PLEX Cards and its Techniques as Source of Inspiration When Designing for Playfulness[END_REF][START_REF] Olsson | Reflections on Experience-Driven Design: a Case Study on Designing for Playful Experiences[END_REF].

To this end, the framework needs to be developed further by populating the experience categories with rich descriptions of a large set of experiences based on user studies of existing ubicomp systems and prototypes. The relationships between designable factors and features that affect UX in different ways need to be established. A recent example of a related framework on user experience of augmented reality is presented in [START_REF] Olsson | User Expectations and Experiences of Mobile Augmented Reality Services[END_REF]. In addition, further aspects affecting ubicomp UX such as relevant aesthetic and interaction design principles should be linked to the design process.

Experience evaluation. To understand how well the system enables the experience goals and what are its experiential facets in the overall system quality, the developed ubicomp system should be evaluated against the experience targets. By evaluating the system both quantitatively and qualitatively, deep understanding of the experiential qualities of the system can be formed. Such understanding will support the further development of the system.

To this end, the framework needs to be developed to include operationalized, quantifiable evaluation metrics for the specific target experiences. Also actionable and lightweight techniques to elicit qualitative experience feedback from the end users are needed.

The proposed framework can help in gaining elaborate understanding of ubicomp user experiences and focusing on them systematically through the design process. Eventually, the framework can contribute to both empirical research as well as to the development of successful ubicomp systems.

  UBICOMP keywords ( | refers to OR operation): ubiquitous computing | ubiquitous system | ubiquitous service | pervasive system | pervasive computing | calm computing | smart space | smart environment | context-aware system | context-aware service | context-aware application | context-based system | context-based service | context-based application | location-aware system | location-aware service | location-aware application | proximity-based system | proximity-based service | location-based | smart device | smart object | physical computing | tangible computing | mixed reality | wearable system | wearable device | sensor-based system | ambient intelligence | internet of things | everyware AND EMPIRICAL keywords: user study | field study | user trial | user evaluation | empirical study | usability study AND UX keywords: user experience | experience | experiential | product experience | human experience | user acceptance | user perception | user perceptions | human factors | perceived value | customer value | emotion | emotional

Fig. 1 .

 1 Fig. 1. Publication years of the resulting 75 publications matching the search criteria (the 2014 data is based on query performed on August 21st, 2014).

Fig. 2 .

 2 Fig. 2. An initial design and evaluation framework for user experience in ubicomp systems.

Table 1 .

 1 Search logic and keywords.

Table 2 .

 2 Summary of the initial database sources and search results.

	Source	# of results	Search based on
	ACM	31 / 2998	Title + abstract / full text
	IEEE Xplore	57 / 3313	metadata / full text
	Science direct	7 / 894	Title + abstract + keywords / full text
	Scopus	125	Title + abstract + keywords
	Springer Link	798	All article data
	ProQuest	6 / 728	metadata / all data

Table 3 .

 3 Distributions of data gathering methods, user representatives and use periods, and # of participants in the relevant papers.

	Data gathering meth-	#	User representatives	#	Use period	#
	ods
	Questionnaire (paper,	45 System's intended target	20 < 30 min	10
	online)		users
	Interview (incl. group	40 High diversity or anyone	16 30min -2	22
	interviews)		available		hours
	System logging	22 Students	11 Up to 1 day	1
	Observation	12 Colleagues and other	5	1-6 days	7
			internals
	Diaries and probes	5	Groups of (unspecified)	4	1-7 weeks	19
			people
	Experience sampling or	3	Early adopters	3	2 months or	6
	day reconstruction				more
	method

Table 4 .

 4 Types of user experience findings in empirical ubicomp user studies.

	Types of user experience findings	# of papers	Examples
	Usability findings	9	[21] [23] [33]
	User acceptance findings	9	[3] [30] [46]
	Scaled ratings on predetermined UX aspects 14	[2] [18] [52]
	Individual user comments or observations	16	[13] [26] [27]
	Overall summaries about UX	23	[20] [28] [36]
	Description of several UX types	4	[12] [19] [45]

Acknowledgements. We thank Jari Laaksonen for conducting the article database queries.


Yanxia Zhang
email: yazhang@lancaster.ac.uk

Sophie Stellmach

Abigail Sellen
email: asellen@microsoft.com

Andrew Blake
email: ablake@microsoft.com

The Costs and Benefits of Combining Gaze and Hand Gestures for Remote Interaction

Keywords: Hand Gestural Interface, Gaze Interaction, Mid-air Gestures, Remote Interaction, Large Display, Smart Living Room

Gaze has been proposed as an ideal modality for supporting remote target selection. We explored the potential of integrating gaze with hand gestures for remote interaction on a large display in terms of user experience and preference. We conducted a lab study to compare interaction in a photo-sorting task using gesture only, or the combination of gaze plus gesture. Results from the study show that a combination of gaze and gesture input can lead to significantly faster selection, reduced hand fatigue and increased ease of use compared to using only hand input. People largely preferred the combination of gaze for target selection and hand gestures for manipulation. However, gaze can cause particular kinds of errors and can induce a cost due to switching modalities.

Introduction

With advances in sensing technologies, people can now interact in much richer ways with computer systems without the need for physical contact or manipulation of devices. Free hand gestures, such as those enabled by the Kinect depth-sensing camera, have already been shown to be an effective method of input for games and interactive television applications on displays at a distance. This raises the question of whether other input modalities which support interaction at a distance, such as eye gaze, can further enhance the way we interact remotely. One reason for this is that many applications performed on a large screen over a distance often involve manipulating contents of the entire screen space, which can be widely dispersed [START_REF] Vogel | Distant Freehand Pointing and Clicking on Very Large, High Resolution Displays[END_REF]. Because people instinctively look at objects of interest, gaze has been shown to be an efficient modality for targeting remote objects [START_REF] Sibert | Evaluation of eye gaze interaction[END_REF][START_REF] Zhai | Manual and gaze input cascaded (MAGIC) pointing[END_REF][START_REF] Zhang | SideWays: A Gaze Interface for Spontaneous Interaction with Situated Displays[END_REF][START_REF] Stellmach | Look & touch: gaze-supported target acquisition[END_REF][START_REF] Turner | Eye Pull, Eye Push: Moving Objects between Large Screens and Personal Devices with Gaze & Touch[END_REF]. In addition, gaze often precedes a manual action [START_REF] Pelz | The coordination of eye, head, and hand movements in a natural task[END_REF] suggesting that we could exploit the combination of these two modalities for more efficient methods of interaction. For example, it suggests that gaze might be used for selection of a target object followed by hand gestures to operate on that object.

Recent work has highlighted the potential of combining gaze and hand gestures for fast and accurate point and drag interactions [START_REF] Kosunen | Comparing Eye and Gesture Pointing to Drag Items on Large Screens[END_REF], and enabling more attentive and immersive 3D UI interactions [START_REF] Yoo | 3D User Interface Combining Gaze and Hand Gestures for Large-scale Display[END_REF]. While this work focused on examining speed and accuracy, many other aspects of integrating gaze with hand gesture input for remote large screen interactions are still not well understood. To further explore how people perform and perceive the combination, we ask what benefits gaze might bring to gestural interaction from the user's perspective. We also investigate any potential costs that are incurred by adding gaze as a second modality.

In order to address these questions, we constructed a photo sorting task involving the fast assignment of multiple objects spread out across a large screen to different "piles" or destinations. This task, which involves selection of remote target objects followed by a limited set of repetitive actions on those objects, has been shown to be representative of common tasks that users would want to carry out on large screens [START_REF] Vogel | Distant Freehand Pointing and Clicking on Very Large, High Resolution Displays[END_REF].

We compared users' experience of two interfaces for this task: one which represents the status quo for device-free remote interaction such as gaming, namely using gestures only (hand-only), and another which works across modalities (gaze-hand). In the cross modality condition, we assigned gaze and hand gestures in what we surmised would be the optimal way for such a task. Based on the prior work mentioned above, we reasoned that gaze would be best for target selection (of relatively large targets), while hand gestures would be good for expressive but not necessarily precise manipulation. In this case, gestures are used to "fling" photos to different piles, an action which involves coarse-grained (as opposed to fine-grained) control.

User Study: Photo Sorting

In this study, our goal was to examine the effect of integrating gaze into a remote, gestural interaction task in terms of both performance and subjective experience. We hypothesized that: (1) the combination of gaze pointing and hand gestures will give rise to faster task completion than sorting with gesture only; and (2) using gaze will introduce an extra mental load due to the need to work across modalities during the task. We made no a priori predictions about subjective experience.

Task and Design

The photo-sorting task required repetitive selection followed by quick gesture commands on the selected items (see Fig. 1). The participants' task was to classify displayed photos into four categories labeled in diagonal corners (animal, car, flower and food), as accurately and as quickly as possible.

We used a 2x3 within-subjects design for factors technique {gaze-hand, hand-only} and number of pictures presented {1, 4, 8} at a time (Fig. 1). We varied the number of pictures presented in any trial reasoning that this might emphasize any differences between the two techniques: selecting amongst 8 possible targets instead of focusing on one at a time might show bigger advantages for the gaze-hand condition, for example.

In total, each session required a subject sorting 192 pictures, 96 for gaze-hand and 96 for hand-only. The order of presentation of the technique was counterbalanced and the order of the number of pictures presented was randomized among three blocks. Each block consisted of sorting 32 pictures displayed one, four or eight at a time.

Setup and Procedure

Participants and Apparatus. We recruited 15 participants (5 female), with a mean age of 26.7 years (SD=4.5), who had little experience with eye tracking or motion and hand tracking applications. Participants sat on a chair with armrests at a distance of 2.4m in front of a 55-inch (121cm×68.5cm) display (Fig. 2). To capture gaze input, we used a Tobii TX300 at a sampling rate of 60Hz. We positioned a Kinect sensor at a height of 1.8m above the ground and 1.5m in front of the user. For hand tracking, we used a fast random forest-based hand state classification method [START_REF] Keskin | Hand Pose Estimation and Hand Shape Classification Using Multi-layered Randomized Decision Forests[END_REF].

Procedure. After practicing the first technique encountered for approximately five minutes, participants were randomly presented with a set of 32 pictures, either 1, 4 or 8 pictures at a time (Fig. 2(A)). In the hand-only condition, the user's hand position was mapped to a screen cursor; in the gaze-hand condition, the user's gaze was mapped to a screen cursor for selection instead. To confirm to the participant which image was about to be selected, the cursor-overlaid image was highlighted with a gray border after 150ms (this is the "hover state", Fig. 2(B)). Thereafter, in both conditions, the user confirmed the selection by a hand grab gesture, indicated by the border color switching from gray to black ("select state", Fig. 2(C)). (Note that we deliberately decided against using gaze dwell time to confirm selection. Short dwell durations can induce the "Midas Touch" problem [START_REF] Sibert | Evaluation of eye gaze interaction[END_REF][START_REF] Stellmach | Look & touch: gaze-supported target acquisition[END_REF][START_REF] Kumar | EyePoint: Practical Pointing and Selection Using Gaze and Keyboard[END_REF] where users can easily misselect items when they only intend to look at an image, while long dwell durations can yield lower task performance and user satisfaction, as they are slow and disruptive.) The user then sorted the photo (in both conditions) by swiping their hand diagonally (Fig. 2(D)). We used four diagonal hand swipe directions to assign categories. Once a swipe was detected, the photo flew to one of the screen corners with swoosh sound effect. Participants filled out a questionnaire after each technique was completed. During the study, participants were allowed to take a short break after each experiment block. At the end of the entire session, we also conducted interviews to collect feedback. Each session lasted for approximately one hour.

Dependent Variables. We logged three variables (Fig. 2): T the overall selection time, defined as the time between states A to C for single picture trials and from D to C for multiple picture trials; The time T can be further divided into: T 1 the time from initial movement of the cursor, starting from picture being presented to confirmation of the hover state, and T 2 the time between hover confirmation and selection using the grab gesture (B to C). Swipe time (from C to D) was assumed, and was indeed found to be, similar across conditions. In addition, sorting errors were logged.

Results

Overall Selection Time (T) per Picture

The time to select a picture was faster with gaze-hand than hand-only (shown by a significant main effect of technique on T, F 1,14 =32.4, p<0.0001), being on average 0.33s faster per picture (Fig. 3(A)). There was also a significant main effect of the number of pictures on T (F 1.3,18.25 =28.8, p<0.0001), with the overall selection time increasing as the number of pictures increased. There was no interaction between technique and number of pictures on T (F 2,28 =0.346, p=0.71).

3.2

Cursor Moving Time (T 1 ) and Grabbing Time (T 2 ) per Picture Despite the overall difference in T or selection time, it is more instructive to break this down into two components.

Time to move the cursor to a hover state (T 1 ) was significantly faster (by on average 0.61s) with gaze-hand than with hand-only (F 1,14 =308.9, p<0.0001). There was also a significant main effect of the number of pictures on T 1 (F 1.39,19.39 =14.18, p<0.001) but as Fig. 3(B) shows, there was also a significant interaction between technique and number of pictures (F 2,28 =5.99, p<0.01) which suggests this effect is due an increase in T 1 as the number of pictures increased in the hand-only technique.

When we look at T 2 , the time from hover to select, we see a very different story. Here the hand-only technique was faster than gaze-hand (F 1,14 =47.33, p<0.0001), being on average 0.25s faster per picture (Fig. 3(C)). Number of pictures again gives rise to a main effect (F 2,28 =21.87, p<0.0001) but again it is made more difficult to interpret due to a significant technique by number of pictures interaction (F 2,28 =6.06, p<0.01). This indicates that this time it is the gaze-hand condition that accounts for this increase and not the hand-only condition.

Sorting Accuracy and Error Analysis

We analyzed the sorting accuracy per block (N=32) (see Table 1). The success rate is generally high for all trials. Error analysis revealed that users made significantly more mistakes in the 8-picture trials in the gaze-hand condition than hand-only (t(14)=2.902, p<0.012). There were many kinds of errors. Most were incorrectly sorted images due to swiping in the wrong direction (categorization errors). Some were due to the way the system was configured, or the peculiarities of the interaction. For example, this included triggering a swipe response when returning the hand to the resting position or not completely releasing a grabbed picture.

However, the lower accuracy in the 8-picture condition of the gaze-hand technique is probably explained by difficulty in coordinating gaze and gesture. For example, in the 4-and 8-picture trials, participants occasionally performed a swipe as their eyes prematurely moved ahead to the next image -the eyes "jumping the gun" before the hand had finished its work, which is similar to the synchronization problem identified in prior work [START_REF] Kumar | EyePoint: Practical Pointing and Selection Using Gaze and Keyboard[END_REF]. Further, the logged gaze data revealed that neighboring pictures in the gaze-hand condition, largely in 8-picture trials, distracted some participants. During selection, their gaze shifted between the selected and the adjacent image, resulting in incorrectly sorting the wrong object. Participants' feedback in post-task interviews suggested that both of these kinds of errors occurred when they tried to speed up their performance, which in turn caused them to slow down to be more careful.

User Experience and Preferences

The questionnaire data indicate only two significant differences across techniques. First, hand fatigue was rated significantly higher with the hand-only technique than gaze-hand (Z=-2.751, p<0.01). Most participants (10/15) commented on this. For example, one said, "holding the arm high in mid-air is tiring" and "[with gaze] I could rest my hand and use my hand only for sorting".

Second, participants felt that the hand-only technique was easier to learn than the gaze-hand technique (Z=-2.309, p<0.05). Though participants were positive about the combination of gaze for selection and hand for grasping and throwing, they felt the gaze-hand combination was something that required time to get used to. Despite this, when we tested for learning effects over trials, we found no evidence of differences between gaze-hand and hand-only techniques.

Finally, participants rated which technique they preferred and the perceived speed and accuracy in the sorting task (Fig. 4). Thirteen users preferred the gaze-hand technique: although they found both techniques "intuitive", some felt that the gaze-hand technique was "less demanding" (4/15) and "easier to use" (10/15), primarily due to less physical fatigue and fast and accurate gaze selection.

Further explanation for this can be found in the interview data. For example, one participant said, "I am pretty confident with the gaze to select pictures, as what I looked at was what I wanted to select". In contrast, users felt that positioning the cursor on top of a picture using the hand was trickier, as it required constant checking and it was difficult to control the cursor accurately on a 2D screen while moving their hands freely in 3D space.

Discussion

The Benefits of Combining Gaze and Hand Gestures

Our results show improvements in both speed and user experience when we add gaze to hand gesture input in this remote interaction task. Previous work showed that gaze is faster than the hand for pointing and positioning a single target [START_REF] Kosunen | Comparing Eye and Gesture Pointing to Drag Items on Large Screens[END_REF]. Our study expands on this showing that the speed advantage of gaze persists when used for target selection and when combined with gesture in a single technique. More than this, it persists in the context of multiple targets and indeed becomes more pronounced as the number of targets increases. This was shown in our analysis of T 1 , the time to move the cursor, which increased significantly for the hand-only technique with more targets, but which did not increase when gaze directed the movement of the cursor. This explains the overall speed advantage for the task of image selection.

The majority of participants also perceived the gaze-hand technique as faster than hand-only, and users' subjective feedback further confirmed the fact that gaze complements gestural interaction when it comes to indicating objects of interest on a large screen. Here, users explained that hand gestures alone were slow, tiring, and inaccurate compared to incorporating gaze into the technique.

Taken together, this suggests that the combination of gaze and hand gestures is particularly well suited for applications that involve repetitive manipulation of multiple objects. Here, gaze selection can be both faster and reduce hand fatigue, especially when the task requires frequent pointing. Of course, arm fatigue could be even further reduced by attending to the ergonomics of the hand gestures too, such as designing them to require minimal effort, providing arm support and so on.

The Costs of Combining Gaze and Hand Gestures

Although participants indicated that they preferred the gaze-hand condition and reported little trouble with eye-hand coordination, the performance data (T 2 ) shows that it actually took longer to switch from gaze-hover to hand-select, than it did to transition from hand-hover to hand-select. Further, this effect was made worse as the number of targets to choose from increased. This may be because multiple targets increased visual distraction, delaying the ability for participants to confirm that the hovered object was indeed the intended target. This was not the case with selection within modality. In other words, there is a cross-modality speed of performance cost here, which may only be amplified as the interface becomes more complex. Added to this, the analysis of errors provides some indication of accuracy issues induced by the mixed modality interaction. Sometimes it was clear that participants were visually distracted by adjacent pictures, leading to errors in selection. Other times, especially in the 8-picture condition, participants sometimes moved their gaze too quickly in advance of completing a gesture, causing errors. Participants were very aware of these synchronization errors and said that this was further amplified when they wanted to be fast. In other words, while gaze was often a fast way to reach the next target, the cost was that users had to wait for the hand to finish the previous manipulation. Solving this problem through good design starts by recognizing this problem.

Participants' feedback also suggests that working across modality was more difficult to learn than the single modality technique. While we might expect that these cross-modality costs would disappear with practice, we found no obvious learning effects during the course of the study. It may be that longer-term use is needed to investigate these aspects.

However, it is notable that despite the costs we outline above, participants said that overall they preferred the gaze-hand technique and expected it would be "more productive" in the long run. So clearly, these judgments are weighing up the benefits against the costs we have highlighted.

Conclusion

In this paper, we investigated both the costs and benefits of combining gaze and hand gestures for remote interaction. Our work contributes to a better understanding of how users perceive this input combination, and shows that gaze can complement free-hand gestural interfaces if the task is designed appropriately. Despite the costs, users mainly prefer the combination of gaze and gesture, but designers must take account of the kinds of errors that users can make and expect that such techniques may be initially perceived as more difficult to learn.

Fig. 1 .

 1 Fig. 1. User interface (left) and study design (right)

Fig. 2 .

 2 Fig. 2. Our setup: a user sat in front of a large screen using a combination of gaze and hand gestures for photo sorting. After acquiring a photo by either gaze or hand (B), users grabbed (C) and swiped it to diagonal directions (D).

Fig. 3 .

 3 Fig. 3. Different average performance time for each technique and number of pictures presented at a time (mean ± 95% CI).

Fig. 4 .

 4 Fig. 4. Participants' technique preference, perceived speed, and accuracy in sorting as rated at the end of the experiment.

Table 1 .

 1 Average sorting accuracy (mean±std) of the 15 participants made for each block.

	Gaze-hand	Hand-only	Paired samples t-test
	One 92.1%±6.0% 93.3%±3.9%	t(14)=1.000, p=.334
	Four 91.7%±5.7% 93.3%±4.1%	t(14)=1.035, p=.318
	Eight 90.0%±7.4% 95.4%±3.1%	t(14)=2.902 p=.012