Augmented Reality (AR) has a great potential of becoming a pervasive tool in users' everyday life. The technology is progressing rapidly and the research community is actively working on studying its uses, promising a future where users could take advantage of AR in their daily lives. Such a wide adoption of AR raises important questions regarding users' experience. This thesis identifies three specific problems regarding the usability of AR in everyday use. The first research question looks at low-level aspects of interaction with respect to users' motor performance and comfort when manipulating virtual content displayed in AR. We propose ARPads as a way to enable indirect input and thus reducing users' fatigue without losing speed and precision in comparison with direct manipulation of AR content. The second research question is about the integration of wearable AR within users' existing panorama of digital technologies. We focus on the combination of AR with handheld devices and propose a design space for enhancing phone-centric interactions with AR as well as quantitative results regarding the gained performance in the specific case of AR-enhanced widgets for phones. The last research question investigates high-level usability aspects when AR is always-on and thus can interfere with users' physical environment and social interactions. We introduce the concept of AR de-augmentation, giving users agency over the display of AR content to selectively remove specific AR content from their field of view.

FRENCH SUMMARY

Ce manuscrit présente les recherches que j'ai effectuées dans le cadre de ma thèse de doctorat autour des usages de la Réalité Augmentée (RA) dans la vie de tous les jours. J'y présente trois questions de recherche : QR 1 Comment reconsidérer nos interactions avec la Réalité Augmentée portative pour que celles-ci soient appropriées pour des interactions prolongées ?

QR 2 Peut-on énoncer des recommandations sur l'augmentation des appareils mobiles avec la Réalité Augmentée portative ?

QR 3 Comment un utilisateur peut-il garder le contrôle sur le contenu qu'il perçoit en Réalité Augmentée ?

Le manuscrit est organisé en trois parties répondant respectivement à chacune de ces trois questions de recherche.

Pour répondre à notre première question (QR 1 ), nous présentons le projet ARPads. Un ARPad n'a pas de réalité physique, mais représente un plan d'interaction flottant dans l'air dans lequel les mouvements de l'utilisateur sont traduits par des mouvements de curseur dans une fenêtre de RA affichée dans l'environnement. Ce contrôle indirect permet à l'utilisateur d'adopter une posture plus confortable pour manipuler du contenu affiché en RA par opposition au contrôle direct qui force l'utilisateur à maintenir ses bras en l'air en direction du contenu de RA. Après avoir défini les critères de conceptions de ces ARPads en prenant en compte leur position et leur orientation relatives à l'utilisateur, nous avons implémenté et évalué certaines conceptions prometteuses. Nos résultats montrent que le contrôle indirect peut atteindre des performances similaires à celles du contrôle direct, tout en limitant la fatigue perçue par l'utilisateur et identifient les conceptions d'ARPads les plus efficaces. Pour notre deuxième question (QR 2 ), nous nous concentrons sur la combinaison entre les appareils mobiles et la RA. Suite à un atelier de conception mené avec des utilisateurs d'appareils mobiles, nous avons défini un espace de conception d'applications centrées sur le smartphone pouvant bénéficier d'une association avec la RA. En particulier, nous avons Part II PROJECTS 2. ENHANCING SMARTPHONE-CENTRIC INTERACTION WITH AUGMENTED REALITY This chapter is based on the paper "AR-enhanced Widgets for Smartphone-centric Interaction" [19] that has been presented in MobileHCI '21. The co-authors of this paper are Emmanuel Pietriga and Caroline Appert.

identifié deux dimensions principales : la fonction et la position du contenu holographique par rapport au contenu de premier intérêt affiché sur l'appareil mobile. Ce premier travail montre une grande richesse en termes de créativité pour proposer des améliorations possibles pour les appareils mobiles. Cependant, le manque de données quantifiables sur leur performance effective nous empêche encore de poser un avis objectif sur les réels avantages d'une telle association. Après avoir prototypé quelques exemples d'application pour démontrer leur faisabilité, nous nous sommes concentrés sur l'évaluation d'un cas spécifique : les composants d'interface (widgets) traditionnels des smartphones. En particulier, nous avons comparé différentes distributions de l'interface utilisateur entre la RA et le smartphone pour les widgets suivants : déclencheurs (boutons), ajustement paramétrique (slider) et sélection en 2 dimensions.

Part I INTRODUCTION

When talking about Augmented Reality (AR), several images come to our minds. Movies [133,80], Series [201], Anime [39], Books [194], Video Games [START_REF][END_REF] and even Art [1] have contributed to create a mental model of what AR is. Most people imagine users seeing holograms that they can interact with using mid-air gestures.

Besides this fictional vision, researchers have worked on providing a more formal definition of AR. In particular, Augmented Reality falls within the scope of Mixed Reality, which Milgram [START_REF]a n d K i s h i n o , F . A taxonomy of mixed reality visual displays[END_REF] has described as a continuum in 1994. However, this continuum, illustrated in Figure 0.1, does not specify clear boundaries of what AR encompasses, but broadly defines it as a technology that enhances the real environment with virtual information. When virtual elements are getting more important than real ones, Milgram no longer talks of AR but rather of Augmented Virtuality (AV). The use of a continuum suggests that there exist different levels of AR depending on the importance of the real environment in the system. Azuma [START_REF]A survey of augmented reality[END_REF] complements this definition by identifying three properties that characterize AR: (1) combining real and virtual, (2) being registered in three dimensions, and (3) being interactive in real-time. [START_REF]a n d K i s h i n o , F . A taxonomy of mixed reality visual displays[END_REF] AR is no longer part of fiction only. There is a range of technologies that can actually run AR systems. Among these, we can identify two broad categories depending on the technology's form factor: (1) Stationary setups and (2) Mobile setups. Stationary setups are integrated into users' environment and make it possible to augment a specific area. For example, Figure 0.2-a) illustrates HoloDesk [START_REF] H I L L I G E S | Holodesk: Direct 3d interactions with a situated see-through display[END_REF], a system that projects a virtual layer onto a fixed transparent screen in front of the user. With such a system, users can interact with virtual elements by placing their hands under this transparent screen. Mobile setups are usually more lightweight than stationary setups. They are portable, making them better suited to diverse tasks and contexts. Mobile setups can be (1) Handheld or (2) Wearable. By definition, Handheld devices (e.g. tablet, smartphone) must be held. This not only occupies users' hands but also forces users to maintain a potentially uncomfortable posture. This makes these devices unsuitable for long-lasting tasks. For example, Figure 0.2-b) illustrates how the user should hold their smartphone to scan the real environment when using the application Pokémon Go [START_REF]Ni a n t i c[END_REF]. On the opposite, Wearable devices, also called Head-Mounted Displays (HMD), can be permanently worn by users (e.g. headsets, glasses). For example, Microsoft has released multiple scenarios of AR use cases with their Hololens. Figure 0.2-c) illustrates such a scenario for a work environment where the user is manipulating a virtual object with two hands as they would do with a physical object. Besides their form factor, Wearable AR devices can also be separated into two other categories, depending on their rendering method. On the one hand, Augmented Reality can be obtained using Video See-Through (VST) devices, which video record the real environment and modify the video stream to include virtual objects and thus render an augmented view to the user (e.g. Varjo -XR -3 [203], smartphones). However, such devices raise an important issue: if the VST device stops working, the user gets completely blind. The second way of obtaining AR relies on Optical See-Through (OST) devices, which superimpose a virtual layer on the environment by projecting it on a transparent or semi-transparent screen (e.g. Google Glasses [START_REF]Go o g l e. Google glasses[END_REF], Microsoft HoloLens [START_REF]ro s o f t. Microsoft hololens[END_REF]). Even if the performances of VST devices are higher, we strongly believe that OST devices have a great potential to get widely adopted in users' everyday life. Indeed, users are not completely cut off from their environment as these devices do not "replace" reality, but rather supplement it.

Mixed Reality (MR)

Looking at their history, they have evolved a lot and have recently started to get close to a technology that could actually get adopted at large. If OST devices still benefit from the novelty effect of new technologies, researchers have been actually studying them for a while. The first prototype of an AR Optical See-Through device has been presented in 1968 by Sutherland [START_REF] S U T H E Rl A N D | A head-mounted three dimensional display[END_REF] (cf. Figure 0.3 -a)). Even if the system was crude, it already integrated specific sensors for tracking the user's head position and rotation, which were used to change the perspective of a virtual element displayed through cathode tubes and overlaid with reality thanks to a mirror-based system. Since 1968, there have been major improvements regarding AR devices' performances, not only regarding their tracking and rendering capabilities but also regarding their form factor. Indeed, while Sutherland's system could only be used within a specific room, Feiner et al. introduced a portable system called the Touring Machine [START_REF] F E I N E R | A touring machine: Prototyping 3d mobile augmented reality systems for exploring the urban environment[END_REF] (cf. Figure 0.3 -b)) in 1997. It consisted of an OST tied to a backpack containing a full computer, providing users with virtual in-context annotations on real buildings when walking in their university campus. Electronic miniaturization then made it possible to reduce HMD's volume and weight, while significantly increasing their performances. Not only are they lighter and more aesthetic, but they are faster and have a higher display resolution. These improvements have led to the Microsoft HoloLens 2 [START_REF]ro s o f t. Microsoft hololens[END_REF] (cf. Figure 0.3 -c)) in 2017.

Current technologies such as the Microsoft HoloLens now allow users to interact with various modalities (e.g. gaze, touch, voice), while remaining relatively lightweight (566g).

Their price also suggests that AR HMDs are getting more and more affordable [START_REF] M C G U Ff I N | Augmented reality knowledge work : Towards a research agenda[END_REF].

However, this evolution has not yet driven to a final product for daily use. They are still quite uncomfortable and conspicuous. But, a future direction for AR devices includes metasurfaces such as contact lenses [START_REF]Augmented reality in a contact lens[END_REF][START_REF]Augmented reality in a contact lens: It's the real deal[END_REF][START_REF] M O J O. Mojo | [END_REF] (cf. [START_REF] S U T H E Rl A N D | A head-mounted three dimensional display[END_REF]; b) 1997 -the Touring Machine of Feiner et al. [START_REF] F E I N E R | A touring machine: Prototyping 3d mobile augmented reality systems for exploring the urban environment[END_REF]; c) 2017 -Microsoft HoloLens [START_REF]ro s o f t. Microsoft hololens[END_REF]; d) On going -AR contact lenses [START_REF]Augmented reality in a contact lens[END_REF] Besides technological advances, some researchers also argue that Augmented Reality has the potential of becoming a pervasive tool in users' everyday life [START_REF] M C G U Ff I N | Augmented reality knowledge work : Towards a research agenda[END_REF]. Indeed, it has already proved useful for specific use cases in both academia and industry. In the field of education, AR has the potential of providing a 3D immersive environment that facilitates learning with a higher recalling, understanding, and engagement [START_REF] B I L L I N G H U R S T | A survey of augmented reality[END_REF][START_REF]Why should my students use ar? a comparative review of the educational impacts of augmented-reality[END_REF]. For example, it can be helpful for users' spatial orientation and their understanding of 3D relief [START_REF] A N D A S E N S I O | Augmented reality as a digital teaching environment to develop spatial thinking[END_REF].

The higher level of engagement can also profit culture at large by revitalizing historical elements [START_REF] P E De Rs E N | More than meets the eye: The benefits of augmented reality and holographic displays for digital cultural heritage[END_REF] or enhancing art [1]. In Industry, AR can be used to support workers in assembly tasks [START_REF] F U N K | Working with augmented reality? a long-term analysis of in-situ instructions at the assembly workplace[END_REF].

Wearable AR can also assist people with some visual impairments. For example, the ChromaGlasses [START_REF] L A N G L O T Z | Computational glasses for compensating colour blindness[END_REF] proposes to correct images by exaggerating some colors in the AR virtual overlay to lower the mental demand of color blinds and increase their level of confidence while describing colors. Other types of overlays can also be considered for mitigating other visual disabilities. For example, an AR overlay can provide a magnifying view of the scene, increase the visibility of obstacles with higher contrast, provide audio feedback of what is situated within the scene, etc. [START_REF] M I N H T I K E | a mb o l c h il a r , P . Augmented reality glasses as an orientation and mobility aid for people with low vision: A feasibility study of experiences and requirements[END_REF]. AR could also be used as a tool for therapy as it can bring new fully virtual elements in users' field of view (FoV), exposing users to e.g. a phobia, while feeling safer than in a real situation [START_REF] Lc A N I Z | Using augmented reality to treat phobias[END_REF]. AR can also train people for specific situations in emergency management and clinical environment [START_REF] S C H L O S S E R | Head-worn displays for emergency medical services[END_REF] as such situations could not be reproduced in real life. But AR is not limited to these specific uses and more casual use cases for daily life have been described in the literature too. AR can e.g. enhance gaming by providing real-life accessories and metaphors for interaction [START_REF] P I E K A R S K I | The outdoor augmented reality gaming system[END_REF], making users more engaged. Some HCI researchers are also envisioning the use of AR in a large range of domestic contexts. For example, AR can assist in caring for a baby, cooking, alerting of dangerous elements in a room to prevent injuries, or enhancing home devices [START_REF] K N I E R I M | Exploring the potential of augmented reality in domestic environments[END_REF].

In summary, Augmented Reality has shown its usefulness in many fields, and promises a large range of yet non-explored other use cases for helping users with their daily lives. However, echoing Koelle [START_REF] Ko E L L E | All about acceptability? identifying factors for the adoption of data glasses[END_REF], I argue that there is still a range of questions regarding the usability of AR to address before actually considering it as a viable technology for everyday use. I detail below the three research questions that my thesis addresses.

The first question is about the means of interacting with wearable AR. In most existing systems, users have to involve their bodies when interacting with virtual object as they would do when interacting with physical objects in a real life context (Figure 0.2-c) illustrates such a case). But AR is very different from reality as interactable elements are fully virtual and thus have no tangibility. The early stage of maturity of AR has encouraged the design of short-term interactions, mostly based on real life metaphors where users touch virtual objects. However, some studies suggest that large movements in the air are tiring and not suitable for continuous use [START_REF] B Ac H Y N S K Y I | Performance and ergonomics of touch surfaces: A comparative study using biomechanical simulation[END_REF][START_REF] H I N C A P I É -R A M O S | Consumed endurance: A metric to quantify arm fatigue of mid-air interactions[END_REF]. Furthermore, interacting with large gestures can be socially awkward, especially with personal devices, as external people cannot see the virtual content displayed in the personal view. Therefore, before AR gets widely adopted, there is a need for reconsidering interaction with wearable AR. My first research question is:

How to reconsider interaction with wearable AR to better support long-lasting tasks?

The second question revolves around the place of wearable AR within our already large set of digital devices. Like smartwatches have not replaced smartphones, wearable AR devices will likely co-exist with other digital devices. As Augmented Reality typically turns the air into an interactive space, it offers promising prospects for enhancing other devices that are constrained by their limited interactive space. Smartphones are especially good candidates for this as they have a very limited size while offering good computing performance, good input precision, and high display resolution. Normand et al. [START_REF] A N D M C G U Ff I N | Enlarging a smartphone with ar to create a handheld vesad (virtually extended screen-aligned display)[END_REF] gave a first overview of the possible smartphone-based applications that could benefit from being combined with an AR headset. Yet, there are no clear guidelines for designers and developers about successful ways of combining a personal handheld device with a wearable AR device. My second research question is:

Can we provide guidelines on how a wearable AR device can enhance a handheld device?

Augmented Reality definitely has the potential of changing users' routines by blending the boundaries between real and virtual content [START_REF]A survey of augmented reality[END_REF]. Despite all its advantages, this technology is very intrusive as it almost directly modifies users' perception of their environment. Virtual and real environments are superimposed, which can create confusion for users due to visual interferences between the two environments. The last question regards how users could be able to differentiate real and virtual elements, be it for their safety or for ethical considerations. Before integrating AR in a daily context, there is a need to provide users with "safeguards", allowing them to take advantage of AR without being misled or endangered by it. My third research question is:

How can users be in control of what Augmented Reality tells them?

While these are broad and complex questions, this manuscript reports on projects I conducted during my PhD to provide preliminary answers. First, we investigate the concept of mid-air pads (ARPads) as an alternative to gaze or direct hand input to control a cursor in windows anchored in the environment. An ARPad is a virtual plane within which users can move their hands to control a cursor displayed in the headset screen. Contrary to direct input, an ARPad is not spatially aligned with the headset screen, meaning that users can perform mid-air movements in a more relaxed, comfortable posture. We investigate a design space for ARPads, which takes into account the position of the pad relative to the user's body, and the orientation of the pad relative to that of the headset screen. Our study suggests that 1) indirect input can achieve the same performance as direct input while causing less fatigue than hand raycast, 2) an ARPad should be attached to the wrist or waist rather than to the thigh, and 3) the ARPad and the screen should have the same orientation.

Then, I present an investigation of the combination of an AR headset with a handheld device. Following a user-centered approach, we started with a workshop that we organized with smartphone users to brainstorm about the possible enhancements that AR could bring to mobile devices. Our observations drove the definition of a design space of smartphone+AR applications based on two dimensions: the spatial location of the augmented content around the smartphone, and the function of that augmented content with respect to the primary content displayed on the phone. We then prototyped some applications to populate interesting cells of this design space with examples. Finally, in order to provide enhancements that are readily applicable to existing smartphone applications, we focused on existing smartphone widgets and designed several ways of enhancing them with AR. We propose several solutions for manipulating discrete and continuous widgets (1D and 2D) with a varying amount of interaction offloaded in the air. We compare these solutions against their state-of-the-art touch-only counterparts with a series of exploratory studies in which participants had to perform three tasks: command trigger, parameter value adjustment, and precise 2D selection. We then derive guidelines from our empirical observations.

Finally, I introduce the concept of De-augmenting AR, which allows users to define arbitrary areas that are freed from augmented content. In the De-augmenting AR project, we rationalize the need to give users agency over the visibility of this virtual content.

De-augmentations let users selectively remove AR elements from their field of vision, only targeting those that actually interfere with their perception of the physical world.

We build a taxonomy that captures the different facets of de-augmentation, discuss a variety of scenarios that illustrate its usefulness, demonstrate its practical feasibility, and discuss multiple questions that it raises about the design of future AR systems.

As described above, this thesis offers interesting response elements over the proper way to integrate AR in users' everyday life. However many questions remains. Inspired by our work and the related litterature, I discuss some of these in a last chapter, putting forward directions for future research on this topic.

MID-AIR INDIRECT INPUT FOR AUGMENTED REALITY

This chapter is based on the paper "ARPads: Mid-air Indirect Input for Augmented Reality" [START_REF] B R A S I E R | Arpads: Mid-air indirect input for augmented reality[END_REF] that has been presented in ISMAR ' 20. The co-authors of this paper are Olivier Chapuis, Nicolas Ferey, Jeanne Vezien, and Caroline Appert.

Augmented Reality (AR) headsets make it possible for users to get private, in-situ user interfaces. They can offer personalized exhibits in museums [START_REF] P E De Rs E N | More than meets the eye: The benefits of augmented reality and holographic displays for digital cultural heritage[END_REF], deliver specific instructions in an educational context [START_REF] F R E Y | Pointing in spatial augmented reality from 2d pointing devices[END_REF], or display data visualizations related to objects in the environment [START_REF] C O R D E I L | DXR: A toolkit for building immersive data visualizations[END_REF]. Interacting with virtual objects displayed in the headset is typically achieved through gaze and hand gestures performed in front of the headset.

For example, the Microsoft Hololens requires users to move their head to adjust the cursor position over the object of interest, and then raise their arm to perform an air-tap hand gesture in the tracking area located ahead of the headset.

Both gaze input and mid-air gestures have drawbacks. Mid-air gestures can quickly get tiring when the arms are in an upward position [START_REF] B Ac H Y N S K Y I | Performance and ergonomics of touch surfaces: A comparative study using biomechanical simulation[END_REF][START_REF] H I N C A P I É -R A M O S | Consumed endurance: A metric to quantify arm fatigue of mid-air interactions[END_REF], while gaze-based input suffers from precision issues [START_REF] Wa R E | An evaluation of an eye tracker as a device for computer input[END_REF] and is prone to accidental selections [START_REF]What you look at is what you get: Eye movement-based interaction techniques[END_REF][START_REF]Eye tracking in advanced interface design[END_REF]. In the case of AR headsets that estimate gaze based on head position and orientation, the situation is even worse as the object to select needs to be put in the viewport's center, making peripheral (but potentially interesting) objects leave the user's field of view. Furthermore, large gestures performed in mid-air can be socially awkward [START_REF] A N D I R A N I | Are you comfortable doing that? acceptance studies of around-device gestures in and for public settings[END_REF]. So can unnatural head movements. This might result in a mismatch between displayed content that is intended for private consumption and exaggerated movements to interact with this content.

We investigate the concept of ARPads as a means to replace or complement the default input channels of AR headsets. An ARPads can be seen as a virtual plane in which users perform small scale movements to control the cursor displayed in the headset's screen, like a trackpad does with a cursor displayed on a laptop's screen. 1 As already demonstrated in the context of remote control of distant vertical screens [START_REF] Vog E L | Subtle arms-down mid-air interaction[END_REF], this indirect way of controlling a cursor enables users to adopt more relaxed postures and perform gestures that are subtle, and thus more acceptable in public [START_REF] Q U I G L E Y | Exploring mixed-scale gesture interaction for AR applications[END_REF].

We investigate a design space for such indirect input in the context of AR headsets. We consider two dimensions: where the gestures are performed relative to the user's body (wrist, waist or thigh), and what the main orientation of those gestures is (horizontal, vertical). We implement specific designs in this space in an AR headset that has a limited-size viewport (i.e., a Microsoft Hololens 1). We conduct a study to evaluate these designs as well as two direct input baselines (gaze and hand raycast) on both discrete and continuous input tasks. Our study results indicate that: 1) indirect input can achieve the same performance as direct input while causing less fatigue than hand raycast; 2) an ARPad is more efficient when positioned close to users' wrist or waist rather than to their thigh; and 3) users perform better with an ARPad when its orientation matches that of the headset screen.

Our contributions are: 1. a design space mid-air indirect input techniques for AR; 2. a study of these input techniques compared to their state-of-the-art counterparts.

Related Work

Indirect input is a fundamental concept in HCI that received a lot of attention in the literature about pointing. We only discuss here on works that are closely related to our contribution, focusing on interaction with head-mounted displays and mid-air gestures for interacting with portable devices.

Interacting with Head-mounted Displays

Usability is a key aspect to consider if we want to facilitate the adoption of AR glasses [START_REF] Ko E L L E | All about acceptability? identifying factors for the adoption of data glasses[END_REF].

Many projects in the literature aim at improving interaction with such devices. As mentioned above, interaction with AR headsets usually relies on gaze input. While fast, gaze input suffers from precision issues [START_REF] Wa R E | An evaluation of an eye tracker as a device for computer input[END_REF]. Gaze input thus benefits from the addition of other modalities [START_REF] E N S | Pinpointing: Precise head-and eye-based target selection for augmented reality[END_REF] such as a mouse in a desktop setting [START_REF] A N D I H D E | Manual and gaze input cascaded (MAGIC) pointing[END_REF] or a handheld touch device in a mobile setting [START_REF] A N D Dac H S E Lt | Look & Touch: Gaze-supported target acquisition[END_REF]. Gaze input also suffers from the Midas Touch problem [START_REF]What you look at is what you get: Eye movement-based interaction techniques[END_REF][START_REF]Eye tracking in advanced interface design[END_REF], which can cause accidental activation commands in the absence of a proper selection mechanism. Selection with gaze is usually performed using a dwell action. It can also rely on an additional device [START_REF]What you look at is what you get: Eye movement-based interaction techniques[END_REF][START_REF] Wa R E | An evaluation of an eye tracker as a device for computer input[END_REF]. When targets are moving, selection can also be achieved with eye pursuit [START_REF] O E C Hs N E R | Interaction in virtual reality using smooth pursuit eye movements[END_REF][START_REF] Iu Ms O Mb O O N | l i n g h u r s t , M . Exploring natural eye-gaze-based interaction for immersive virtual reality[END_REF].

Mid-air gestures have also received a lot of attention as a means to interact with headmounted displays. They remove the need for additional devices and allow users to perform direct manipulation of objects that are within arm's reach [START_REF] M I N E | Moving objects in space: Exploiting proprioception in virtual-environment interaction[END_REF]. Mid-air gestures can be complemented with another modality such as speech [START_REF] Iu Ms O Mb O O N | l i n g h u r s t , M . Grasp-Shell vs Gesture-Speech: A comparison of direct and indirect natural interaction techniques in augmented reality[END_REF] or gaze [START_REF] A N D G E L L E R S E N | Gaze + Pinch interaction in virtual reality[END_REF]. Commercial solutions like the Microsoft Hololens rely on gaze for pointing and air tap gestures for selection. In this specific context of AR headsets, optical sensors for tracking hand gestures are usually mounted on the headset, looking in the direction of users' sight (e.g., [START_REF] Ya N G | Compact, low power 3D gesture sensing for interaction with head mounted displays[END_REF][START_REF] F E I N E R | Head-worn, rgb-d camerabased, bare-hand user interface with visually enhanced depth perception[END_REF]). This means that users have to perform gestures at a relatively high height, which quickly causes discomfort and fatigue [START_REF] H I N C A P I É -R A M O S | Consumed endurance: A metric to quantify arm fatigue of mid-air interactions[END_REF][START_REF] Fo R Sb E Rg | Image plane interaction techniques in 3D immersive environments[END_REF].

When users do not need to keep their hands free, an external device can be used as a controller for Head-mounted Displays. This external device can be a smartwatch [START_REF] P I E Tro S Z E K | Watchcasting: Freehand 3d interaction with off-the-shelf smartwatch[END_REF], a smartphone [START_REF]a n d B i l l i n g h u r s t , M . Using a hhd with a hmd for mobile ar interaction[END_REF], a tablet [START_REF] Vog E L | Exploring the design space for using a multi-touch tablet in virtual reality[END_REF], or a custom one [START_REF] S E Rr A N O | On-body tangible interaction: Using the body to support tangible manipulations for immersive environments[END_REF]. The tangibility of an external device usually improves comfort and precision. For instance, a smartphone can act as a trackpad for pointing content displayed on a distant screen [START_REF] A N D L A N K | Pointing at a distance with everyday smart devices[END_REF] or in AR [START_REF] C H E N | Understanding viewportand world-based pointing with everyday smart devices in immersive augmented reality[END_REF]. A spatially tracked mobile device can control 6 degrees of freedom (DOF) for manipulating spatial content [START_REF] A N D M C G U Ff I N | Integrating augmented reality with a desktop gui and smartphone interaction[END_REF][START_REF] Tat Z G E R N | Enabling smartphones for 3d interaction on mobile head-mounted displays[END_REF]. Further adding multitouch input, users can even perform advanced manipulations in VR [START_REF] Vog E L | Exploring the design space for using a multi-touch tablet in virtual reality[END_REF]. Conversely, while portable devices can address weaknesses of HMDs in terms of input, HMDs can also address weaknesses of portable devices in terms of output [START_REF]Bishare: Exploring bidirectional interactions between smartphones and head-mounted augmented reality[END_REF][START_REF] K R A N Z | Challenges in mobile multi-device ecosystems[END_REF]. For instance, an AR display can enlarge a smartphone's display to accommodate a large information space [START_REF] A N D M C G U Ff I N | Enlarging a smartphone with ar to create a handheld vesad (virtually extended screen-aligned display)[END_REF] or to distribute widgets across displays [START_REF] H E I N I S C H | Multifi: Multi fidelity interaction with displays on and around the body[END_REF].

For tasks performed in the office, users can lay a tablet on their desk and use its soft keyboard for text entry in VR [START_REF] W I T Z A N I | Text entry in immersive head-mounted display-based virtual reality using standard keyboards[END_REF]. They can also use a mouse [START_REF] F R E Y | Pointing in spatial augmented reality from 2d pointing devices[END_REF] or a physical keyboard [START_REF] W I T Z A N I | Text entry in immersive head-mounted display-based virtual reality using standard keyboards[END_REF]. In a mobile context, a smartwatch can be used to interact with remote content: for raycast pointing [START_REF] P I E Tro S Z E K | Watchcasting: Freehand 3d interaction with off-the-shelf smartwatch[END_REF], for indirect pointing [213,[START_REF]Bishare: Exploring bidirectional interactions between smartphones and head-mounted augmented reality[END_REF] or for text input [START_REF] A N D L E E | Typing on a smartwatch for smart glasses[END_REF]. All these projects advocate using an external input device to interact with HMDs. We argue that bare-hand input can complement such techniques or replace them when an external device is not available. Bare-hand input also has the advantage of making HMDs stand-alone devices, and of leaving users' hands free for other actions.

A hands-free alternative consists in using the user's body as an input device with largeamplitude gestures performed on the whole body [START_REF] L E C Ol I N E T | Impact of semantic aids on command memorization for on-body interaction and directional gestures[END_REF] or with finger gestures performed on specific parts of the body (e.g., on the forearm [START_REF] H A Rr I S O N | Appropriating the body as an input surface[END_REF], on the hand palm [START_REF] D E Z F U L I | Palm-based interaction with head-mounted displays[END_REF], on the abdomen [START_REF] Vo | Belly gestures: Body centric gestures on the abdomen[END_REF], and even on the face [START_REF] S H I N | Designing socially acceptable hand-to-face input[END_REF][START_REF] S E Rr A N O | Exploring the use of hand-to-face input for interacting with head-worn displays[END_REF][START_REF] Yam A S H I Ta | Turning your cheek into an input surface by embedded optical sensors on a head-mounted display[END_REF]). Wearable devices also support portability. Dobbelstein et al. have proposed wearables for interacting with headset screens with, e.g., a touch-enabled belt [START_REF] D O B B E Ls T E I N | Belt: An unobtrusive touch input device for head-worn displays[END_REF] and pocket [START_REF] D O B B E Ls T E I N | Pocketthumb: A wearable dual-sided touch interface for cursor-based control of smarteyewear[END_REF]. Finally, foot taps have also been investigated recently as a means to perform menu selection with head-mounted displays [START_REF] S C H M I T Z | Mind the tap: Assessing foot-taps for interacting with head-mounted displays[END_REF]. However, these latter solutions require custom sensors, have limited input capabilities and might raise questions regarding social acceptance.

Overcoming Fatigue of Mid-air Gestures

In light of the above, mid-air gestures remain a very good candidate for interacting with headset screens. However, tracking such gestures in a portable setting is not trivial. Some research prototypes embed cameras in clothes, shoes or jewels. For example, the Gesture Pendant [START_REF] Ta R N E R | The gesture pendant: A self-illuminating, wearable, infrared computer vision system for home automation control and medical monitoring[END_REF] is a wearable device used to control home environments with mid-air gestures. SixthSense [START_REF]Sixthsense: A wearable gestural interface[END_REF] extends the Gesture Pendant by combining a projector and a camera in a pendant. It can project digital content in the physical environment in front of the user, and capture gestures that are performed at chest height. ShoeSense [START_REF] M Ü L L E R | A new perspective on gestural interaction and wearable applications[END_REF] augments a shoe with a Leap Motion to track large-amplitude arm and hand gestures to control wearable applications. Such wearable tracking equipment has the advantage of allowing users to perform mid-air gestures with arms in a position lower than when optical sensors are mounted on the headset itself.

OwnerShift [START_REF] F E U C H Tn E R | Facilitating overhead interaction in virtual reality with an ownership-preserving hand space shift[END_REF] is a recent technique for Virtual Reality (VR) headsets that enables users to progressively relax their posture during long-lasting interactions. Two optical sensors are mounted on the headset itself, one oriented along users' sight, the other oriented downward. Users see a virtual representation of their hands, which gets progressively shifted up so as to encourage users to shift their real hands down, smoothly transitioning from overhead to waist. OwnerShift builds upon the set of techniques that play with the rubber hand illusion [START_REF] B O T V I N I C K | Rubber hands 'feel' touch that eyes see[END_REF]. For example, the Go-go Interaction Technique virtually extends users' arms to reach distant objects without walking [START_REF] P O U P Y R E V | The Go-go interaction technique: Non-linear mapping for direct manipulation in VR[END_REF]. The converse approach also works: instead of virtually extending users' body, virtual objects can be brought closer in motor space while preserving their positions in visual space [137]. However, AR gives less freedom than VR as it makes it challenging to modify the representation of users' body.

Gunslinger [START_REF] Vog E L | Subtle arms-down mid-air interaction[END_REF] addresses the fatigue problem of mid-air gestures in a more radical manner.

Relying on what has been later called at-your-side gestures [START_REF] Wa | Exploring at-your-side gestural interaction for ubiquitous environments[END_REF] and earlier used for zooming actions in [START_REF] Wagn E R | Mid-air pan-and-zoom on wall-sized displays[END_REF], Gunslinger allows users to remote control a large display using small-scale movements that are performed in a relaxed posture. One Leap Motion controller is attached to each thigh, tracking both hands when arms are along the body. Users point with their dominant hand following a trackpad metaphor, and activate commands with specific postures of their non-dominant hand. Gunslinger's support for pointing is very similar to a horizontal ARPad attached to users' thigh in our design space, with one notable difference: Gunslinger works in relative mode (with clutching actions) while ARPads work in absolute mode.

Extending Input Space in the Air

Extending the interaction space in the air is a strategy that has been implemented for devices other than head-mounted ones. It was initially investigated for small devices because of the limited size of their interaction surface, which causes occlusion and precision issues. For example, the Gesture Watch [START_REF] Ly O N S | The gesture watch: A wireless contact-free gesture based wrist interface[END_REF] allows users to perform mid-air gestures above the watch. Gestures are performed with the dominant hand over the watch worn on the non-dominant arm (the watch is equipped with an array of proximity sensors). The Abracadabra wristwatch [START_REF] H A R R I S O N | Wireless, high-precision, and unpowered finger input for very small mobile devices[END_REF] extends a watch's sensing area above the surface in the same way, but differs in the sensing technology: the finger from the dominant hand wears a magnet, making it trackable by the watch's magnetometers. Here again, those techniques share some similarities with ARPads when they are located on users' wrist with a horizontal orientation.

Building upon the concept of continuous interaction space, that considers a touch surface and the space around it as a continuum [START_REF] G R E E Nb E Rg | The continuous interaction space: Interaction techniques unifying touch and gesture on and above a digital surface[END_REF], Chen et al. equip a smartphone with a depth camera in order to extend the vocabulary of possible commands with air+touch gestures [START_REF] C H E N | Interweaving touch & in-air gestures[END_REF]. The combination of touch and mid-air input has also proven efficient for performing large amplitude slide gestures on a smartphone [START_REF] R E K I K | Large display pointing in a continuous interaction space around a mobile device[END_REF] or on a smartwatch [START_REF] Pa R K | Designing touch gestures using the space around the smartwatch as continuous input space[END_REF].

BODYPOSITION

Waist

Thigh

Wrist

Horizontal Vertical

PADORIENTATION

Chest

90° x-axis rotation parallel

Vertical (rotated)

90° y-axis rotation

ARPad Design Space

Mid-air indirect input, as a concept, is not novel. But all possible designs have not been systematically investigated. This section describes our design space for ARPads. This space encompasses techniques from the literature, as well as novel ones that build upon prior work on mid-air gestures for portable devices.

An ARPad is an imaginary pad on which users can move their hands to control the position of a cursor displayed in the headset screen. Like imaginary interfaces [START_REF] G U S Ta F S O N | Imaginary interfaces: Spatial interaction with empty hands and without visual feedback[END_REF], users operate ARPads with their bare hands and without any visual support. However, ARPads differ from the concept envisioned by Gustafson et al. where imaginary interfaces are operated with direct hand gestures. An ARPad is rather an imaginary device, which is used for indirect control of a distant screen. The number of possibilities for implementing such a mid-air pad is large. Our design space, illustrated in Figure 1. The first dimension, B o dyP os it i o n, is directly motivated by previous work on mid-air gestures for mobile contexts (discussed in the previous section). Tracking mid-air gestures while leaving users free to move in their environment requires attaching optical sensors to them. Depending on which body part the sensors are attached to, users will be able to perform gestures at different locations relative to their body. In our design space, dimension B o dyP os it i o n represents the body part close to which users perform mid-air gestures. Here, we specifically consider three body parts: Thigh, Waist and Wrist (Figure 1.2). Thigh is an obvious candidate as it has already been used for cursor control in the Gunslinger project [START_REF] Vog E L | Subtle arms-down mid-air interaction[END_REF]. Although not focusing on indirect pointing, mid-air gestures relative to users' Wrist have also been considered for improving interaction with smartwatches (e.g., [START_REF] H A R R I S O N | Wireless, high-precision, and unpowered finger input for very small mobile devices[END_REF][START_REF] Ly O N S | The gesture watch: A wireless contact-free gesture based wrist interface[END_REF]). In projects where mid-air gestures were tracked by a pendant [START_REF] Ta R N E R | The gesture pendant: A self-illuminating, wearable, infrared computer vision system for home automation control and medical monitoring[END_REF][START_REF]Sixthsense: A wearable gestural interface[END_REF], users were performing gestures in front of their Chest. However, considering empirical results about the fatigue of mid-air gestures when arms are held high in the air [START_REF] H I N C A P I É -R A M O S | Consumed endurance: A metric to quantify arm fatigue of mid-air interactions[END_REF], Waist is a more promising option to study. When gestures are tracked at waist level, users can keep their arms low and should experience less fatigue.

The second dimension, Pa d O r ie n tat i o n, describes the orientation of the 2D plane within which the pad stands. We consider Vertical pads, which correspond to cases where the ARPad's orientation matches that of the headset screen when users look straight ahead. We also consider Horizontal pads, which correspond to cases where the ARPad is rotated 90 • around the x-axis of the headset screen. This configuration is particularly interesting, as we can expect it to be familiar to users who have experience with a physical trackpad on a laptop computer. While we limit our first investigation to these two specific values, other values might be worth investigating. For example, for an ARPad located close to users' Thigh, we could also consider the case where the ARPad is rotated 90 • around the y-axis of the headset screen in order to allow for movements parallel to the sagittal plane that might be comfortable. We could even go beyond 2D planes and consider curved surfaces that might better fit the trajectory of limbs around joints.

ARPad Prototypes

We implemented the six ARPads identified above using a Microsoft HoloLens and a Leap Motion controller. We relied on Unity (2017. 2.4.f1) and the MixedRealityToolkit (2017. 4.3.0) to implement our own AR cursor. We reused the IPointingSource interface from MRTK, and registered the cursor in the FocusManager instance as the main one.

The pointer position gets updated each frame depending on what has been sensed with the Leap Motion (without being affected by head motion). Object picking in the AR scene is achieved through raycasting from this pointer.

We rely on a Leap Motion controller for tracking users' hands. Depending on the pad's orientation (B o dy P o s i t i o n), we attach the Leap Motion to users with pieces of velcro tape, as illustrated in Figure 1.2. As opposed to Gunslinger [START_REF] Vog E L | Subtle arms-down mid-air interaction[END_REF], which implements The LeapMotion provides a 3D coordinate system whose origin is the device itself. We transform coordinates from this cuboid to the ARPad's plane by ignoring the axis that is orthogonal to that plane. This means that an ARPad is a cuboid inside which all movements are projected on a plane rather than an actual plane on which users would have to perform unnatural co-planar movements.

Informed by pilot studies, we implemented a control display ratio of 1cm:1. , which is uniform on both the x-and y-axes (Figure 1.4) . This means that a 1cm hand movement will make the cursor move 1.38 • in visual angle.

We use an external computer for communication between the Leap Motion and the HoloLens. As the Leap Motion cannot be directly plugged to the HoloLens, it sends data to the computer via USB, which in turn sends them to the HoloLens via sockets.

Experiment

Our study aims at assessing 1) how indirect input performs in comparison with direct input, and 2) what ARPad designs perform best.

Direct Input Baselines

HeadGaze -An AR headset cursor is usually controlled through gaze, allowing users to directly look at the object they want to select. In current headsets such as the Microsoft HoloLens, gaze orientation is actually estimated based on head orientation. Our first baseline (HeadGaze) corresponds to this default cursor control technique.

HeadHand -In our second baseline, users directly control the cursor with their hand.

The cursor position results from a raycast along the vector headset-hand. As opposed to indirect techniques, HeadHand implements a 1:1 control display ratio to make the cursor position match that of the hand. This corresponds to the Image-Plane technique in [START_REF] N Ac E N Ta | A comparison of ray pointing techniques for very large displays[END_REF].

Hypotheses

Our experiment compares the eight techniques illustrated in Figure 1. [START_REF] A N D L E E | Typing on a smartwatch for smart glasses[END_REF]. Our hypotheses regarding their comparative performance were as follows:

H 1 Users experience more fatigue with HeadHand than with ARPads. This first hypothesis is the actual motivation for using indirect input as an alternative to hand direct input. Freehand interaction when holding arms up in the air are tiring [START_REF] H I N C A P I É -R A M O S | Consumed endurance: A metric to quantify arm fatigue of mid-air interactions[END_REF]. ARPads let users keep their arms in positions that are more comfortable than HeadHand does.

H 2 Users perform better with direct techniques than with ARPads. Visual and motor spaces are not spatially aligned with ARPads. Such an indirection can degrade user performance, as is the case for a typing task on a software keyboard in Virtual Reality [START_REF] W I T Z A N I | Text entry in immersive head-mounted display-based virtual reality using standard keyboards[END_REF]. We expect that this indirection will also be cognitively demanding when pointing with an ARPad, and that it may affect user performance.

H 3 Vertical ARPads perform better than horizontal ones. The level of indirection between the motor space and the visual space is even higher when both spaces differ in their orientation. This might degrade user performance even more.

H 4 An ARPad at Wrist position performs better than at Waist or Thigh positions.

When the ARPad is located on the Wrist, users can move their non-dominant hand to adjust the plane's location and orientation. This enables bimanual actions where the non-dominant hand sets the context for precise actions performed with the dominant one. Such an asymmetric division of labor might prove efficient [START_REF]Asymmetric division of labor in human skilled bimanual action: The kinematic chain as a model[END_REF].

Participants

16 participants (11 men and 5 women) volunteered for the experiment. All of them were right-handed. Our experiment started with a question about their prior experience with AR and VR headsets. Twelve participants reported that they never to rarely used AR headsets, and the other four answered sometimes to daily. Eleven participants answered never to rarely for VR headsets, and five of them sometimes to daily.

Apparatus

Our experiment ran on a Microsoft HoloLens 1 (30 A Leap Motion controller has some limitations regarding the range within which it can accurately track users' hands. In particular, the hand must not be too close to the device (i.e., distance should be at least 8cm). We made users aware of this limit by providing some feedback. When users' hand is getting close to the limit, the border of the AR window progressively turns red, indicating that they should adjust their hand position.

The operator instructed the participants about how to interpret this visual feedback at the beginning of the experiment.2

Tasks

As in several recent studies (e.g., [START_REF] F E U C H Tn E R | Facilitating overhead interaction in virtual reality with an ownership-preserving hand space shift[END_REF][START_REF] E N S | Pinpointing: Precise head-and eye-based target selection for augmented reality[END_REF][START_REF] S C H M I T Z | Mind the tap: Assessing foot-taps for interacting with head-mounted displays[END_REF]), our tasks focus on 2D interaction. This does not limit ARPads to 2D interactions, however. An ARPad can also be used for interacting with 3D content same as mouse input can control 3D software on desktop workstations. Participants perform pointing tasks in diverse directions by acquiring a series of targets that are laid out in a circular manner. As illustrated in Figure 1.5, participants acquire eight targets successively. The next target to acquire is colored white. It turns green as soon as the cursor enters it. The participant then has to keep the cursor inside the target for 500ms (dwell) to make it disappear. This makes the next target to acquire turn white.

We chose to rely on dwell actions for target selection so as not to involve any additional input device or gesture recognition engine that could have introduced some noise.

All targets in a series have the same size. Some series feature Large targets (2.36 • ), Medium targets (1.18 • ) or Small targets (0.59 • ). Targets are laid on a 7.78 • -diameter circle. 3 The difficulty of the pointing task is thus an inverse function of target size.

For each elementary pointing task, we measure acquisition time (i.e., the time interval between the last target's disappearance and the time at which the current target is selected), as well as the number of errors (i.e., how many times the cursor leaves the target before it gets selected). As the cursor is located at the center of the scene at the beginning of a series, its distance to the first target is smaller than the diameter of the ring on which targets are laid out. We thus ignore the first target acquisition of all series.

3 One visual degree corresponds to 3.39cm in the AR vertical window displayed at 194 cm from the headset, and to 1.73cm in motor space for indirect techniques. Our conditions thus correspond to 8 cm-, 4 cm-or 2 cm-targets laid out on a 26.5 cm-diameter circle in AR window, and to 1.73 cm-, 0.87 cm-or 0.43 cm-targets on a 5.74 cm-diameter circle in motor space for indirect techniques. Participants have to follow a circular target (0.59 • in diameter) that moves in a quasirandom manner within a square of 11.66 • ×11.66 • (Figure 1.6). Participants are instructed to follow this target in order to keep the cursor as close as possible to it. We limited the duration of pursuit tasks to 30 seconds. The target trajectory seemed random to participants, but was actually pre-computed in order to balance the difficulty across participants and techniques. In our experiment design, users had to perform a series of three pursuit tasks with each of the eight techniques. We thus generated three target paths in advance using a sum of four sinusoids for both the x-and -y directions with parameter values reported in [START_REF] F E U C H Tn E R | Facilitating overhead interaction in virtual reality with an ownership-preserving hand space shift[END_REF]. We used the same three paths for each participant × technique block.

For each pursuit task, we consider the mean angular distance between the target and the cursor (we incrementally compute this value at each frame).

Design and Procedure

We follow a within-subject design with the primary factor T e c hn i q u e. Presentation order between the couple of subblocks was also counterbalanced across participants and body positions. Finally, within each subblock, the presentation order of tasks was always the same: a tutorial session, followed by all pointing tasks and then all pursuit tasks.

In the tutorial session of a block, the operator quickly introduced the technique to be tested. When the technique was an ARPad, the operator used a cardboard sheet to illustrate where the pad stood and what its orientation was. Participants then had to point at eight targets, one at each corner of the screen and one at the middle of each side.

If needed, participants were free to perform this training session again at will, before actually starting the block. Participants then completed 2×3 pointing series (one series actually consists of eight elementary pointing tasks as detailed above). Pointing series were grouped by three, which were always presented in an increasing order of difficulty: Large, Medium, and Small targets. The first group of three is considered practice, and the other one is used for analysis. Finally, participants performed three pursuit tasks as described above. Here, the first one is considered practice while the other two are for analysis.

We took particular care not to have a transfer of fatigue from one technique to the other.

As Figure 1.7 illustrates, our experiment was divided into two sessions of 1h30mn each, performed on consecutive days. Each session was also divided into two sub-sessions of 30 minutes, separated by a 30-minute break to let participants rest. They could rest longer if they wanted to. In all cases, they started a B o dy P o s it i o n block only after having self-assessed themselves as not tired. We counterbalanced presentation order for the two Pa d O r ie ntat i o n conditions within a block so as not to have always the same transfer direction from one orientation to the other.

Participants had to answer several questionnaires during the course of the experiment. The first one had to be filled in at the very beginning of the procedure. It consisted of the consent form and two questions about their prior experience with head-mounted displays. Then, at the end of each B o dyP os it i o n condition, they had to rate the mental demand, the physical demand and the perceived performance of the technique for each of the two pad orientations (using definitions and associated 20-point Likert scales from NASA-TLX). Collecting participants' comparative feedback between techniques is somewhat conflicting with the aim of minimizing transfer of fatigue between conditions. As sessions were run across two separate days, participants might have had difficulties recalling conditions that they tested on the first day. To help participants remember each technique, conditions were illustrated with the pictograms used in Figure 1. [START_REF] A N D L E E | Typing on a smartwatch for smart glasses[END_REF]. They were displayed in each instruction message all along the experiment, and were printed in the questionnaires.

Results

We use repeated measures a n ovas and post-hoc paired t-tests for quantitative analyses (with aggregated data, after having checked that there is no strong violation of normality).

We use a rt a n ovas and post-hoc paired Wilcoxon signed-rank tests for qualitative analyses. To decrease the probability of Type I errors, we apply Bonferroni-Holm corrections. Our graphs represent data distribution using notched box plots (notches show the 95% confidence interval for the median). We use R [START_REF] R C O R E T E A M | R: A language and environment for statistical computing[END_REF] to run our analyses.

a n ovas rely on the ez [START_REF] Aw R E N C E | Easy Analysis and Visualization of Factorial Experiments[END_REF] and ARTool [START_REF] Artool | Aligned Rank Transform for Nonparametric Factorial ANOVAs[END_REF] Finally, we analyze participants' feedback through questionnaires.

Pointing Task

Direct and Indirect Techniques (Time). Figure 1.8 shows pointing time (Time) for each T e c hn i q u e, and Figure 1.9 shows a breakdown of these data per Ta rg e t W i d t h. An a n ova for model Time ∼ T e c hn i q u e × Ta rg e t W i d t h reveals a significant effect of T e c hn i q u e (F 7,105 = 9.02, p < 0.001, η 2 G = 0.16), and of Ta rg e t W i d t h (F 2,30 = 9340, p < 0.001, η 2 G = 0.69) on Time. Moreover, the a n ova reveals a significant T e c h n i q u e ×Ta rg e t W i d t h interaction effect (F 14,210 = 3.28, p < 0.001, η 2

G = 0.06).
Unsurprisingly, Ta rg e t W i d t h has a significant effect on Time. The larger the target, the easier the task, as reflected by the shorter pointing time.

Direct and Indirect Techniques (P recision).

To select a target, participants had to keep the cursor for at least 500ms within the target's boundaries (dwell). We counted a T argetLeave (T L) event each time the cursor went out of the target, before successfully selecting it. The number of such events provides an indicator of users' precision with the different techniques, as it increases with the difficulty of stabilizing the cursor.

For this analysis, we consider the full set of collected data (direct and indirect techniques).

With averages of 1.71 and 1.36 respectively, HeadGaze and HeadHand have a significantly higher number of T L events than indirect techniques (p's < 0.001), which have an average of T L=0.58 by elementary pointing task. Vertical ARPads at the Thigh position have a slightly higher number of such events (0.74) than other ARPads, and actually significantly more than ARPads at the Wrist location (p's = 0.013), which have an average of T L=0. [START_REF] G Ro S S M A N | Candid interaction: Revealing hidden mobile and wearable computing activities[END_REF].

This suggests that the two direct techniques, HeadGaze and HeadHand, suffer from precision issues, which probably explains why their comparative performance decreases with small targets.

Figure 1.10 shows how collected data fit Fitts' law for each technique. We can observe that the point for small targets is systematically above the linear regression line, and even more above the line that would pass through the other two points (large and medium targets). This indicates a "small target effect" [START_REF]a n d D r agi c ev i c , P . Effects of motor scale, visual scale, and quantization on small target acquisition difficulty[END_REF], with small targets being more difficult to acquire than what Fitts' law predicts. This effect is stronger for the direct techniques than it is for the indirect techniques, and it is particularly strong for HeadGaze. This accounts for the low adjusted r 2 for the direct techniques, as well as the high (i.e., bad) slope of their regression lines. Again, these results are consistent with the precision issues that users encounter with direct techniques. Interestingly, we can also notice that WristHorizontal has the lowest (i.e., best) slope, but a very high intercept. This suggests that users might have a bit of trouble in initiating a movement with this technique, but that it likely better scales with the task difficulty.

Pursuit Task

For the pursuit task, our measure is the average angular distance between the target and the cursor (AngularDistance) over a 30s trial. Direct and Indirect Techniques (AngularDistance). An a n ova for the model Angu-larDistance ∼ T e c hn i q u e reveals a significant effect of T e c hn i q u e (F 7,105 = 7.65, [START_REF]A survey of augmented reality[END_REF] shows the comparative performance of techniques. The best performing techniques are HeadGaze, WristVertical, HeadHand, and WaistVertical. Corrected t-tests (n = 28) do not reveal any significant difference between these four techniques. This suggests that HeadHand performs comparatively better for continuous tasks than it does for pointing tasks.

p < 0.001, η 2 G = 0.27). Figure 1.
Pairwise t-tests also reveal a few interesting differences. First, the two ARPads at Thigh perform significantly worse than the two direct techniques and than WristVertical (p's < 0.01). Second, WaistHorizontal performs significantly worse than both WristVertical (p = 0.015) and HeadGaze (p = 0.031). each score is assigned a color, and the color patch's size is proportional to the ratio of times this score was given. Right (green) is better.

Indirect Techniques only (AngularDistance). To analyze the effect of

Qualitative Results

Figure 1.12 shows participants' qualitative feedback for each technique along the following scales: perceived performance, physical demand, and mental demand. a rt a n ovas reveal a significant effect of T e c hn i q u e for each of these scales: Participants' perceived performance is consistent with quantitative data. They feel most efficient with WristVertical, WaistVertical and HeadGaze. However, few of the differences are statistically significant. Only WristVertical and HeadGaze are graded significantly better than ThighVertical (p = 0.040 and p = 0.014).

Grades associated with physical demand are in line with H 1 : HeadHand is the most tiring technique on average. HeadHand is actually significantly more demanding than all other techniques (p's < 0.038) except WristHorizontal (p = 0.098). Regarding physical demand, the only other significant differences are about HeadGaze, which is graded as less tiring than WristHorizontal (p = 0.005) and WristVertical (p = 0.022). Average grades for Wrist techniques are actually high. This is interesting: while participants were fast and precise with WristVertical, they also found it quite tiring.

Regarding mental demand, participants found it particularly demanding to interact with ThighVertical, which has been graded significantly worse than both direct techniques (p's < 0.037), WaistVertical (p = 0.013), and WristVertical (p = 0.011). Also, for Waist and Wrist, horizontal ARPads have been graded significantly more demanding than vertical ones (p = 0.021 and p = 0.030, respectively).

Regarding Pa d O r ie ntat i o n, we also asked participants about their preferred orientation for each B o dyP os it i o n. We also asked them about their preferences between the two direct techniques: they all preferred Gaze over Hand.

The results of the above two paragraphs are in line with H 3 . The larger the difference between the visual space and the motor space, the less usable an ARPad is (more tiring and less liked). However, the Thigh seems to be an exception: ThighVertical received a very poor grade for mental demand and there is no clear Pa d O r ie ntat i o n preference between the two ARPads. This echoes participants' feedback during the experiment: seven of them mentioned having trouble understanding how the cursor's position was mapped to hand movements in both Thigh conditions. This might be because of ARPads' lateral location in comparison with Wrist and Waist positions which are more centered on the sagittal plane. For example, one participant mentioned that they were expecting vertical ARPads at the Thigh to be parallel to the sagittal plane (i.e., Vertical (rotated) in our design space -Figure 1.1). We designed it this way to be consistent across body positions, but it might be interesting to reconsider our design for the Thigh position. For example, we could position the Leap Motion to make it track in front of the thigh as opposed to sideward, as proposed by two participants.

Implications for Design

Data collected in our experiment support most of our hypotheses, and bring additional insights from which we can derive guidelines about implementing indirect input for AR headsets.

Direct Input vs. Indirect Input

First, users actually perform better on average with direct gaze input than with any of the other techniques we have tested. However, well-designed ARPads allow users to reach a similar level of performance overall. In particular, our study showed that direct input suffers from precision issues, and that indirect input might be better suited to interaction with small objects.

Second, as we hypothesized, direct hand input is very tiring. However, our study also showed that direct hand input (HeadHand) performs quite well for continuous tasks, better than it does for pointing tasks. It thus remains a good candidate to consider for brief manipulations. However, it quickly causes fatigue, so its use should be limited to manipulations that do not last too long. To give a rough estimate, pursuit tasks in our experiment lasted only 30s but already caused much fatigue. Direct hand input should definitely be considered neither for long manipulations of objects nor for repetitive tasks.

Guidelines for the Design of Indirect Input

The data collected yield guidelines for practitioners regarding ARPads's position as well as their orientation.

Users' performance with Thigh was poor in comparison to Waist and Wrist. Comparison between Waist and Wrist is more nuanced. Participants performed slightly better with ARPads located at Wrist for discrete tasks. However, qualitative feedback also showed that they found them more tiring than ARPads at Waist position. Furthermore, one limitation of the Wrist position is that it prevents users from performing bi-manual interactions for, e.g., controlling two cursors. ARPads at Wrist are good candidates for short-duration interactions, and they might be particularly relevant for some situations such as, e.g., remote control in a seated position. But, overall, Waist seems to be the best compromise. Users were efficient and felt comfortable with this body position.

Participants performed better and expressed a clear preference for Vertical ARPads over their Horizontal counterparts. Despite experience with trackpads on laptops, a difference in orientation between visual and motor spaces seems to have a negative impact.

Conclusion and Future Work

AR usually relies on bare-hand input. However, this can cause fatigue when performing long-lasting tasks. Relying on device-based techniques is an interesting option that, however, prevents users from keeping their hands free. In this chapter, we answer our first research question by studying ARPads, which are mid-air indirect input techniques for AR. We contribute guidelines for designing such indirect input that we derive from an empirical investigation of design space for ARPads. Our study compared different designs for ARPads and measured their comparable performances with direct input baselines. In our study, gaze input performed very well but suffered from precision issues. Participants' performance with ARPads suggest that well-designed indirect input can achieve the same level of performance as direct gaze input. In addition, ARPads are particularly relevant for long-lasting or precise interactions as they minimize fatigue and cope with precision issues related to gaze input.

We do not advocate for replacing direct input with indirect input, however. Direct input actually performed well in our study and is probably more intuitive. We rather believe that both types of input should co-exist, allowing users to freely choose the type of input that best suits their context and task. For example, both types of input could be combined in bi-manual techniques where one hand performs direct object selection, and the other hand performs long or fine parameter adjustments using indirect input. Users could also decide to rely on indirect input when they want their interactions to be more subtle [START_REF]Supporting subtlety with deceptive devices and illusory interactions[END_REF] or rely on direct input when, on the contrary, they want them to be transparent to others [START_REF] G Ro S S M A N | Candid interaction: Revealing hidden mobile and wearable computing activities[END_REF]. Supporting the coexistence of both requires implementing mode switching techniques. Such mode switches could be implicit, relying on hand location relative to headset orientation, in the spirit of the Gaze-Shifting technique for pen input [START_REF] C H O N G | Gaze-Shifting: Direct-Indirect input with pen and touch modulated by gaze[END_REF].

Finally, this preliminary study suggests directions for future work. First, ARPads tested in this study implement absolute control with a headset that has a limited-size viewport.

Considering absolute control was necessary to test the effect of an ARPad's position on users' performance without any confound from a posture recognition engine (for e.g., clutching actions for relative control). A larger viewport would require considering one or several of the following strategies: increasing the CD gain, enlarging ARPads, or implementing a clutching mechanism.

Second, complementing ARPads with subtle audio or haptic feedback could help users get a better sense of where the pad stands in the air. This is particularly relevant when relying on technology such as the Leap Motion, which has tracking limitations.

Third, our informal observations suggest that users do not adopt planar movements when interacting with an ARPad. A fine-grained analysis of users' movements could help refine the pads' orientation, or even consider pad surfaces that are not planar. Finally, in our experiment, participants were interacting with graphical objects that were displayed in a world-anchored window that was in front of them. It would be interesting to study cases where users are not well positioned in front of the window they are interacting with.

With such interaction, AR could be used for a wider range of daily tasks as it is no longer limited to short-lasting tasks. In this chapter, we have focused on motor performance and comfort when interacting with AR, without reflecting upon its usefulness in a daily context. However, such low-level considerations are a necessary prerequisite, as daily tasks often involve long-lasting operations. Now that we have presented a first investigation over this topic, we can investigate interesting use cases in a daily life context.

Introduction

Augmented Reality Head-Mounted Displays (ARHMDs) have capabilities that distinguish them from other mobile devices: they render stereoscopic imagery seamlessly superimposed on what users see in their field of view, and they enable manipulation of this imagery via freehand gestures coupled with head-and gaze-tracking. As such, ARHMDs have much potential to complement smartphones and smartwatches, which render imagery on a much smaller physical surface but with a much higher display quality, while also enabling tactile input. Combined together, these different devices are shaping a future where users interact with Mobile Multi-Device Environments [START_REF] K R A N Z | Challenges in mobile multi-device ecosystems[END_REF]. In such environments, users can select the best device for a task based on its unique strengths, but they should also be able to use those devices together as one single, powerful, and seamless interactive system.

The combination of ARHMDs with other personal devices has opened up a large design space. Of particular interest is the combination of smartphones and ARHMDs, which can be made to work in tandem, as explored at length in the BISHARE design space [START_REF]Bishare: Exploring bidirectional interactions between smartphones and head-mounted augmented reality[END_REF]. For instance, smartphones can be used as high-precision input devices to manipulate spatial AR content [START_REF] Tat Z G E R N | Enabling smartphones for 3d interaction on mobile head-mounted displays[END_REF], while ARHMDs can enlarge a smartphone's display capacity [START_REF] H E I N I S C H | Multifi: Multi fidelity interaction with displays on and around the body[END_REF][START_REF] A N D M C G U Ff I N | Enlarging a smartphone with ar to create a handheld vesad (virtually extended screen-aligned display)[END_REF] or show 3D content in the air. These are only two examples from a much richer design space, in which many possibilities remain to be explored. In this chapter, we explore a specific part of the BISHARE design space [START_REF]Bishare: Exploring bidirectional interactions between smartphones and head-mounted augmented reality[END_REF], enhancing smartphones with ARHMDs.

The BISHARE design space identifies high-level dimensions of smartphone+AR HMD combinations, covering a very large set of designs but learning out details. We introduce a lower-level design space to specifically describe the case where the AR HMD is used to enhance a smartphone-centric application. Our two main dimensions are (1) the function of the holographic content, and (2) its location around the smartphone's screen. We populate this design space with examples from the literature and novel ideas. Then, we study a suite of AR-enhancements for phone-centric interaction. Acknowledging the strong legacy bias associated with smartphones, which makes users resort to well-known interaction styles whenever possible [START_REF] H O L Z | Cross-device taxonomy: Survey, opportunities and challenges of interactions spanning across multiple devices[END_REF][START_REF]Bishare: Exploring bidirectional interactions between smartphones and head-mounted augmented reality[END_REF], AR-enhanced widgets are augmentations of those traditionally found on this type of device. We make use of AR capabilities to distribute output by offloading widgets "in the air" around the phone,1 as well as to distribute input between touch and mid-air gestures. Both distribution strategies free up precious screen real-estate on the phone, and limit situations where content gets occluded as well, since users interact partly "around the device" [START_REF] J O N E S | Around device interaction for multiscale navigation[END_REF][START_REF] Ro S H A Nd E L | Towards using embedded magnetic field sensor for around mobile device 3d interaction[END_REF]. Additional input degrees of freedom may also help users perform tasks that require precision.

These UI distribution strategies that consist of offloading widgets in the air when applicable, and making indirect control possible through movements in the air (as a complement or replacement to touch). Figure 2.1 illustrates the potential advantages of such an approach: 1) more widgets always visible in the air and more space on the phone's screen to display content in high resolution, and 2) indirect control with hand movements in the air to avoid potential occlusion and to enable higher-precision control.

We discuss design considerations about applying such a strategy to phone widgets, and report on a series of exploratory studies to identify viable designs and estimate their performance.

Our contributions are: 1. a design space of smartphone + ARHMDs applications; 2. a series of detailed designs for AR-enhanced phone widgets; 3. an exploratory study of these widgets compared to their state-of-the-art, touch-only counterparts.

Related Work

Phone-centric distributed UIs constitute only two cells in a larger design space of bidirectional interactions between smartphones and head-mounted augmented reality [START_REF]Bishare: Exploring bidirectional interactions between smartphones and head-mounted augmented reality[END_REF], which is itself only one item in the cross-device interaction taxonomy [START_REF] H O L Z | Cross-device taxonomy: Survey, opportunities and challenges of interactions spanning across multiple devices[END_REF]. We refer the reader to these two recently-published and complete articles, and only focus here on work that directly relates to distributed UIs in a mobile context: interacting above and around handheld or wearable devices; augmenting mobile 2D input; multi-device mobile systems involving augmented reality technology.

Interacting Above and Around Mobile Surfaces

While direct touch remains a very effective way to interact with a mobile device, barehand interaction around the device has several interesting properties. It enables a greater physical input volume beyond the small touchscreen [START_REF] A N D H O D G E S | Multi-"touch" interaction around small devices[END_REF][START_REF] J O N E S | Around device interaction for multiscale navigation[END_REF][START_REF] Pa R K | Designing touch gestures using the space around the smartwatch as continuous input space[END_REF][START_REF] H A R R I S O N | Wireless, high-precision, and unpowered finger input for very small mobile devices[END_REF], sometimes with higher expressiveness [START_REF] C H E N | Interweaving touch & in-air gestures[END_REF][START_REF] O F E K | Interaction around unmodified mobile devices using sunglasses[END_REF]. It partially addresses problems of content occlusion [START_REF] A N D H O D G E S | Multi-"touch" interaction around small devices[END_REF][START_REF] J O N E S | Around device interaction for multiscale navigation[END_REF][START_REF] H A R R I S O N | Wireless, high-precision, and unpowered finger input for very small mobile devices[END_REF][START_REF] O F E K | Interaction around unmodified mobile devices using sunglasses[END_REF].

It provides possibilities to lay out widgets beyond the touchscreen [START_REF] C H E N | Skin buttons: Cheap, small, low-powered and clickable fixed-icon laser projectors[END_REF], or organize multiple pieces of content relative to the user's body and quickly access them [START_REF] C H E N | Extending a mobile device's interaction space through body-centric interaction[END_REF][START_REF] A N D I R A N I | Twohanded around-device interaction for pane switching on smartphones[END_REF].

One of the challenges is to accurately track the fingers relative to the device and to segment gestures. While many systems use external motion tracking systems for prototyping purposes [START_REF] J O N E S | Around device interaction for multiscale navigation[END_REF][START_REF] Pa R K | Designing touch gestures using the space around the smartwatch as continuous input space[END_REF][START_REF] A N D I R A N I | Twohanded around-device interaction for pane switching on smartphones[END_REF], some others actually try to embed the tracking solution in the mobile setup. The early SideSight system [START_REF] A N D H O D G E S | Multi-"touch" interaction around small devices[END_REF] used IR proximity sensors mounted on the device, tracking finger gestures performed on the surface upon which the device was laid. Skin Buttons [START_REF] C H E N | Skin buttons: Cheap, small, low-powered and clickable fixed-icon laser projectors[END_REF] also use IR sensors to detect taps on icons projected from a smartwatch on the surrounding skin. Air+Touch [START_REF] C H E N | Interweaving touch & in-air gestures[END_REF] uses a miniature depth camera attached to the phone. Similar to SideSight, GlassHands [START_REF] O F E K | Interaction around unmodified mobile devices using sunglasses[END_REF] recognizes surface gestures around the phone, but achieves this without additional sensors, by tracking the reflection of the user's phone and hands in her sunglasses with the phone's front-facing camera. Ether-Toolbars [START_REF] R E K I K | Ether-toolbars: Evaluating off-screen toolbars for mobile interaction[END_REF] explore a similar approach, but uses a reflective surface mounted above the camera. Tracking can also rely on magnetic sensing. Ketabdar et al. [START_REF] Ro S H A Nd E L | Towards using embedded magnetic field sensor for around mobile device 3d interaction[END_REF] detect changes in the magnetic field around the phone produced by the circular or linear movements of a finger ring. Nenya [START_REF] A N D W H I T E | Subtle and eyes-free mobile input with a magnetically-tracked finger ring[END_REF] also uses a ring, but detects twisting and sliding gestures, which are mapped to selection and click, respectively. Abracadabra [START_REF] H A R R I S O N | Wireless, high-precision, and unpowered finger input for very small mobile devices[END_REF] has users wear a magnet below their finger to interact around a smartwatch.

One downside of mid-air around device interaction is fatigue [START_REF] H I N C A P I É -R A M O S | Consumed endurance: A metric to quantify arm fatigue of mid-air interactions[END_REF]. Jones et al. [START_REF] J O N E S | Around device interaction for multiscale navigation[END_REF] report that participants performed equally well with mid-air and with touch interactions, and that they found mid-air interactions enjoyable but tiring as well. Participants also raised concerns about arm fatigue with AirPanes [START_REF] A N D I R A N I | Twohanded around-device interaction for pane switching on smartphones[END_REF]. This suggests that the use of mid-air gestures should be restricted to occasional, non-lasting interactions that complement touch interactions. Air+Touch [START_REF] C H E N | Interweaving touch & in-air gestures[END_REF] explores such an approach, interweaving touch events and short in-air gestures. Empirical data about user performance and fatigue is not reported, unfortunately.

Augmenting Mobile 2D Input

Because we consider AR as a means of enhancing input with mobile devices, our work also relates to mobile interaction techniques that add more input degrees of freedom to trigger commands or adjust parameter values. Grossman et al. track a pen in close proximity to a portable screen, letting users invoke so-called Hover Widgets [START_REF] H I N C Kl E Y | Hover widgets: Using the tracking state to extend the capabilities of pen-operated devices[END_REF] to trigger commands with small 2D gestures performed in-situ. The Hover Cursor [START_REF]Hover cursor: Improving touchscreen acquisition of small targets with hover-enabled pre-selection[END_REF] tracks fingers hovering the phone's screen, in this case to facilitate the selection of small targets by offsetting the pointer, thus avoiding occlusion of the target of interest. Earlier, Shift [START_REF] Vo G E L | d i s c h , P . Shift: A technique for operating pen-based interfaces using touch[END_REF] had explored a complementary approach to the well-known fat-finger problem, offsetting a copy of the area under the finger when in contact with the screen to keep that region visible and enable precise selection.

The above techniques can be seen as augmenting the capabilities of the screen itself.

Other techniques have investigated a more indirect way of augmenting 2D widgets, using the mobile device's side buttons. SidePress [START_REF] S P E L M E Z A N | Side pressure for bidirectional navigation on small devices[END_REF] uses two buttons on one of the phone's sides, that can sense different pressure levels. The buttons can be used for any bidirectional selection and navigation task. Going further, the Power-up Button [START_REF] S P E L M E Z A N | Controlling widgets with one power-up button[END_REF] combines pressure and proximity sensors to let users control 2D widgets with the thumb holding the device.

Multi-device Mobile Systems using AR

AR-enhanced phone-centric interactions are part of the broader category of Distributed User Interfaces in which the system is aware of the relative spatial location of all devices involved: smartphones, tablets, and combinations thereof [168,[START_REF] B E Ng L E R | Surfaceconstellations: A modular hardware platform for ad-hoc reconfigurable cross-device workspaces[END_REF][START_REF] A N D K R A N Z | Ad hoc mobile multi-display environments through head tracking[END_REF]. These often focus on use cases where one or more users temporarily combine several devices in a stationary context, typically laying them out on a table. Gluey [START_REF] S E Rr A N O | Developing a head-worn display interface to unify the interaction experience in distributed display environments[END_REF] is a conceptual interactive system where devices, which can be in different locations, are combined thanks to an ARHMD, using the latter for tracking and as a means to transfer content between devices.

ARHMDs also enable distributed user interfaces adapted to a mobile context. As mentioned earlier, ARHMDs and mobile touchscreens have very different capabilities, that can complement one another effectively when made to work in tandem, as observed empirically with MultiFi [START_REF] H E I N I S C H | Multifi: Multi fidelity interaction with displays on and around the body[END_REF]. The BISHARE design space [START_REF]Bishare: Exploring bidirectional interactions between smartphones and head-mounted augmented reality[END_REF] organizes smartphone+ARHMD combinations into two broad categories: Phone-centric and HMD-centric. The latter category covers cases where the handheld device enhances spatial interaction with AR content. For instance, the handheld device becomes an alternative or a complement to bare-hand input for AR/VR content manipulation [START_REF]a n d B i l l i n g h u r s t , M . Using a hhd with a hmd for mobile ar interaction[END_REF][START_REF] Vog E L | Exploring the design space for using a multi-touch tablet in virtual reality[END_REF] and navigation [START_REF] M E Nz N E R | Above surface interaction for multiscale navigation in mobile virtual reality[END_REF], with applications to, e.g., CAD [START_REF] A N D M C G U Ff I N | Integrating augmented reality with a desktop gui and smartphone interaction[END_REF] and gaming [START_REF] Tat Z G E R N | Enabling smartphones for 3d interaction on mobile head-mounted displays[END_REF]. The former category rather includes cases where the HMD is used to enhance interaction with the handheld device. The AR-enhanced interactions we focus on fall under this category, and more specifically in subcategories phone-centric distributed UI (D2P) and phone-centric distributed input (D1P).

Such phone-centric combinations, in which the ARHMD is mostly used to add more input and output capabilities to the phone, have gained attention in the HCI literature. VESAD [START_REF] A N D M C G U Ff I N | Enlarging a smartphone with ar to create a handheld vesad (virtually extended screen-aligned display)[END_REF] is an example of augmenting a phone's visual output. The technique virtually extends the phone's screen by displaying additional 2D content in AR around the phone, keeping that content spatially aligned in the same plane. A VESAD proved more efficient than a phone only on a manual classification task. However, Eiberger et al. [START_REF] K R I S T E N Ss O N | Effects of depth layer switching between an optical see-through head-mounted display and a body-proximate display[END_REF] report that the combination of a physical display with AR is not always beneficial. In their study about the integration of information from different depth layers, participants had to perform a visual search task using a combination of HMD and smartwatch, or using an HMD only. Their performance was significantly lower in the HMD+smartwatch case, in terms of both speed and error.

In the above examples, the AR imagery consists of an expansion of what is shown on the mobile device. But AR can also show complementary information of a different nature, as illustrated by WatchThru [START_REF] W E N I G | Expanding smartwatch displays with mid-air visuals and wrist-worn augmented reality[END_REF] and mobile true-3D displays [START_REF] S E Rr A N O | Identifying suitable projection parameters and display configurations for mobile true-3d displays[END_REF]. MultiFi [START_REF] H E I N I S C H | Multifi: Multi fidelity interaction with displays on and around the body[END_REF] goes one step beyond, making interactive widgets run across devices and ARHMD, adapting both their input and output based on the respective fidelity of devices involved: screen resolution and physical size, direct vs. indirect input. For instance, the smartphone's high-resolution screen can be used as the primary display for an item in focus, while the ARHMD will provide lower-resolution information about surrounding items in the same list. Or the phone's touchscreen can be turned into a fullscreen keyboard, the edited text being shown in mid-air. MultiFi essentially focuses on how to make widgets span devices with different input/output characteristics. Interaction remains limited to basic touch and spatial pointing. Finally, VESAD can also be used to show additional windows or widgets around the phone. The authors briefly suggest this possibility [START_REF] A N D M C G U Ff I N | Enlarging a smartphone with ar to create a handheld vesad (virtually extended screen-aligned display)[END_REF], but report neither on the prototyping nor the evaluation of such AR-based enhancement techniques.

Extending interactive surfaces to the air can involve input (e.g., mid-air movements), output (e.g., AR imagery) or both -though they have so far been mostly studied separately. Such strategies are promising research avenues. However, the actual, quantifiable pros and cons of applying these extension strategies to distribute UI phone controls in an actual ARHMD+phone set-up have not been explored. In this chapter, we contribute a design space to help interface developers explore the set of AR-enhancements to mobile devices. Then, we discuss some specific phone-centric widgets' UI design, implementation and performance aspects individually and, where possible, in combination with a representative set of phone controls.

Workshop

As a first step into the investigation of AR-enhanced phone interactions, we adopted a user-centered design approach with a workshop in order to observe how users would like AR to enhance their interactions with their mobile devices.

We organized a workshop with 13 people (4 females, 9 males) aged from 20 to 33. All of them were familiar with mobile devices, 11 of them had knowledge about Human-Computer Interaction, 3 had already worked with Augmented Reality. Participants were split into 4 groups of 3 to 4 people. The groups remained the same for the whole workshop.

The workshop lasted 2 hours and was organized into 2 parts. The first part was a brainstorming session about what phone applications or use cases would take advantage of AR enhancements. The second part was a storyboarding session where participants had to create a storyboard to detail one of the applications/use cases they had proposed during the brainstorming session.

Brainstorming

The brainstorming session lasted 45 minutes. The moderator asked the three following questions, one after the other: To think about issues, participants were allowed to use their phones during the workshop.

Q 1 What
The moderator also encouraged them to reflect on specific issues that they personally faced in the past (critical incidents). Some of the issues they raised are related to hardware limitations while others are more related to UI design, for which AR could provide some solutions.

They identified many issues about applications that relate to physical limitations. (1) screens of smartphones are too small for many applications. For example, when reading a book, users often zoom in and out.

(2) Screen luminosity can have a detrimental effect on users' eyes.

(3) The processor speed is often limited. (4) There is a lack of physical buttons, and software controls on the screen consume a lot of valuable display real estate.

(5) The battery duration is sometimes short. ( 6) Memory is limited. [START_REF] P I E T R I G A | Custom-made tangible interfaces with touchtokens[END_REF] There is a lack of haptic feedback due to the flat screen. ( 8) Sensors are often inaccurate, which can be a problem while doing e.g. health monitoring. ( 9) Internet connection can be poor.

Some other issues more relate to users' perception and psychology. [START_REF] B E N Ko | Haptic retargeting: Dynamic repurposing of passive haptics for enhanced virtual reality experiences[END_REF] Showing some content on the phone to another user means getting physically close to that user, and potentially invading their personal space. [START_REF]A survey of augmented reality[END_REF] When objects such as books are digitalized, users might miss the physical feeling of it. ( 12) Smartphones are addictive. [START_REF] M Ü L L E R | A new perspective on gestural interaction and wearable applications[END_REF] Notifications while focusing on a task can be disturbing. ( 14) As the smartphone is a mobile device, users might get disturbed by the external environment. ( 15) Representation of the real world through the smartphone may be misleading, for example, when a user places an order, the product's physical features might be actually different from what the user has in mind. ( 16) Phones typically accommodate a single window, which makes multitasking difficult.

Some of these findings echo mobile devices' issues described by the Nielsen Norman group.

In particular, one of their reports [START_REF] N I E L S E N N O Rm A N G | ro u p. Mobile user experience: Limitations and strengths[END_REF] lifts some of the issues that were also mentioned by our participants. It also mentions one more which is [START_REF] B O T V I N I C K | Rubber hands 'feel' touch that eyes see[END_REF] the inaccuracy of touch interaction. This issue is described as resulting from, first, the small size of the screen, which causes a rise of the interaction's cost and makes the user relies on short-term memory and, second, interruptibility which is inherent to those devices and leads to a fragmentation of users' attention. Both of these causes have been explicitly proposed by our participants.

Q 3 What AR-based solution can we think of to address such problems?

Participants had to answer this question by giving concrete examples about how they would solve the listed issues using AR. There was only one rule imposed: the phone should remain the primary device and should not be turned into a control device for an AR-only application. Following this principle, users imagined several AR-based solutions that can be categorized by action. One of the most cited ideas during the workshop session was (1) to relate the content displayed on the phone to the physical world. Participants mentioned, for example, that they would like to see the physical aspect of a digital book when reading it. They also proposed the use of AR to (2) enlarge the phone's screen; (3) add interaction, either to free the phone's screen from large buttons or keyboards, or to add an new layer of interaction; and (4) add content. They proposed this content to be either independent from the phone's content (e.g. notifications) or dependent from it (e.g. an additional information layer). Being able to (5) abstract the data displayed on the phone was also perceived as useful by the participants. In particular, such abstraction can give a new perspective of the content (e.g. an overview of a map or an outline of a text document). Finally, they discussed having multiple views of the phone's screen, either for sharing one view with another user or to avoid uncomfortable postures that result from the fact that the phone's screen is colocated with their hand. Indeed, users either have to look down toward their hand or maintain their arm in an upper posture.

One solution can be to [START_REF] A N D F E K E T E | Orthozoom scroller: 1d multi-scale navigation[END_REF] duplicate the content of the screen and move the screen's copy to a convenient location.

StoryBoarding

During this part, participants had to select and illustrate one of the given AR-based solutions that they had proposed or that has been proposed by another group. We asked participants to provide two storyboards per group. The first one should be about the issue they decided to focus on. The second one should present the AR-based solution, showing how AR could be useful in this situation.

Design Space

The workshop reported above revealed that there is a large range of possibilities for combining an AR headset and a mobile phone. Analyzing solutions proposed by our participants, we created a design space for AR enhancements to mobile phones. We focused on how AR could complement mobile devices, excluding the cases where mobile devices are used as a controller to manipulate AR content, which is an approach that has already been explored by others [START_REF] A N D Wo O | Graphical menus using a mobile phone for wearable ar systems[END_REF][START_REF] A N D M C G U Ff I N | Integrating augmented reality with a desktop gui and smartphone interaction[END_REF]213]. Our design space, which is illustrated in Figure 2.4, features two main dimensions: (1) the location of the holographic content, and (2) its function. We populated it with both projects and ideas that we found in the HCI literature (in bold) to demonstrate its descriptive power and then used it to generate ideas (in italic). Some of these ideas led to actual prototypes that we describe in section 2.4.3.

Dimensions of the Design Space

In this section, we detail all the dimensions of our design space. For each dimension, we give a textual description as well as a pictogram in which orange represents the holographic content and blue represents the mobile device.

Dimension LOCATION

This first dimension describes the location of the holographic content. It can be relative to: (1) the handheld device, (2) the user, or (3) the environment.

Relative to the handheld device

The holographic content is anchored to the handheld device. It can have the same orientation than that of the device's screen (i.e., coplanar) or not (i.e., not coplanar). When LOCATION is coplanar, we distinguish cases depending on whether the holographic content is above, behind, or on the side or the device.

Behind (Coplanar)

The holographic content is located behind the handheld device. The user can see only one of the two contents at a time.

Above (Coplanar)

The holographic content is superimposed on the handheld device content. It can have a varying level of transparency.

Side (Coplanar)

The holographic content is located somewhere around the handheld device.

Not coplanar

The holographic content is displayed with a different orientation than that of the handheld device's screen.

Relative to the user

The holographic content is anchored to the user, meaning that spatial manipulations of the handheld device do not affect the location of the holographic content.

Relative to the user The holographic content is anchored to the user. Its location is defined as a function of the user's location.

Relative to the environment

The holographic content is spatially registered in the environment. It can be anchored to an object or it can be floating in the environment at a fixed location.

Anchored The holographic content is anchored to a specific object in the environment. Manipulating that object will have an impact on the location of the holographic content.

Floating The location of the holographic content is defined as a 3D position in the world. It appears as floating in the air at a fixed position.

Dimension FUNCTION

This second dimension focuses on the function of the holographic screen relative to the mobile device. The holographic layer can be used to display additional data, to provide users with more powerful ways of interacting with the content displayed on the handheld device, or to enhance the presentation of this content.

Add data

The holographic view contains data that is not on the handheld device. Such data can either be: Dependent or Independent from the handheld device's content.

Dependent The holographic screen displays data that relate to what is displayed on the handheld device's screen. In that case, the holographic content is dynamically updated as the content displayed on the handheld device changes or as the user navigates it.

Independent The holographic screen displays data that are not related to the data displayed on the handheld device's screen.

Enhance presentation AR can be used to enhance the presentation of the content displayed on the handheld device, without providing additional data.

Enlarge The holographic content is used to provide a larger viewport over the handheld device's content.

Duplicate A copy of the handheld device's screen is displayed in AR.

Abstract The holographic screen presents the same content as the handheld device but with a different level of abstraction (e.g., an outline of the content displayed on the handheld device).

Relate to physical world

The holographic layer is used to display virtual cues that make the metaphor with the physical world stronger.

Add interaction AR can also provide a more diverse range of controls to interact with the handheld device.

Add interaction

The holographic screen features widgets and controllers to manipulate the content displayed on the handheld device.

Populating the Design Space with Existing Solutions

To populate this design space2 , we first sorted out AR-enhanced solutions that we could find in the HCI literature. We only considered papers that present a handheld device-centric application. Indeed numerous papers rather present AR applications that use a smartphone or a tablet as a tool for interacting.

Using AR to enlarge a limited device's screen is a concept that has been explored by Feiner and Shamash in early 1991. They describe Hybrid User Interfaces [START_REF] F E I N E R | Hybrid user interfaces: Breeding virtually bigger interfaces for physically smaller computers[END_REF], proposing to enlarge a computer's screen with AR content. This allows users to keep track of windows that are outside the screen (Enlarge × Side). Feiner also presented the Touring Machine [START_REF] F E I N E R | A touring machine: Prototyping 3d mobile augmented reality systems for exploring the urban environment[END_REF] in 1997. This project links anchored AR content related to their university's buildings with their description written on a tablet (Dependent × Anchored). More recently, researchers have resumed investigating a possible synergy between mobile devices and AR. For example, Budhiraja et Al. [START_REF]d B i l l i n g h u r s t , M . Interaction techniques for hmd-hhd hybrid ar systems[END_REF] organizes the different ways of combining a smartphone with an AR system into the following three categories: a user can either interact with the phone only, use the phone as input for AR, or use the phone to show additional data related to the AR scene. This latter category corresponds to the phone-centric case we are interested in and falls within the Dependent × Anchored cell of our design space. Wenig et al. presented WatchThru [START_REF] W E N I G | Expanding smartwatch displays with mid-air visuals and wrist-worn augmented reality[END_REF], an application that displays AR content above a smartwatch by adding a piece of plexiglass on top of the watch. Authors illustrate their concept with e.g. a GPS application where the AR screen is used to show the next direction to the user. This application belongs VESAD [START_REF] A N D M C G U Ff I N | Enlarging a smartphone with ar to create a handheld vesad (virtually extended screen-aligned display)[END_REF] Personal Cockpit [START_REF] F I N N E G A N | The personal cockpit: A spatial interface for effective task switching on head-worn displays[END_REF] Dependent

VESAD [148] DesignAR [173] Multi- Fi [60] Task Selection BISHARE [218] BISHARE [218]

Locate photos in space

Touring Machine [START_REF] F E I N E R | A touring machine: Prototyping 3d mobile augmented reality systems for exploring the urban environment[END_REF] Interaction techniques for HMD-HHD [START_REF]d B i l l i n g h u r s t , M . Interaction techniques for hmd-hhd hybrid ar systems[END_REF] Enlarge Desig-nAR [START_REF] R E I P S C H L Äg E R | Immersive 3d-modeling combining augmented reality with interactive displays[END_REF] VESAD [START_REF] A N D M C G U Ff I N | Enlarging a smartphone with ar to create a handheld vesad (virtually extended screen-aligned display)[END_REF] Multi-Fi [START_REF] H E I N I S C H | Multifi: Multi fidelity interaction with displays on and around the body[END_REF]

Hybrid User

Interfaces [ VESAD [START_REF] A N D M C G U Ff I N | Enlarging a smartphone with ar to create a handheld vesad (virtually extended screen-aligned display)[END_REF] BISHARE [START_REF]Bishare: Exploring bidirectional interactions between smartphones and head-mounted augmented reality[END_REF] Multi-Fi [START_REF] H E I N I S C H | Multifi: Multi fidelity interaction with displays on and around the body[END_REF] Virtual joystick Fig. 2.4: Our design space for combining Augmented Reality with mobile devices. The first dimension corresponds to the position of the holographic content. The second dimension represents the function of the AR screen with respect to the mobile device.

. to the Abstract × Not Coplanar cell of our design space. With Multi-Fi [START_REF] H E I N I S C H | Multifi: Multi fidelity interaction with displays on and around the body[END_REF], users can e.g. navigate in a menu by moving their phone close to an element in the 3D environment, or navigate a map displayed on a smartwatch, having the context in AR (Add interaction × Floating). In the VESAD project [START_REF] A N D M C G U Ff I N | Enlarging a smartphone with ar to create a handheld vesad (virtually extended screen-aligned display)[END_REF], Normand and McGuffin present concept applications that use the space around the phone to display more content.

For example, they propose to provide AR tooltips around the smartphone's screen widgets (Dependent × Side). They also study different possibilities for enlarging the smartphone's screen (Enlarge × Side). more recently, BISHARE [START_REF]Bishare: Exploring bidirectional interactions between smartphones and head-mounted augmented reality[END_REF] organized a wider design space of bidirectional interactions between AR and smartphones. While this design space covers a larger range of applications including the use of smartphones as a tool for AR-only applications, some cells fit within our design space. For example, they suggest adding AR controls or independent data around the smartphone which corresponds to our design space's respective cells: Add interaction × Side and Independent × Side. DesignAR [START_REF] R E I P S C H L Äg E R | Immersive 3d-modeling combining augmented reality with interactive displays[END_REF] is another tool for combining a tablet and an AR HMD. More than providing the holographic screen with more controls (Add interaction × Side), DesignAR [START_REF] R E I P S C H L Äg E R | Immersive 3d-modeling combining augmented reality with interactive displays[END_REF] proposes to enlarge the tablet's screen to make data accessible in 3D and then, falls within the scope of Enlarge × Front. Finally, techniques such as the one presented in Personal Cockpit [START_REF] F I N N E G A N | The personal cockpit: A spatial interface for effective task switching on head-worn displays[END_REF] could be use in association with mobile devices, showing some independent content to the user (Independent × Relative to the user).

Generating novel solutions

We can observe that some cells have not been explored yet, opening spaces for creativity.

In this section, we present novel solutions that our design space inspired us. To generate these solutions, we came back to some problems that participants raised during the workshop. We then systematically considered the cells of our design space to think of possible ways of address these issues. We came up with a set of solutions that are displayed in italic in Figure 2. [START_REF]Supporting subtlety with deceptive devices and illusory interactions[END_REF]. Some of these solutions are still conceptual while others have led to actual prototypes that we developed.

Concept applications

Locate photos in space This application would display photos from the phone's gallery at the exact position where they have been taken from. Here, the holographic content is dependent on the smartphone's one, and photos are floating in the 3D space (Dependent × Floating). Ergonomic view This application would allow users to get a view of their current screen in front of their eyes while holding a good posture for the neck, preventing them from holding their head down. The holographic screen is a copy of the actual phone's screen, located along with the user's sight (Duplicate × Relative to the user). We illustrate this concept application in Figure 2. [START_REF] A N D F E K E T E | Orthozoom scroller: 1d multi-scale navigation[END_REF]. Context sharing This application would allow others to quickly get what the user is doing by glancing at the back of their phone. A context icon corresponding to the task the user is doing would be displayed behind the phone (e.g., musical note when listening to music, envelope when reading messages) (Abstract × Behind). Without revealing the actual content on the handheld device, this application would give a bit of context to others so they know whether they can interrupt the smartphone's user or not. We illustrate this idea in Figure 2.7.

or Contextual electric switchboard This application would display the floor plan of a house.

When in front of heir electrical board, a user could select a room and get an AR link that relates that room to a specific switch (Relate to physical world × Anchored).

Prototypes

We also implemented actual prototypes for some ideas that we considered particularly promising. The first three prototypes were developed with a Samsung Galaxy J5 combined with a Microsoft HoloLens 1. As for software, we used Unity 2018. [START_REF]Supporting subtlety with deceptive devices and illusory interactions[END_REF].9f1 and Vuforia 8. 3.8. The smartphone was equipped with two Vuforia markers on top, one with the same orientation as the smartphone's screen and one on its back. In this section, we also present two projects that have been developed by interns that we supervised. Both have been coded using a Microsoft HoloLens 2 and a Samsung Galaxy A50.

Back of device mirror This prototype displays an AR copy of the handheld device's screen on the back of the device. It allows a user to keep track of what is displayed on their device's screen. The typical scenario involves showing a digital scene to a friend. When the user faces the back of his handheld device, they keeps a live view of what is displayed on the screen. For now, the duplicated AR view is not interactable but this is an improvement that could be implemented in the future. In our design space, this application falls within the following cell: Behind × Duplicate. Figure 2.9 illustrates this application.

Task Selector This application allows a user to switch between the different apps that are opened on their handheld device. A stack of all the opened apps is displayed on top of the smartphone. When a user looks at this stack (i. move their gaze cursor in this menu from an app to another. By performing a tap gesture, the user selects a screen, which changes the currently running task of the smartphone.

Then the apps get re-arranged into a stack. In our design space, this application belongs to cell Side × Add interaction. Figure 2.10 illustrates it. This prototype refines and actually implements an idea that had remained conceptual in the VESAD [START_REF] A N D M C G U Ff I N | Enlarging a smartphone with ar to create a handheld vesad (virtually extended screen-aligned display)[END_REF] article.

Ambilight The Ambilight application allows the user to have a smoother virtual transition between the mobile device's screen and the environment. It got inspired by an actual technology for TVs that relies on LEDs that are located on the side of the TV to display a colored halo around it. Here, we provide a fully virtual Ambilight by displaying a color gradient in AR around the smartphone. To achieve that effect, we capture the handheld device's screen as an image and apply a gaussian blur to it. Then we remove the center of the image to let the smartphone's screen appears. In our design space, it falls within the Side × Enlarge cell. This application does not involve any interaction from the user, it is purely visual. Figure 2.11 illustrates it.

Fig. 2.12: HoloLens 1 view of the AR-extended screenshots application. To generate such AR content, the participant has to: 1) click on a "crop" button, on the smartphone's screen;

2) use the touch interface to circle the area of interest in the picture; 3) click on a "done" button to send this area to the HoloLens.

Internship projects

AR-extended clipboard I had the opportunity of helping to supervise the 24H internship of Mariam Diallo and Mamoun Elkhettar, both students from Polytechnique (France).

They designed and implement a prototype of an application for saving selections of a smartphone's screen in AR, using a HoloLens 1. To do so, they developed two applications. They used Android Studio for the smartphone application and Unity for the HoloLens application. Both could communicate using a TCP/IP protocol. The smartphone was equipped with a marker, detected by the HoloLens using Vuforia. The AR-extended screenshots make it possible for a user to offload content displayed on the smartphone's screen in AR. When entering the smartphone's application, the user can browse their phone's gallery. Once the picture is selected, the user can select the area of interest of this picture, by circling it using the touch interface. Once the finger leaves the display, the picture is shown as cropped and can be offloaded by clicking on a button. This project falls within the Enlarge × Side cell of our design space. AR-extended Annotations I also co-supervised the internship of Francesco Riccardo Di Gioia, Master student from Université Paris-Saclay (France). We have investigated ways to streamline content annotation on a handheld device by combining it with an Augmented Reality Head-Mounted Display (ARHMD). Francesco designed a full prototype of an application that provides users with annotations offloading in the AR space around a mobile device and also makes it possible to annotate documents not only with traditional means such as text entry and inking, but with content captured on the spot using the ARHMD's camera as well. This prototype works with a Samsung Galaxy A50 or an Ipad Pro combined with a Microsoft HoloLens 2. Regarding software, he developed this prototype using Unity and Vuforia. Both the smartphone and the tablet were equipped with three Vuforia markers, all with the same orientation as their screen to make the Vuforia tracking more robust. This application allows users to select content on different media (i.e. 1) Text documents, 2) Maps, and 3) Videos). The selection can be a media unit (resp. highlighted text, pins, and frames) or a free-form drawing mark.

The prototype allows users to annotate with: 1) typed or handwritten texts, 2) photos, 3) phone screenshots, and 4) videos. Annotations are offloaded in the air, around the device. They are visually linked to their scope as the same color is used for the content selection and the outline of the annotation. These colors are automatically generated by the application, which relies on a color palette for categorical encoding from Color Brewer [START_REF] Rcolorbrewer | ColorBrewer Palettes[END_REF] to make them easily distinguishable. A user study to evaluate this system's usability is in progress.

AR-enhanced Widgets: Design Considerations

Our design space shows that there are many possibilities for enhancing phone-centric interactions with AR. However, as mentioned earlier in this chapter, the quantifiable pros and cons of distributing a UI across a mobile device and the air are unknown. As a first step toward providing quantitative assessments, we considered the case of standard phone widgets and studied in depth how they could take advantage of AR enhancements.

We first identified three low-level, generic interactions3 typically performed using widgets that could be at least partially offloaded to the air, either their input, their output, or both: command trigger/item selection; parameter value adjustment; and precise 2D selection. We then prototyped different AR-enhanced techniques, discarding those that we perceived as clearly ineffective. Among the retained techniques, some raised specific questions or required some parameters to be fine-tuned empirically, in which case we ran preliminary experiments to inform these choices. This section reports on the main considerations involved in making design and implementation decisions.

Phone-centric Interaction Space

One of the first decisions we had to make was to bound the input space around the phone.

In HMD-centric distributed UIs [START_REF]Bishare: Exploring bidirectional interactions between smartphones and head-mounted augmented reality[END_REF], the phone acts as a controller for AR content that can be anywhere in the user's physical environment. It can be tethered to the phone, or free-floating. In phone-centric distributed UIs, the phone holds the primary content. We thus focus on the direct surroundings of the phone, considering widgets and input areas tethered to the phone.

We adopted the rule of thumb that interaction should take place not further than 10cm away from the phone. We found that this distance allowed for comfortable posture and movements, especially when the user's upper arms are along their trunk. It avoids large arm movements in the air, which can be both tiring [START_REF] H I N C A P I É -R A M O S | Consumed endurance: A metric to quantify arm fatigue of mid-air interactions[END_REF] and socially awkward [START_REF] A N D I R A N I | Are you comfortable doing that? acceptance studies of around-device gestures in and for public settings[END_REF]. It also avoids large eye movements and limits problems of divided attention between the phone and the AR display.

Legacy Bias

Past experience with interactive systems influences the way users apprehend a novel system, making them expect the novel system to work the same way as the systems they are familiar with. Brudy et al. [START_REF] H O L Z | Cross-device taxonomy: Survey, opportunities and challenges of interactions spanning across multiple devices[END_REF] warn interaction designers about this so-called legacy bias [START_REF] Da N I E L E S C U | Reducing legacy bias in gesture elicitation studies[END_REF] in the context of cross-device interaction. This led us to consider how people interact with smartphones, and find ways to augment their experience rather than radically change it. The three generic interactions we identified have a straightforward correspondence with UI widgets on smartphones:

• command trigger and item selection are typically achieved with buttons, checkboxes, or menus;

• parameter value adjustments are typically achieved with sliders;

• precise 2D selection is achieved by pointing at a precise location inside widgets such as images, text areas, maps.

Keeping the issue of legacy bias in mind, we favor mid-air interactions which are as close as possible to their touch-only counterparts, striving for a unified set of interaction techniques across modalities. In our design, offloading widgets from the phone to the air does not noticeably change their appearance. Interaction with these widgets is directly inspired by the paradigm developed for the Microsoft Hololens 2 for hand input. Users touch discrete widgets with their index tip, and perform continuous input by adopting a pinch posture and moving their hand (air-tap-and-hold). These can be seen as the mid-air counterparts to tap and slide interactions on the phone's screen.

Three-state Mid-air Input

Touch input is direct, with focus implicitly given to the widget being touched. Mid-air input is more indirect. It calls for a three-state model that will enable the selection of a widget before it gets manipulated. When adopting a pinch or hover hand posture, users control an invisible pointer located halfway in-between the thumb's and index finger's tips. The hover posture is used to select which widget has focus. When the pointer enters a widget, that widget gets highlighted with a blue halo. If the user brings his fingers closer to actually adopt a pinch posture, the widget will then own focus and subsequent hand movements will control it. A widget that already has focus can be controlled by initiating an air-tap-and-hold interaction either on the widget itself or anywhere in empty space.

This third state also enables us to provide users with a straightforward way to customize how the UI is distributed around and on the phone. An air-click event 4 immediately followed by an air-tap-and-hold interaction 5 actually grabs the highlighted widget. Users are then free to drop it wherever they want by releasing their pinch posture. This works both for widgets displayed on the phone and widgets that are already in the air.

Mid-air Input Accuracy

The three-state model above addresses the issue of expressive power, but it does not address that of input accuracy. While indirect mid-air input can contribute to improving accuracy (less occlusion, larger widgets), the absence of physical support and haptic feedback mean that movements are typically less precise (because of, e.g., hand tremor). Furthermore, from a design perspective, movements in the air need to be segmented in order to isolate control gestures from other movements. With the three-state model, we rely on the pinch posture for segmenting continuous input from other movements. But releasing control necessarily entails relaxing this pinch posture. As opposed to lifting up a finger from a touchscreen, relaxing a posture is a movement, which thus cannot be detected immediately. The system needs to collect a sufficient portion of the relaxing movement before actually releasing control, which inevitably causes an involuntary movement of the cursor. However, since cursor control is indirect with mid-air input, we can account for such inaccuracy by adjusting the CD gain. Setting it to a large-enough value, involuntary movements will not cause the cursor to move.

We ran a preliminary experiment involving six participants to quantify the amplitude of such involuntary movements, which we identify as ∆ release . Participants controlled a cursor on the phone with mid-air movements. They had to put it precisely on a very small target (1.5mm), and then switch from a pinch to a release posture. We measured ∆ release , the offset between the target position and the cursor's actual position at the time the system detects this change of hand posture. The design, procedure and results are detailed in supplemental material 6 . Averaging all collected values, we obtain

∆ release = 1.4 ± 2.2 < 4mm.
We use this 4-mm maximum bound for ∆ release in the design of interactions that involve mid-air input by enabling users to adjust the CD gain up to 4mm : cursorU nit so that the inaccuracy inherent to the segmentation of air movements will be compensated for, at least when the CD gain is maximum.

Effective use of the Air as a Display

The above experiment was coupled with another one aimed at gathering empirical data about users' widget placement preferences. Indeed, widgets can be placed anywhere in the air around the phone. But some sides will generate occlusion, or require adopting uncomfortable postures, depending on which hand is holding the phone. In this second preliminary experiment, participants had to interact with buttons and sliders displayed in the air on each of the phone's four sides. The design, procedure and results are detailed in supplemental material.

Overall, a majority of participants ( 5) ranked the right side as their preferred location for interacting with Sliders, the top side coming second. Results were more nuanced for Buttons. Half of the participants ranked the top side first, and the other half the right side. The area below the phone requires users to adopt uncomfortable postures, at least with the limited vertical field of view of current ARHMDs. Our findings are consistent with the recommendations from Normand and McGuffin [START_REF] A N D M C G U Ff I N | Enlarging a smartphone with ar to create a handheld vesad (virtually extended screen-aligned display)[END_REF], who report that some study participants had trouble interacting with the AR content around the phone when they had to cross their arms (for instance when they had to interact with virtual elements on the phone's left side using their right hand).

AR-enhanced Widgets: Design and Comparative Evaluation

We designed AR-enhanced widgets that cover a representative sample of phone interactions: Command trigger (E 1 ), Parameter adjustment (E 2 ), and 2D selection (E 3 ). For each of these interactions, we designed several variations of the corresponding widget: one that offloads the widget to the air, one that extends the input space to the air, one that does both. Specific design choices were guided by the considerations detailed in the above section. We systematically compared those designs against state-of-the-art touch-only input techniques in an exploratory study.

We conducted one study per interaction. The three studies were conducted sequentially, always in this order. Participants had to take a break of at least 15 minutes between them. During these breaks, they were asked to rate each technique along all six NASA-TLX dimensions, and then to rank the techniques according to their preferences.

Screen Screen

O: I:

Air Air O: I:

Offloading buttons to the air has the benefit of freeing screen real-estate on the phone. But the lack of haptic feedback with air buttons will likely make input more difficult. Furthermore, many users have a lot of prior experience with traditional touch buttons. However, we do not know how much this impacts user performance with air buttons.

Task Three circular buttons, horizontally aligned, with a spacing of 3mm between them, are displayed either 5mm below the screen's top edge (Screen -Screen), or 5mm above it (Air -Air). Button diameter varies according to task D i f f i c u lt y: 2cm (Low), 1.5cm (Medium) or 1cm (High). When a button gets highlighted, participants have to select it as fast and as accurately as possible. Buttons in a sequence are always highlighted in the same order: left, right, middle.

Design We follow a within-subject design with two factors: T e c hn i q u e ∈ {Screen -Screen, Air -Air} and D i f f i c u lt y ∈ {Low, Medium, High}. Trials are blocked by T e c hn i q u e. Block presentation order is counterbalanced across participants with a Latin Square. Each T e c h n i q u e block consists of 12 trials, i.e., four replications per T e c hn i q u e × D i f f i c u lt y. The presentation order of those 12 trials is counterbalanced across blocks and participants with a Latin Square. Each block starts with three practice trials presented in order of increasing difficulty.

Results

Our main measure is the Time between two button activations. We filter out interactions with the first (leftmost) button in a sequence as there is no control over the initial position of the participant's hand. We observe a large difference between techniques, with average time 0.4±0.25s for Screen -Screen and ∈ 1.2±0.8s for Air -Air.

Before conducting any analysis, we check that the collected data follow a normal distribution with a Shapiro-Wilk test. As we observe a violation of normality in one of the conditions (Screen -Screen × Medium, p < 0.05), we log-transform our measure data to avoid having a strong violation of the normality test. 10 An analysis of variance (anova) of T e c hn i q u e × D i ff ic u lt y on Time reveals a simple effect of both T e c hn i q u e (F 1,11 = 181, p = 0.001, η 2 G = 0.9) and D i f f ic u lt y (F 2,22 = 12, p = 0.001, η 2 G = 0.13). Screen -Screen is significantly faster than Air -Air (0.4s vs 1.28s on average), and the three levels of difficulty are all different from each other (p's < 0.05). Comparing the techniques' relative performance across participants also reveals a high variability with, e.g., one participant performing 4.82 times better with Screen -Screen than with Air -Air, and another one 1.88 times only. Interestingly, the participant whose relative performance difference between techniques is the smallest has strong experience with AR headsets. This suggests that users might be able to trigger buttons in the air relatively quickly, but this seems to require some training, and even then performance with on-screen buttons remains significantly better.

Qualitative assessments based on NASA-TLX scales are in line with quantitative observations: average grades on all scales are in favor of Screen -Screen. Wilcoxon signed-rank tests reveal that all these differences between techniques are significant (p's < 0.05 and Z s ∈ [-2.6, -2.4]), with the exception of the frustration scale (p = 0.6). Two participants found interacting in the air fun (2 participants), but participants mostly mentioned the advantages of interacting on screen, such as high reliability (6 participants) and haptic feedback [START_REF] A N D P I E T R I G A | High-precision magnification lenses[END_REF]. They also reported that touch requires low amplitude movements (8) compared to air input, which requires traversing the button and then exiting it with movements that are large-enough to be tracked reliably. Finally, only two participants preferred Air -Air over Screen -Screen.

Overall, Screen -Screen outperforms Air -Air in terms of both completion time and comfort. These results are not in favor of offloading phone buttons for activating discrete commands.

Experiment E 2 Parameter Adjustment

In the second experiment, participants had to select a numerical value using a slider. We include high precision tasks which involve value ranges that exceed typical smartphone screen resolution -causing quantization problems [START_REF] A N D P I E T R I G A | High-precision magnification lenses[END_REF] -and users' motor abilities in terms of fine-grained control. The touch-only baseline, Screen -Screen (CD ), implements OrthoZoom control [START_REF] A N D F E K E T E | Orthozoom scroller: 1d multi-scale navigation[END_REF]. This enables participants to perform very precise adjustments, with sub-pixel accuracy. We compare this baseline to three AR-enhanced slider designs (see also Figure 2.17 Screen -Screen (CD ). Users interact with OrthoZoom sliders as they do with traditional touch sliders. The only difference is that input precision increases progressively ( ) as the orthogonal distance between the slider and the finger increases. CD gain is set to 1:1 when the finger is less than 2cm away from the slider's track, and linearly increases up to 1:sliderU nit mm at 10cm from the slider, with sliderU nit mm being the size of a slider unit in mm. Above this maximal distance, a 1mm-displacement thus corresponds to a 1-unit adjustment on the slider.

Screen -Air (CD ).

The slider is still displayed on the phone, but input takes place in the air. The slider is manipulated with air-tap-and-hold interactions (Section 2. [START_REF] A N D P I E T R I G A | High-precision magnification lenses[END_REF]

.3).

As with OrthoZoom, input precision is a function of the distance between the fingers' location when the interaction started and the current fingers' location. But in this case, the distance considered is different: precision increases progressively ( ) as the hand moves away from the phone's plane towards the user's head. CD gain is set to 1:1 when that distance is short (<2cm), and linearly increases up to 4: sliderU nit mm when it reaches 10cm. Above this 10-cm distance, the hand movement must be of at least 4mm to change the slider's value. As discussed in Section 2. 5.4, this gives the tolerance required to enable users to release mid-air pinch postures safely without changing the slider's value. The idea of gaining precision for sliders and scrollbars as the distance in the air increases was mentioned in previous work about tabletop interaction [START_REF] C H E N | Interweaving touch & in-air gestures[END_REF][START_REF] G R E E Nb E Rg | The continuous interaction space: Interaction techniques unifying touch and gesture on and above a digital surface[END_REF]. But to our knowledge, it has not actually been tested. Harrison and Hudson [START_REF] H A R R I S O N | Wireless, high-precision, and unpowered finger input for very small mobile devices[END_REF] explore a similar idea for radial selection on a smartwatch, but do not report how CD gain is adjusted, and study their technique in a very different situation, where participants used an external hardware button attached to a table to validate selections.

Air -Air (CD

). The slider is displayed in the air, on the right-hand side of the phone. Interaction is exactly the same as with Screen -Air (CD ).

Screen -Screen+Air (CD ).

The slider is displayed on the phone. Input is hybrid, taking place both on the screen and in the air. Users perform coarse adjustments with slide interactions. They can also perform fine adjustments with air-tap-and-hold interactions.

There are only two levels of precision (symbol means dual-precision). CD gain is set to 1:1 for finger slides on the touchscreen, and to 4: sliderU nit mm for movements in the air.

Design properties Each of these four designs has its own advantages relative to the others, but it is difficult to infer their overall performance ranking. First, screen input benefits from haptic feedback, and from the strong experience users have acquired with touch devices. This might be an advantage in favor of Screen -Screen (CD ). Second, some designs are consistent regarding where input and output take place, while others are not. Consistency might be preferable and thus be an advantage for Screen -Screen (CD ) and Air -Air (CD ). But, at the same time, their control requires continuously adjusting two degrees of freedom (slider value, CD gain). They might thus be more cognitively demanding than a clear distinction between only two levels of precision (coarse and fine). In that respect, Screen -Air (CD ) might have some advantages over the other three techniques. In the end, these pros and cons make it difficult to know the cost of offloading input, output, or both in the air for slider control.

Task A 5cm horizontal slider is displayed on the phone for all Screen -* designs (11cm above the bottom edge of the screen) and in the air for the Air -Air design (offset 2cm from the phone's right edge). Depending on task D i ff ic u lt y, the slider has 100 (Low), 500 (Medium) or 1000 (High) units. The value to select, G oa l, is displayed in the upper part of the phone. Participants have to select the G oa l as fast as possible. The trial ends as soon as participants select the goal value (no finger on screen, hand in release posture). The slider knob's size depends on the current CD gain (half its default size at maximum precision), providing feedback to users about input precision.

Design We follow a within-subject design with two primary factors: T e c hn i q u e ∈ {Screen -Screen (CD ), Screen -Air (CD ), Air -Air (CD ), Screen -Screen+Air (CD )} and D i ff ic u lt y ∈ {Low, Medium, High}. Trials are blocked by T e c hn i q u e. Block presentation order is counterbalanced across participants with a Latin Square. We introduce a G oa l factor that can take four possible values for ecological purposes. It corresponds to the value participants are instructed to select. Each T e c hn i q u e block consists of 12 trials, i.e., one per T e c h n i q u e × D i f f i c u lt y× G oa l. Presentation order of trials within a block is counterbalanced across blocks and participants with a Latin Square. Each block starts with three practice trials, presented in order of increasing difficulty.

Results

We first check the normality of the collected task completion Time measures with Shapiro-Wilk tests. We observe that the data are not normally distributed under a few conditions (Screen -Screen+Air (CD ) × Low: p < 0.001, Screen -Screen+Air (CD ) × Medium: p < 0.05 and Screen -Screen (CD ) × Medium: p < 0.05). Applying a log-transformation solves the issue. We thus perform our analyses on a log-transformed time measure. An anova of T e c h n i q u e and D i f f i c u lt y on Time reveals a simple effect of both T e c hn i q u e (F 3,33 = 23, p = 0.001, η 2 G = 0.5) and D i ff ic u lt y (F 2,22 = 118,

p = 0.001, η 2 G = 0.6).
Unsurprisingly, all levels of D i ff ic u lt y significantly differ from each other: completion time increases as the task gets more difficult (Figure 2.18-a). Regarding differences between techniques, pairwise t-tests reveal that all techniques significantly differ from each other with all p's < 0.01, except for Screen -Screen+Air (CD ) and Screen -Screen (CD ), which have a p-value close to 0.05 (p = 0.04).

Sorting techniques by performance, we get: Screen -Screen+Air (CD ) (6.25s), Screen -Screen (CD ) (6.65s), Air -Air (CD ) (8.64s), Screen -Air (CD ) (11.1s). Compared to experiment E 1 , the drop in performance when offloading input and output in the air is much lower with sliders than with buttons. Interestingly, Screen -Screen+Air (CD ) actually performs slightly better than Screen -Screen (CD ), the OrthoZoom touch-only baseline. This suggests that air control can even be better than touch for indirect control. However, this is only the case when air control is implemented according to a dual-precision strategy. On the opposite, continuous precision level control is less efficient in the air than on screen.

Qualitative assessment. We perform Wilcoxon signed-rank tests to compare pairs of techniques along each of the NASA-TLX scales. The difference between Screen -Air (CD ) and Air -Air (CD ) is systematically significant, except for physical demand. Participants' comments reflect this preference for Air -Air (CD ) over Screen -Air (CD ): six of them spontaneously said that they like interacting in the air; and five of them reported having issues of divided attention when output is on the phone and input in the air, i.e., when there is a mismatch between where input and where output take place. Screen -Air (CD ) and Screen -Screen (CD ) also differ significantly, with ratings systematically in favor of Screen -Screen (CD ) on all scales except mental demand. Prior experience actually using an equivalent of Screen -Screen (CD ) was reported by three participants and might explain this.

Screen -Air (CD ) and Screen -Screen+Air (CD ) also differ significantly along all scales but physical demand and effort. Users found it easier to switch between two levels of precision with Screen -Screen+Air (CD ), as opposed to the continuous precision adjustments of the other (CD ). The former strategy (CD ) also avoids some usability problems. One issue raised by four participants about continuous CD gain adjustment (CD ) relates to clutching. When releasing a pinch posture in mid-air to immediately initiate a new one (e.g., to avoid leaving the tracking volume or to adopt a more comfortable position), CD gain is reset to its default value. These participants commented that they would have preferred the system to set the CD gain to its last value before release instead. Implementing such behavior is not trivial however, as it requires distinguishing between clutch actions and actual successive-but-distinct uses of the same widget (for which the current behavior makes more sense). It also raises usability issues, such as the higher chance for the phone to "be in the way" of the user's hand if resuming slider manipulation too close to the screen. Interestingly, clutching issues are not observed only for air control. Three participants reported sometimes reaching the phone's left or right edge when seeking a high level of precision with OrthoZoom, which forced them to clutch (i.e., releasing control and resuming on the slider's track itself).

Finally, seven participants reported that the switching cost between touch and air is high. Overall, participants' ranking of techniques is not completely in line with actual performance. As Figure 2.18-b shows, preferences are split and the fastest techniques are not systematically the preferred ones. In particular, Screen -Screen+Air (CD ) performed best in terms of completion time, but only 17% of participants ranked it first.

The novelty effect probably played in favor of Air -Air (CD ), which got ranked first by more than half of the participants. On the contrary, there is a clear consensus against Screen -Air (CD ).

Experiment E 3 2D Selection

In the third experiment, participants have to point at a precise location in a 2D workspace. Output (O) necessarily takes place on the phone, as the display resolution of current ARHMDs is too low to render 2D targets small-enough for our purposes. We thus focus on where input (I) takes place. Mid-air input has the potential to offer an occlusion-free alternative to touch for precise pointing tasks such as, e.g., positioning a caret in a text or delineating a region in an image or a map. But it will also likely suffer from significant precision issues. We compare input performed in the air to a touch-only technique and to a hybrid technique: Screen -Screen (CD ) serves as our baseline condition. Direct touch in the 2D canvas lets users position the pointer coarsely. They can then control it indirectly but much more precisely (higher CD gain) by initiating a slide gesture from a smaller rectangular area at the bottom of the screen (2cm above the edge). This is a direct adaptation of a technique available on Android and iOS smartphones for caret positioning, invoked by dwelling on the soft keyboard's space bar or by applying a higher force level anywhere on the keyboard (with pressure-sensitive screens). 11 Our adaptation behaves differently though, changing the CD gain to enable very high precision pointing. The original technique found on most smartphones keeps a 1:1 CD gain when entering indirect pointing mode. 12 Thus, it does address finger occlusion issues, but it does not enable high-precision pointing. Our implementation is a dual-precision version (CD ) which sets CD gain to 1:precision mm when entering indirect mode, where precision mm represents the size of the smallest element that users might want to acquire. In this experiment, we set it to 0.5mm, meaning that a 1mm finger displacement moves the pointer by 0.5mm. [START_REF] J O N E S | Around device interaction for multiscale navigation[END_REF], which compared different designs for pan & zoom navigation using mid-air gestures around the phone. Although the three input dimensions are not exactly the same in the two studies, their technique and ours are both based on integral and continuous input in a 3D mid-air volume, with movements in the x-y plane for controlling a 2D space (pan vs. cursor position) and movements along the z-axis for scaling this 2D input (zoom vs. CD gain).

Screen -Air

Screen -Screen+Air (CD ) is similar to Screen -Screen+Air (CD ) for sliders, but enables 2D selection. Users perform coarse adjustments with direct touch input, and fine-grained adjustments with air-tap-and-hold interactions. Precision-level control is dual (CD ), with the CD gain set to 1:1 for touch input on the phone's screen, and to 4: precision mm for mid-air input.

Design properties Again, each of these designs have supposed advantages, but it is difficult to anticipate their comparative performance. The difference between Screen -Screen (CD ) and Screen -Screen+Air (CD ) mostly lies in the input modality.

Screen -Screen (CD ) might benefit from touch input properties such as haptic feedback and prior experience. But at the same time, in the above-mentioned study by Jones et al. [START_REF] J O N E S | Around device interaction for multiscale navigation[END_REF] the 1-Button-Simultaneous -which resembles Screen -Air (CD ) -performed as well as multitouch gestures did. This latter result suggests that offloading the control in the air with a continuous control strategy might not have a high cost. The dual-precision control in Screen -Screen+Air (CD ) could make this cost even lower considering how it improved air control for parameter adjustment in experiment E Error bars represent the 95% confidence interval.

target diameter is 4mm (Low), 2mm (Medium) or 1mm (High). The next target to acquire is colored black, while other targets are colored grey. As targets can be very small, the background turns green when the crosshair cursor is inside the target. As soon as participants select the black target (no finger on screen, hand in release posture), the next target turns black. Cursor size is inversely proportional to the current CD gain, providing participants with a rough indication of its value, as in Experiment E 2 .

Design We follow a within-subject design with two primary factors: T e c hn i q u e ∈ {Screen -Screen (CD ), Screen -Screen+Air (CD ) and Screen -Air (CD )} and D i ff ic u lt y ∈ {Low, Medium, High}. Trials are blocked by T e c hn i q u e. Block presentation order is counterbalanced across participants with a Latin Square. Each trial is a series of eight pointing tasks, and is replicated twice. A T e c hn i q u e block consists of 6 trials (i.e., 6×8 pointing tasks). Presentation order for trials within a block is counterbalanced across blocks and participants with a Latin Square. Each block starts with three practice trials, presented in order of increasing difficulty.

Results

Before running analyses, we filter out the first pointing task in each series of eight, as the initial cursor position is not controlled when a series starts. We additionally remove one outlier that corresponds to a trial during which the HoloLens actually stopped working for a few seconds. Shapiro-Wilk tests do not reveal any violation of normality.

An anova of T e c h n i q u e and D i ff i c u lt y on Time reveals a simple effect of both T e c hn i q u e (F 2,22 = 42, p = 0.001, η 2 G = 0.65) and D i ff ic u lt y (F 2,22 = 39, p = 0.001, η 2 G = 0.41), as well as an interaction effect (F 4,44 = 6, p = 0.001, η 2 G = 0.14). As illustrated in Figure 2.19, the difference between Screen -Screen (CD ) and Screen -Screen+Air (CD ) gets larger as difficulty increases, and is not even significant when D i ff ic u lt y = Low (p = 0.7). The third design, Screen -Air (CD ), performed very poorly in comparison with the other two. Sorting techniques by performance, we get: Screen -Screen (CD ) (1.67s), Screen -Screen+Air (CD ) (2.53s), Screen -Air (CD ) (6.18s). Offloading input to the air has a performance cost. But this cost is limited if air control is implemented with a dual-precision mode, and could be counterbalanced by the advantages that air control has over touch: no occlusion, and no need to reserve space on screen for toggling between precision modes.

Qualitative assessment. Quantitative measures match participants' qualitative evaluation of techniques, in which they systematically rated Screen -Air (CD ) as significantly worse than the other two along all NASA-TLX scales, with the exception of Screen -Screen+Air (CD ) for Mental Demand (p = 0.07). This poor performance of Screen -Air (CD ) is consistent with the findings from Experiment E 2 . It again proved difficult to control position and input precision with a movement in the air while the output was on phone. Interestingly, this is different from what Jones et al. [START_REF] J O N E S | Around device interaction for multiscale navigation[END_REF] observed. Several elements might be at play. First, their technique was used for multi-scale navigation, which affects the whole graphical scene rather than just the cursor's position and size. The stronger visual feedback associated with their task might be more effective for continuous control. Second, zoom control is relative in their case, while precision control is absolute with Screen -Air (CD ). As discussed in E 2 , implementing relative control when clutching is not trivial, a problem which does not exist in the case of multi-scale navigation. One last difference is about the delimiter for mid-air gestures. Jones et al. rely on a physical button on the phone, which might be faster than the pinch posture that we use. However, participants in their study also complained about this physical button delimiter which they found uncomfortable to use.

Finally, participants' preferences are distributed across Screen -Screen (CD ) and Screen -Screen+Air (CD ) with eight participants ranking Screen -Screen (CD ) first, the other four preferring Screen -Screen+Air (CD ). They consistently ranked Screen -Air (CD ) third.

Summary of Findings

We compared different distributed UIs for three tasks that are representative of touch interaction with smartphones. While we observe an input performance drop with air controls in many cases, this drop is compensated for in certain situations since offloading widgets to the air has interesting properties. Widgets displayed in the air are readily accessible when they would otherwise be several steps away in the hierarchy. Some phone screen real-estate gets freed up to show more content. We derive the following guidelines from these empirical results. First, discrete controls (e.g., buttons, checkboxes) should not be offloaded to the air. Users have much trouble activating discrete controls with air gestures, at least with the default MRTK behavior used on the HoloLens 2.

Second, dual-precision air control (CD ) can effectively complement touch for parameter adjustments and 2D selections. Our observations reveal that users' performance and preferences are split between touch and dual-precision air control for on-screen widgets.

Participants were faster in experiment E 2 with dual-precision air control than with touch control for sliders on screen. In experiment E 3 , they performed equally well with touch and air controls (both dual-precision) for low-difficulty cursor positioning (4-mm precision). Air control performance degraded at higher difficulty, but the technique remains interesting nevertheless. First because it supports a three-state model of input which gives interaction designers more expressive power. But also because it is indirect and thus avoids visual occlusion issues -which were not operationalized in our task.

Occlusion can be partly addressed in a touch-only context by featuring an area on the phone's screen to trigger indirect control as our baseline condition does in experiment E 3 , but this takes some screen real-estate. Thus, as touch and air controls can coexist, there are only benefits in supporting both.

Finally, frequently-used sliders can be good candidates for offloading to the air. While participants were slower with air sliders than they were with screen sliders, the performance drop of Air -Air (CD ) illustrated in 2.18 (about 1-to-2 seconds) can be considered reasonable when put in perspective. Touch sliders are faster, but they consume a lot of screen space and accessing them often involves navigating menus (2.1-(a)). Air sliders are slower, but can be accessed very quickly without navigating the menu hierarchy. In addition, they do not generate any visual occlusion, as opposed to high-precision sliders such as OrthoZoom which extends control space beyond the widget's visual footprint.

Interaction designers should thus consider this trade-off when making a choice between touch and air sliders for parameter adjustments.

Figure 2.20 illustrates how these guidelines can be applied to the design of our image editing example from Figure 2.1. The toolbar remains on the phone's screen as offloading buttons in the air would cause too much of a drop in performance. Frequently-used sliders, which were accessible after navigating a menu, are offloaded in the air and thus brought to the application's top layer. The second button in the toolbar ( ), which was used to invoke those sliders, is thus no longer necessary. This frees a bit more space in the phone's toolbar to accommodate additional controls or enlarge other buttons. Finally, 2D selection can be performed directly on the phone, or indirectly in the air (dual-precision mode) if occlusion is an issue.

Limitations and Future Work

Studies as ours necessarily depend on the characteristics of the apparatus. Even using a state-of-the-art ARHMD, there are limitations, related primarily to the field of view and tracking accuracy. Our study gives a sense of the comparative performance of different widget designs with current AR technology. As this technology improves, comparative performance results might evolve. Novel sensing and display capabilities might also enable enhanced widget designs. Beyond initial empirical results, our contribution also includes a series of AR-enhanced widget designs, along with a set of tasks representative of phone interactions to test them. These can serve as a basis for future studies.

The observed performance drop of air control relative to touch should be interpreted keeping in mind that we compared AR-enhanced techniques to OrthoZoom and to a dual-precision touch pointer. These state-of-the-art touch-only techniques are hard to beat, even more so given the experience people have accumulated with touch interfaces over the years. Several participants reported that they found touch input more reliable, but also that they were feeling increasingly efficient with air input as they were proceeding with the experiment. It would be interesting to conduct longer studies in order to quantify learning aspects and identify performance envelopes [START_REF] B E N Ko | Performance envelopes of virtual keyboard text input strategies in virtual reality[END_REF]. Follow-up studies should also consider more elaborate tasks to evaluate the impact of visual occlusion and the overhead caused by having to navigate in the widget hierarchy in touch-only conditions.

Because of the covid-19 pandemic, the studies were run with a restricted pool of participants, including the authors themselves. Although this is unusual, we believe that our results have reasonable validity. First, running our statistical analyses without the authors does not change the outcome. Second, our contribution is an exploration of various AR-enhanced widget designs. There is no notion of the authors' solution and thus no bias in favor or against any specific design. Our pool also includes both participants with much ARHMD experience and participants who had never even seen one before. Despite this great variability, we were able to observe significant -and sometimes quite large -effects. However, we insist that our results remain exploratory, and should not be generalized without further investigation.

Finally, there are likely opportunities to improve air controls in terms of interaction design. For example, we used the default MRTK behavior for air buttons, but we thought about other mechanisms to activate them. For instance, by crossing them with finger movements coplanar to the phone, whose edges would provide haptic feedback. While participants performed better with sliders displayed on the phone but controlled in the air than with sliders fully in the air, several of them indicated that they preferred the latter. They did not like the input-output mismatch of the former. Adapting a dual-precision strategy for sliders fully in the air could also improve performance.

Conclusion

To help the wide adoption of AR HMD in users' everyday life, we should work towards advancing relevant use cases. In this chapter, we have discussed the benefits of AR for enhancing other everyday devices such as smartphones. We have first contributed a design space of smartphone-centric applications enhanced with AR. This design space described two dimensions which are the FUNCTION of the holographic content and its LOCATION. We populated this design space with examples from the existing HCI literature, demonstrating its descriptive power. We also provided novel ideas of AR-enhanced smartphone-centric applications, proving that this design space can also be used to generate new concepts and that AR could bring benefits to many phone-centric interactions. While some of these ideas remain conceptual, some led to actual implementations, demonstrating the feasibility of such applications.

We have also conducted a preliminary investigation, comparing different distributions of input and output between the holographic screen and the device's screen for common smartphone-based widgets. In particular, our study reported on command triggering, parameter adjustment, and 2D selection. All AR-enhanced widgets were compared to their touch-only counterparts. Our results suggest that, while touch-only widgets performed better in most cases, some of the proposing AR-enhanced distributions did reach similar performances, showing the practicability of such distributions. We do not push for the use of AR to offload all smartphone-based widgets, but rather suggest that frequently used widgets could benefit from being permanently accessible in AR, as this would save some time to access them and some valuable screen-estate.

In this chapter, we have provided practical use-cases for AR to complement daily devices as much as valuable guidelines for future AR-enhanced, smartphone-centric applications regarding the UI distribution across the two environments. These suggest that AR could be used on a daily basis, being integrated into users' digital environment.

DE-AUGMENTING AUGMENTED REALITY

This chapter is based on a work I have done in collaboration with Caroline Appert, Anastasia Bezerianos, Olivier Chapuis, and Emmanuel Pietriga. It is currently under review for CHI '22.

Introduction

Introducing AR in users' daily life is not only a matter of appealing use cases. The technology's form and performance should be good enough to be acceptable. Wearable Augmented Reality (AR) devices are improving on several fronts in that regard. Their field of vision is getting wider. Display resolution is increasing. Enhanced tracking capabilities enable better input techniques. Clumsiness is still an issue, however. Helmets and glasses remain overly conspicuous and uncomfortable for people to consider wearing unless necessary. Yet we can anticipate progress on this front too as AR-enabling eyewear technology evolves. Miniaturization and progress in terms of autonomy should eventually make it possible to embed AR technology in prescription glasses. AR displays are also getting integrated in domain-specific pieces of equipment in which they are considered socially acceptable already [START_REF] Ko E L L E | t look at me that way! understanding user attitudes towards data glasses usage[END_REF]: motorcycle helmets, safety glasses, hard hats. Near-eye displays [START_REF] B E N Ko | Combining an optically see-through near-eye display with projector-based spatial augmented reality[END_REF][START_REF] B A N G | Toward the next-generation vr/ar optics: a review of holographic near-eye displays from a human-centric perspective[END_REF] can involve radically different technologies such as metasurfaces [START_REF] H O N G | Metasurface eyepiece for augmented reality[END_REF][START_REF] Z H A N G | Metasurfaces for near-eye augmented reality[END_REF] and other types of contact lenses [START_REF]Augmented reality in a contact lens[END_REF][START_REF]Augmented reality in a contact lens: It's the real deal[END_REF], though these are still in the early stages of development.

As mentioned earlier in this manuscript, both academia and industry are anticipating such improvements and are contemplating an increasingly diverse range of use cases for AR.

Beyond the well-known ones [START_REF] B I L L I N G H U R S T | A survey of augmented reality[END_REF] such as gaming [START_REF] P I E K A R S K I | The outdoor augmented reality gaming system[END_REF], education [START_REF]Why should my students use ar? a comparative review of the educational impacts of augmented-reality[END_REF] or industry [START_REF] F U N K | Working with augmented reality? a long-term analysis of in-situ instructions at the assembly workplace[END_REF], use cases now include domestic use [START_REF] K N I E R I M | Exploring the potential of augmented reality in domestic environments[END_REF], therapy [START_REF] Lc A N I Z | Using augmented reality to treat phobias[END_REF], emergency management [START_REF] S C H L O S S E R | Head-worn displays for emergency medical services[END_REF][START_REF] S C H L O S S E R | Head-worn displays for emergency medical services staff: Properties of prehospital work, use cases, and design considerations[END_REF], assisting people with visual impairments [START_REF] L A N G L O T Z | Computational glasses for compensating colour blindness[END_REF][START_REF] M I N H T I K E | a mb o l c h il a r , P . Augmented reality glasses as an orientation and mobility aid for people with low vision: A feasibility study of experiences and requirements[END_REF], a large range of sports & leisure activities, spatial orientation assistance at large [START_REF]Landscape interpretation with augmented reality and maps to improve spatial orientation skill[END_REF], as well as advertising [151].

Developments on the technological side and on the usage side go hand in hand, and suggest a future in which AR has become pervasive. Becoming inconspicuous, the technology will get socially acceptable beyond the workplace [START_REF] Ko E L L E | All about acceptability? identifying factors for the adoption of data glasses[END_REF][START_REF] R I X E N | Exploring augmented visual alterations in interpersonal communication[END_REF]. Becoming practical in many contexts, it will allow for the actual realization of many of the envisioned use cases, as well as new, unanticipated ones. Removing these barriers to adoption will likely entice people to wear AR on a very regular basis, if not all the time. Perhaps more insidiously, some future forms of AR displays -such as contact lenses -could actually be impractical to put on & off continually, leading their users to keep wearing them even when not actually needed, which might in turn entice them to leave AR turned on.

This can be seen as a virtuous or a vicious circle depending on the perspective we take on it; but one that leads part of society to always-on AR, regardless. Matsuda [START_REF] Da | Hyper-reality[END_REF] as well as May-raz & Lazo [START_REF] Sight | Short film[END_REF] both propose striking depictions of this in their futuristic short films. Whether we embrace such a future or not, always-on AR raises multiple questions.

Many relate to ethics [START_REF] R I X E N | Exploring augmented visual alterations in interpersonal communication[END_REF][START_REF] Ro E Sn E R | Security and privacy for augmented reality systems[END_REF][START_REF] G O N Z A L E Z -L I E N C R E S | The ethics of realism in virtual and augmented reality[END_REF], some others relate to practical concerns, and yet others to both. One of them stems from the visual interference that AR can cause with respect to people's perception of, and interaction with, the real world. The more often AR content is shown, the more often it will somehow be "in the way", with consequences that range from mere user annoyance to serious physical injuries. AR designers simply cannot anticipate all situations, and there is a need for users to be in control of what is in their field of vision.

We argue that to achieve proper user agency, user control over AR content display cannot be limited to a simple on/off toggle. Some AR content might be interfering, while other content might be critical to the task at hand or even just convenient to have, thus calling for solutions to selectively remove AR elements from the field of vision.

In this chapter, we introduce the concept of de-augmenting augmented reality. After positioning our work in the literature, we rationalize the need to give users control over virtual content display through three scenarios. We then build a taxonomy that captures the different facets of de-augmentation. Does the de-augmentation affect a specific area of the user's field of vision or a specific region in the physical world? Is it triggered automatically based on rules or on-demand using, e.g., deictic gestures? Does it hide, composite or geometrically transform the virtual content to mitigate visual interferences?

We demonstrate the concept's technical feasibility, using off-the-shelf technology to build a proof-of-concept prototype. Finally, we discuss the many avenues for future work that it opens in terms of both societal concerns (safety, privacy, ethics) and practical concerns (interactive system design).

Background

De-augmenting augmented reality relates to the more general problem of displaying virtual content so that it effectively coexists with what users need to perceive from the physical environment. While this problem arises across the entire reality-virtuality continuum [START_REF]a n d K i s h i n o , F . A taxonomy of mixed reality visual displays[END_REF], some questions are of more particular importance in the context of AR.

As AR devices typically display content as a semi-transparent overlay on the user's field of vision, virtual elements can augment the physical world. But they can also visually interfere with the perception of that physical world. Even if not directly interfering from a perceptual perspective, virtual content can compete with cues from the environment (and conversely), causing problems of user attention, interpretation and general awareness of one's physical surroundings. Such issues will be even more critical in mobile contexts of use where the physical environment is highly dynamic and unpredictable, making the coexistence of the physical and virtual difficult to effectively automate. In this section, we give an overview of concepts and techniques from prior work that have inspired our reflections on de-augmentation and informed our taxonomy: user attention in a mobile context; techniques that alter our perception of reality; as well as augmented virtuality techniques, that bring some elements from the physical world back in a VR (Virtual Reality) environment.

Mobile Technologies and User attention

Compared to stationary hardware such as desktop workstations, mobile devices are often used in much more dynamic and unpredictable environments. Paying too little attention to the physical surroundings in such contexts can cause safety issues. The mobile technology literature reports on studies about this type of attention tunnelling [START_REF] G O N C A Lv E S | Impact of task on attentional tunneling in handheld augmented reality[END_REF], which has actually been investigated in the specific case of smartphone-based AR applications such as the Pokémon Go game. For instance, Gruenefeld et al. [START_REF] Ru E N Ef E L D | Guiding smombies: Augmenting peripheral vision with low-cost glasses to shift the attention of smartphone users[END_REF] designed glasses equipped with low-cost LED displays placed on their sides. Those LEDs emit light in the periphery of smartphone users' field of vision to warn them about potential physical hazards. Kang et al. [START_REF]Obstacle detection and alert system for smartphone ar users[END_REF] analyse a smartphone's camera feed to detect forward obstacles. They test different modalities, finding that audio and tactile cues are more efficient than visual ones. CrashAlert [START_REF] H I N C A P I É-R A M O S | Enhancing peripheral alertness for eyes-busy mobile interaction while walking[END_REF] implements a similar concept, this time using a depth camera to detect potential obstacles based on their distance to the phone. Different kinds of visual alerts on the phone were observed to be successful at making users adapt their behavior and avoid obstacles.

While all of the above projects study contexts in which users focus on their smartphone, Jung et al. [START_REF] Da | Ensuring safety in augmented reality from trade-off between immersion and situation awareness[END_REF] address similar physical safety situations, but focus on users wearing AR headsets in the street. They implement a vehicle position estimation technique that is then used to enhance ground traffic situation awareness, providing visual indications about vehicles that are too close. Regardless of the specifics of how the AR wearer is notified about potentially hazardous situations, these represent one obvious case in which de-augmentation can be beneficial. As discussed later, such situations are not the only ones that can benefit from it.

Altering Reality

Part of the above safety issues when wearing AR displays are due to the AR content either distracting users or actually hiding safety-critical elements in the physical environment.

But such visual occlusion is actually not the only source of concern about the perception of the physical environment. AR also has the potential to alter our perception of reality, which can be a major issue when the display covers the whole field of vision or at least a significant part of it.

AR technologies can display content that is obviously virtual, such as floating windows [START_REF] F I N N E G A N | The personal cockpit: A spatial interface for effective task switching on head-worn displays[END_REF]. Such elements are unlikely to be mistaken for physical elements, but they can hinder the visibility of people or physical objects that users interact with. For instance, the recent HoloDoc system [START_REF] A Nn E T T | Enabling mixed reality workspaces that harness physical and digital content[END_REF] builds on the seminal Digital Desk prototype [START_REF] W E L Ln E R | Interacting with paper on the digitaldesk[END_REF], providing a desk environment where virtual objects coexist with multiple physical ones. The latter can be large, such as printed documents, or fairly small, such as pens and erasers whose manipulation can be hindered because of the visual interference caused by the virtual content.

AR technologies are not limited to the display of such obviously-virtual content, however.

With proper spatial registration, AR content can be seemingly integrated into the physical environment, going as far as changing the appearance of objects or even people. While this yields exciting perspectives, it also raises ethical concerns which, at the very least, call for a mechanism that lets users remove augmentations on demand. The SceneCtrl system [START_REF] Ya N G | Mixed reality enhancement via efficient scene editing[END_REF] relies on scene reconstruction and texture patches to enable users wearing a Hololens to delete or move a real object in their field of vision. Annexing Reality [START_REF]Annexing reality: Enabling opportunistic use of everyday objects as tangible proxies in augmented reality[END_REF] is another system that makes use of actual objects in the user's environment to create tangible proxies for manipulating virtual objects. The system scans the environment for identifying the best match between a physical object and the virtual object to manipulate.

The virtual object is used as a texture applied to the physical object, actually modifying its appearance. Lindlbauer et al. [START_REF] L I N D Lb Au E R | Changing the appearance of real-world objects by modifying their surroundings[END_REF] also demonstrate that, beyond direct texturing, AR can be used to modify the surroundings of a physical object (e.g., to display virtual extensions or cast a shadow). Their system can modify an object's perceived size, color, or shape. HoloFace [100] goes one step further, using face tracking to change people's face. While the paper focuses essentially on the technical realization, such possibilities obviously raise ethical concerns. On this front, Rixen et al. [START_REF] R I X E N | Exploring augmented visual alterations in interpersonal communication[END_REF] study the potential of color alterations applied to clothes or objects, focusing more particularly on interpersonal communication. They identify limits to what users will accept both as the person altering others and as the person being altered.

As graphics rendering and spatial registration technologies improve, it will become increasingly difficult to discern the virtual from the physical. VR technologies can actually be used to implement mixed-reality experiences that go even further, literally creating "a world in which the physical laws ordinarily governing space, time, mechanics, material properties, etc. no longer hold" [START_REF]a n d K i s h i n o , F . A taxonomy of mixed reality visual displays[END_REF]. For instance, the Remixed Reality system [START_REF] L I N D Lb Au E R | Remixed reality: Manipulating space and time in augmented reality[END_REF] relies on a set of depth cameras positioned to make a live 3D model of a room, which can then be manipulated before being displayed in the headset. This technique makes it possible to give users the impression that they are flying or walking on walls. Mise-Unseen [START_REF] Winckler | Mise-unseen: Using eye tracking to hide virtual reality scene changes in plain sight[END_REF] makes it possible to perform such changes in the environment in plain sight without users actually noticing. The system relies on eye-tracking to model where users' visual attention is in order to make changes in areas that they are not focusing on.

In such cases where the world seen by users does not match the world in which they actually are, methods to ensure safety must be implemented. Chaperoning techniques can for instance display a grid visualization of nearby obstacles (such as the HTC Vive's Chaperone system1 ) or indicate a safe trajectory for users to follow [START_REF] K A N A M O R I | Walking assist method for vr zombie[END_REF]. Some research projects also rely on geometrical transformations applied to the displayed world in order to play on users' perception and make them adopt safe walking trajectories [START_REF] Ko H N | Redirected walking[END_REF] and gestures [START_REF] B E N Ko | Haptic retargeting: Dynamic repurposing of passive haptics for enhanced virtual reality experiences[END_REF]. However, while those techniques can address collision issues, there is a much more diverse type of safety and ethical issues to consider [START_REF] G O N Z A L E Z -L I E N C R E S | The ethics of realism in virtual and augmented reality[END_REF] when considering alterations of the real world, as Rixen et al. [START_REF] R I X E N | Exploring augmented visual alterations in interpersonal communication[END_REF] have started investigating. Here again, de-augmentation capabilities might be beneficial to address some of those concerns as we will discuss later.

Augmented Virtuality

Virtual reality fully replaces the real environment with a computer-generated 3D scene. While this helps make the experience immersive, the resulting isolation from the real world can again raise safety concerns, and prevents users from interacting with people and physical objects [START_REF] B O L A N D | A dose of reality: Overcoming usability challenges in vr head-mounted displays[END_REF]. This has led researchers to explore the concept of Augmented Virtuality where some elements from reality get integrated back into the artificial scene so as to, e.g., avoid collisions with physical obstacles; improve awareness of the social environment; and make it possible to interact with physical things -for instance, to grab a glass of water and drink.

In one of their studies, McGill et al. [START_REF] B O L A N D | A dose of reality: Overcoming usability challenges in vr head-mounted displays[END_REF] investigated different views onto reality to enable users immersed in VR to interact with physical objects (e.g., type on a keyboard or grab food). They observed that a large window onto the real world conveyed a lesser sense of presence than local views which, for instance, only integrate users' hands in the virtual scene, as in [START_REF] Av V Ed U T O | I'm in vr! using your own hands in a fully immersive mr system[END_REF]. These results are in line with those of Budhiraja et al.'s study [START_REF] J O N E S | r s y t h , D . Where's my drink? enabling peripheral real world interactions while using hmds[END_REF] in which VR users had to interact with a cup. The technique showing only hand and cup performed better than the one which cut a larger rectangular window onto the real world. Kanamori et al. [START_REF] K A N A M O R I | Obstacle avoidance method in real space for virtual reality immersion[END_REF] compared three methods to enable users to perceive real objects while playing in VR: the HTC Vive's Chaperone system that displays the outline of objects in range of the headset's camera; a visualization that displays a 3D point cloud of objects close to the user; and an visualization that actually displays real objects in the VR scene. The latter method proved the most efficient for a task that required participants to walk and reach a specific physical object such as a cup. Overall, these results suggest that, in the context of VR, seeing some reality is necessary to enable interactions with physical objects and enable safe locomotion in the physical environment, but that the amount of elements brought from the real world should be limited so as not to impact immersion too much.

The RealityCheck system [START_REF] H O L Z | Blending virtual environments with situated physical reality[END_REF] recently investigated an application-independent technical solution for the real-time reconstruction of people and objects in the physical space, and their graphical compositing with the virtual environment. This is interesting, as it paves the way for a generic solution to integrate physical elements back in VR. The problem has received less attention in AR than in VR, perhaps because it has been considered less critical so far. The work of Budhiraja et al. [START_REF]d B i l l i n g h u r s t , M . Interaction techniques for hmd-hhd hybrid ar systems[END_REF] is an exception. It investigates techniques that combine AR in a headset with a mobile device (as in, e.g., [START_REF] H E I N I S C H | Multifi: Multi fidelity interaction with displays on and around the body[END_REF][START_REF] A N D M C G U Ff I N | Enlarging a smartphone with ar to create a handheld vesad (virtually extended screen-aligned display)[END_REF][START_REF]Bishare: Exploring bidirectional interactions between smartphones and head-mounted augmented reality[END_REF]).

One of these technique actually de-augments the phone when users look down at it, so as to prevent virtual objects displayed in the headset from interfering with the phone's screen.

As AR technologies evolve, they will be used in a variety of daily tasks that involve interactions with physical objects and people. Users' needs in terms of de-augmentation will thus increase. Our contribution is a series of scenarios that illustrate these new needs, and a taxonomy of generic de-augmentation operations. As opposed to VR, which requires the system to decide which elements of reality to show, AR users may still have some perception (even if altered) of the physical environment, giving them agency over what to de-augment, when, and how.

Illustrative Scenarios

Taking a leap into the future, we now describe various situations in which AR has become pervasive. We illustrate why users would sometimes want to de-augment AR in those situations, and how they could do it.

All scenario figures are using artificial color schemes for the sake of clarity: real-world elements are colored with shades of grey, while AR content is colored mostly with shades of orange. De-augmented areas are outlined with a blue stroke.

Workplace

Kim is working on the design of a hydrogen car with her laptop at her workstation in a large room she shares with co-workers. A physical model of the car's backbone lies in the center of this room. When looking at it through their AR glasses, all workers can see the latest virtual iteration on the car's dashboard, properly aligned with the physical backbone. Kim and her co-workers gather around it to discuss some aspects of the car's interior design.

Like many others, Kim closed her laptop before joining the meeting. She still wants to keep some awareness about incoming messages -both professional and personal -but she does not want those to interfere with the meeting. She has thus configured silent notifications, displayed in AR, world-anchored above her laptop, so that they are not constantly in her way but only show when she glances at the laptop (Figure 3.1-a). The team is discussing the optimal placement for the seats and shift lever, considering comfort, aesthetics and mechanical constraints. They also have to make a preliminary choice about two different steering wheel designs, for which they have 3D printed models.

As they want to see how each one integrates with the virtual dashboard, Kim triggers a de-augmentation of the wheel based on its bounding polygon. As illustrated in Figure 3.1b & 3.1-c, this effectively cuts holes in the AR display, resulting in a hybrid representation that seamlessly combines virtual and physical elements.

After an intense discussion, Kim suggests they take a break and go get a coffee. By company policy, AR is not allowed in the cafeteria to ensure a healthy, relaxing break promoting truly-social face-to-face interaction. Management had originally considered asking people to remove their eye-ware in the cafeteria, but quickly realized that there were legitimate reasons to keep wearing it in there too. For instance, Kim has her AR display integrated with her prescription glasses. Jo, one of her co-workers, recently bought the latest technological innovation: true-color AR contact lenses. As any other lenses, those would be quite impractical to remove just to have a coffee break. Management eventually opted for another solution: people are still allowed to keep their eye-ware, but the devices are automatically instructed to de-augment everything that is not safety-critical when entering the cafeteria's area.

After the break, everyone goes back to their workstation. On leaving the cafeteria, personnel-wide de-augmentation restrictions are removed. Kim gets back to her desk and opens her laptop. The AR notifications floating above it are getting in her way now, however. She points at her laptop and performs a swipe gesture to de-augment them.

The system infers that the computer screen is the area to de-augment as the gesture was initiated while pointing at it. It also infers from the context and the explicit nature of this gesture that Kim does not want the AR content that falls in this area to be removed, but rather shifted outside it. As a result, some of the AR notifications get offset upward, as illustrated in Figure 3.1-d.

Home

Alex is an engineer working from home, actually developing a prototype AR application. They just deployed it from their desktop workstation and are in the process of testing it when their phone alarm goes off. This automatically triggers a de-augmentation of the area around the phone, enabling Alex to check it. They see that it is time to prepare dinner for the family. They exit the AR application they were testing in debug moderesetting their AR eye-ware to normal mode. In this mode, incoming AR notifications appear on the left side of their field of vision. Alex likes this setting as they can check messages while they move around the house.

They start cooking dinner and launch an AR cooking application to get virtual instructions about how to prepare a new dish they have been wanting to try for a while. Meanwhile they receive a notification from their partner that they will be home soon (Figure 3.2-a).

In the kitchen, the induction cooking plates and the smart oven are always de-augmented. This is both for safety and practical reasons, as it enables direct monitoring of the cooking process. Figure 3.2-b illustrates two de-augmentations in the oven area: the AR recipe gets partially hidden where it intersects the oven; and the AR notifications (here the cooking plate countdown at 11 minutes) get shifted downward, right below the oven.

Alex's partner has come home. Dinner will soon be ready. Alex goes upstairs fetch their kids (Figure 3.2-c). One of the kids is playing an AR game that involves catching butterflies that try to hide behind clouds that float throughout the room. As Alex enters, their face gets automatically de-augmented from the child's perspective (Figure 3.

2-d).

This way they can talk to each other without AR interfering and adversely impacting social interactions [START_REF] H E Rr E R A | Social interaction in augmented reality[END_REF]. The oven is always de-augmented, but the two pieces of AR content get de-augmented differently. The notifications get shifted downard, while the recipe gets cropped. c) Alex is heading upstairs. Meanwhile one of their kids is playing an AR game about catching butterflies that try to hide in the clouds. d) Alex enters the kid's room, and when the kid looks at them, their face gets automatically de-augmented, cropping AR clouds and butterflies.

Outdoors

Jen is taking a stroll in the park. She notices a poster advertising a live music concert when her partner, Sam, calls her (Figure 3.3-a). He is at the bakery and wants to know what type of bread Jen would prefer (Figure 3.3-b). Once the matter is settled, Jen asks Sam to remain on the call, as she thinks the concert is by his favorite band.

Jen got closer to the poster while she was talking with Sam. She can now read the details, but Sam's video feed is in the way. Jen points at the poster (Figure 3.3-c) to de-augment it temporarily with a gesture. She confirms the band and finds out when they will be performing (Figure 3.3-d). She asks Sam whether he would like to go, and suggests one of the three performance dates. Sam seems to remember that they are already invited to a party on that day. Jen checks her calendar in AR, and confirms that it conflicts with their friend's party indeed. Having both her calendar and the concert poster in her field of vision, she finds another date that works. The timer has now expired, automatically canceling the temporary de-augmentation. Jen and Sam keep chatting, with Jen asking Sam to book tickets as soon as he is home.

De-augmentation Taxonomy

As illustrated in the scenarios above, there is a variety of ways to invoke de-augmentation operations, and a variety of ways to actually de-augment AR content. We already saw that a de-augmentation can be triggered by the user or by an external event, and that it can have a very limited scope or, on the contrary, that it can apply globally. In this section, we first define what a de-augmentation operation is. We then organize these operations into a taxonomy. The overall goal of this taxonomy is to help designers of AR systems think about situations that call for de-augmentation and design techniques to support such operations accordingly.

A de-augmentation is defined as an operation that alters virtual content in order to improve the user's perception of the real world. The definition is kept vague on purpose, as the concept of de-augmentation could be relevant not only for visual content, but possibly for modalities such as audio and haptics as well. As we focus here on visual de-augmentations, we can also give a more specific definition of a visual de-augmentation: an operation that alters virtual content in the user's field of vision in order to improve their visual perception of the real world.

We decompose a visual de-augmentation operation into three components:

• what triggers the de-augmentation;

• what is its scope, i.e., what part of the real-world gets de-augmented and for how long; • how is the de-augmentation achieved, i.e., how is the virtual content's rendering altered.

De-augmentation Triggers

AR de-augmentations happen in response to a trigger. We distinguish three main types of triggers. The first two are specified by users themselves: either on-demand by invoking a User Command; or automatically, the system triggering User Rules that match the current situation. The last type of trigger is specified by an External Source independent from the user. These can be rules hardcoded for safety or privacy reasons, or imposed by authorities who have jurisdiction in the current context.

User Command

A de-augmentation can be triggered with a User Command invoked explicitly. Such commands can involve any input modality. In the workplace scenario, Kim performs a swipe gesture to de-augment her laptop's screen. In the outdoors scenario, Jen points at the poster advertising the concert. While these examples involve hand movements (gestures), any other interaction technique that works well with AR technologies can be considered, such as voice or gaze. One could for instance envision giving users some control on the level of (de-)augmentation based on how they look through their glasses. Just the same as the wearer of prescription glasses equipped with progressive lenses does not get the same amount of correction depending on whether they look through the lower, middle, or upper part of their eye-ware, the wearer of AR glasses could have some control on virtual content (de-)augmentation. This would only give them a relatively basic level control, however.

Depending on the other characteristics of the de-augmentation to be specified, more expressive input might be required, for instance to define a specific scope or a specific way to de-augment the virtual content. Simple deictic pointing combined with computer vision techniques can for instance be used to de-augment specific objects. Freeform gesture tracking enables users to delineate arbitrarily-shaped spatial scopes such as, e.g., a sketch on a whiteboard or an irregular area on the floor. We can even think of using other modalities such as a handheld device (e.g., a smartphone) or tangibles. For example, drawing inspiration from Butz et al. [START_REF] B E S He R S | Of vampire mirrors and privacy lamps: Privacy management in multi-user augmented environments[END_REF], a tangible could play the role of a lamp that de-augments the region it illuminates. Tangible posts could also be used to delimit a large area, in the spirit of the ShareSpace system [START_REF] Ya N G | Facilitating shared use of the physical space by both vr head-mounted display and external users[END_REF].

Finer control on the de-augmentation's properties can also be provided with traditional UI widgets in AR, for instance to adjust specific rendering parameters such as the transparency of de-augmented areas. However, we can expect that advanced and timeconsuming de-augmentation specifications would rarely occur, elaborate configurations being better managed as automatically-triggered rules when possible.

User Rules

Contrary to user commands, User Rules

do not require explicit actions from the user. They are triggered by the AR system automatically, whenever they apply to the current context. In the workplace scenario, a de-augmentation is triggered to let Kim see the steering wheel because she has configured the system so that it will de-augment the area of her field of vision that corresponds to any object she holds in her hands. This is a general rule, that she applies in any context, that would also let her see the glass or cup she intends to drink from [START_REF] B O L A N D | A dose of reality: Overcoming usability challenges in vr head-mounted displays[END_REF].

Such user rules remove the need to take explicit action to trigger a de-augmention, but as illustrated just above, they also remove the need to specify a scope for the deaugmentation. Another example consists of de-augmenting people's face in the household, as Alex and their family have done in the Home scenario. More elaborate examples include Alex's phone getting de-augmented because an alarm went off; or a window in their house getting de-augmented whenever they stand less than one meter from it as they enjoy relaxing by looking at the stunning view over the mountains they get from there.

We can also envision rules that would fire at regular intervals. For instance, previous work has pointed at the actual risk associated with too much AR assistance, that could make everyday life too facile [START_REF] K N I E R I M | Exploring the potential of augmented reality in domestic environments[END_REF], and make dependency on technology too strong [START_REF] Ro G E R S | Moving on from weiser's vision of calm computing: Engaging ubicomp experiences[END_REF]. People who want to self-regulate their use of AR or just make sure that they stay in touch with (totally-unaltered) reality could configure a rule to de-augment everything for a few seconds every five minutes.

External Source

While many rules will have been configured, or at least activated, by the users themselves, we can also envision that some rules would be imposed by External Sources . These de-augmentations would be beyond users' individual control. As mentioned earlier, authorities having jurisdiction in a given area might enforce a no-or limited-AR policy for safety, privacy, or security reasons. Buildings such as police stations or any type of restricted area where sensitive activities are performed are obvious candidates, but trafic-heavy streets (at least when about to cross them) might also be considered as such. The workplace scenario illustrates another sort of trigger from an external source, where AR content gets de-augmented when entering the cafeteria by company policy, except for safety-critical information.

De-augmentation Scope

The second defining characteristic of a de-augmentation is its scope, which can actually be broken down in two (Figure 3.4): the spatial scope, and the temporal scope. The spatial scope can be defined by physical entities (people, objects) no matter where they are; or it can be defined as a specific region, no matter what entities it contains. The temporal scope is orthogonal to the spatial scope. It can be either permanent or temporary.

Spatial Scope: Entity

De-augmenting an entity means de-augmenting virtual content that visually interferes with the perception of this entity, regardless of its actual location in the physical world. Typical examples include people (face or entire body); physical objects; as well as physical displays such as computer screens or smartphones. An entity's location can change over time, implying that the system must be able to track it in the user's field of vision in order to dynamically adapt the de-augmented area.

Alex's smartphone and face (home scenario), or the concert poster (outdoors scenario) are examples of de-augmentations whose spatial scope is an entity. A scope can consist of multiple entities. In collaborative AR work contexts [START_REF] B I L L I N G H U R S T | A survey of augmented reality[END_REF] such as the one described in our workplace scenario, multiple entities of different nature get de-augmented simultaneously: users share the same overall virtual scene, but de-augment not only physical objects (such as the 3D-printed steering wheel) but their collaborators' face as well to preserve the directness and naturalness of face-to-face interpersonal communication [START_REF] H E Rr E R A | Social interaction in augmented reality[END_REF].

Spatial Scope: Region

De-augmenting a region means de-augmenting virtual content that visually interferes with the perception of a region in the user's field of vision, regardless of what physical entities it contains. A region's location does not change over time but its contents can change, without impacting the de-augmentation. A region can be either world-anchored or FoV-anchored. A world-anchored region is fixed in the physical world, regardless of the user's location and viewing direction. A FoV-anchored region is defined relative to the user's field of vision, as if it were attached to the eye-ware.

The cafeteria in the workplace scenario is an example of world-anchored-region scope. The oven and cooking plate de-augmentations in the home scenario can actually be considered as instances of world-anchored-region scopes. On a conceptual level, these are entity de-augmentations. But because those entities are stationary for most practical purposes, handling them as fixed regions removes the need for dynamic object tracking. FoV-anchored-region scopes are perhaps less common. They can be useful to guarantee the absence of visual interference in part of the user's field of vision regardless of any other consideration. They could also serve purposes such as before/after comparisons: with half of their field of vision showing virtual content but not the other, users could compare, e.g., actual vs. virtual clothes or face with/without makeup.

Temporal Scope: Temporary

A de-augmentation whose temporal scope is temporary will only last some pre-defined duration, after which it will automatically get cancelled, the virtual content appearing back.

Jen's de-augmentation of the concert poster in the outdoors scenario is temporary, as she only needs to look at it for a few seconds. Having to cancel the de-augmentation explicitly would be cumbersome. Similarly, Alex's smartphone could get de-augmented for three or four seconds when notifications pop up. Temporary de-augmentations can also be useful to lower the level of engagement [START_REF] A N D M U R R Ay-S M I T H | Focused and casual interactions: Allowing users to vary their level of engagement[END_REF], as an AR game player might want to in order to eat a snack for instance.

Temporal Scope: Permanent

A de-augmentation whose temporal scope is permanent lasts until it is explicitly cancelled, either by the user or by an outside event that results in a change of context. For instance, the rule that triggered the de-augmentation might no longer apply.

The de-augmentation in the cafeteria from the workplace scenario, the oven and cooking plates in the home scenario as well as Alex's face are all examples of permanent deaugmentations. Even if they do not necessarily last very long, the steering wheel and the laptop screen de-augmentations in the workplace scenario are considered permanent as they are active as long as those entities remain in the user's field of vision.

De-augmentation Rendering

Once triggered in a specific scope, a de-augmentation is achieved by altering the virtual content that falls within that scope. The simplest de-augmentation consists of not showing anything in the corresponding area of the user's field of vision, as if cutting a hole in the virtual content. But alternative rendering strategies can also be considered (Figure 3.4).

Pixel Compositing

Cutting holes in a layer to reveal hidden objects has been experimented in many different situations, including window management [START_REF] Ta N | Manipulating arbitrary window regions for more effective use of screen space[END_REF], map comparisons [START_REF] P I E T R I G A | An evaluation of interactive map comparison techniques[END_REF], or layered 3D models [START_REF] P I E T R I G A | Drilling into complex 3d models with gimlenses[END_REF]. Drilling holes in the AR layer can be as straightforward as applying a binary image mask to the rasterized virtual content right before display in the eye-ware. But this is just one particular option in the broader category of pixel-based image compositing strategies. Pixels that fall within the de-augmentation's scope or in its surroundings can have their alpha channel adjusted without necessarily making them fully transparent. Content-aware techniques [START_REF] A N D F E I N E R | Interacting with hidden content using content-aware free-space transparency[END_REF]207] are also quite interesting in our context: they could be used to adjust the level of transparency of individual pixels based on the comparative importance of the virtual and real content in a given area. Most of the de-augmentations in our three scenarios are examples of compositing-based de-augmentations.

Geometric Transform

Virtual content can also be removed from a given scope in the user's field of vision by applying transformations to elements in the AR layer. One of the simplest transformations consists of applying affine transforms to individual elements that intersect the deaugmentation's scope. With a combination of translations and downscaling operations, elements from the AR layer can be prevented from visually interfering with elements to be de-augmented. Notifications in both the workplace and home scenarios are examples of such de-augmentations.

Beyond affine transforms, more advanced transforms that distort the geometry of elements in the AR layer can be considered. For example, content-aware focus+context techniques [START_REF] P I E T R I G A | Content-aware adaptive lenses[END_REF] could be adapted to distort elements in the virtual layer so as to better integrate them with the de-augmentation scope's shape. Desktop techniques can also be a source of inspiration, based on the metaphor of rolling or folding layers as if they were paper [START_REF] A N D Ro U | ss e l , N . Copy-and-paste between overlapping windows[END_REF][START_REF]Combining crossing-based and paper-based interaction paradigms for dragging and dropping between overlapping windows[END_REF][START_REF] E L M Q V I S T | Space folding for multi-focus interaction[END_REF].

Proof-of-concept Prototype

Many of the situations described in our scenarios take a more or less distant leap in the future, making the assumption that advancements in eye-ware display technology has led to the broad diffusion of AR. It is still interesting to assess the technical feasibility of de-augmentation with current AR devices as the principles should essentially remain the same. We developed a proof-of-concept prototype to illustrate and test several of the de-augmentations from our taxonomy. This prototype runs on a Microsoft HoloLens 2 (OS Build 10.0.19041.1131), and was developed with Unity 2019. [START_REF]Supporting subtlety with deceptive devices and illusory interactions[END_REF].11f1 and the Microsoft Mixed Reality Toolkit (MRTK 2.5.4).

The prototype, illustrated in Figure 3.5 and demonstrated in the companion video, lets users create multiple de-augmentations by invoking a menu. They first specify the type of spatial scope: world-anchored or FoV-anchored. They can then select among multiple options to delimit it.

The first technique to define a world-anchored scope relies on MRTK's SceneUnderstanding feature to detect surfaces in the physical environment. Users point at the surface2 they want to de-augment and perform an air tap to actually select it. Other techniques involve drawing primitives such as lasso or rectangular selection tools. The interaction sequence differs depending on the drawing tool considered, seeking to make interaction as simple as possible. With the lasso tool, users can draw arbitrary shapes; the system will then sample points along the drawn curve and reconstruct a 3D surface from them. With other tools, users only have to specify two points in 3D space. The system will constrain the bounding box of the selected geometric primitive (cuboid, ellipsoid) so that two of its faces remain parallel to the user's sagittal plane. Regardless of how these 3D scopes are specified, pixels that fall in their projection in the user's field of vision will be de-augmented, as illustrated in Figure 3 Users define FoV-anchored scopes by delimiting an area on a 2D surface coplanar to the headset's screen. Usual drawing tools are again used for this purpose, including lasso, rectangle and ellipse. An additional eraser tool lets users delimit a scope by holding one hand flat in front of them and moving it, as if they were wiping the virtual plane. As in the world-anchored case, pixels that fall in those areas will be de-augmented. The areas remain tied to the headset, however: the de-augmentations remain fixed in the user's field of vision.

Beyond manually-specified de-augmentations, our prototype also supports some simple User Rules aimed at demonstrating automatically-triggered de-augmentations. These rules de-augment faces, the wearer's hands, and objects that have been tagged. Hand tracking is readily available in the Hololens. Face recognition relies on OpenCV 4. 1.2 and HoloLens2ForCV [START_REF] Ng U R Ea N U | Hololens 2 research mode as a tool for computer vision research[END_REF]. Objects are tracked by attaching Vuforia tags to them and declaring them in the system.

Our prototype can render de-augmentations based on pixel compositing. Pixels that fall in a scope are made fully transparent -thus revealing the physical world -using ColorMask shaders, one for each de-augmented area in the user's field of vision. The shader removes any color in the area and forces the shape to be rendered first in the queue, right in front of the near plane, so that the de-augmentation mask will apply to all objects behind it in the virtual scene.

Discussion

Technological advances let us foresee a future where AR eye-ware has become so lightweight and inconspicuous that reality can be augmented most of the time, if not all the time. The purpose of this chapter is not to advocate for such a future, but rather to highlight the need to give users some agency over virtual content, enabling them to de-augment augmented reality. While toggling the full AR display off is a special case of de-augmentation, we are rather interested in locally-bounded de-augmentations, that remove virtual content in some parts of the users' field of vision, where it actually interferes with their perception of reality only.

De-augmentation, as a concept, is different from hybrid representations in VR that bring elements from the real world back into the user's virtual world. VR completely occludes the real world. Research on hybrid representations has consequently focused on integrating elements from the users' physical surroundings based on the assumption that the system would be in charge of detecting what elements from the real world should be integrated into the virtual scene. Because AR superimposes content on a virtual, transparent layer in the user's field of vision, it does not completely occlude the real world. It is thus inherently hybrid, and raises different challenges. In AR, users can still perceive the world around them. Even in a crowded field of vision, they will see the real world through the virtual content -albeit degraded. While this content can impede users' perception of their physical surroundings -one of the very reasons for enabling de-augmentations -this ability to see through the virtual layer is key to giving users agency over how to mitigate visual interference. The challenge lies in providing users with easy means to de-augment, both manually (interaction techniques) and automatically (context-dependent rules). Another challenge comes from the versatility of AR, which has different use cases than VR. Perhaps a broader range of them, but most importantly, many that involve mobile contexts of use which are much more unpredictable than the typical VR room. Virtual content will be "in the user's way" in a variety of situations, all of which cannot possibly be anticipated by AR system designers and engineers.

In this chapter, we set forth a series of scenarios that illustrate when de-augmentations might be relevant. Examples developed in these scenarios included: seeing faces when talking to people while staying immersed in a dense AR environment (kid playing a game in the Home scenario); combining information from the physical and the virtual world while on a call (concert poster, calendar and remote video feed in the Outdoors scenario); guaranteeing the visibility of safety-critical appliances (oven and cooking plates in the Home scenario); immersing physical objects in their virtual surroundings (steering wheel inside the car in the Workplace scenario); enforcing social rules in public spaces (cafeteria in the Workplace scenario).

More scenarios calling for de-augmentation capabilities exist. These are the ones that informed our taxonomy of AR de-augmentations. This taxonomy represents the main contribution of this work. It aims at providing a common framework to talk about de-augmentation. It also aims at helping designers integrate de-augmentation in their AR experiences. The taxonomy characterizes a de-augmentation based on three elements: who or what triggers it; what is its scope (both spatial and temporal); and how is the de-augmentation achieved visually. The goal is not only to organize de-augmentation techniques, but to inspire future designs as well. The taxonomy itself will likely need to be expanded, or at least refined, as new AR usage scenarios emerge. For instance, the External Source trigger category is fairly broad. There might also be some specific triggers that do not fit nicely in one of the categories. Let us consider the case where a de-augmentation should be triggered when somebody waves at an AR user. The trigger is essentially from an External Source, but also requires that a User Rule exist on the AR user's side to authorize such de-augmentations initiated by third parties. This might be acceptable when the de-augmentation originates from relatives, friends and colleagues. For example, the parent in the Home scenario might have had to actually wave at their kid instead of being automatically de-augmented, if only to prevent unwanted de-augmentations when just passing by. But such third-party triggers might not be acceptable when they originate from mere acquaintances or bystanders. This raises the broader question of which external sources might have control over a user's AR de-augmentations, when, and how. Yang et al. [START_REF] Ya N G | Facilitating shared use of the physical space by both vr head-mounted display and external users[END_REF] investigated how to enable both VR and non-VR users to share the same physical space. Non-VR users could "protect" themselves from VR users wearing an HMD by positioning virtual shields (visible in VR) in the room. The intention was primarily to avoid physical collisions, but this work draws attention on the issue of enabling both XR and non-XR users to be in the same space. In particular, what sort of control should XR and non-XR users have over the virtual content? What sort of control should authorities have in areas under their jurisdiction? The notion of sensitive areas with restricted physical access, or where taking pictures is not allowed, could be extended to limit the use of AR. Safety-related de-augmentations such as automatically removing virtual content interfering with emergency exits would seem to make sense, but what if the user had special needs that made the virtual content key to their safety? Could these external triggers be overridden by the user's system? These questions have safety and societal dimensions (privacy, ethics) that obviously go well beyond the design of a taxonomy of de-augmentations. But the concept of de-augmentation seems relevant to the broader matter of AR-related laws and regulations. Keeping in mind that in many scenarios a full AR deactivation will be neither desirable nor necessary, and drawing a parallel with accept vs. block lists: while some cases are best handled by preventing an augmentation in the first place, other cases are best handled by de-augmenting part of what has been augmented in the first place.

Future Work

Beyond safety and societal considerations and the corresponding needs in terms of regulations and protocols, there are also more practical questions that pertain more directly to interaction design. One of them relates to the cost of specification and trigger of de-augmentations. User rules activate automatically based on the user's context, but necessarily have an initial setup cost. De-augmentations based on user commands are specified manually on the spot. Their creation cost is not necessarily very high, as they should be fairly simple (typically applied directly to a given region or entity). But manually creating a de-augmentation will require users to interrupt their primary task, which could be tedious and frustrating.

Streamlining the de-augmentation process from an interaction design perspective is a key issue. The prototype framework introduced in Section 3.5 features examples of both user commands and predefined user rules. Implemented with current off-the-shelf technology, this prototype primarily aims at demonstrating that de-augmentation is not just a concept, but an actual possibility already. The interaction techniques developed to manually create de-augmentations are very crude and only enable simple de-augmentations. Much work remains to be done to lower the cost of de-augmenting with user commands. This interaction design work, however, will be heavily dependent on the input and tracking capabilities of future AR devices. The specification of user rules is maybe less dependent on those capabilities, at least in terms of input modalities. But implementing effective rule systems will require multiple sensors in order to infer the user's context and activate de-augmentations accordingly.

Contextual information could actually be used to assist users in the specification of the rules themselves, in order to minimize the setup cost. For instance, the system could learn from users' past actions: if a user frequently de-augments windows in their house when they are less than one meter from them, the system could suggest adding a rule to automatically de-augment any window within this perimeter. The system might also learn to apply different rendering strategies to de-augment particular pieces of virtual content, for example making notifications semi-transparent instead of masking them based on the inferred user's degree-of-interest in this type of content. Contextual information could also help create de-augmentations manually. As mentioned earlier, our prototype lets users select surfaces detected by the HoloLens 2 as de-augmentation scopes. More elaborate object detection and scene understanding capabilities would help streamline the interactive de-augmentation process. This would mitigate the earlier-mentioned interruptions in the user's primary task, which is key to making de-augmentation an effective solution for managing visual interferences caused by virtual content.

Taking a more prospective look at future uses of de-augmentation, the concept might find interesting applications as we get closer to the ultimate goal of "generat[ing] virtual objects that are so realistic that they are virtually indistinguishable from the real environment [START_REF]A survey of augmented reality[END_REF], possibly helping locally differentiate the virtual from the real. Another direction for future work will be to investigate the concept of de-augmentation applied to other AR modalities such as sound and haptics. Technological advances (audio spatialization, noise-cancelling techniques and ultrasound-based haptics just to mention a few) yield exciting opportunities to manipulate these signals. Here again, and as per the definition of de-augmentation given in Section 3.4, the idea would rather be to investigate the feasibility and benefits of partially de-augmenting the sounds and haptic feedback generated by the AR system -as opposed to fully deactivating them -so as to improve users' perception of the real world across more senses than vision alone.

Conclusion

Technology advance promises a future where Augmented Reality content can be blended with the real environment. This definitely affects users' perception which can be tricky, especially for security and social concerns. In this chapter we have presented the concept of de-augmenting AR as a way for users to choose what they want their focus to be on, being constantly in control of what they perceive. We have first illustrated the concept of de-augmenting with three scenarios that illustrate how de-augmention could appear in three different daily contexts: 1) at work, 2) outdoor, and 3) at home. We introduced a taxonomy of de-augmentions and thus, delineating the main components in the design of de-augmenting techniques, which are: 1) the triggering, 2) the scope, and 3) the rendering.

We implemented an actual prototype combining several tools for de-augmentation. Our prototype shows the feasibility of letting users decide how to organize their field of view, deciding on what the elements of interest are in what they view. While we argue that AR eyewear could be integrated into users' daily lives,we also point at the risk for users to lose control over their perception of the real world with the advance of AR technology. There is a real need for involving users in the blending process between both the real and the virtual environments.

Part III CONCLUSION AND PERSPECTIVES

Supported by the HCI literature as well as my own research, I strongly argue that Augmented Reality has a great potential of being adopted by users in different daily contexts. However, we are not there yet. As discussed in this manuscript, it is not only related to the maturity of the technology, it is also because of usability and utilisability issues [START_REF] Ko E L L E | All about acceptability? identifying factors for the adoption of data glasses[END_REF]. Such issues range from low-level ones such as the fatigue that is inherent to mid-air interaction, to high-level ones such as the acceptability regarding safety and social aspects. This thesis is an investigation of how AR could be reconsidered to be adapted to daily situations. We identified three specific research questions and reported on research projects to answer them. Each of these questions relates to one aspect of the adoption at large of AR technology: 1) users' motor performance when interacting with AR ontent; 2) integration of AR technology in users' digital environment; 3) users' agency regarding where AR content should be displayed. This thesis' contributions are in terms of interaction design, empirical results and conceptual frameworks. This chapter gets back to each individual research question, provides a summary of contributions and opens questions for future work.

RQ 1 How to reconsider interaction with wearable AR to better support long-lasting tasks?

To answer this research question, we present the ARPads project, studying the performance of mid-air indirect input for Augmented Reality as a means to reduce fatigue when interacting with AR content. We characterize a design space for such interactions, based on the location and orientation of those pads relative to the user's body. We report on a study, comparing six of these ARPads to their state-of-art counterparts for continuous and discrete interactions. Our study shows that some ARpads can reduce users' fatigue while reaching time and precision performances that are similar to or better than the state-of-art counterparts. We also provide guidelines for developers to implement such ARPads.

Our current implementation of ARPads relies on absolute input. This means that the absolute position of the user's hand whithin the ARPad is mapped to the same absolute location whithin the cursor's space, whith a Control-Display ratio that is the same over the whole surface. Future investigations could explore the use of a relative mapping between the user's hand movements and the cursor displayed in the AR screen. While absolute input makes it fast to reach a specific point in the interactive area [START_REF] A N D I R A N I | Arc-pad: Absolute+relative cursor positioning for large displays with a mobile touchscreen[END_REF], relative input techniques could bring new advantages. In particular, they make it possible to change the Control-Display ratio, which is often useful to perform precise movements [START_REF] N A N C E L | a n d B e au d o u i n -L a fo n , M . Mid-air pointing on ultra-walls[END_REF]. The Control-Display ratio can also be defined as a function of the UI semantics [START_REF] B L A N C H | Semantic pointing: Improving target acquisition with control-display ratio adaptation[END_REF] in order to facilitate interaction with some specific UI elements. Finally, relative mapping can also take advantage of transfer functions that consider precision as a function of movement speed [START_REF] A N D Ro U Ss E L | No more bricolage! methods and tools to characterize, replicate and compare pointing transfer functions[END_REF]. A promising direction could be to consider a combination of both absolute and relative input such as HypridPointing [START_REF] Fo Rl I N E S | Hybridpointing: Fluid switching between absolute and relative pointing with a direct input device[END_REF], which could make ARPads benefit from both relative and absolute mapping advantages. Such improvements would open up the applicability of ARPads to larger interactive spaces than the medium-sized AR window that we tested in our study.

Another interesting perspective for ARPads relates to their shape. Our experiment only considers 2D rectangle pads, with an aspect ratio that is identical to the virtual window displayed in AR. In future work, we would like to explore 3D indirect mid-air input. The 3D AR volumes could either have a basic, cuboid shape or even a more complex one that fits the geometry of the specific object to manipulate. In both cases, this future work would likely require more design work regarding feedback about the shape and the location of the virtual interactive volume. In our project, we displayed a transparent, red layer over the user's field of view when the user's hand leaves the ARPad's virtual plane . This solution might be not efficient enough to help users to have a clear representation of the 3D AR volume boundaries. To help users get a clear mental representation of the control volume, we could think of using e.g. haptic [START_REF] L O N G | br am a n i a n , S . Rendering volumetric haptic shapes in mid-air using ultrasound[END_REF] or audio feedbacks to better convey the boundaries of this volume.

RQ 2 Can we provide guidelines on how a wearable AR device can enhance a handheld device?

We address this research question by studying potential enhancements to phone-centric interactions. We report on a workshop with users and describe a design space to organize users' suggestions for enhancing phone interactions with AR. This design space features two dimensions that relate to the location and the function of the AR content with respect to the phone's content. After prototyping some of the examples to showcase promising cells in this design space, we focus on the AR enhancement of traditional smartphone-based widgets. We run a study to define how the user interface could be distributed between a handheld device and AR. This study shows that handheld devices could benefit from being AR-enhanced, be it for users' performance, or for freeing up the phone's screen estate.

In our project, we only focus on how a user can interact with an AR-enhanced interface for smartphones, without considering the prior step of configuring this AR-enhanced interface. Such customization could be either automatically generated based on users' past interactions, or entirely controlled by users themselves. Automatically generating an ARenhanced interface for smartphones could rely on dedicated algorithms like SUPPLE [START_REF] Supple | Automatically generating user interfaces[END_REF].

However, prior research suggested that there is a need for keeping the user in control of the adaptation process [START_REF] Nd E R S O N | fm a n , S . Automatically personalizing user interfaces[END_REF][START_REF] K N I E W E L | Achieving user participation for adaptive applications[END_REF]. An interesting approach could be hybrid with a system like Skillometers [START_REF] S C A R R | Skillometers: Reflective widgets that motivate and help users to improve performance[END_REF] that would provide means for users to reflect on their activity and identify what gain they could get from some customization. Studying how users could decide on whether they want to offload a widget or not is an interesting avenue for future work. Some researchers have already worked on such adaptable user interfaces [START_REF] S T U E Rz L I N G E R | User interface façades: Towards fully adaptable user interfaces[END_REF] on desktop, but this needs to be explored in the context of AR+smartphone combination. We would also like to continue working on AR combination with interaction modalities other than touch-only. In particular, we have started to work on the combination of AR with tangible controllers such as TouchTokens [START_REF] M O R A L E S G O N Z Á L E Z | Guiding touch patterns with passive tokens[END_REF] or WallTokens [START_REF] Walltokens | Surface tangibles for vertical displays[END_REF]. Tokens could benefit from this combination as they are physical objects whose shape cannot be changed on the fly [START_REF] P I E T R I G A | Custom-made tangible interfaces with touchtokens[END_REF][START_REF] Wa N G | Customizable passive tangible controllers on and around conventional mobile devices[END_REF]. Using a standard physical placeholder for tokens and changing the way they look in AR could make them easy to customize. More than just preventing users from having to manipulate too many tokens and interaction designers from creating too many token instances, this solution could preserve the advantages of physicality like haptic feedback while overcoming its drawbacks like resistance to change. We have already implemented an initial prototype for users to define the appearance of one token using AR (illustrated in Figure 3.6). Users can simply turn their hand holding the token to invoke a menu that proposes different visual appearances (and thus functions) that the token can take. The user can then customize the token by sliding their finger from the desired menu item to the physical token.

2) Prototypes

← Back of device mirror ← Task selector Ambilight → → Discrete controls (e. g. , buttons, checkboxes) should not be offloaded to the air.

→ Dual-precision air control (CD ) can effectively complement touch for parameter adjustments and 2D selections.

→ Discrete controls (e. g. , buttons, checkboxes) should not be offloaded to the air. 3) Experiment Le manuscrit est organisé en trois parties répondant respectivement à chacune de ces trois questions de recherche.

4) Results and Guidelines

Pour répondre à notre première question (QR 1 ), nous présentons le projet ARPads. Un ARPad n'a pas de réalité physique, mais représente un plan d'interaction flottant dans l'air dans lequel les mouvements de l'utilisateur sont traduits par des mouvements de curseur dans une fenêtre de RA affichée dans l'environnement. Ce contrôle indirect permet à l'utilisateur d'adopter une posture plus confortable pour manipuler du contenu affiché en RA par opposition au contrôle direct qui force l'utilisateur à maintenir ses bras en l'air en direction du contenu de RA. Après avoir défini les critères de conceptions de ces ARPads en prenant en compte leur position et leur orientation relatives à l'utilisateur, nous avons implémenté et évalué certaines conceptions prometteuses. Nos résultats montrent que le contrôle indirect peut atteindre des performances similaires à celles du contrôle direct, tout en limitant la fatigue perçue par l'utilisateur et identifient les conceptions d'ARPads les plus efficaces. Enfin, pour notre troisième question (QR 3 ), nous nous projetons dans un futur dans lequel le champ de vision des utilisateurs contient fréquemment du contenu virtuel, et proposons une solution concrète pour que celui-ci n'interfère pas avec leur perception du monde réel. Pour cela, nous introduisons le concept de de-augmentation de la RA. Il s'agit de permettre aux utilisateurs de retirer volontairement certains éléments holographiques de leur champ de vision. Nous commençons par définir une taxonomie des opérations de dé-augmentation qui s'articule autour de trois dimensions : l'objet de la dé-augmentation, son déclencheur, et son rendu graphique. Nous illustrons ensuite ce concept au travers de trois scénarios démontrant l'utilité de telles opérations dans quelques situations particulières. Finalement, nous proposons une implémentation pratique de ce concept avec un prototype opérationnel, dont nous détaillons les différentes interactions qui permettent à l'utilisateur de définir une zone dé-augmentée. Ce dernier projet, plus théorique, ouvre des questionnements et des perspectives sur l'orientation des futurs systèmes de RA. The manuscript is divided into three parts respectively addressing each of these three research questions.

To answer our first research question ((RQ 1 )), we present the ARPads project. An ARPad has no tangibility but represents a floating interaction plane on which users can move their hands to control a cursor displayed in an AR window. Such indirect input allows users to keep a comfortable posture while interacting with AR, as opposed to direct input that forces users to keep their arms in an upward position towards the content displayed in AR in front of them.

After exploring a design space of these ARPads regarding their position and orientation relative to the user, we implement and empirically evaluate some promising combinations. Our results show that indirect input can achieve the same performance as direct input while limiting users' fatigue. From these results, we derive guidelines for future implementations.

Regarding our second research question (RQ 2 ), we focus on the association of wearable AR and mobile devices. We adopt a user-centered approach with a workshop organized with users of mobile devices.

Based on the feedback derived from this workshop, we define a design space of smartphone-centric applications that could benefit from this association.

In particular, we identify two main dimensions: the function and the location of the AR content relative to the main content displayed on the mobile device's screen. This first contribution highlights the creative ways to enhance mobile devices with AR. However, not much can be asserted regarding such enhancements without actual measures about their performance. After prototyping some use cases to show their feasibility, we evaluate some UI component distributions between a phone's screen and AR.

Finally, we answer our last question (RQ 3 ) by anticipating a future where users' field of view is frequently augmented with AR content. We introduce the concept of AR de-augmentation as a means to prevent this AR content from interfering with users' perception of the real world. AR de-augmentation gives users agency over the AR content. We first define a taxonomy of such de-augmentation operations regarding three aspects: its scope, its trigger, and its rendering. Then, we illustrate the concept of AR de-augmentation with three scenarios to demonstrate its usefulness in some specific use cases. Finally, we implement a working prototype in which we detail some interactions that allow users to define a deaugmented area. This last project is more theoretical and projective, opening questions and perspectives on future research directions.

Fig. 0. 1 :

 1 Fig. 0.1: Virtuality Continuum of Milgram[START_REF]a n d K i s h i n o , F . A taxonomy of mixed reality visual displays[END_REF]

Fig. 0. 2 :

 2 Fig. 0.2: The three forms factor of AR devices: a) Stationary AR setup (e.g. HoloDesk [75]), b) Mobile AR setup (e.g. Pokemon Go [145] using a smartphone), c) Wearable AR setup (e.g. Commercial for Microsoft HoloLens [127]).

Fig. 0. 3 :

 3 Fig. 0.3: Evolution of Augmented Reality devices. a) 1968 -Head-Mounted Display of Sutherland [197]; b) 1997 -the Touring Machine of Feiner et al. [50]; c) 2017 -Microsoft HoloLens [127]; d) On going -AR contact lenses [150]

Fig. 1 . 1 :

 11 Fig. 1.1: Our design space is structured along two dimensions: where the ARPad is located relative to the user's body (B o dy P o s i t i o n), and how the ARPad is oriented (Pa d O r i e n tat i o n). Blue points indicate the six specific configurations that we study.

Figure 1 . 3 -

 13 Figure 1.3-left shows where the six ARPad designs that we investigate stand in our design space. These designs result from crossing values {Thigh, Waist, Wrist} for BodyP o s i t i o n with values {Horizontal, Vertical} for Pa d O r i e n tat i o n.

Fig. 1 . 2 :

 12 Fig. 1.2: Waist, Wrist and Thigh body positions for the Leap Motion

Fig. 1 . 3 :Fig. 1 . 4 :

 1314 Fig. 1.3: (Left) Indirect input conditions: the six ARPad specific configurations from our design space {Thigh, Waist, Wrist} × {Horizontal, Vertical}. (Right) Direct input conditions: Gaze and Hand raycast..

Fig. 1 . 5 :Task 1 :

 151 Fig. 1.5: The pointing task. (a) The target to acquire is colored white. (b) It turns green when the cursor enters it. Users have to keep the cursor inside it for 500ms to make it disappear.

Fig. 1 . 6 :

 16 Fig. 1.6: The pursuit task. (a) The cursor is green when it is close to the target. (b) As it is getting further from the target, it turns reddish.

Task 2 :

 2 Continuous, trajectory-based interactions

Figure 1 . 3

 13 gives an overview of the eight T e c hn i q u e conditions. Each indirect technique (i.e., ARPad) is a combination of the two secondary factors B o dyP os it i o n and Pa d O r ie ntat i o n, which correspond to the respective dimensions in our design space. For the two direct techniques, we consider the value of B o dyP os it i o n to be Head but the other secondary factor rather corresponds to the modality: Gaze or Hand.

Fig. 1 . 7 :Figure 1 . 7

 1717 Fig. 1.7: Experiment timeline

Fig. 1 . 10 :Fig. 1 . 11 :

 110111 Fig. 1.10: Linear regression analysis with respect to Fitts' law (Movement Time as a function of Index of Difficulty) for all techniques.

2 G

 2 Pa d O r ie ntat i o n, we filter out data for indirect techniques only (as we did for pointing tasks above). An a n ova for model AngularDistance ∼ B o dyP os it i o n × Pa d O r i e n tat i o n reveals a significant effect of Pa d O r i e n tat i o n (F 1,15 = 10.8, p = 0.005, η 2 G = 0.10) and a significant B o dy P o s i t i o n × Pa d O r i e n tat i o n interaction (F 2,30 = 6.63, p = 0.004, η = 0.09), but no effect of B o dyP o s it i o n (F 2,30 = 1.70, p = 0.200, η 2 G = 0.04). Pairwise t-tests reveal where the interaction comes from: Vertical ARPads perform better than Horizontal ones only for Waist (p = 0.033) and Wrist (p = 0.011) body positions. Whatever their orientation, ARPads located at the thigh perform particularly poorly for continuous tasks. But, overall, our study suggests that Vertical ARPads are more efficient than Horizontal ones (H 3 ).

Fig. 1 . 12 :

 112 Fig.1.12: Participants' grades for the following three NASA-TLX scales: perceived performance, physical demand, and mental demand. The plot relies on color coding and binning: each score is assigned a color, and the color patch's size is proportional to the ratio of times this score was given. Right (green) is better.

F 7 ,Fig. 1 . 13 :

 7113 Figure1.12 shows participants' qualitative feedback for each technique along the following scales: perceived performance, physical demand, and mental demand. a rt a n ovas reveal a significant effect of T e c hn i q u e for each of these scales: F 7,105 = 4.97, p < 0.001 for perceived performance, F 7,105 = 9.70, p < 0.001 for physical demand, and F 7,105 = 6.69, p < 0.001 for mental demand.

Figure 1 .

 1 [START_REF] M Ü L L E R | A new perspective on gestural interaction and wearable applications[END_REF] shows participants' answers. While preferences are unclear between the two Thigh techniques (p = 0.210), they preferred Vertical for both Waist (p < 0.001) and Wrist (p = 0.021).

Fig. 2 . 1 :

 21 Fig. 2.1: Offloading controls of an image editing application running on a smartphone. Left) UI Widgets take significant screen real-estate when displayed on the phone; and precise selections are difficult with touch input because of finger occlusion. Right) Default UI distribution: 1) output: widgets are displayed in the air, freeing screen real-estate on the phone to show more content; and 2) input: the pointer can be controlled indirectly with movements performed in the air, causing less occlusion of the image being edited.

Fig. 2 . 2 :

 22 Fig. 2.2: Evolution of the paperboard during the brainstorming session: a) the moderator introduces the first question Q 1 and gives a couple of answers, b) the moderator populates with answers from participants. c) the moderator populates with problems raised with Q 2 , d) the moderator populates with solutions proposed with Q 3 .

Figure 2 . 3 Fig. 2 . 3 :

 2323 Fig. 2.3: Example of a storyboard created during the workshop: a) description of the issue encountered with a mobile phone, and b) illustration of a solution using an AR Head-Mounted Display

Figure 2 . 5

 25 illustrates this concept application.

Fig. 2 . 5 :Fig. 2 . 6 :

 2526 Fig. 2.5: Rough sketch representing the Locate photos in space application. Photos are displayed where they were taken from.

Fig. 2 . 7 :

 27 Fig. 2.7: Rough sketch representing the Context sharing application. A holographic icon is displayed behind the smartphone to indicate what task the user in currently doing. On the left, the user is taking pictures using the camera. On the right, the same user is reading emails.

Fig. 2 . 8 :

 28 Fig. 2.8: Rough sketch representing the Contextual electric switchboard application. The user selects one room of a map displayed on their smartphone. A holographic link identifies which switch of the electrical panel relates to that room.

Figure 2 Fig. 2 . 9 :

 229 Fig. 2.9: Interaction steps with the Back of device mirror prototype: a) the user navigates the smartphone's gallery; b) the user selects a picture in the smartphone's gallery; c) the user shows the picture to their friend, keeping track of what their friend sees thanks to the duplicated view behind the smartphone; d) the duplicated view disappears, letting the user know that the smartphone went into sleeping mode.

Fig. 2 . 10 :Fig. 2 . 11 :

 210211 Fig. 2.10: Interaction steps with the Task selector prototype: a) all the opened apps form a stack; b) the user gazes at the stack; c) the user selects the calendar app; and d) the smartphone's view gets replaced with the calendar app.

Fig. 2 . 13 :

 213 Fig. 2.13: Example scenario of the AR-extended annotations application. Here, the user: a) selects an area to annotate on a text file; b) decides to annotate the selected area with a picture; c) takes a picture from the HoloLens' point of view by clicking on the corresponding AR button; d) returns to the tablet, observing the new annotation displayed on the right of the device's screen.

Figure 2 .Fig. 2 . 14 :

 2214 Fig. 2.14: The interpretation of users' hand movements in the air depends on the distance ∆ T I between the thumb's tip and the index finger's tip. a) Out of Range (∆ T I > 4cm): hand movements have no effect; b) Tracking (2cm < ∆ T I < 4cm): hand movements control which widget is in focus; c) Dragging (∆ T I < 2cm): hand movements actually control the widget in focus.

Fig. 2 . 16 :

 216 Fig. 2.16: Command Trigger experiment (E 1 ): completion time by D i ff ic u lt y × T e c hn i q u e.Error bars represent 95% confidence intervals.

  ) that vary in where output (O) and input (I) take place, and in how users control input precision (CD):

Fig. 2 . 17 :

 217 Fig. 2.17: The four techniques tested in Experiment E 2 : a) Screen -Screen (CD ), b) Screen -Air (CD ), c) Air -Air (CD ), d) Screen -Screen+Air (CD ), performing a coarse selection on the touchscreen (left), then a precise adjustment in the air (right). The two hand icons provide basic feedback about HoloLens left & right hand tracking status and never interfere with widgets displayed in the air.

Fig. 2 . 18 :

 218 Fig. 2.18: Parameter Adjustment experiment E 2 . a) Completion time by D i ff ic u lt y × T e c hn i q u e. Error bars represent 95% confidence intervals. b) Participants' preferences. Each bar represents the rank distribution for one technique.

2 .TaskFig. 2 . 19 :

 2219 Fig. 2.19: 2D Selection experiment E 3 . Completion time by D i ff ic u lt y × T e c hn i q u e.Error bars represent the 95% confidence interval.

Fig. 2 . 20 :

 220 Fig. 2.20: Redesign of the image editing application from Figure 2.1 following the guidelines derived from our empirical results. Buttons remain on the smartphone. Sliders for parameter adjustments performed frequently get offloaded. 2D selection can be performed directly on the phone, or indirectly in the air (dual-precision mode) if occlusion is an issue.

Fig. 3 . 1 :

 31 Fig. 3.1: Some de-augmentations from the Workplace scenario. a) Car dashboard and exterior sketches shown to all coworkers in AR, with Kim's desk and laptop closed in the background. Notifications are displayed in the air only to Kim. No de-augmentation is active. b) De-augmenting the first 3D-printed steering wheel, that Kim is holding in her left hand. The wheel can be seen in the context of the dashboard without the latter visually interfering with it. c) Similar de-augmentation applied to the other steering wheel. The de-augmentation's scope matches its shape. d) De-augmenting Kim's laptop screen by shifting some notifications upward.

Fig. 3 . 2 :

 32 Fig. 3.2: Some de-augmentations from the Home scenario. a) Alex has started cooking in the kitchen. The recipe and notifications appear in AR. No de-augmentation is active. b)The oven is always de-augmented, but the two pieces of AR content get de-augmented differently. The notifications get shifted downard, while the recipe gets cropped. c) Alex is heading upstairs. Meanwhile one of their kids is playing an AR game about catching butterflies that try to hide in the clouds. d) Alex enters the kid's room, and when the kid looks at them, their face gets automatically de-augmented, cropping AR clouds and butterflies.

Fig. 3 . 3 :

 33 Fig. 3.3: Some de-augmentations from the Outdoors scenario. a) Incoming call from Sam. b) a concert poster catches Jen's attention. c) Jen points at the poster to de-augment it while talking with Sam. d) The poster is now de-augmented. Sam's video feed is cropped.

Fig. 3 . 4 :

 34 Fig. 3.4: The three characteristics of a visual de-augmentation operation: Trigger, Scope and Rendering.

Fig. 3 . 5 :

 35 Fig. 3.5: De-augmentions performed with our prototype, performed on the HandInteractionExamples scene from MRTK. a) User invoking a hand menu with their hand flat in the Hololens' viewport. The user can choose how to delimit the scope of a de-augmentation to be manually created. b) Example of a world-anchored cuboid de-augmentation to reveal a 3D printer. c) Example of a world-anchored freeform flat de-augmentation to reveal part of a whiteboard.

Fig. 3 . 6 :

 36 Fig. 3.6: Initial prototype of AR-enhancement for tangible tokens. a) The user has already selected an orange star as the appearance of the physical, transparent WallToken [37]. b) Then, the user turns their hand to display the menu that proposes other possible appearances of the token. c) The user moves one menu item to the physical token by sliding their finger toward it. d) The appearance of the token is updated.

Fig. 3 . 8 :

 38 Fig. 3.8: Summary of chapter 2's contributions.

  Pour notre deuxième question (QR 2 ), nous nous concentrons sur la combinaison entre les appareils mobiles et la RA. Suite à un atelier de conception mené avec des utilisateurs d'appareils mobiles, nous avons défini un espace de conception d'applications centrées sur le smartphone pouvant bénéficier d'une association avec la RA. En particulier, nous avons identifié deux dimensions principales : la fonction et la position du contenu holographique par rapport au contenu de premier intérêt affiché sur l'appareil mobile. Ce premier travail montre une grande richesse en termes de créativité pour proposer des améliorations possibles pour les appareils mobiles. Cependant, le manque de données quantifiables sur leur performance effective nous empêche encore de poser un avis objectif sur les réels avantages d'une telle association. Après avoir prototypé quelques exemples d'application pour démontrer leur faisabilité, nous nous sommes concentrés sur l'évaluation d'un cas spécifique : les composants d'interface (widgets) traditionnels des smartphones.

RQ 1 RQ 2 RQ 3

 123 Title: Using Augmented Reality in Everyday Life Keywords: Human-Computer Interaction, Augmented Reality Summary: This manuscript describes the research topics I explored during my Ph.D. regarding the uses of Augmented Reality (AR) in everyday life. I tackle the three following research questions: How to reconsider interaction with wearable AR to better support long-lasting tasks? Can we provide guidelines on how a wearable AR device can enhance a handheld device? How can users be in control of what Augmented Reality tells them?

  • × 17.5 • screen) and a Leap Motion controller (running Orion 4.0.0+52238). Communication between the Leap Motion and the HoloLens was enabled by a Microsoft Surface Book 2 through WiFi / UDP.The experiment scene was rendered in an AR vertical window of 47.5 × 84.5 cm. It was positioned at a distance of 194cm from the headset when starting an experiment session (i.e., its visual size is 24.57 • × 13.96 • ). The window then remained fixed in space if users moved their head. The control display ratio for ARPads in this specific case is thus 1cm:4.62cm. A 1cm hand movement on an ARPad makes the cursor move1.38 • in visual angle, which is 4.62cm in the window (Figure1.4).

  packages. Post-hoc tests rely on functions t.test and wilcoxsign_test of the coin [78] package, and the p.adjust function. Charts rely on the ggplot2 library [212] for box plots and the likert package for Likert graphs.

	We first analyze data collected in pointing tasks, then in pursuit tasks. For both tasks,
	we analyze the effect of T e c hn i q u e to get a comparison between the different ARPads
	and with the direct baselines (i.e., Head condition). We then analyze the effect of
	Pa d O r ie ntat i o n and B o dyP os it i o n for indirect techniques only in order to
	understand how each of the two dimensions of the design space contributes to their
	performance. These results help to interpret differences between indirect techniques. Such
	analyses for indirect techniques only consider a subset of data, as Pa d O r i e ntat i o n
	is not crossed with B o dy P os it i o n in the Head condition. Our experimental design
	ensures a balanced number of measures across the different groups that we compare in
	all cases. However, when we analyze data for indirect techniques only, results might
	have been slightly impacted by the presentation order as the counterbalancing strategy
	applies to the whole set of technique conditions. We evaluate this impact as low, as
	the global Latin Square still ensures a balanced presentation order between pairs of
	indirect techniques. The only impact might come from the presentation of baseline direct
	techniques in-between two indirect conditions.

Overall, these results partially support H 2 , which states that direct techniques outperform indirect ones. HeadGaze

  HeadGaze has the best completion Time on average. It is significantly faster than the two Thigh techniques (p's < 0.001), than WaistHorizontal (p = 0.007), and than HeadHand (p < 0.001). We then look at the interaction with Ta rg e t W i d t h. For large and medium targets, we have the same results (with similar p's). However, for small targets, actually performs very well, but HeadHand does not perform well. However, the ARPad that has the best average completion time (Wrist × Vertical) is not significantly slower than HeadGaze. This suggests that indirect input can achieve performance close to that of direct input with gaze.

	Small	Thigh	Waist	Medium	Wrist	Head	Large
	4000
	2000 3000 WaH WaV WrH Pointing Time (ms) Fig. 1.9: Notched box plots for the pointing time per T e c hn i q u e × Ta rg e t W i d t h condi-TH TV WrV HG HH TH TV WaH WaV WrH WrV HG HH TH TV WaH WaV WrH WrV HG HH 0 1000 2000 3000 Pointing Time (ms) * * * * * * * * * * * * * * * * * * * 1000 tion.
	0 TH: ThighHorizontal, TV: ThighVertical, WaH: WaistHorizontal, WaV: WaistVertical, WrH: * * * * * * * WristHorizontal, WrV: WristVertical, HG: HeadGaze, HH: HeadHand
	Horizontal Vertical	Horizontal Vertical	Horizontal Vertical	Gaze	Hand
	Fig. 1.8: Notched box plot for the pointing time per T e c hn i q u e, grouped by B o dyP os it i o n,
	then by Pa d O r i e ntat i o n for indirect techniques or modality for direct techniques.
	Pointing time includes dwell time for selection.
	Indirect Techniques only (Time). Considering data for indirect techniques in isolation,
	we can analyze the effect of Pa d O r i e n tat i o n with an a n ova according to model:
	Time ∼ B o dyP os it i o n × Pa d O r i e n tat i o n × Ta rg e t W i d t h. We observe HeadGaze is only significantly faster than ThighHorizontal (p = 0.017) and HeadHand the same significant effect of Ta rg e t W i d t h (F 2,30 = 279, p < 0.001, η 2 G = 0.65), but (p = 0.033). This suggests that HeadGaze suffers from an increase in pointing difficulty, we also observe a significant effect of both B o dyP os it i o n (F 2,30 = 12.6, p < 0.001, which is in line with previous work that discusses the precision issues users face with η 2 G = 0.10) and Pa d O r ie ntat i o n (F 1,15 = 10.1, p = 0.003, η 2 G = 0.04). We also have gaze input [83, 84, 208]. one significant interaction effect: B o dyP os it i o n × Ta rg e t W i d t h (F 4,60 = 4.02,
	WristVertical ranks second regarding average completion Time. WristVertical is signifi-p = 0.003, η 2 G = 0.03).
	cantly faster than the two Thigh techniques (p's = 0.010). Looking closer at the interaction
	with Ta rg e t W i d t h, we can see that, for small targets, WristVertical becomes signifi-
	cantly faster than HeadHand (p = 0.021). Here again, we can see that users have relatively
	more trouble with direct techniques than with indirect techniques for tasks that require
	more precision. As HeadHand is implemented based on a raycast along the headset-hand
	vector, it might also have suffered from involuntary head movements.
	WaistVertical ranks third regarding average completion Time. But it is significantly
	faster than ThighHorizontal only (p = 0.013). Furthermore, WaistVertical's advantage is
	for large targets only (p = 0.008). Tests do not reveal any significant difference for medium
	and small.

Our study suggests that participants are significantly faster with Vertical ARPads than with Horizontal ones (H 3 ). Moreover

	, t-tests (corrected n = 3)
	reveal a strict order of performance between the three body positions: Wrist outperforms
	Waist (p = 0.04), which in turn outperforms Thigh (p = 0.04). Regarding the B o dyP o -
	s i t i o n × Ta rg e t W i d t h interaction effect, t-tests (correction n = 9) show that the
	superiority of Wrist over Waist is significant for small targets only. But, overall,

ARPads seem to perform best when they are positioned at users' wrist (H 4 ).

  can we do with a mobile phone? Q 2 What are the problems users can face in such cases?

Q 3 What AR-based solutions can we think of to address such problems? Figure

2

.2 illustrates how the paperboard evolved during the whole brainstorming session. After each question, participants had 15 minutes to work in groups and provide answers. The moderator populated the paperboard with answers given from participants. Answers to Q 1 were written down in black. They are the varying use cases participants could think of. Problems identified when answering Q 2 were marked down on pink post-it notes close to their related use case. Finally, solutions coming from Q 3 were reported on green post-it notes that were positioned close to the problem they address. The moderator explicitly said to the participants that they could either build on previous answers or propose novel elements if they wanted to.

Q 1 What can we do with a mobile phone?

Altogether, participants grouped their ideas into categories that we can summarize as follows: (1) Media consumption, (2) Localization, (3) Gaming, (4) Security, (5) Social Network, (6) Sport, (7) Shopping, (8) Information, (9) Network Management,

[START_REF] B E N Ko | Haptic retargeting: Dynamic repurposing of passive haptics for enhanced virtual reality experiences[END_REF]

Communication, (11) Time, (12) Tools, (13) Design,

[START_REF] B E N Ko | Combining an optically see-through near-eye display with projector-based spatial augmented reality[END_REF]

Wallet, (15) General.

Q 2 What are the problems users can face in such cases?

  (CD ) is similar to Screen -Air (CD ) for sliders (in Experiment E 2 ), but enables 2D selection. Users control the cursor with air-tap-and-hold interactions. Precision gets higher as the hand moves away from the start position of the air-tap-andhold gesture. The CD gain is set to 1:1 when the distance is low (<2cm), and progressively increases up to 4: precision mm at a distance of 10cm and above. In our study, users are able to point with sub-millimeter precision (precision mm = 0.5mm). Such as the ARPads presented in the first chapter of this thesis, this technique implements indirect 2D input with movements in the air. It is conceptually close to the 1-Button-Simultaneous technique from Jones et al.'s study

  the 11th Annual ACM Symposium on User Interface Software and Technology (New York, New York, USA, 1998), UIST '98, Association for Computing Machinery, pp. 171-172. of the 27th Annual ACM Symposium on User Interface Software and Technology (New York, New York, USA, 2014), UIST '14, Association for Computing Machinery, pp. 519-525. Usages de la Réalité Augmentée pour la Vie Quotidienne Mots clés: Interaction Homme-Machine, Réalité Augmentée Résumé: Ce manuscrit présente les recherches que j'ai effectuées dans le cadre de ma thèse de doctorat autour des usages de la Réalité Augmentée (RA) dans la vie de tous les jours. J'y présente trois questions de recherche : QR 1 Comment reconsidérer nos interactions avec la Réalité Augmentée portative pour que celles-ci soient appropriées pour des interactions prolongées ? QR 2 Peut-on énoncer des recommandations sur l'augmentation des appareils mobiles avec la Réalité Augmentée portative ? QR 3 Comment un utilisateur peut-il garder le contrôle sur le contenu qu'il perçoit en Réalité Augmentée ?

		Summary of chapter 3's contributions
	Dimension TRIGGER	User command	User rule				External source
	Dimension SCOPE	Entity	Spatial	Region	Temporary	Temporal	Permanent
		?
	RENDERING Dimension	Binary Mask	Alpha Blending Pixel Compositing	Affine	Advanced Geometric Transform
		?		?	?			?

Fig. 3.9: Summary of chapter 3's contributions.

of Titre:

Enfin, pour notre troisième question (QR 3 ), nous nous projetons dans un futur dans lequel le champ de vision des utilisateurs contient fréquemment du contenu virtuel, et proposons une solution concrète pour que celui-ci n'interfère pas avec leur perception du monde réel. Pour cela, nous introduisons le concept de de-augmentation de la RA. Il s'agit de permettre aux utilisateurs de retirer volontairement certains éléments holographiques de leur champ de vision. Nous commençons par définir une taxonomie des opérations de dé-augmentation qui s'articule autour de trois dimensions : l'objet de la dé-augmentation, son déclencheur, et son rendu graphique. Nous illustrons ensuite ce concept au travers de trois scénarios démontrant l'utilité de telles opérations dans quelques situations particulières. Finalement, nous proposons une implémentation pratique de ce concept avec un prototype opérationnel, dont nous détaillons les différentes interactions qui permettent à l'utilisateur de définir une zone dé-augmentée. Ce dernier projet, plus théorique, ouvre des questionnements et des perspectives sur l'orientation des futurs systèmes de RA.

The parallel is not to be considered literally as a trackpad enables relative cursor control (with clutching actions) while our implementation of an ARPads enables absolute control (as a graphics tablet).

Such "Out of range" events occurred rarely in our experiment. For Indirect Techniques, it happened: (i) in

3% of pointing trials; (ii) between 0.05% and 1.8% of total time per pursuit trial.

The concept of such widgets actually floating in the air around the phone has been popularized in science fiction shows such as, e.g., The Expanse. Querying the Web for "the expanse hand terminal" yields artists' renderings that accurately illustrate the idea.

When creating this design space, we noticed that some dimensions' values could not be crossed. In particular Duplicate ×Front cell. This would mean superimposing two identical layers, which makes no sense. This cell is colored in black in our design space.

Navigation interactions (scrolling, panning, zooming) represent another category, that usually does not involve widgets on a smartphone and has been investigated elsewhere (see, e.g.,[START_REF] J O N E S | Around device interaction for multiscale navigation[END_REF]).

An air-tap event followed by an air-release event less than

500ms later.[START_REF] A N D P I E T R I G A | High-precision magnification lenses[END_REF] The air-tap event should happen less than 500ms after the last air-click event.[START_REF] A N D F E K E T E | Orthozoom scroller: 1d multi-scale navigation[END_REF] Study data are available as supplemental material online at https://ilda.saclay.inria.fr/ ARwidgets4phones/

A quick sanity check shows that significant effects and differences are overall the same when running the anova and post-hoc tests on the non-corrected Time measure. The log correction tends to make differences more nuanced but, overall, the general observations remain the same.

We set the dimensions of this area according to the dimensions of the space bar on the phone used for the experiment (31.77mm×7.75mm), offsetting it 1cm upward to make room for downward movements.

Some smartphones implement a transfer function that makes the CD gain decrease as the finger accelerates to enable coarse adjustments from this mode.

https://www.vive.com

The system identifies the surface closest to the head-hand ray.

ACKNOWLEDGMENTS

Of course, my first acknowledgments go to my supervisor, Caroline Appert. Thank you Caroline for being an amazing supervisor, for your time, support, pedagogy, and most important, thank

AR HMD Microsoft HoloLens 2

SMARTPHONE

Samsung Galaxy A50

Face Mask FFP2 Hairnet Fig. 2.15: Illustration of a participant during the experiment.

Participants and Apparatus 12 participants (5 women, 7 men, 32 ± 11.1 year-old on average), all right-handed, took part in the study. 7 They were seated on a chair without armrest. We used a Samsung Galaxy A50 and a Microsoft HoloLens 2. The two devices communicate on a dedicated wireless network using TCP. We track the phone's location with Vuforia, 8 using three markers offset by 8cm from the phone's top edge, as illustrated in Figure 2.17.

Experiment E 1 Command Trigger

In this first experiment, participants are asked to push buttons, either on the phone or in the air around the phone. We describe techniques based on where the output (O) and input (I) take place, respectively. Technique Screen -Screen, our baseline, displays buttons on the phone and expects touch input. Technique Air -Air displays buttons in the air and expects direct input in the same place. 9

7 About COVID-19 and experiments: all experiments reported in this article were run during the summer of 2020. Throughout all studies, we recruited participants from a very restricted pool of 12 volunteers: six relatives of the first author who live in the same house, and three volunteer co-workers. The sanitary protocol involves hand disinfection, device cleaning, wearing hair protection under the ARHMD and a FFP2 safety mask. Figure 2.15 illustrates the setup for the experiment.

8 https://developer.vuforia.com 9 A button is a graphical component that implements the HandInteractionTouch script from the Microsoft Mixed Reality Toolkit (MRTK). To trigger an event, the index finger's tip must enter and then leave the component.

RQ 3 How can users be in control of what Augmented Reality tells them?

To provide a first answer to this question, we present the concept of de-augmenting Augmented Reality. We describe a taxonomy of de-augmentations regarding their triggering, scope, and rendering. We illustrate our taxonomy through three scenarios in order to demonstrate how this concept would help integrate AR in different daily contexts. We also contribute a prototype that shows the technical feasibility of implementing deaugmentation operations that users have control over. This project is a preliminary investigation of letting users control what falls in their field of vision when using AR. But AR can be more than visual only. Multimodal interfaces have been studied for a long time [START_REF]A design space for multimodal systems: Concurrent processing and data fusion[END_REF][START_REF] S E Rr A N O | The openinterface framework: A tool for multimodal interaction[END_REF][START_REF] L A L A N N E | Fusion engines for multimodal input: A survey[END_REF]. They can take advantages of users' multiple sensory channels to enhance user experience. Researchers have already explored a wide range of augmented modalities [START_REF]A design space for multimodal systems: Concurrent processing and data fusion[END_REF][START_REF] S E Rr A N O | The openinterface framework: A tool for multimodal interaction[END_REF][START_REF] L A L A N N E | Fusion engines for multimodal input: A survey[END_REF] that include haptic [START_REF] P E Z E N T | Explorations of wrist haptic feedback for ar/vr interactions with tasbi[END_REF], smell [START_REF] C O Bd E N | A toolkit to enable smell-based experience design[END_REF], taste [START_REF] A N D O B R I S T | Gustatory interface: The challenges of 'how' to stimulate the sense of taste[END_REF], sound [START_REF] D U E N S E R | d B i l l i n g h u r s t , M . Spatial sound localization in an augmented reality environment[END_REF] or temperature [START_REF] R A N A S I N G H E | Enhancing sense of presence in virtual reality by simulating real-world environmental conditions[END_REF]. Envisioning a future where AR is multisensory, we should enlarge the scope of applications for our de-augmentation concept. As a future project, we could think of letting users de-augment one or several AR feedback channels. More than just helping the user be in control of their various perceptual channels, this could make AR systems more accessible for people with disabilities. As an example, people with autism spectrum disorders can be especially sensitive to loudness [START_REF] B Ru N E Au | Increased perception of loudness in autism[END_REF], and people with headaches can encounter osmophobia [START_REF] S I Lva-N É T O | Osmophobia and odour-triggered headaches -review of the literature and main research centres[END_REF]. Such channels' de-augmentations could be tailored to suit these users' specific needs.