Unlike desktop computing, Augmented reality (AR) blurs the boundaries between the physical and digital world by superimposing virtual information to the real environment. It is possible to create an immersive AR experience either by wearing a head-mounted display or using a projector. This thesis explores interaction challenges associated with these two types of augmented reality displays.

Head-mounted AR displays (AR-HMDs) are constantly improving in terms of display (i.e., the field of view), tracking, interaction techniques, and portability. Currently available input techniques (such as hand-tacking, head/eye-gaze, and voice) in AR glasses are relatively easy to use for some tasks (e.g., grasping and moving an object). However, they lack precision and are not suitable for prolonged usage. Therefore, tasks that require accuracy become difficult to perform. In this research, we consider one such task -text selection that needs character level precision.

On the other hand, projection-based AR, usually called as spatial augmented reality (SAR), augments physical objects directly with digital content using projectors. In SAR, digital augmentation is generally pre-defined, and the user often acts as a passive viewer. A way to enhance the interactivity in SAR environment is to make graphical widgets (pop-up windows, menus, labels, interactive tools, etc.) more accessible. Unfortunately, embedding those widgets within the SAR scene is challenging.

In this dissertation, we explored new interaction techniques to address those challenges mentioned above. Our two main contributions are -First, we investigated the use of a smartphone as an interactive controller for selecting text in AR displays. We developed four eyes-free, one-handed text selection techniques for AR-HMDs using a smartphone: continuous touch (where smartphone touchscreen acted as a trackpad pad), discrete touch (where smartphone touchscreen was used to move the cursor character by character, word by word, and line by line), spatial movement (smartphone was used as an air-mouse), and raycasting (smartphone was used as a laser pointer). We also compared them in a user study.

Second, we extended the physical space of SAR by providing D graphical widgets in mid-air using projection on a drone-mounted panel. Users were able to control the drone position and interact with the projected information dynamically with a handheld controller. We present three possible ways to embed widgets using a drone in SAR: displaying annotations in mid-air, providing interactive tools, supporting different viewpoints. We also describe the implementation details of our approach.

These explorations aim at extending the interaction space in immersive AR applications.

Exploration de la sélection de texte à l'aide d'un smartphone dans les casques de RA

En général, la sélection de texte dans les casques RA peut être effectuée à l'aide de diverses modalités d'entrée, notamment le suivi de la main, le regard et la visée de la tête, les commandes vocales [] et les manettes de contrôle. Cependant, ces techniques ont leurs limites. Par exemple, le suivi de la main ne permet pas d'atteindre la précision d'un caractère [], manque de retour haptique [], et provoque la fatigue du bras []

Résumé étendu en Français

Les environnements informatiques de bureau reposent sur un affichage des données au travers d'un écran monoscopique D. L'idée de sortir de ce paradigme d'affichage et d'interaction WIMP standard (fenêtres, icônes, menus et pointeur), et de mélanger les espaces physiques qui nous entourent avec des informations numériques, a vu le jour en lorsqu'Ivan Sutherland a décrit sa vision de l'affichage ultime : "L'affichage ultime serait, bien sûr, une pièce dans laquelle l'ordinateur peut contrôler l'existence de la matière. [...] Avec une programmation appropriée, un tel affichage pourrait littéralement être le pays des merveilles dans lequel Alice est entrée". Trois ans plus tard, il présentait le tout premier casque de réalité augmentée (AR-HMD), qui constituait un premier pas vers cet écran ultime []. Depuis le "Sword of Damocles" de Sutherland, nous avons beaucoup progressé en termes de techniques d'affichage, de suivi et d'interaction au cours des cinq dernières décennies. Récemment, les casques optiques où l'augmentation se fait par transparence (comme HoloLens , Magic Leap , Nreal Light) sont sortis des laboratoires de recherche pour devenir des objets grand public.

Dans le passé, les dispositifs de RA étaient surtout utilisés pour effectuer des tâches très spécifiques (qui impliquaient principalement la sélection et la manipulation d'objets D, la visualisation de données immersives, etc.) dans des domaines d'application spécifiques tels que la réparation et la maintenance, la formation, les jeux, les opérations d'entrepôt, les soins de santé, etc. Cependant, on peut s'attendre à ce que nous utilisions ce type d'affichage pour l'informatique à usage général à l'avenir. En gardant cela à l'esprit, les chercheurs ont récemment commencé à explorer l'utilisation de casques de RA pour le travail impliquant plusieurs fenêtres de travail. En particulier, pour ce type de travail, les fenêtres virtuelles en D des casques contiennent souvent des informations textuelles (par exemple, un document PDF, une feuille Excel ou une navigation sur le Web), et les utilisateurs doivent fréquemment effectuer des opérations de saisie et d'édition de texte []. Lorsqu'ils sont à leur bureau, ils peuvent profiter d'un clavier standard et d'une souris/trackpad pour effectuer ces tâches efficacement. Mais lorsqu'ils sont en déplacement, la saisie et l'édition de texte ne sont pas aisées car il n'existe pas de techniques de saisie équivalentes au clavier et à la souris/trackpad. Récemment, la saisie de texte pour les casques de RA a attiré l'attention de la communauté des chercheurs [-]. En revanche, les recherches liées à l'édition du texte qu'un utilisateur a déjà tapé restent rares []. L'édition de texte est une tâche complexe, et la première étape consiste à sélectionner le texte pour l'éditer. Par conséquent, dans la première partie de cette thèse, nous nous concentrerons sur cette partie de sélection du texte.

Au-delà des casques optiques de RA, il est également possible d'augmenter notre espace physique directement avec du contenu numérique en utilisant des projecteurs.

Cette technique est connue sous le nom de réalité augmentée spatiale (RAS). Dans la RAS, nous pouvons utiliser un seul projecteur statique ou orientable, ou plusieurs projecteurs pour augmenter la surface d'affichage potentielle et améliorer la qualité de l'image. Contrairement aux casques de RA, les affichages au travers de projecteurs sont toujours détachés des utilisateurs et intégrés dans les environnements en RAS. Cela permet à plusieurs utilisateurs de visualiser la même augmentation avec des indices de profondeur naturels (à l'exception de la projection stéréoscopique dépendante de la vue, qui est généralement réglée pour un utilisateur unique). Ils peuvent toucher et interagir directement avec les surfaces physiques, ce qui fournit un retour haptique passif et améliore considérablement leur compréhension des informations présentées. En outre, ils peuvent se déplacer librement autour de la maquette physique et découvrir les augmentations sous différents angles de vue et positions. Les utilisateurs peuvent lire les expressions faciales des autres pendant les tâches collaboratives car leurs visages ne sont plus couverts par des casques. De plus, la collaboration n'est pas limitée par le nombre de casques disponibles et interconnectés, mais plutôt par le nombre de personnes qui peuvent être accueillies dans l'espace disponible. Malheureusement, l'une des principales limites des environnements SAR est que le contenu visuel ne peut être affiché que sur des supports physiques. Par conséquent, l'intégration de widgets d'interface utilisateur tels que les menus et les fenêtres contextuelles dans la RAS devient un défi. Ces widgets doivent être positionnés sur les objets physiques augmentés, ce qui entraîne un encombrement visuel qui affecte l'expérience globale de l'utilisateur. La géométrie et le matériau de la scène physique rendent même parfois impossible l'affichage de widgets lisibles []. Par conséquent, dans la deuxième partie de cette thèse, nous examinerons comment intégrer des éléments d'interface utilisateur graphique dans la scène RAS.

Contributions de la thèse

pendant une interaction prolongée. Le regard et la visée de la tête souffrent du problème du 'Midas Touch', qui provoque l'activation involontaire de commandes en l'absence d'un mécanisme de sélection approprié [-]. De plus, les mouvements fréquents de la tête dans l'interaction tête-regard augmentent le mal des transports []. L'interaction vocale peut ne pas être socialement acceptable dans les lieux publics [], et elle peut perturber le flux de communication lorsque plusieurs utilisateurs collaborent. Dans le cas d'un contrôleur portable dédié, les utilisateurs doivent toujours transporter du matériel spécifique supplémentaire.

Récemment, des chercheurs ont exploré l'utilisation d'un smartphone comme entrée pour les casques de RA en raison de sa disponibilité (il peut même être l'unité de traitement des casques []), de sa familiarité, de son acceptabilité sociale et de son aspect tangible [-]. Il ne fait aucun doute qu'il existe un énorme potentiel pour la conception de nouvelles applications inter-appareils avec la combinaison d'un écran RA et d'un smartphone. Dans le passé, les smartphones ont été utilisés pour interagir avec différentes applications fonctionnant sur des casques, comme la manipulation d'objets D [], la gestion de fenêtres [], la sélection de menus graphiques [] et ainsi de suite. Cependant, nous n'avons pas connaissance d'une recherche ayant étudié la sélection de texte dans un affichage RA à l'aide d'un smartphone disponible dans le commerce. Par conséquent, nous proposons quatre techniques de sélection de texte sans contact visuel pour les casques de RA utilisant un smartphone comme contrôleur d'entrée -le toucher continu, le toucher discret, le mouvement spatial et le rayon virtuel.

Pour sélectionner du texte avec succès en utilisant l'une des techniques proposées, l'utilisateur doit suivre la même séquence d'étapes à chaque fois. Tout d'abord, il déplace le curseur, situé dans la fenêtre de texte d'un écran de RA, jusqu'au début du texte à sélectionner (c'est-à-dire le premier caractère). Ensuite, il effectue un double tapotement sur le téléphone pour confirmer la sélection de ce premier caractère. Il peut voir sur l'écran du casque que le premier caractère a été mis en évidence en jaune. En même temps, elle passe en mode de sélection de texte. Ensuite, il continue à déplacer le curseur jusqu'à la position finale du texte en utilisant l'une des techniques présentées ci-dessous. Pendant que le curseur se déplace, le texte est également mis en évidence simultanément jusqu'à la position actuelle du curseur. Enfin, il termine la sélection du texte par un second double-tap.

Continuous Touch:

Cette technique utilise le smartphone comme un trackpad (voir Figure . .(a)). Il s'agit d'une technique de pointage indirect où l'utilisateur déplace son pouce sur l'écran tactile pour changer la position du curseur sur l'affichage RA. Pour la mise en correspondance entre l'affichage et l'écran tactile, nous avons utilisé un mode relatif avec embrayage. Comme l'embrayage peut dégrader les performances, un gain de contrôle de l'affichage (CD) a été appliqué pour le minimiser.

Discrete Touch: Cette technique s'inspire de la sélection de texte avec les raccourcis clavier disponibles dans les systèmes Mac [] et Windows []. OS. Dans ce travail, nous avons essayé d'émuler quelques raccourcis clavier. Nous avons particulièrement envisagé d'imiter les raccourcis clavier liés à la sélection de texte au niveau des caractères, des mots et des lignes. Par exemple, sous Mac OS, le fait de maintenir la touche enfoncée et d'appuyer sur la touche ou étend la sélection de texte d'un caractère vers la droite ou la gauche. Tandis que maintenir la touche + enfoncée et appuyer sur ou permet aux utilisateurs de sélectionner le texte d'un mot vers la droite ou la gauche. Pour sélectionner le texte jusqu'au caractère le plus proche situé à la même position horizontale sur la ligne supérieure ou inférieure, l'utilisateur doit maintenir la touche enfoncée et appuyer sur la touche ou respectivement. Dans le cadre d'une interaction tactile discrète, nous avons reproduit tous ces raccourcis à l'aide de gestes de balayage directionnel (voir Figure. .(b)). Le glissement vers la gauche ou la droite permet de sélectionner le texte aux deux niveaux -mot et caractère. Par défaut, il fonctionne au niveau du mot. Les utilisateurs effectuent un tap long qui agit comme un bouton de basculement pour passer de la sélection au niveau du mot à celle du caractère. D'autre part, un glissement vers le haut ou vers le bas permet de sélectionner le texte à une ligne au-dessus ou au-dessous de la position actuelle. L'utilisateur ne peut sélectionner respectivement qu'un seul caractère/mot/ligne à la fois avec son geste de balayage.

Mouvement spatial:

Cette technique émule le smartphone en tant que souris tenu en l'air [, ] pour les casques de RA. Pour contrôler la position du curseur sur l'écran du casque, l'utilisateur tient le téléphone devant son torse, place son pouce sur l'écran tactile, puis déplace le téléphone dans l'air avec de petits mouvements de l'avant-bras dans un plan perpendiculaire à la direction du regard (voir Figure. .(c)). Pendant le déplacement du téléphone, les données de position suivies en coordonnées XY sont traduites en mouvement du curseur en coordonnées XY dans une fenêtre D. Lorsqu'un utilisateur souhaite arrêter le mouvement du curseur, il lui suffit de lever son pouce de l'écran tactile. Les événements de toucher et relâcher avec le pouce définissent le début et l'arrêt du mouvement du curseur sur l'écran RA. L'utilisateur détermine la vitesse du curseur en déplaçant simplement le téléphone plus rapidement ou plus lentement en conséquence. Nous avons appliqué un gain de CD entre le mouvement du téléphone et le déplacement du curseur sur la fenêtre de texte.

Raycasting: Le raycasting est une technique d'interaction populaire dans les environnements RA/RV pour sélectionner des objets virtuels D [, ]. Dans ce travail, nous avons développé une technique de raycasting basée sur un smartphone pour sélectionner du texte affiché sur une fenêtre D dans le casque (voir Figure. .(d)). Un smartphone suivi DoF a été utilisé pour définir l'origine et l'orientation du rayon. Dans l'affichage du casque, l'utilisateur peut voir le rayon en ligne droite qui apparaît depuis le haut du téléphone. Par défaut, le rayon est toujours visible pour les utilisateurs du caque de RA tant que le téléphone est correctement suivi. Pour orienter le rayon, l'utilisateur doit effectuer de petits mouvements angulaires du poignet pour pointer le contenu textuel. Lorsque le rayon touche la fenêtre de texte, l'utilisateur y voit le curseur. Par rapport aux autres méthodes proposées, le raycasting ne nécessite pas d'embrayage car il permet de pointer directement la cible. L'utilisateur confirme la sélection de la cible sur l'affichage RA en fournissant une entrée tactile (c'est-à-dire un double-tap) à partir du téléphone.

Nous avons évalué ces quatre techniques dans le cadre d'une étude auprès de participants où les utilisateurs devaient sélectionner du texte à différents niveaux de granularité. Nos résultats suggèrent que le toucher continu, dans lequel nous avons utilisé le smartphone comme un trackpad, a surpassé les trois autres techniques en termes de temps d'exécution de la tâche, de précision et de préférence des utilisateurs.

Extension des espaces physiques en Réalité Augmentée Spatiale à l'aide d'un drone

Nous avons proposé DroneSAR pour fournir des widgets graphiques interactifs en RAS, comme un menu flottant dans l'air, en utilisant la projection sur un panneau monté sur un drone, tandis que les utilisateurs émettent des entrées via un contrôleur manuel pour interagir avec la scène et positionner le drone de manière dynamique. Cette approche présente plusieurs avantages. Premièrement, par rapport à la projection directe sur un objet, la qualité de la projection ne dépend pas de la géométrie et du matériau de la scène physique, ce qui garantit une bonne visualisation des widgets. Deuxièmement, les utilisateurs peuvent se concentrer sur la zone d'intérêt sans diviser leur attention avec une deuxième zone d'interaction (c'est-à-dire un panneau tenu dans la main, une tablette, etc.) et ils peuvent se déplacer librement dans l'espace. Troisièmement, ils peuvent positionner les widgets à des emplacements D spécifiques, qui peuvent être éloignés. La technique proposée leur permet de voir les widgets dans leur contexte spatial D. Les utilisateurs auront l'impression que le contenu projeté sur le drone est toujours sémantiquement lié aux surfaces physiques augmentées. Enfin, plusieurs utilisateurs sont capables de percevoir la même information en même temps, ce qui favorise le travail collaboratif. Nous avons présenté trois façons possibles d'intégrer des widgets à l'aide d'un drone -afficher des annotations dans l'air, fournir des outils interactifs, soutenir différents points de vue. De nombreuses autres fonctionnalités pourraient être imaginées, où DroneSAR permet de faire évoluer des applications de bureau standards vers le domaine des environnements en réalité augmentée spatiale.

Affichage des annotations en l'air: L'ajout d'annotations dans la RAS enrichira l'expérience de l'utilisateur, mais le placement des étiquettes associées au monde physique augmenté n'est pas trivial en raison de sa surface de projection non plane et texturée. Pour résoudre ce problème, DroneSAR permet de projeter des annotations virtuelles sur le drone, indépendamment de la surface de projection. Tout en affichant l'étiquette en l'air, les utilisateurs peuvent positionner le drone à côté de l'objet physique à l'aide d'un contrôleur portable pour créer un lien entre l'annotation et la région d'intérêt (ROI) dans l'espace physique. Ils ont également la possibilité de positionner le drone de manière automatique définie par l'application. En outre, notre système permet aux utilisateurs d'interagir avec ces étiquettes projetées à l'aide des boutons de saisie du contrôleur. S'il s'agit d'un texte ou d'une image, ils peuvent utiliser le trackpad du contrôleur pour modifier son orientation. Dans le cas d'une vidéo, ils peuvent la lire ou la mettre en pause avec le bouton de déclenchement. Pour afficher les étiquettes, nous avons implémenté un widget dédié. Comme décrit dans la Figure . Fournir des outils interactifs: Dans le domaine de la réalité augmentée spatiale, les utilisateurs agissent souvent comme des spectateurs passifs. Il serait intéressant de leur fournir des outils interactifs pour jouer avec l'augmentation virtuelle sur les objets physiques de façon dynamique. Inspirés par les "dynamic shader lamps" [], nous avons augmenté le panneau du drone avec plusieurs outils virtuels. Les utilisateurs peuvent sélectionner un outil en le pointant à l'aide d'un contrôleur. Une fois sélectionné, le contrôleur devient le mandataire de cet outil et lui permet d'effectuer une opération spécifique sur le contenu augmenté. Par exemple, un utilisateur peut sélectionner un outil de mesure dans le menu principal du panneau du drone, illustré sur la Figure .(A). Comme l'illustre la Figure .(B), les participants tracent une ligne sur la maison augmentée à l'aide du bouton de déclenchement du contrôleur, et la longueur mesurée s'affiche sur le panneau du drone. On peut facilement l'étendre à une application de peinture où le panneau du drone sera enrichi de différents outils (palette de couleurs, coup de pinceau, etc.).

En outre, au lieu de fournir une palette d'outils virtuels, le drone lui-même peut agir comme un proxy pour un outil particulier. En déplaçant le drone à l'aide d'un contrôleur, les utilisateurs accomplissent la fonction de cet outil. Pour illustrer cela, nous fournissons un outil lié à la source lumineuse de la scène. Dans ce cas, le drone agit comme un proxy de la source de lumière virtuelle. Les utilisateurs peuvent modifier de manière interactive la position de la lumière, ce qui serait difficile à réaliser sans le retour d'information sur la position en l'air que fournit le drone. L'apparence de la maison est modifiée en conséquence lorsqu'ils déplacent la lumière de droite à gauche (voir la Figure .(A &B)). On obtient ainsi une visualisation tangible d'un objet non physique qui s'inspire du projet Urp[].

Support de différents points de vue: Une autre caractéristique intéressante de DroneSAR est d'afficher une vue D interactive de l'objet augmenté observé à proximité de la zone d'intérêt. En effet, même si les environnements RAS présentent divers avantages intéressants, leur caractère physique implique également de fortes limitations par rapport aux environnements purement virtuels. Il n'est pas possible de voir les objets physiques augmentés depuis une vue de dessus ou de derrière, et l'échelle des objets reste toujours fixe. Inspirés par le concept de One Reality [] qui combine le RAS et la RV pour ajouter de la flexibilité aux mondes physiques, nous proposons une approche où le DroneSAR est utilisé comme une visionneuse interactive D contextuelle. Les participants peuvent voir l'objet physique augmenté sous différents angles et à différentes échelles en utilisant le trackpad et le bouton de déclenchement du contrôleur tout en restant ancrés dans l'environnement physique. Ainsi, ils peuvent facilement faire le lien entre l'objet réel-augmenté et son homologue virtuel (voir la Figure .(C)).

En résumé, certaines tâches de base (telles que la saisie de texte, l'édition de texte, la sélection de menus, l'interaction avec des widgets graphiques, etc.) sont simples à réaliser sur notre environnement informatique de bureau mais difficiles à accomplir dans un environnement immersif de RA. Pour surmonter ces difficultés, nous avons contribué à améliorer l'espace d'interaction des deux formes les plus courantes de réalité augmentée immersive. Certes, les techniques d'interaction proposées ne sont pas des solutions génériques. Elles dépendent du contexte de l'application. Néanmoins, nous n'avons fait qu'effleurer la surface de ce qui est possible de faire avec un smartphone et un drone dans le contexte de la réalité augmentée immersive. .). Moreover, while users are leaving their desk, they can take their office with them, and interfaces adapt dynamically (e.g., register windows to the head/body/world, avoid occlusion by changing windows layout/transparency, switch the primary interaction technique, etc.) according to their walking and physical environment (see Figure . Particularly, for productivity work, these D virtual windows in AR-HMDs often contain textual information (e.g., pdf/word document, excel sheet, or web browsing), and users need to perform text input and editing operations frequently Normally, text editing is a complex task, and the first step is to select the text to edit it. Therefore, in this thesis, we will focus on this text selection part.

List of Figures

Beyond AR-HMDs, it is also possible to augment our physical space directly with digital content using projectors. This technique is known as spatial augmented reality (SAR). In SAR, we can use a single static or steerable and multiple projectors to increase the potential display area and enhance the image quality. For monoscopic projections, there is no need to track the user's viewpoint to render suitable graphics.

Tracking might be required for view-dependent renderings in some applications, such as accurate lighting effects. To experience view-dependent stereoscopic projections, users need to wear light-weight D shutter glasses with attached markers for DOF tracking. However, it is still possible to achieve multiple perspective views without wearing any glasses in a dyadic SAR scenario [], but it requires each user to stay opposite to one another for maintaining correct display registrations.

Unlike HMDs, displays are always detached from the users and integrated into the environments in SAR. This enables multiple users to view the same augmentation with natural depth cues (except view-dependent stereoscopic projection, usually a single-user setup). They can touch and interact directly with physical surfaces, which provides passive haptic feedback and greatly enhances their understanding of the presented information. In addition, they can freely move around the physical mock-up and experience the augmentations from different angles and positions. Users can read each other's facial expressions during collaborative tasks as their faces are not covered with HMDs anymore. Further, collaboration is not limited by the amount of equipment available, rather by the number of people who can be accommodated in the available space. augments an environment around a conventional display with projected visualizations to expand the visual display area as well as to create a magical gaming experience (see Figure .). RoomAlive [] takes this concept to the next level to transform an entire room into an immersive augmented display and allow multiple users to interact with the projected content directly in a playful way. Furthermore, recent work focuses on augmenting deformable surfaces [] and movable objects [] instead of limiting SAR only to stationary surfaces. Note that, in this thesis, we consider a most commonly used collaborative SAR set up like Shader Lamps [].

Compared to head-mounted AR display, SAR provides an unlimited field of view (theoretically), a scalable resolution, and an easier eye accommodation (because virtual objects are typically rendered on the physical surfaces). Despite all these advantages, bringing interactivity in the SAR environment is not straightforward. Users often need graphical widgets to interact with the SAR scene. Unfortunately, supporting UI control elements in spatial augmented reality is challenging due to rigid mapping between the physical surface and virtual content. Therefore, this dissertation will look into how to embed widgets elements in the SAR scene.

.

Research Questions

Overall, this thesis aims to enrich the interaction space of augmented reality by addressing two main research questions, which we have described below.

In AR-HMDs, we can see a text document (e.g., a word/pdf file, web pages) in full size, but we can not select text efficiently. Performing text selection, which requires character level accuracy, in on-the-go situations is cumbersome because currently available input techniques of an AR-HMD (hand tracking, head/eye-gaze, and voice) are not as efficient as a standard keyboard and mouse/trackpad. For example, hand tracking often lacks precision [] and is not suitable for long-term use []. Both eye-gaze and head-gaze have 'Midas Touch' problem (i.e., unintended target selection) [].

Further, voice interaction is not socially acceptable in public places []. Therefore, in this work, we asked ourselvesis there a better way (i.e., interaction technique) which will allow users to select text fast and accurately in an AR-HMD?.

On the other hand, in spatial augmented reality, digital augmentation is always pre-defined, and the user often acts as a passive viewer. A way to enhance the interactivity in SAR environment is to make graphical widgets (pop-up windows, menus, labels, interactive tools, etc.) more accessible. Previous works attempted to provide widget elements in SAR on the surface of a For instance, when UI elements are located on the table, users always have to come close to it to access widgets while viewing the scene from a distance. In other cases, their hands are occupied to carry a panel/tablet, and they have to divide their attention between augmented scenes and tools at their hands. By displaying widgets in mid-air like floating menus in AR-HMDs, we might overcome these issues. Then, physical augmentation and widgets will be in the user's view.

Unfortunately, displaying virtual content in the air is not feasible in SAR as it always requires a physical surface for projection. Therefore, in this research, we asked ourselveshow to provide contextual graphical widgets in mid-air in SAR?

.

Research Contributions

The research contributions of this thesis are the following:

First, we developed four text selection techniques for AR-HMDs using a smartphone as an input device because of its availability, familiarity, social acceptability, and tangibility. All these techniques are eyes-free and one-handed. Our proposed four techniques are -continuous touch (where smartphone touchscreen acts as a trackpad pad), discrete touch (where smartphone touchscreen is used to move the cursor character by character, word by word, and line by line), spatial movement (smartphone is used as an air-mouse), and raycasting (smartphone is used as a laser pointer). Next, we compared these techniques in a user study where users have to select text at various granularity levels. Our results suggest that continuous touch outperformed the other three techniques in terms of task completion time, accuracy, and user preference.

Second, we proposed DroneSAR to provide interactive graphical widgets in SAR like a floating menu in mid-air using projection on a drone-mounted panel, while users issue eyesfree input via a hand-held controller to interact with the scene and positioning the drone dynamically. Drones can be positioned quickly with an acceptable accuracy around the augmented scene. As a result, users can freely move in the space, and they can access widgets when needed from anywhere in the scene. We present three possible ways to embed widgets using a drone in SAR -displaying annotations in mid-air, providing interactive tools, supporting different viewpoints. We also describe the implementation details of our approach.

.

Organization of the Thesis

This dissertation is organized in the following way. Chapter briefly covers the overall interaction space of AR-HMDs and SAR. Then, in Chapter , we describe our proposed smartphone-based text selection techniques in AR-HMDs. Next, Chapter presents our proposed DroneSAR system to support contextual graphical widgets in mid-air in a SAR scene. Finally, Chapter summarizes this thesis and provides future directions for further exploration.

. Publications

All main contributions of this thesis have been published at international peer-reviewed conferences. Here are our publication details -

Related Work

In this chapter, we will first briefly review the overall interaction space of head-mounted augmented reality displays (AR-HMDs). Then, we will take a quick look into spatial augmented reality (SAR) interaction space.

. Interaction Space in AR-HMDs

The overall interaction space of an AR-HMD consists of input techniques (hand-tracking, head-gaze, eye-gaze, body, pen/stylus, mobile devices, and multimodal) and extending the output (i.e., display). The following subsections briefly describe this space.

Hand Tracking Based Interaction

In current AR-HMDs, hands are the primary input modality as it allows natural user interaction (i.e., intuitive and easy to learn), thanks to RGB and depth cameras installed on the device. Using hands, users can manipulate D virtual objects as well as interact with D user interface elements (windows, virtual keyboard, menus, etc.) Researchers also explored the potential of hand tracking for tap and swype based text input [] as well as ten finger mid-air typing [][] in mixed reality devices. While hand tracking, available in modern HMDs, allows for easy access to mid-air gestures, the accuracy of those spatial tracking solutions is still significantly lower than dedicated lab-based external tracking systems []. Moreover, it is not precise, suffers from "gorilla-arm" effects for extended periods of use, ). Then users directly manipulate digital content affixed to that surface, which act as a virtual touchscreen.

Head-Gaze Based Interaction

Head movements are deliberate and accurate. To acquire a target with a head-gaze, users first position the cursor on the target with head movements, then confirm the target selection with another modality (e.g., button-press on a clicker, hand gesture, dwell timing). Yu et al. [] first investigated the feasibility of head pointing based text typing for HMDs by proposing three techniques -DwellType (user selects a letter by pointing to it and dwelling over it for ms), TapType (user selects a letter by pointing to it and tapping a button), and GestureType (user performs word-level input using a gesture typing style). They found that users can learn to type with their heads quickly, and it was not as fatiguing as it may seem at a first impression. Overall, GestureType outperformed the other two techniques.

In general, the confirmation techniques used in head-based pointing have their limitations. If the user's hands are busy, a button pressing on a clicker or performing a hand gesture might not be suitable. It is difficult to determine the optimal dwell time [] as longer dwell time slows down the interaction, while shorter dwelling causes unintentional selection. Moreover, pre-defined dwell time makes interaction stressful because users must always be very focused and act carefully to avoid unwanted false selection. This "inside-outside" design rejects the false positives of unintentional head movements and speeds up the selection process. They showed several potential applications such as text input, menu selection in HMDs using this approach.

Eye-Gaze Based Interaction

Eye-gaze input is ergonomic in nature and allows fast pointing, but it often suffers from the well-known 'Midas Eye-gaze cues (e.g., fixations, saccades, or blinks) play an important role (predicting another person's intention, complex social signal, etc.) to achieve successful face-to-face communication. Similarly, researchers conducted several studies to

Body Based Interaction

In the past, researchers have utilized our body as an interaction surface to leverage our proprioception capabilities. This proprioception (i.e., the sensation of relative position, orientation, and movement of our body parts to each other) allows building user interfaces in an eyes-free way. Moreover, our body surface offers natural tactile cues when it is touched.

With the proliferation of low-cost sensors and advancement

Pen Based Interaction

On a desktop, we can easily point at small targets precisely with a mouse or trackpad. But, achieving a similar level of accuracy with current input techniques (hand-tracking, head/eye-gaze, voice) in AR-HMDs is cumbersome. Recently, ). They found that the D pen significantly outperformed modern VR controllers in all evaluated measures, and its throughput is comparable to the mouse. Participants also liked the D pen more than the controller. Further in this direction, Batmaz et al. [] studied Fitts' task performance when a pen-like input device is held in a power or a precision grip for interaction with objects at different depths (see Figure .). Results favored precision grip irrespective of the distance of targets from the user.

Researches also investigated the use of pens for drawing and modeling in AR. Gasques et al. [] developed a rapid interactive prototyping tool called PintAR (see Figure .). With this tool, users first sketch digital content using a pen and tablet. Then, they rely on an AR-HMD to place their sketches in the real world and manipulate it directly.

In SymbiosisSketch [], the authors proposed a hybrid sketching system that combines drawing in mid-air (D) and https://massless.io/ † https://holo-light.com/products/stylus-xr/ ‡ https://www.logitech.com/en-roeu/promo/vr-ink.html . Interaction Space in AR-HMDs Using AR, we can sketch directly on an existing physical object. This is particularly helpful in personal fabrication for novice designers. They can quickly and easily create simple D models that should attach to or align with existing objects (e.g., a snap-on handle for a drinks container). Sketching

Related Work

directly on physical objects always provides natural haptic feedback whenever the pen touches them. Wacker et al. [] studied the users' performance of a stroke when drawing with a pen around physical vs. virtual objects in AR (see Figure .). They found that tracing physical objects was % more accurate but took longer than tracing virtual objects.

Mobile Devices Based Interaction

By combining handheld devices and HMDs, researchers try to make the most of the benefits of both []. The handheld device brings a D high-resolution display that provides a multi-touch, tangible, and familiar interactive surface.

. Interaction Space in AR-HMDs Whereas, HMDs provide a spatialized, D, and almost infinite workspace. With MultiFi [], Grubert et al. showed that such a combination is more efficient than a single device for pointing and searching tasks (see Figure .). For a similar setup, Zhu and Grossman proposed a design space for crossdevice interaction between HMD and phone (see Figure .

Multimodal Interaction

Researchers explored rich interaction opportunities by combining multiple input channels. Complementing the strengths of multiple channels can lead to enriched user experiences.

Regarding multimodal inputs, the synergy between speech and gestures is a widely used input combination. When used as unimodal input, speech can be beneficial for abstract tasks, whereas gestures can be beneficial for direct pointing and manipulation. In Their results showed that gesture-only interaction outperformed multimodal technique substantially for most tasks (except object scaling). This indicates that multimodality per se is not always beneficial for interaction, but needs to be carefully designed to suit the task at hand. Further, Chen et al. [] empirically investigated a set of techniques for disambiguating the effect of freehand interactions while manipulating virtual objects. They compared three input modalities (speech, head-gaze, and foot tap) paired with three different timings (before, during, and after an interaction) in which options become available to resolve ambiguity. The results indicated that using head-gaze for disambiguation during an interaction with the object achieved the best performance followed by speech and foot-tap. Prior work also looked into entering text using speech and hand-tracking [] which is illustrated in Figure .. In recent times, researchers are trying to seamlessly combine eye-gaze with other modalities (such as mid-air gestures and head movements) for designing novel interfaces in AR/VR-HMDs. For example, Pfeuffer et al. [] described the Gaze + Pinch technique which integrates eye-gaze with hand gestures for performing several tasks such as object selection, manipulation, scene navigation, menu interaction, and image zooming (see Figure .).

Extending Display Space

While AR displays are effective in many ways by augmenting the physical world with digital information, it has a limited field of view (FOV), and the isolated user experience (i.e., virtual environment is only visible to the HMD user) makes collaboration with external users difficult. Previous research combined an AR-HMD with spatial augmented reality (SAR) [], mobile devices (e.g., smartphone [], tablet [][]), interactive surfaces [], and large displays (see Figure .) [] to extend its display space and support external collaboration. Recently, AR-HMDs have also been combined with an actuated head-mounted projector to share AR content with co-located non-HMD users and enable them to interact Summary of AR-HMDs Interaction Space: Previous work has proposed several interesting ways to enrich the interaction space of AR-HMDs. Although we have discussed both input and output space briefly, we will mainly focus on the input interaction for our purpose in this thesis. Researchers have tried to accomplish different tasks (such as object selection and manipulation, interactive mid-air sketching, menu selection, mode switching, text input, gaze cues visualization in remote collaboration, windows management, interactive data visualization and navigation, etc.) with different input techniques. But, the text selection task in head-mounted AR glasses didn't receive much attention from the community so far (note that we have mentioned those work that considered text selection in AR-HMDs in the specific related work section of Chapter ). In this dissertation, we are particularly interested to investigate smartphone as an input device for selecting text in AR glasses. Overall, previous research showed smartphone provides a good alternative input for augmented reality display in various tasks. Still, it is not clear for text selection -only tactile interactions should be used on the smartphone, or it should also be tracked in mid-air to provide spatial interactions. We have tried to address this question in this thesis.

. Interaction Space in SAR

An advantage of spatial augmented reality (SAR) over AR-HMDs is that users can see the virtual content directly without wearing head-mounted displays. It enables multi-user collaboration seamlessly. However, the best way to interact with SAR setup is still an open problem.

To bring interactivity in the SAR scene, Bandyopadhyay et al. [] initially extended the concept of shader lamps [] with dynamic tracking to allow the users to paint onto physical objects using a tracked brush (see Figure .). Ware and Rose [] found that users' ability to perform virtual object manipulations (such as rotating a virtual object) improves with physical handles. Although these experiments were conducted in the context of virtual reality, the results are applicable to the SAR environment undoubtedly. As users' hands are free (they don't need to hold a display in their hands) and they can see the real world in SAR, they can take advantage of physical objects as a part of the user interface elements like tangible user interfaces (TUIs) []. These objects can also act as display surfaces. Inspired by this, researchers tried to combined TUI and SAR together. For example, Jones et al. [] developed a novel surface interaction engine that enables users to build their own Related Work physical world with almost any material suitable for the projection (like a set of wooden blocks), map virtual content onto their physical construction, and play directly with the surface using a stylus (see Figure .). Their setup provides end-users a uniquely immersive, tangible SAR experience. They also designed a set of surface adaptive D GUIs (like a radial menu) to show possible avenues for structuring interaction on complex physical surfaces. The user selects a menu item with a stylus from a surface adaptive radial menu that adapts to the surface that it is displayed on.

Marner et al. [] proposed the concept of Physical-Virtual

Tools (PVTs) in SAR by projecting application-specific GUIs on a physical tool that is carried by the user in the projection area. This allows to overload a single tool with several functions to interact with SAR. Figure .) shows an example of PVTs for an airbrushing application. The user holds the stencil tool in his left hand, and the system projects the controls widgets on it. The airbrush tool, held in the right hand, is augmented with projection to provide currently selected paint color, spray angle, and brush type information. PVTs, build on the physical nature of TUIs, are invaluable in a large-scale SAR system where there is no fixed location available for displaying traditional user interfaces (UIs). Moreover, users can interact with the system using this tool from anywhere in the scene as they always have quick access to control widgets.

We can also utilize unique physical affordances of everyday objects (e.g., markers, stapler, coffee-mugs, etc.) beyond a dedicated tangible system to enhance interaction with the SAR system. For instance, users immediately pick up a marker from their desk and move/rotate in different directions to use it as an improvised joystick to control D content. Henderson and Feiner called On the other side, Schmidt et al. [] presented a floor-based user interface (UI) that allows multiple users to explore a SAR environment with both monoscopic and stereoscopic projections (see Figure .). MirageTable [] combined a depth camera, a curved screen, and a stereoscopic projector to enable the user to perform freehand interaction with virtual D objects in a tabletop SAR scenario (see Figure .).

Other techniques explored mobile devices and a standard mouse to interact with the SAR scene. For example, Hartmann and Vogel [] investigated three smartphone-based pointing techniques to select virtual objects in SAR (see Figure .). These methods are -viewport pointing (use the phone's camera as a viewport to interact with remote targets), raycast pointing (a virtual ray emits from the phone's front end towards the target and then the user taps a button on the screen to make a selection), and tangible pointing (a target is selected by directly touching it with the mobile phone). From the experiments, the authors found that raycast is fastest for high and distant targets, tangible is fastest for targets in close proximity to the user, and viewport performance is in between. Similarly Due to the rigid mapping between physical and virtual parts in spatial augmented reality, the virtual scene cannot be explored in different scales and points of view. To overcome this issue, previous works fused multiple mixed reality

Summary of SAR Interaction Space:

Previous research proposed several interesting ways to interact with the SAR scene, but they have limitations too.

Techniques that provide graphical widgets on the table (like

Related Work dynamic shader lamps [], opportunistic controls interface [][])

or on the physical surfaces of the mock-ups [] restrict users to be always close to the scene. This restriction might not be suitable in some scenarios like museums. Handbased interaction [] in SAR also has the same problem. With a standard mouse [] indirect interaction is possible, but a surface is required to keep it (this limits our mobility in space). In the case of hand-held tool [], users need to hold it all the time, as well as explicit attention switching is required to select UI elements projected on that panel. Floor projected UI and mobile devices [][] also have the focus shifting issue []. On the other hand, using other displays (VR-HMDs or transparent screens) [, ] to experience physical augmentation from different perspectives inhibits multi-user face-to-face collaborations in SAR.

To overcome these limitations, we proposed to provide interactive graphical widgets in mid-air using projection on a drone (this gives a sense of extending the space in SAR), while users issues eyes-free input via a hand-held controller (see Chapter ). This combination of input and output allows users to maintain face-to-face collaborations; they can move in space freely and access interactive widgets instantly from anywhere without explicitly switching their attention.

Exploring Smartphone-enabled Text

Selection in AR-HMDs

Chapter Summary: Text editing is essential and at the core of most complex tasks, like writing an email or browsing the web. Efficient and sophisticated techniques exist on desktops and touch devices, but are still under-explored for AR glasses. Performing text selection, a necessary step before text editing, in AR display commonly uses techniques such as hand-tracking, voice commands, eye/head-gaze, which are cumbersome and lack precision. In this work, we explored the use of a smartphone as an input device to support text selection in AR-HMDs because of its availability, familiarity, and social acceptability. We proposed four eyes-free text selection techniques, all using a smartphone -continuous touch, discrete touch, spatial movement, and raycasting. We compared them in a user study where users have to select text at various granularity levels. Our results suggested that continuous touch, in which we used the smartphone as a trackpad, outperformed the other three techniques in terms of task completion time, accuracy, and user preference.

. Introduction

Text input and text editing represent a significant portion of our everyday digital tasks. We need it when we browse the web, write emails, or just when we type a password. Because of this ubiquity, it has been the focus of research on most of the platforms we are using daily, like desktops, tablets, and mobile phones. The recent focus of the industry on AR-HMDs, with the development of devices like the Microsoft HoloLens and Magic Leap † , made them more and more accessible to us, and their usage is envisioned in our future everyday life. The lack of a physical keyboard and mouse/trackpad with such devices makes text input difficult and an important challenge in AR research. While text input for AR-HMDs has been already well-studied [-], limited research focused on editing text that a user has https://www.microsoft.com/en-us/hololens † https://www.magicleap.com/en-us already typed. Normally, text editing is a complex task and the first step is to select the text to edit it. This work will only focus on this text selection part. Such tasks have already been studied on desktop [] with various modalities (like gaze+gesture [], gaze with keyboard []) as well as touch interfaces []. On the other hand, no formal experiments were conducted in AR-HMDs contexts.

Generally, text selection in AR-HMDs can be performed using various input modalities, including notably hand-tracking, eye/head-gaze, voice commands [], and handheld controller []. However, these techniques have their limitations. For instance, hand-tracking suffers from achieving character level precision [], lacks haptic feedback [], and provokes arm fatigue [] during prolonged interaction. Eye-gaze and head-gaze suffer from the 'Midas Touch' problem, which causes unintended activation of commands in the absence of a proper selection mechanism [-]. Moreover, frequent head movements in head-gaze interaction increase motion sickness []. Voice interaction might not be socially acceptable in public places [], and it may disturb the communication flow when several users are collaborating. In the case of a dedicated handheld controller, users always need to carry extra specific hardware.

Recently, researchers have been exploring to use of a smartphone as an input for AR-HMDs because of its availability (it can even be the processing unit of HMDs []), familiarity, social acceptability, and tangibility [-]. Undoubtedly, there is a huge potential for designing novel cross-device applications with a combination of an AR display and a smartphone. In the past, smartphones have been used for interacting with different applications running on AR-HMDs, such as manipulating D objects [], windows management [], selecting graphical menus [] and so on. However, we are unaware of any research that has investigated text selection in an AR display using a commercially available smartphone. In this work, we explored different approaches to select text when using a smartphone as an input controller. We proposed four eyes-free text selection techniques for AR display. These techniques, described in Section , differ with regard to the mapping of smartphone-based inputs -touch or spatial. We then conducted a user study to compare these four techniques in terms of text selection task performance.

The main contributions of this research are -() design and development of a set of smartphone-enabled text selection techniques for AR-HMDs; () insights from a person comparative study of these techniques in text selection tasks.

. Specific Related Work

In this section, we review previous work on text selection and editing in an AR display as well as on a smartphone. We also review research that combined handheld devices with large wall displays.

Text Selection and Editing in AR glasses

A very few research focused on text editing in AR-HMDs. Ghosh et al. presented EYEditor to facilitate on-the-go textediting on a smart-glass with a combination of voice and a handheld controller [] (see Figure .). They used voice to modify the text content, while manual input is used for text navigation and selection. The use of a handheld device is inspiring for our work; however, voice interaction might not be suitable in public places. Lee et al. [] proposed two force-assisted text acquisition techniques where the user exerts a force on a thumb-sized circular button located on an iPhone 7 and selects text which is shown on a laptop emulating the Microsoft Hololens display (see Figure .). They envision that this miniature force-sensitive area ( mm ⇥ mm) can be fitted into a smart-ring. Although their result is promising, a specific force-sensitive device is required. In this work, we follow the direction of the two papers previously presented and continue to explore the use of a smartphone in combination with an AR-HMD. While use of a smartphone for text selection is still rare, it has been investigated more broadly for other tasks in DUIs.

Combining Handheld Devices and Large Wall Displays

The use of handheld devices as input was investigated in combination with large wall displays. It is a use case close to the one presented in this work as text is displayed inside a D virtual window. Campbell et al. studied the use of a Wiimote as a distant pointing device []. With a pointing task, the authors compared its use with an absolute mapping (i.e., raytracing) to a relative mapping and showed that participants were faster with the absolute mapping. Vogel and Balakrishnan [] designed three freehand distant pointing techniques for a very large high-resolution displayabsolute position finger ray casting, relative pointing with clutching, and a hybrid technique using ray casting for quick absolute coarse pointing combined with relative pointing when more precision is desired. These techniques are illustrated in Figure .. They found raycasting was faster, but only with large targets and when clutching was necessary. In their study, participants had a lower accuracy with an absolute mapping. This lower accuracy for an absolute mapping with spatial interaction was also shown when compared with distant touch interaction of the handheld device as a trackpad, with the similar task []. Jain et al. also compared touch interaction with spatial interaction, but with a relative mapping, and found that the spatial interaction was faster but less accurate []. The accuracy result was confirmed by a recent study from Siddhpuria et al. in which the authors also compared the use of absolute and relative mapping with the touch interaction and found that the relative mapping was faster []. These studies were all done for a pointing task and overall showed that using the handheld device as a trackpad (so with a relative mapping) is more efficient (to avoid clutching, one can change the transfer function []). In their paper, Siddhpuria et al. highlighted the fact that more studies needed to be done with a more complex task to validate their results. To our knowledge, this has been done only by Baldhauf et al. with a drawing task, and they showed that spatial interaction with an absolute mapping was faster than using the handheld device as a trackpad without any impacts on the accuracy []. In this work, we take a step in this direction and use a text selection task. Considering the result from Baldauf et al. [], we cannot assume that touch interaction will perform better.

Text Selection on Handheld Devices

Text selection has not been yet investigated with the combination of a handheld device and an AR-HMD, but it has been studied on handheld devices independently. Using a touchscreen, adjustment handles are the primary form of text selection techniques. However, due to the fat-finger problem [], it can be difficult to modify the selection by one character. A first solution is to allow users only to select the start and the end of the selection as it was done in TextPin (see Figure .), in which it was shown to be more efficient than the default technique []. Fuccella et al. [] and Zhang et al. [] proposed to use the keyboard area to allow the user to control the selection using gestures and showed it was also more efficient than the default technique (see Figure .). Ando et al. adapted the principle of shortcuts and associated different actions with the keys of the virtual keyboard that was activated with a modifier action performed after. In the first paper, the modifier was the tilting of the device (see Figure .) [], and in a second one, it was a sliding gesture starting on the key (see Figure . In this work, we choose to focus on commercially available smartphones, and we will not explore in this work the use of deformable or fully touch-sensitive ones. Compared to the use of shortcuts, the use of gestures seems to lead to good performance and can be performed without looking at the screen (i.e., eyes-free), which avoids transition between the AR virtual display and the handheld devices.

. Designing Smartphone-Based Text

Selection in AR-HMDs AR-HMDs are wearable see-through displays that can be used in a wide range of contexts: users can be seated alone in their office, standing up in a meeting room presenting to collaborators, or even walking outside. Text selection techniques need to adapt to this diversity. The following two use cases scenario demonstrate situations in which the combination of a smartphone and an AR-HMD can be an appropriate solution considering the context.

Collaborative Meeting: Alice is doing a meeting with her colleagues in an immersive environment using Spatial ‡ . She is wearing an AR-HMD and, thus, is not limited by the D flat screen anymore. She can move around in the physical space and use it to organize the content of her presentation. To present some interesting statistics, Alice wants to highlight them in her document by selecting them. Using a smartphone-based technique, which is in addition eye-free and uni-manual, she can perform this text selection without interrupting her presentation (e.g., she can keep eye contact with her colleague, use deictic gestures while talking). ‡ https://spatial.io/

.

Designing Smartphone-Based Text Selection in AR-HMDs

Walking in a park: Bob is walking in a park wearing an AR-HMD. He is browsing the upcoming conference website to take a look at the conference schedule. The AR-HMD display allows Bob to have a large viewport to look at the website, but also to keep an eye on what is happening in the real world, and make sure there is no one or nothing in front of him. Using a smartphone-based technique, which in addition is not cognitively or physically demanding, Bob can select interesting paper titles and copy them in his favorite notepad application, while still walking and being careful of the world around him.

Design Criteria

We considered the following design requirements:

Single-handed Input: Although the two-handed technique tends to outperforms uni-manual input [, ], users often have a situation where only one hand is available to hold the phone on-the-go scenario [, ]. In this work, we decided to investigate uni-manual interaction (i.e., holding the smartphone in portrait mode with the dominant hand and use the thumb for touch input) which will allow users for more casual interaction. They can use their non-dominant hand to interact with the real world, such as holding a tool or inspecting an item.

Eyes-free Interaction: Frequent visual attention switching between an AR-HMD and the smartphone creates distractions to the user and might lead to higher error rates [].

To maintain focus on the text selection task in AR-HMDs, we chose to provide eye-free input from the smartphone. Like prior work [], we took advantage of one-handed touch and spatial inputs instead of soft buttons, which demand constant visual attention from the user. The smartphone also offers vibration feedback to confirm successful actions.

Minimal

Proposed Techniques

Previous work used a smartphone as an input device to interact with virtual content in AR-HMDs mainly in two ways -touch input from the smartphone and tracked the smartphone spatially like AR/VR controller. Similar work on wall displays suggested that using the smartphone as a trackpad would be the most efficient technique, but this was tested with a pointing task. With a drawing task (which could be closer to a text selection task than a pointing task), spatial interaction was actually better [].

Inspired by this, we propose four eyes-free text selection techniques for AR-HMDs -two are completely based on mobile touchscreen interaction, whereas the smartphone needs to be tracked in mid-air for the latter two approaches to use spatial interactions. For spatial interaction, we choose a technique with an absolute mapping (Raycasting) and one with a relative one (Spatial Movement). The comparison between both in our case is not straightforward, previous results suggest that a relative mapping would have better accuracy, but an absolute one would be faster. For touch interaction, we choose not to have an absolute mapping; its use with a large virtual window could lead to a poor accuracy [], and just have a technique that uses a relative mapping.

In addition to the traditional use of the smartphone as a trackpad (Continuous Touch), we propose a technique that allows for a discrete selection of text (Discrete Touch). Such discrete selection mechanism has shown good results in a similar context for shape selection []. Overall, while we took inspiration from previous work for these techniques, they have never been assessed for a text selection task.

To select text successfully using any of our proposed techniques, a user needs to follow the same sequence of steps each time. First, she moves the cursor, located on the text window in an AR display, to the beginning of the text to be selected (i.e., the first character). Then, she performs a double tap on the phone to confirm the selection of that first character. She can see on the headset screen that the first character got highlighted in yellow color. At the same time, she enters into the text selection mode. Next, she continues moving the cursor to the end position of the text using one of the techniques presented below. While the cursor is moving, the text is also getting highlighted simultaneously up to the current position of the cursor. Finally, she ends the text selection with a second double-tap.

Continuous Touch

In continuous touch, the smartphone touchscreen acts as a trackpad (see Figure . .(a)). It is an indirect pointing technique where the user moves her thumb on the touchscreen to change the cursor position on the AR display. For the mapping between display and touchscreen, we used a relative mode with clutching. As clutching may degrades performance [], a control-display (CD) gain was applied to minimize it (see Subsection ).

Discrete Touch

This technique is inspired by the text selection with keyboard shortcuts available in both Mac [] and Windows [] OS.

In this work, we tried to emulate a few keyboard shortcuts. We particularly considered imitating keyboard shortcuts related to the character, word, and line-level text selection. For example, in Mac OS, hold down and pressing or extends text selection one character to the right or left. Whereas hold down + and pressing or allows users to select text one word to the right or left. To perform text selection to the nearest character at the same horizontal location on the line above or below, a user needs to hold down and press or respectively. In discrete touch interaction, we replicated all these shortcuts using directional swipe gestures (see Figure . .(b)). Left or right swipe can select text at both levels -word as well as character. By default, it works at the word level. Users perform a long-tap which acts as a toggle button to switch between word and character level selection. On the other hand, up or down swipe selects text at one line above or one line below from the current position. The user can only select one character/word/line at a time with its respective swipe gesture.

Note that, to select text using discrete touch, a user first positions the cursor on top of the starting word (not the starting character) of the text to be selected by touch dragging on the smartphone as described in the continuous touch technique. From a pilot study, we observed that moving the cursor every time to the starting word using discrete touch makes the overall interaction slow. Then, she selects that first word with the double-tap and uses discrete touch to select text up to the end position as described before.

Spatial Movement

This technique emulates the smartphone as an air-mouse [, ] for head-mounted AR displays. To control the cursor position on the headset screen, the user holds the phone in front of her torso, places her thumb on the touchscreen, and then she moves the phone in the air with small forearm motions in a plane that is perpendicular to the gaze direction (see Figure . .(c)). While moving the phone, its tracked positional data in XY coordinates get translated into the cursor movement in XY coordinates inside a D window. When a user wants to stop the cursor movement, she simply lifts her thumb from the touchscreen. Thumb touch-down and touch-release events define the start and stop of the cursor movement on the AR display. The user determines the speed of the cursor by simply moving the phone faster and slower accordingly. We applied a control display gain between the phone movement and the cursor displacement on the text window as described in Subsection .

Raycasting

Raycasting is a popular interaction technique in AR/VR environments to select D virtual objects [, ]. In this work, we developed a smartphone-based raycasting technique for selecting text displayed on a D window in AR-HMDs (see

Figure. .(d))

. A 6 DoF tracked smartphone was used to define the origin and orientation of the ray. In the headset display, the user can see the ray in a straight line appearing from the top of the phone. By default, the ray is always visible to users in an AR-HMD as long as the phone is being tracked properly. In raycasting, the user needs to do small angular wrist movements for pointing on the text content using the ray. Where the ray hits on the text window, the user sees the cursor there. Compared to other proposed methods, raycasting does not require clutching as it allows direct pointing to the target. The user confirms the target selection on the AR display by providing a touch input (i.e., double-tap) from the phone.

Implementation

To prototype our proposed interaction techniques, we used a Microsoft HoloLens ( ◊ screen) as an AR-HMD device and a OnePlus as a smartphone. For spatial movement and raycasting interactions, real-time pose information of the smartphone is needed. An OptiTrack § system with three Flex-cameras was used for accurate tracking with low latency. To bring the hololens and the smartphone into a common coordinate system, we attached passive reflective markers to them and did a calibration between hololens space and optitrack space.

In our software framework, the AR application running on HoloLens was implemented using UnityD (.) and § https://optitrack.com/ Exploring Smartphone-enabled Text Selection in AR-HMDs To render text in HoloLens, we used TextMeshPro. A Windows workstation was used to stream tracking data to HoloLens. All pointing techniques with the phone were also developed using UnityD. We used UNet library for client-server communications between devices over the WiFi network.

For continuous touch and spatial movement interactions, we used a generalized logistic function [] to define the CD-gain between the move events either on the touchscreen or in the air and the cursor displacement in the AR display:

⇠⇡(E) = ⇠⇡ "0G ⇠⇡ "8= 1 + 4 ⌫(E + 8= 5 ) + ⇠⇡ "8= (.)
⇠⇡ "0G and ⇠⇡ "8= are the asymptotic maximum and minimum amplitudes of CD gain and ⌫ is a parameter proportional to the slope of the function at E = + 8= 5 with + 8= 5 a inflection value of the function. We derived initial values from the parameters of the definitions from Nancel et al.

[], and then empirically optimized for each technique. The parameters were not changed during the study for individual participants. The values are summarized in Table ..

In discrete touch interaction, we implemented up, down, left, and right swipes by obtaining touch position data from the phone. We considered a 700 msec time window for detecting a long-tap event after doing a pilot test with four users from our lab. Users get vibration feedback from the phone when they perform long-tap successfully. They also receive vibration haptics while double-tapping to start and end the text selection in all interaction techniques. Note that, there is no haptic feedback for swipes. With each swipe movement, they can see that texts are getting highlighted in yellow color. This acts as visual feedback by default for touch swipes. ¶ https://github.com/microsoft/MixedRealityToolkit-Unity https://docs.unity3d.com/Manual/UNet.html

.

Experiment

In the spatial movement technique, we noticed that the phone moves slightly during the double-tap event. This results in a slight unintentional cursor movement. To reduce that, we suspended cursor movement for 300 msec when there is any touch event on the phone screen. We found this value after doing trial and error with different values ranging from 150 msec to 600 msec.

In raycasting, we applied the = C Filter [] with = and min-cutoff = . at the ray source to minimize jitter and latency, which usually occur due to both hand tremor and double-tapping []. We tuned these two parameters by following the instruction mentioned in the = C Filter implementation website . We set the ray length to meters by default. The user sees the full length of the ray when it is not hitting the text panel.

.

Experiment

To assess the impact of the different characteristics of these four interaction techniques we perform a comparative study with a text selection task while users are standing up. Particularly, we are interested to evaluate the performance of these techniques in terms of task completion time, accuracy, and perceived workload.

Participants and Apparatus

In our experiment, we recruited unpaid participants (P-P) ( males + females) from a local university campus. Their ages ranged from to years (mean = ., SD = .). Four were left-handed. All were daily users of smartphones and desktops. With respect to their experience with AR/VR technology, participants ranked themselves as an expert because they are studying and working on the same field, participants were beginners as they played some games in VR, while others had no prior experience. They all had either normal or corrected-to-normal vision. We used the apparatus and prototype described in Subsection .

https://cristal.univ-lille.fr/ casiez/euro/

Task

In this study, we ask participants to perform a series of text selections using our proposed techniques. Participants were standing up for the entire duration of the experiment. We reproduce different realistic usage by varying the type of text selection to do, like the selection of a word, a sentence, a paragraph, etc. Figure . shows all the types of text selection that were asked to the participants. Concretely, the experiment scene in HoloLens consisted of two vertical windows of . cm ⇥ . cm positioned at a distance of cm from the headset at the start of the application (i.e., its visual size was 31.75 ⇥ 18.1806 ). The windows were anchored in the world coordinate. These two panels contain the same text. Participants are asked to select the text in the action panel (left panel in

Study Design

We used a within-subject design with factor: I T (Continuous Touch, Discrete Touch, Spatial Movement, and Raycasting) ⇥ T (shown in Figure .) ⇥ participants = trials. The order of I T was counterbalanced across participants using a Latin Square. The order of T is randomized in each block for each I T (but same for each participant).

Procedure

We welcomed participants upon arrival. They were asked to read and sign the consent form, fill out a pre-study questionnaire to collect demographic information and prior AR/VR experience. Next, we gave them a brief introduction to the experiment background, hardware, the four interaction techniques, and the task involved in the study. After that, we helped participants to wear HoloLens comfortably and complete the calibration process for their personal interpupillary distance (IPD). For each block of I T, participants completed a practice phase followed by a test Exploring Smartphone-enabled Text Selection in AR-HMDs session. During the practice, the experimenter explained how the current technique worked, and participants were encouraged to ask questions. Then, they had time to train themselves with the technique until they were fully satisfied, which took around minutes on average. Once they felt confident with the technique, the experimenter launched the application for the test session. They were instructed to do the task as quickly and accurately as possible in a standing condition. To avoid noise due to participants using either one or two hands, we asked to only use their dominant hand.

At the beginning of each trial in the test session, the text to select was highlighted in the instruction panel. Once they were satisfied with their selection, participants had to press a dedicated button on the phone screen to get to the new task. They were allowed to use their non-dominant hand only to press this button. At the end of each block of I T, they answered a NASA-TLX questionnaire [] on iPad, and moved to the next condition.

At the end of the experiment, we asked participants a questionnaire in which they had to rank techniques by speed, accuracy, and overall preference and performed an informal post-test interview.

The entire experiment took approximately minutes in total. Participants were allowed to take breaks between sessions during which they could sit and encourage to comment at any time during the experiment. To respect COVID-safety protocol, participants wore FFP mask and maintained a -meter distance with the experimenter at all times.

Measures

We recorded completion time as the time taken to select the text from its first character to the last character, which is the time difference between the first and second double-tap. If they selected more or less characters than expected, the trial was considered wrong. We then calculated the error rate as the percentage of wrong trials for each condition. Finally, as stated above, participants filled a NASA TLX questionnaire to measure the subjective workload of each I T, and their preference was measured using a ranking questionnaire at the end of the experiment.

.

Result

Hypotheses

In our experiment, we hypothesized that: H. Continuous Touch, Spatial Movement, and Raycasting will be faster than Discrete Touch because a user needs to spend more time for multiple swipes and do frequent mode switching to select text at the character/word/sentence level.

H. Discrete Touch will be more mentally demanding compared to all other techniques because the user needs to remember the mapping between swipe gestures and text granularity, as well as the long-tap for mode switching.

H. The user will perceive that Spatial Movement will be more physically demanding as it involves more forearm movements.

H. The user will make more errors in Raycasting, and it will be more frustrating because double-tapping for target confirmation while holding the phone in one hand will introduce more jitter [].

H. Overall, Continuous Touch would be the most preferred text selection technique as it works similarly to the trackpad which is already familiar to users.

. Result

To test our hypothesis, we conducted a series of analyses using IBM SPSS software. Shapiro-Wilk tests showed that the task completion time, total error, and questionnaire data were not normally distributed. Therefore, we used the Friedman test with the interaction technique as an independent variable to analyze our experimental data. When significant effects were found, we reported post hoc tests using the Wilcoxon signed-rank test and applied Bonferroni corrections for all pair-wise comparisons. We set an = . in all significance Exploring Smartphone-enabled Text Selection in AR-HMDs tests. Due to a logging issue, we had to discard one participant and did the analysis with instead of participants.

Task Completion Time

There was a statistically significant difference in task completion time depending on which interaction technique was used for text selection [" 2 () = ., p < .] (see Figure

Error Rate

We found significant effects of the interaction technique on error rate [" 2 () = ., p < .] (see Figure .). Post hoc tests showed that Raycasting [M = ., SD = .] was more error-prone than Continuous Touch [M = ., SD = .], Discrete Touch [M = ., SD = .], and Spatial Movement [M = ., SD = .].

Questionnaires

For NASA-TLX, we found significant differences for mental demand [" 2 () = ., p = .], physical demand [" 2 () = ., p < .], performance [" 2 () = ., p < .], frustration [" 2 () = ., p < .], and effort [" 2 () = ., p < .]. Post hoc tests showed that Raycasting and Discrete Touch . Result had significantly higher mental demand compared to Continuous Touch and Spatial Movement. On the other hand, physical demand was lowest for Continuous Touch, whereas users rated significantly higher physical demand for Raycasting and Spatial Movement. In terms of performance, Raycasting was rated significantly lower than the other techniques. Raycasting was also rated significantly more frustrating. Moreover, Continuous Touch was least frustrating and better in performance than Spatial Movement. Figure . shows a bar chart of the NASA-TLX workload sub-scales for our experiment.

For ranking questionnaires, there were significant differences for speed [" 2 () = ., p < .], accuracy [" 2 () = ., p < .], and preference [" 2 () = ., p < .]. Post hoc test showed that users ranked Discrete Touch as the slowest and Raycasting as the least accurate technique. The most preferred technique was Continuous Touch whereas Raycasting was the least. Users also favored Discrete Touch as well as Spatial Movement based text selection approach. Figure . summarises participants responses for ranking questionnaires.

. Discussion & Design Implications

Our results suggest that Continuous Touch is the technique that was preferred by the participants (confirming H). It was the least physically demanding technique and the less frustrating one. It was also more satisfying regarding performance than the two spatial ones (Raycasting and Spatial Movement). Finally, it was less mentally demanding than Discrete Touch and Raycasting. Participants pointed out that this technique was simple, intuitive, and familiar to them as they are using trackpad and touchscreen every day. During the training . Discussion & Design Implications session, we noticed that they took the least time to understand its working principle. In the interview, P commented, "I can select text fast and accurately. Although I noticed a bit of overshooting in the cursor positioning, it can be adjusted by tuning CD gain". P said, "I can keep my hands down while giving input to select text in AR display. This gives me more comfort".

On the other hand, Raycasting was the least preferred technique and led to the lowest task accuracy (confirming H). Participants (irrespective of experienced and inexperienced) were also the least satisfied with their performance using this technique. This can be explained by the fact that it was the most physically demanding and the most frustrating. Finally, it was more mentally demanding than Continuous Touch and Spatial Movement. In their comments, participants reported about the lack of stability due to the one-handed phone holding posture. Some participants complained that they felt uncomfortable to hold this OnePlus phone in one hand as it was a bit bigger compared to their hand size. This introduced even more jitter for them in Raycasting while double-tapping for target confirmation. P commented, "I am sure I will perform Raycasting with fewer errors if I can use my both hands to hold the phone". Moreover, from the logged data, we noticed that they made more mistakes when the target character was positioned inside a word rather than either at the beginning or at the end, which was confirmed in the discussion with participants.

As we expected, Discrete Touch was the slowest technique (confirming H), but was not the most mentally demanding, as it was only more demanding than Continuous Touch (rejecting H). It is also more physically demanding than Continuous Touch, but less than Spatial Movement and Raycasting. Several participants mentioned that it is excellent for the short word to word or sentence to sentence selection, but not for long text as multiple swipes are required. They also pointed out that performing mode switching with a long-tap of msec was a bit tricky and lost some time there during text selection. Although they got better with it over time, still they are uncertain to do it successfully in one attempt. To improve this mode switching, one participant suggested using a triple tap for mode switching instead of a long-tap.

Finally, contrary to our expectation, Spatial Movement was not the most physically demanding technique, as it was less demanding than Raycasting but more than Continuous Touch Exploring Smartphone-enabled Text Selection in AR-HMDs and Discrete Touch (rejecting H). It was also less mentally demanding than Raycasting and led to less frustration. However, it led to more frustration and participants were less satisfied with their performance with this technique than with Continuous Touch. According to participants, with this technique, moving the forearm needs physical effort undoubtedly, but they only need to move it for a very short distance which was fine for them. From the user interview, we came to know that they did not use much clutching (less than with Continuous Touch). P mentioned, "In Spatial Movement, I completed most of the tasks without using clutching at all".

Overall, our results suggest that between touch and spatial interactions, it would be better to use touch for text selection, which confirms findings from Siddhpuria et Such difference can be explained by the fact that with text selection, there is a minimum of two levels of discretization (characters and words), which makes it mentally demanding. It can also be explained by the high number of words (and even more characters) in a text, contrary to the number of shapes in Jain et al. experiment. This led to a high number of discrete actions for the selection, and thus, a higher physical demand. However, surprisingly, most of the participants appreciated the idea of Discrete Touch. If a tactile interface is not available on the handheld device, our results suggest to use a spatial interaction technique that uses a relative mapping, as we did with Spatial Movement. We could not find any differences in time, contrary to the work by Campbell et al. [], but it leads to fewer errors, which confirms what was found by Vogel and Balakrishnan []. It is also less physically and mentally demanding and leads to less frustration than an absolute mapping. On the technical side, a spatial interaction technique with a relative mapping can be easily achieved without an external sensor (as it was done for example by Siddhpuria et al. []). . Limitations There were two major limitations. First, we used an external tracking system which limits us to lab study only. As a result, it is difficult to understand the social acceptability of each technique until we consider the real-world on-the-go situation. However, technical progress in inside-out tracking ‡ ‡ means that it will be possible, soon, to have smartphones that can track themselves accurately in D space. Second, some of our participants had difficulties holding the phone in one hand because the phone was a bit bigger for their hands. They mentioned that although they were trying to move their thumb faster in continuous touch and discrete touch interactions, they were not able to do it comfortably due to the afraid of dropping the phone. This bigger phone size also influenced their raycasting performance particularly when they need to do a double-tap for target confirmation. Hence, using one phone size for all was an important constraint in this experiment.

. Conclusion

In this research, we investigated the use of a smartphone as an eyes-free interactive controller to select text in augmented reality head-mounted display. We proposed four interaction techniques: two that use the tactile surface of the smartphone (continuous touch and discrete touch), and two that track the device in space (spatial movement and raycasting). We evaluated these four techniques in a text selection task study. The results suggested that techniques using the tactile surface of the device are more suited for text selection than spatial one, continuous touch being the most efficient. If a tactile surface was not available, it would be better to use a spatial technique (i.e. with the device tracked in space) that uses a relative mapping between the user gesture and the virtual screen, compared to a classic raycasting technique that uses an absolute mapping.

Augmentation using AR-HMDs is good in many cases. But still, users need to wear displays. On the other hand, spatial augmented reality (SAR) allows us to augment our physical space without wearing any displays. That's why we are interested in SAR, which is the focus of the next chapter.

Extending Physical Spaces in SAR using Projection on a Drone

Chapter Summary: Spatial Augmented Reality (SAR) transforms real-world objects into interactive displays by projecting digital content using video projectors. SAR enables colocated collaboration immediately between multiple viewers without the need to wear any special glasses. Unfortunately, one major limitation of SAR is that visual content can only be projected onto its physical supports. As a result, embedding User Interfaces (UI) widgets such as menus and pop-up windows in SAR is very challenging. We are trying to address this limitation by extending SAR space in mid-air. In this work, we propose DroneSAR, which extends the physical space of SAR by projecting digital information dynamically on the tracked panels mounted on a drone. DroneSAR is a proof of concept of novel SAR User Interface (UI), which provides support for D widgets (i.e., label, menu, interactive tools, etc.) to enrich the SAR interactive experience. We also describe the implementation details of our approach.

. Introduction

Spatial Augmented Reality (SAR) [] transforms physical surfaces into augmented surfaces by projecting digital content directly onto them. Compared to see-through augmented reality, SAR allows multiple users to observe D augmented objects with natural depth clues without being instrumented. This opens many opportunities in architecture [], education [], museum[], and so on.

Unfortunately, one of the main limitations of the SAR environment is that, contrary to see-through AR technologies, visual content can only be displayed onto physical supports. As a consequence, embedding UI widgets such as menus and pop-up windows in SAR becomes challenging. These widgets need to be positioned onto the augmented physical objects, which results in a visual clutter that affects the overall user experience. The geometry and material of the physical scene even sometimes make it impossible to display legible UI widgets []. We have tried to address these limitations by Extending Physical Spaces in SAR using Projection on a Drone extending SAR space in the air. In the traditional SAR, it is not possible to display mid-air information unless using dedicated optical systems [, ] or body-tracked anamorphic illusions []. In this work, we used a flying display within the SAR environment to display mid-air content. The house is augmented using projection, and the main menu is composed of a set of virtual tools projected on the drone panel. (B) A user selected the 'measuring tool' application using a controller. Then, the user positions the drone at the desired location in the D space (i.e., on top of the house) and draws a line shown in blue color on the augmented house to measure its width. Finally, the measured length is displayed on the drone panel.

We proposed DroneSAR, a tracked drone mounted with two rectangular white panels on which it is possible to display digital information on the fly (see Figure . and Figure .). Drones have the advantage of being flexible, as they can be positioned quickly with an acceptable accuracy around any augmented space. This allows us to extend the augmentation space and creates opportunities for new applications. In particular, DroneSAR makes it possible to embed D interactive widgets within the SAR experience.

The concept of extending the SAR space around the physical objects can be achieved with alternative approaches such as holding mobile devices surrounding the physical objects or adding extra projection screens around the real objects. However, our proposed solution has several benefits from its counterparts. For example, in the case of mobile devices, users need to divide their attention between the augmented objects and the phone display. With drones, the augmentation takes place in the relevant D physical space, which can be at a distance from the observer. Regarding the use of extra projection screens around the objects, this makes the physical environment static, whereas the projection on a drone is more dynamic by bringing the screen where we need it. Using a robotic arm carrying a display could be an option, but it requires a complex motion planning setup, whereas the drones are much more flexible in navigating inside a space.

In our implementation, we intentionally chose to use projection rather than equipping drones with LCD screens. This allows us to use smaller drones, which are cheaper, safer, and less noisy. Furthermore, it does not require sending synchronized video streams to the individual displays, and the rendering of the visual content remains uniform over the all augmented scene. In summary, our contributions in this work are -(i) the exploration of the DroneSAR framework and its related interaction techniques and (ii) a concrete implementation and description of the technical details of this approach.

. Specific Related Work

Mid-air Imaging

It is convenient to share information between multiple people by displaying it in the air than by using traditional flat displays. In the literature, different aerial presentation systems have been widely explored. For example, Yagi et al. [] developed a novel fog display system that enables users to observe a D shape of virtual objects from multiple viewpoints. It consists of one cylindrical fog screen and multiple surrounding projectors (see Figure .). It also allows users to touch virtual objects with their hands directly. In MisTable [], authors combined a tabletop with a fog display to create a new interaction volume above its horizontal surface. Tokuda et al. [] further investigated a shape-changing fog display that can support one or two users interacting with either D or D content. On the other hand, Barnum et al.

[] developed a layered .D water drop display by synchronizing the timing of dripping water drops with a camera and a projector (see Figure .). In Pixie Dust [], Ochiai et al. created a volumetric display by controlling dust (i.e., very small particles) in the air with ultrasound (see Figure .). However, all these techniques (fog, water droplet, dust) have insufficient resolution and intensity. Laser plasma emission based mid-air imaging has also been studied, but this method is highly expensive, complex, and requires safety precautions [][]. Moreover, researchers developed optical imaging systems for anchoring real objects with virtual content in mid-air [, ], but they have a limited interactive zone.

In our approach, we used a drone-mounted paper panel as a supporting surface for projecting virtual content in the air.

Drone as a Mid-air Display

Researchers have studied drones as a self-levitating floating display to share information among multiple people.

. Specific Related Work However, such a complex system does not allow direct user interaction at a room scale. In BitDrones [], the authors considered each nano-quadcopter as a voxel, and by combing multiple of them, it would be possible to create high-resolution D tangible displays in the future (see Figure .). They also used drones to carry widgets elements. In summary, many authors explored drones as a promising Extending Physical Spaces in SAR using Projection on a Drone approach to display mid-air information. We also continued to pursue this exploration. On the other hand, none of these works investigated the drone as an extension of the augmented physical scene in SAR environments, as we did.

. DroneSAR

The overall motivation behind DroneSAR is to extend and enhance the projection space around the augmented physical scene, as illustrated in Figure .. To do so, we mounted a small projection screen on a drone whose position can be controlled in real-time either by the system or by the user. This drone panel acts as a D planar surface along the display continuum []. It adds physical space to the scene when needed without modifying the actual geometry of the physical objects. It allows displaying virtual content that would be difficult to display in the SAR scene otherwise. . DroneSAR Embedding widgets within a SAR is challenging, as mentioned in the introduction section. Prior works proposed to provide widget elements in SAR either on the surface of a table [], on a tracked panel [], on the floor [], or via a handheld tablet device []. These approaches solve the problem partially. However, they incline to disconnect the UI elements from the observed augmented scene.

In our approach, we can display visual content on a flat screen almost anywhere around the physical objects. This approach has several advantages. First, compared to the direct projection on an object, the projection quality does not depend on the geometry and material of the physical scene, which ensures good visualization of the widgets. Second, the user can concentrate on the region of interest without dividing their attention with a second area of interaction (i.e., mobile phone, tablet, etc.). Third, they can position the widgets at specific D locations, which can be at a distance. The proposed technique allows them to see the widgets in their D spatial contexts. Users will have the impression that projected content on the drone is always semantically linked to the augmented physical surfaces. Finally, several users are able to perceive the same information at the same time; this favors collaborative work. This work describes three possible ways to support for D widgets in the SAR context to enhance the interactive experience. However, many other functionalities could be imagined, where DroneSAR brings some of the standard desktop applications within the realm of SAR environments.

Displaying Annotations in Mid-air

In mobile or head-mounted AR applications, view management is an important part of designing intuitive user interfaces. This is about the spatial layout of D virtual annotations (i.e., text, image, video) in the view plane for real-world objects to show in-situ information to users.

In a similar way, adding annotations in SAR will enrich the user experience, but the placement of labels associated with the augmented physical world is not trivial because of its non-planar and textured projection surface. To address this, DroneSAR allows projecting virtual annotations on the drone, independently of the projection surface. While displaying the label the air, users can position the drone next to the physical object using a handheld controller to create a link between the annotation and the region of interest (ROI) in the physical space. They also have the flexibility to position the drone automatically defined by the application. Moreover, our system enables users to interact with those projected labels with the controller input buttons. If it is a text or an image, they can use the controller trackpad to modify its orientation. In the case of video, they can play or pause it with the trigger button. To display labels, we implemented a label widget. As described in Figure .(A), when the label 'chimney' needs to be displayed, the drone automatically (i.e., in a system-defined way) comes close to the house chimney and hovers there. In the same way, to point at a specific location in mid-air, we projected a cursor image on the drone panel, and using the trackpad, users change its orientation (see Figure .

Providing Interactive Tools

In SAR, users often act as passive viewers. It would be interesting to provide interactive tools to them to play with the virtual augmentation on physical objects dynamically. Inspired by 'dynamic shader lamps' [], we augmented the drone panel with several virtual tools. Users can select a tool by pointing at it using a controller. Once selected, the controller becomes the proxy of that tool and enables it to perform a tool-specific operation on the augmented content. For example, a user can select a measuring tool from the drone panel main menu, which is shown in Figure .(A). As illustrated in Figure .(B), the participants draw a line on the augmented house using the controller trigger button, . DroneSAR and the measured length is on the drone panel. It can be easily extended to a painting application where the drone panel will be augmented with different tools (e.g., color palette, brushstroke, etc). Furthermore, instead of providing a GUI of virtual tools, the drone itself can act as a proxy of a particular tool too. By moving the drone with a controller, users accomplish that tool function. To exemplify this, we provide a light source tool. In this case, the drone acts as a proxy of the virtual light source. Users can interactively modify the position of the light using a grab gesture, which would be difficult to perform without the feedback of the mid-air position that the drone provides. The appearance of the house is modified accordingly when they move the light from right to left (see Figure .(A & B)). This provides a tangible visualization of a non-physical object which is inspired by the 'Urp' project [].

Supporting Different Viewpoints

Another interesting feature of DroneSAR is to display an interactive D view of the observed augmented object close to the area of interest. Indeed, even if SAR environments have various interesting advantages, their physicality also implies strong limitations compared to purely virtual environments. It is not feasible to see the augmented physical objects from the top or back view, and the scale of the objects always remains fixed. Inspired by the concept of One Reality [] that combines SAR and VR for adding flexibility in physical worlds, we propose an approach where DroneSAR is used as a contextual D interactive viewer. The participants can see the house from various angles and at different scales by using the controller trackpad and trigger button while keeping anchored in the physical environment. Hence, they can easily link the real-augmented object and its virtual counterpart (see Figure .

. Implementation

Our system was comprised of a projector, a small lightweight drone, a controller, and a motion tracking system; the technical components were accompanied by a physical mock-up for demonstration purposes. In the following, we describe the individual components and how they are interconnected to implement the overall system ( see Figure .).

DroneSAR System

All components of our system were controlled from an application created using UnityD ., running on a Windows workstation with an Intel i-processor, GB of RAM, and an NVIDIA GeForce GTX . Each of the physical elements of the scene was digitized manually using OnShape . This application handles SAR augmentation, drone navigation, and user interaction.

Tracking System

The tracking was performed in a secondary Windows PC, running Motive . software † over the Ethernet. It samples with Hz at sub-millimeter accuracy. The setup was comprised of Flex-cameras placed above the interaction volume, covering an interaction space of m x m x m, and tracking all dynamic elements, including the drone. The drone can then hover anywhere inside this volume.

In order to support a comfortable interaction with the projected contents, we used a HTC VIVE controller, which was tracked by two VIVE lighthouses. The calibration between Optitrack space and HTC VIVE space was computed using a least-squares fitting algorithm to optimize transformation parameters (translation and rotation) [][]. This was done using four custom calibration objects (see Figure .). The resulting HTC/Optitrack calibration has an error under mm for the whole interaction volume. To avoid infrared interference, we also synchronized the OptiTrack cameras with the HTC lighthouses ‡ .

Projector Calibration

We used an LG PFG projector to augment our physical world with virtual information. To maximize the accuracy over the projection volume, the projector was manually calibrated by measuring its intrinsic parameters under controlled conditions. This was achieved by placing the projector perpendicular to a flat vertical surface, and then measuring the distance from the lens to the surface, the dimensions of the projected image, and the vertical offset between the center of the lens and the center of the projected image. The extrinsic information was obtained via OptiTrack.

Hardware

We chose a commercially available Parrot mambo quadcopter § as it is less noisy than bigger drones and safe enough to fly in an indoor environment close to people. It was powered by a mAh LiPo battery, providing approximately min of flight time without any attached accessories. To increase its payload capacity, we removed its camera but kept the propeller guards attached for safety reasons. For projection on the drone, we attached two white panels (size: cm x cm) made out of paper on both sides, and the maximum weight of these two panels was grams. We also put five retro-reflective markers on the drone for tracking. The total drone weight was around grams, with a flight time between mins to mins. It was connected to our UnityD application via Bluetooth low energy (BLE) by a middle-ware running on a Raspberry Pi.

Drone Navigation

Drone navigation was controlled using a discrete PID controller to follow trajectories obtained via A* pathfinding algorithm [] over a volumetric grid segmentation of the interaction space. Figure . illustrates overall drone navigation module. The following subsections detail this process.

Space Discretization

To define navigation paths the physical scene, we first discretized the space on a regular grid (cell diameter = cm). Based on the physical object's position, each cell of the grid was flagged as either solid or empty (see Figure .(B)). Once a cell was detected as a solid cell with static content, it did not update anymore, while the rest of the cells were updating in real-time. To prevent the drone from flying under physical objects (e.g., under the table), all cells under a solid one were marked as solid too. We found that the drone airflow interacts differently with the available surfaces, causing more or less turbulence depending on their geometry. This created a required minimum empty volume of cm in diameter to consider a cell safe (see Figure .(C)). Then, we categorized the complete space into 'safe' and 'unsafe' cells.

Path Finding and Following

With a discretization of space, it was then possible to use a navigation algorithm. We utilized a simple volumetric A* algorithm prioritizing vertical movements to obtain the navigation waypoints (see Figure .(D)). Given that the drone was controlled via yaw, pitch, roll commands, we implemented a positional PID corrector (proportional, integral, derivative) to control it with D positions. With this corrector, we continuously reduced the distance error between the drone position and waypoint, and at the same time, we converted the command into yaw, pitch, roll movements. In order to avoid oscillations, we established a dead zone threshold of cm (i.e., the drone was considered at the target location if the distance was under cm).

User Interaction

It has two parts, as described below.

Drone positioning

With the handheld controller, users were able to position the drone almost anywhere they wanted inside the safe area of the tracked volume. Our setup had two modes for this: In automatic mode, the target position was system-defined, and the drone was flying to that target following the pathfinding module without any intervention from the users' side.

On the other hand, guided mode allowed users to position the drone manually by pressing the grab button on the controller. While grabbed, the drone movements were mapped oneto-one to the controller movements. To avoid collisions, the displacement was performed via the pathfinding moduleif users attempted to position the drone beyond its safety boundary, our system warned them via vibration from the controller, while clipping the displacement to the nearest safe position. In our current implementation, users were only allowed to adjust drone rotation along Y-axis.

. Drone Positioning Evaluation

Interacting with the augmented

Users could point and select virtual content projected on the physical objects (i.e., house mock-up) using ray-casting from the controller. However, we found that indirect pointing with the controller trackpad was more comfortable to interact with the graphical widgets projected on the drone panels due to its smaller size and limited drone stability.

. Drone Positioning Evaluation

We evaluated the drone positioning accuracy and how close users can bring the drone to physical objects. For that, we chose four different locations of different surface geometries in our physical mock-up -floor, side of the house, top of the house, and in front of the house (see Figure .). We considered five target positions at the distance of cm, cm, cm, cm, and cm from each surface. When the drone was hovering less than cm away from the target (i.e., dead zone), we recorded its position at Hz for seconds. We repeated this process three times for each target position and computed the mean positional error. Note that we did not consider the rotational error between the target and the drone. When the drone enters the dead zone, the internal PID controller tries to hover the drone inside this zone. The results showed that, on average, the position error was .±. cm. This error happened due to the switching between the positional PID controller and the drone internal PID loop. From Figure ., it is observable that there was no strongly error decreasing pattern when the target distance was increasing. Overall, the minimum positional error was cm while the maximum was cm; the deviation seems to be influenced more by the proximity to the dead zone than to the distance to the obstacle (i.e., higher deviation when the error approaches cm on average). However, we noticed sufficiently stable hovering when the drone was at least cm away from the physical surface. This amount of positional error is acceptable in our application as the virtual augmentation on the drone always follows its real position (tracked drone), not the target location.

. Limitations

We have shown that combing SAR with a drone opens new opportunities to extend the interaction space. However, even when promising, our approach is not without limitations.

The drone can hover almost perfectly in mid-air (with ± cm positional error) when there are no physical objects nearby. This amount of positional error is acceptable as the virtual augmentation on the drone always follows its real position (tracked drone), not the target location. On the other hand, bringing the drone close to the physical objects (e.g., sides or exactly on top of the house) is difficult due to its downwards airflow. Drone hovering was sufficiently stable when it was at least cm away from the physical surface.

The size of the panel attached to the drone is quite small ( cm x cm) as we restricted ourselves to use a lightweight drone for users' safety. The small size of the drone panel restricts us to project only limited content on it. In order to extend the display surface, it could be possible to combine multiple of these drones to create a bigger surface dynamically. We also envision that there will be improvements in the drone payload capacity and battery life with less noise in the coming years. Blimps might also be an alternative option in this direction, trading speed for projection surface and stability.

The drone does not consider the presence while computes a path to reach the goal. In the future, our navigation module should take into account the human position.

Moreover, as we use a front projector, shadows of the user and the drone are inevitable. This could be overcome by using multiple projectors set up [].

.

Conclusion

SAR is strongly linked to the related physical scenes. This makes the user experience with SAR unique, and it provides numerous advantages compared to see-through AR approaches. On the other hand, the physical nature of SAR also induces limitations. We have introduced DroneSAR to overcome some of these limitations. By extending the space on which digital content can be displayed, we have proposed a way to extract the augmentation from the physical constraints. The mid-air drone augmentation is always contextually connected to the physical D augmented scene. In our approach, we have explored a set of interactions where users keep immersed in the augmented scene, and they can benefit from additional displays functionalities. This is a proof of concept of how to extend the physical space of SAR using drone augmentation. Once the technology is stable enough, we will conduct a set of user studies to assess the potential and limits of such an extended SAR environment compared to traditional smartphone or tablet-based augmented reality systems.

Conclusion and Future

Work

In this chapter, we will first revisit our quickly and then present future research directions.

. Revisiting Thesis Contributions

Chapter & Chapter of this dissertation present our two main contributions.

We envision that AR-HMDs are not only designed for manipulating immersive D information but also for interacting with D content (text, image, video). The major advantage of AR-HMDs over desktop computing is that we can take our workspace with us anywhere we want. In contrast, its traditional input techniques (such as hand tracking, head/eye-gaze, and voice) are cumbersome to use in on-thego situations. Inspired by previous research, we agree that smartphone has enormous potential as an input device for AR-HMDs when users are not in their desk. In Chapter , we investigated how to use this smartphone for selecting text in AR-HMDs. We chose text selection as a use-case because a significant portion of our day-to-day productivity work involves text manipulations, and text selection is often the first step before editing. We efficiently perform this text selection task with a keyboard and mouse/trackpad, but it becomes difficult to achieve in AR-HMDs as its current input techniques hardly support character level precision. To address this issue, we proposed smartphone-based text selection techniques -continuous touch, discrete touch, spatial movement, and raycasting. Next, we compared these techniques in a user study where users have to select text at various granularity levels. Our results suggested that continuous touch (where a smartphone was used as a trackpad) outperformed the other three techniques in terms of task completion time, accuracy, and user preference.

In Chapter , we looked into how to embed graphical widgets in spatial augmented reality. Traditional ways of supporting Conclusion and Future Work these widget elements via a handheld tracked panel, mobile devices, on the table surface) either limit users' mobility in the augmented space or need to shift their attention frequently. To overcome these issues, we developed DroneSAR to provide interactive graphical widgets in SAR like a floating menu in mid-air using projection on a drone-mounted panel. Using a handheld controller, users were able to control the drone position and interact with the scene dynamically. In particular, we presented three ways to embed widgets using a drone in a SAR environment -displaying annotations in mid-air, providing interactive tools, supporting different viewpoints.

.

Future Work

In Chapter , we focused on the text selection part. In the future, it would be interesting to explore a more global usage scenario such as a text editing interface in AR-HMDs using smartphone-based input where users need to perform other interaction tasks such as text input and commands execution simultaneously. We also need to compare phonebased techniques to other input techniques like hand tracking, head/eye gaze, and voice commands. Furthermore, we only considered standing condition, but it would be interesting to study text selection performance while walking. Note that, there is text readability issue in HMDs due to the blur motion caused by vertical shock while walking [].

Chapter proposed a novel way to bring a drone into the SAR space. This work can be further extended in a few interesting ways. For example, in our DroneSAR system, the size of the drone panel is quite small. Due to this, we are only able to project limited user interface elements on the drone. In this context, it would be interesting to explore spatial menu concepts in the air like virtual shelves [] and m+pSpaces []. Further, we can think about handsfree interaction where users will directly grab the drone to position it [][] instead of using a handheld controller. They can also perform direct touch interaction on the drone panels as well as on the augmented surfaces for manipulating virtual contents. Lastly, we need to conduct a user study to investigate the benefits of such drone-projected floating widgets compared to traditional approaches.

. Concluding Remarks

We can imagine bringing text selection task in SAR by projecting textual content on the drone panel (but there is a limitation due to its small panel size) and using the smartphone as an input controller.

. Concluding Remarks

To sum up, some basic tasks (such as text input, text editing, menu selection, interacting with graphical widgets, etc.) are straightforward to do on our desktop but hard to achieve in an immersive AR environment. To overcome these challenges, we have contributed to enhancing the interaction space of the two most commonly used forms of augmented reality. Certainly, proposed interaction techniques are not generic solutions at all. It depends on the application context. Nevertheless, we have just scratched the surface of what is possible with a smartphone and a drone in the context of immersive augmented reality.

Beyond our thesis contribution, we are also working on the commands selection interface for AR-HMDs. While manipulating D objects or sketching in AR-HMDs, we often need to issue commands to save the file, change from a brush tool to an eraser tool, changing brush-width, and so on. In a desktop computer, we input commands using D UI elements such as toolbars, pull-down menus, pop-up menus, or function keys on a keyboard. On the other hand, designing menus for supporting commands in AR-HMDs is not trivial as these devices have a limited field of view. Displaying menus in an already narrow FOV creates occlusion to the current content. To avoid that, the traditional approach is to use a hand menu (where the menu is attached to the user's non-dominant hand) or a world-locked floating menu (where the D menu is located in the physical space). In both of these menu layouts, users require to explicitly switch their attention from the ongoing tasks to select a menu item. Hence, we asked this questionis it possible to issue a command rapidly on an AR-HMD without explicitly shifting the user's attention from the primary task? To address this issue, so far, we have developed a very initial proof-of-concept of a novel head-referenced eyes-free menu layout where menu items (buttons and sliders) are located https://docs.microsoft.com/en-us/windows/mixed-reality/ design/hand-menu

Conclusion and Future Work

outside around the FOV. Small dots are shown on FOV border as visual cues to understand the spatial location of each command. After an initial learning phase, users build a spatial memory of command locations outside the display FOV. This is similar to the way we remember shortcuts for commands on the desktop. Then, users can rapidly select a command using their non-dominant hand with a glance instead of explicit attention switching. The initial implementation is working nicely. Currently, we are planning for a user study to understand its feasibility.

  (A), lorsque l'étiquette 'cheminée' doit être affichée, le drone s'approche automatiquement (c'est-à-dire d'une manière définie par le système) de la cheminée de la maison et y fait du surplace. De même, pour pointer un endroit spécifique dans les airs, nous projetons l'image d'un curseur sur le panneau du drone, et à l'aide du trackpad, les utilisateurs modifient son orientation (voir la Figure .(B)). Enfin, DroneSAR peut également afficher une vidéo D dans la scène, comme le montre la Figure .(C).

.

  (A) Collaborating through Augmented Surfaces []; (B) DigitalDesk prototype where users selected a printed number on a piece of paper and put that number into the calculator application []; (C) PaperWindows prototype with three pages []. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (A) The physical model of the Taj Mahal. (B) This model is illuminated with projectors to simulate different materials []. . . . . . . . . . . . . . . . . . . . . IllumiRoom is a proof-of-concept system that augments the physical environment surrounding a television to enhance interactive experiences []. (a) With a D scan of the physical environment, we can (b) directly extend the FOV of the game, (c) selectively render scene elements. . . . . . . . . . . . . . . . . . . . A docking task completed with PinNPivot []: (a) Source and target. (b) DOF manipulation quickly turns the object around. (c) Scaled DOF translation accurately places the spout's tip. (d) A pin is created. (e) It is locked. (f) The object is quickly rotated in DOF. (g) It is accurately rotated in scaled DOF. (h) A second pin is created. (i) It is locked, and a ring appears. (j) The object is rotated in scaled DOF. When the target turns yellow, it indicates a good fit. . . Two example gestures for authoring animations []: (a-b) a gesture manipulating the direction, spread, and randomness of smoke emission; (c-d) a gesture directly bending an object to describe a follow-through behaviour. . . . . . . . . Selected mid-air mode-switching techniques []: (A) non dominant fist; (B) non dominant palm; (C) hand in field of view; (D) touch head; (E) dominant fist; (F) dominant palm; (G) point; (H) orientated pinch; (I) middle finger pinch. . MRTouch enables touch interaction in AR-HMDs []. (a) When a user approaches a surface, MRTouch detects the surface and (b) presents a virtual indicator. (c) The user touch-drags directly on the surface to (d) create a launcher and start an app. (e) In this app, the user uses touch to precisely rotate a D model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A user is entering text using DepthText in a VR-HMD. The user has entered the word "excellent" by performing short forward movements towards the depth dimension (z-axis) with an acceleration speed larger than threshold 0 0 . . . . . Head gestures set for different commands []. The movement of head is indicated by the arrows. "◊" represents the repeating of the action for twice. "s" is an illustration for a dwell. . . . . . . . . . . . . . . . . . . . . . . . . . . . Outline Pursuits [] support selection in occluded D scenes. A: The user points at an object of interest, but the selection is ambiguous due to occlusion by other objects. B: Potential targets are outlined, with each outline presenting a moving stimulus that the user can follow with their eye-gaze. C: Matching of the user's smooth pursuit eye movement completes the selection. Note that outline pursuits can augment manual pointing as shown or support hands-free input using the head or gaze for initial pointing. . . . . . . . . . . . . . . . . . . Three interaction methods proposed by Liu et al. [] to manipulate D object rotation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The eyemR-Vis prototype system, showing an AR user (HoloLens) sharing gaze cues with a VR user (HTC Vive Pro Eye) and vice-versa. . . . . . . . . . . . PalmType [] enables text input for (a) smart glasses using a QWERTY keyboard interface, by using (b) wrist-worn sensors to detect the pointing finger's position and taps, and (c) displaying a virtual keyboard with highlighted keys via the display of the glasses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (a) Leveraging the palm as a gesture interface in PalmGesture []. (b) Tracking bright regions in the infrared camera image. (c) Drawing an email symbol to check emails on Google Glass. . . . . . . . . . . . . . . . . . . . . . . . . . . . . (A) The menu widget concept on forearm. A user is interacting with the menu using (B) touch (C) drag (D) slide and (E) rotate gestures. . . . . . . . . . . . . . (A) Word-gesture and tap based typing around the thigh []. (B) Foot-taps as a direct and indirect input modality for interacting with HMDs []. (C) A user needs to go the North-East direction using DMove technique and a selection is made when the user (nearly) completes the action []. . . . . . . . . . . . . . . Pointing tasks in VR (top) and AR (bottom) using a mouse, a VR controller, and a D pen []. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A participant is holding the pen with (A) a precision grip and (B) a power grip. . PintAR [] in use. (A) Designer sketching an interface element on the tablet using a pen. (B) Designer placing sketched element in the environment using an Air tap. (C) AR interface element placed side-by-side with real display. . . Participants were drawing a stroke around (A) a physical object and (B) a virtual object []. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (A) The overall setup of a stylus-based text input system in a CAVE environment []. (B) The Tilt-Type interface. (C) The Arc-Type interface. . . . . . . . . . . . . MultiFi [] widgets crossing device boundaries based on proxemics dimensions (left), e.g., middle: ring menu on a smartwatch (SW) with head-mounted display (HMD) or right: soft keyboard with full-screen input area on a handheld device and HMD. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The BISHARE [] design space of joint interactions between a smartphone and augmented reality head-mounted display. Each cell contains a single example of joint interaction, but represents a broader class of interaction techniques that may be possible. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (A) Understanding window management interactions using an AR-HMD + smartphone interface []. (B) Enlarging smartphone display with an AR-HMD []. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data visualization using mobile devices and Augmented Reality head-mounted displays []: (a) Envisioned usage scenario; (b) D scatter-plot extended with superimposed D trajectories/paths; (c) D wall visualization in AR aligned with the mobile device; (d) Use of AR for seamless display extension around a geographic map; (e) Combining visualizations with an AR view between the devices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Left: D data spaces can be explored by (a) D panning and (b) zooming relative to their fixed presentation space. Right: A user wearing a HoloLens explores such a D data space with smartphone-based proposed interaction techniques []. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Typing on a midair auto-correcting keyboard with word predictions (left) vs. speaking a sentence and then correcting any speech recognition errors (right) [].

  (B & C)) [][][].Users can also continue on-the-go learning by watching the video in AR-HMDs. Ram and Zhao [] studied different video presentation styles on AR-HMDs to distribute better user's attention between learning and walking. They introduced Layered Serial Visual Presentation (LSVP) style for future AR-HMD based on-the-go video learning. Further,Lu et al. [] proposed Glanceable AR, an information access paradigm for AR-HMDs in walking scenarios. In this paradigm, applications (e.g., weather app, email, calendar, etc.) reside outside FOV to stay unobtrusive and can be accessed by a quick glance (e.g., eye-glance, head-glance, and gaze-summon) whenever needed. In this way, AR-HMDs provide us the unprecedented ability to visualize digital information anywhere seamlessly.

Figure . :

 . Figure .: Comparing physical monitors, virtual AR monitors, and a hybrid combination of both for productivity tasks []: (a) Physical condition had three monitors side-by-side; (b) Virtual condition had three monitors rendered through HoloLens; (c) Hybrid condition combined a central physical monitor with two peripheral virtual monitors.

Figure . :

 . Figure .: (A) activity: doodling, cognitive load: low; (B) activity: brainstorming, cognitive load: medium; (C) activity: reading scientific paper, cognitive load: high. Based on the current context, Mixed Reality interfaces adapt automatically []. For example, the interface shows more elements in more detail in the case of a low cognitive load task (A). Whereas increased cognitive load leads to a minimal UI with fewer elements at lower levels of detail (B to C).

Figure . :

 . Figure .: (A) In an AR workspace, virtual windows are embedded into the user's nearby surfaces []; (B & C) AR interfaces adapt dynamically based on the user's walking and physical environment [].

Figure . :

 . Figure .: (A) Collaborating through Augmented Surfaces []; (B) DigitalDesk prototype where users selected a printed number on a piece of paper and put that number into the calculator application []; (C) PaperWindows prototype with three pages [].

Figure . :

 . Figure .: (A) The physical model of the Taj Mahal. (B) This model is illuminated with projectors to simulate different materials [].

Figure . :

 . Figure .: IllumiRoom is a proof-of-concept system that augments the physical environment surrounding a television to enhance interactive experiences []. (a) With a D scan of the physical environment, we can (b) directly extend the FOV of the game, (c) selectively render scene elements.

Figure . :

 . Figure .: A docking task completed with PinNPivot []: (a) Source and target. (b) DOF manipulation quickly turns the object around. (c) Scaled DOF translation accurately places the spout's tip. (d) A pin is created. (e) It is locked. (f) The object is quickly rotated in DOF. (g) It is accurately rotated in scaled DOF. (h) A second pin is created. (i) It is locked, and a ring appears. (j) The object is rotated in scaled DOF. When the target turns yellow, it indicates a good fit.

Figure . :

 . Figure .: Two example gestures for authoring animations []: (ab) a gesture manipulating the direction, spread, and randomness of smoke emission;(c-d) a gesture directly bending an object to describe a follow-through behaviour.

Figure . :

 . Figure .: Selected mid-air mode-switching techniques []: (A) non dominant fist; (B) non dominant palm; (C) hand in field of view; (D) touch head; (E) dominant fist; (F) dominant palm; (G) point; (H) orientated pinch; (I) middle finger pinch.

Figure . :

 . Figure .: MRTouch enables touch interaction in AR-HMDs []. (a) When a user approaches a surface, MRTouch detects the surface and (b) presents a virtual indicator. (c) The user touch-drags directly on the surface to (d) create a launcher and start an app. (e) In this app, the user uses touch to precisely rotate a D model.

Figure . :

 . Figure .: A user is entering text using DepthText in a VR-HMD. The user has entered the word "excellent" by performing short forward movements towards the depth dimension (z-axis) with an acceleration speed larger than threshold 0 0

Figure . :

 . Figure .: Head gestures set for different commands []. The movement of head is indicated by the arrows. "◊" represents the repeating of the action for twice. "s" is an illustration for a dwell.

Figure . :

 . Figure .: Outline Pursuits [] support selection in occluded D scenes. A: The user points at an object of interest, but the selection is ambiguous due to occlusion by other objects. B: Potential targets are outlined, with each outline presenting a moving stimulus that the user can follow with their eye-gaze. C: Matching of the user's smooth pursuit eye movement completes the selection. Note that outline pursuits can augment manual pointing as shown or support hands-free input using the head or gaze for initial pointing.

Figure . :

 . Figure .: Three interaction methods proposed by Liu et al. [] to manipulate D object rotation.

Figure . :

 . Figure .: The eyemR-Vis prototype system, showing an AR user (HoloLens) sharing gaze cues with a VR user (HTC Vive Pro Eye) and vice-versa.

Figure . :

 . Figure .: PalmType [] enables text input for (a) smart glasses using a QWERTY keyboard interface, by using (b) wristworn sensors to detect the pointing finger's position and taps, and (c) displaying a virtual keyboard with highlighted keys via the display of the glasses.

Figure . :

 . Figure .: (a) Leveraging the palm as a gesture interface in PalmGesture []. (b) Tracking bright regions in the infrared camera image. (c) Drawing an email symbol to check emails on Google Glass.

Figure

  Figure .: (A) The menu widget concept on forearm. A user is interacting with the menu using (B) touch (C) drag (D) slide and (E) rotate gestures.

Figure

  Figure .: (A) Word-gesture and tap based typing around the thigh []. (B) Foot-taps as a direct and indirect input modality for interacting with HMDs []. (C) A user needs to go the North-East direction using DMove technique and a selection is made when the user (nearly) completes the action [].

Figure . :

 . Figure .: Pointing tasks in VR (top) and AR (bottom) using a mouse, a VR controller, and a D pen [].

Figure

  Figure .: A participant is holding the pen with (A) a precision grip and (B) a power grip.

Figure . :

 . Figure .: PintAR [] in use. (A) Designer sketching an interface element on the tablet using a pen. (B) Designer placing sketched element in the environment using an Air tap. (C) AR interface element placed side-by-side with real display.

Figure . :

 : Figure .: Participants were drawing a stroke around (A) a physical object and (B) a virtual object [].

Figure

  Figure .: (A) The overall setup of a stylus-based text input system in a CAVE environment []. (B) The Tilt-Type interface. (C) The Arc-Type interface.

  ). They demonstrated how it could be used to manipulate D objects []. Similarly, Ren et al. [] demonstrated how it can be used to perform windows management (see Figure .(A)). In VESAD [], Normand et al. used AR to directly extend the smartphone display (see Figure .(B)). Other work combined AR-HMDs and tablets for immersive visual data analysis (see Figure .) [][].

Figure . :

 . Figure .: MultiFi [] widgets crossing device boundaries based on proxemics dimensions (left), e.g., middle: ring menu on a smartwatch (SW) with head-mounted display (HMD) or right: soft keyboard with full-screen input area on a handheld device and HMD.

Figure . :

 . Figure .:The BISHARE [] design space of joint interactions between a smartphone and augmented reality head-mounted display. Each cell contains a single example of joint interaction, but represents a broader class of interaction techniques that may be possible.

Figure . :

 . Figure .: Data visualization using mobile devices and Augmented Reality head-mounted displays []: (a) Envisioned usage scenario; (b) D scatter-plot extended with superimposed D trajectories/paths; (c) D wall visualization in AR aligned with the mobile device; (d) Use of AR for seamless display extension around a geographic map; (e) Combining visualizations with an AR view between the devices.

Figure . :

 . Figure .: Left: D data spaces can be explored by (a) D panning and (b) zooming relative to their fixed presentation space. Right: A user wearing a HoloLens explores such a D data space with smartphone-based proposed interaction techniques [].

  , Bolt [] introduced a seminal work put-that-there interface which allows users to place objects inside a media room through a combination of speech and pointing gestures. Later this combination has been used in virtual object selection and manipulation in AR [][][]. Piumsomboon et al. [] studied the use of gestures and speech vs. only gestures for manipulating D objects in AR.

Figure . :

 . Figure .: Typing on a midair auto-correcting keyboard with word predictions (left) vs. speaking a sentence and then correcting any speech recognition errors (right) []. Users correct errors by selecting word alternatives proposed by the speech recognizer or by typing on the virtual keyboard.

Figure . :

 . Figure .: Gaze + Pinch interactions unify a user's eye gaze and hand input: look at the target, and manipulate it (a); virtual reality users can utilise free hand direct manipulation (b) to virtual objects at a distance in intuitive and fluid ways (c).

Figure . :

 . Figure .: Radi-Eye [] in a smart home environment for control of appliances. A: The user turns on the lamp via a toggle selection with minimal effort using only gaze (orange) and head (red) movements. B: Selection can be expanded to subsequent head-controlled continuous interaction to adjust the light colour via a slider. C: Gaze-triggered nested levels support a large number of widgets and easy selection of one of the multiple preset lighting modes. The widgets enabled via Radi-Eye allow a high level of hands-free and at-a-distance control of objects from any position.

  Related Workwith the HMD user and become part of the AR experience (see Figure .) [][][].

Figure . :

 . Figure .: Extending large interactive display visualizations with Augmented Reality []. (A) Two analysts are working on data visualization tasks. (B) Displaying AR Brushing and Linking, Embedded AR Visualizations, and Extended Axis Views. (C) Hinged Visualizations to improve the perception of remote content. (D) Curved AR Screen is providing an overview of the entire display.

Figure . :

 . Figure .: ShARe [] is a modified AR-HMD consisting of a projector and a servo motor attached to its top (a). This allows people in the surrounding to perceive the digital content through projection on a table (b, f) or on a wall (e) and interact via finger-based gestures (c) or marker-based touch (d).

Figure . :

 . Figure .: Dynamic shader lamps for applying virtual paint and textures to real objects simply by direct physical manipulation of the object and a "paint brush" stylus. [].

Figure . :

 . Figure .: Motivating surface interaction examples of the "Build, Map, Play" process proposed by Jones et al. []: (A) a virtual miniature golf game, (B) a two-player tank game, (C) a photo viewer. (D)The user selects a menu item with a stylus from a surface adaptive radial menu that adapts to the surface that it is displayed on.

Figure . :

 . Figure .: Augmented handheld tool is providing virtual widgets for a paint application [].

Figure . :

 . Figure .: Examples of ad hoc controls, in which the wooden cube is the handle for slider interaction []. (A) Video-editing controls on their own. (B) The same controls supplementing the existing standard controls.

Figure . :

 . Figure .: (A) The concept of floor projected UI in a collaborative SAR environment []. (B) Experimental setup with the extended floor UI. (C) Participants discussing a virtual scene. (D) The virtual scene from the master's (the user who controls the perspective) point of view.

Figure . :

 . Figure .: MirageTable is a curved projection-based augmented reality system (A), which digitizes any object on the surface (B), presenting correct perspective views accounting for real objects (C) and supporting freehand physics-based interactions (D).

  , Park et al. [] integrated mobile devices in projection-based AR to afford user interfaces to design interiors effectively. It is described in Figure .(B). Gervais et al. [] found that a standard mouse can be a good fit for pointing in a desktop SAR setup (see Figure .(A)).

Figure . :

 . Figure .: Mobile phone pointing techniques in SAR []: (A) viewport, where targets are captured by a camera-like view; (B) raycasting, where targets are pointed at; (C) tangible, where targets are directly contacted.

Figure . :

 . Figure .: (A) The user is moving a cursor (represented in blue) to a target (represented in red) on an augmented object using a standard mouse []. (B) In DesignAR system, a user is manipulating projected content for interior design using a tablet [].

  (middle)). Then, for further extension, users put HMDs, which provide a virtual replica of the physical scene, taking advantage of the freedom of virtual spaces without losing connection with the environment (see Figure.

  (right)).

Figure . :

 . Figure .: Example scenes to describe One Reality framework []: volcano mock-up made out of sand (top), D printed Toyota engine (bottom). Each scene can be interacted with different display technologies: spatial augmentation (left), see-through displays (middle), and opaque head mounted displays (right).

Figure . :

 . Figure .: Force-assisted text selection technique []: (a) the evaluation setup; (b-c) a force-assisted button interface on an iPhone to select the textual contents on a distal display.

Figure . :

 . Figure .: EYEditor interactions []: User sees the text on a smart glass, sentence-by-sentence. In the Re-speaking mode, correction is achieved by re-speaking over the text and a hand-controller is used to navigate between sentences. Users can enter the Select-to-Edit mode to make fine-grained selections on the text and then speak to modify the selected text.

Figure . :

 . Figure .: Distant freehand pointing and clicking on a very large, high resolution displays []: (A) raycasting (B) relative pointing with clutching (C) hybrid ray-to-relative pointing.

Figure . :

 . Figure .: The workflow of using Text Pin to select text []. (a) Widget A appears at the point of touch with a magnifying lens displaying above it; (b) Widget A consists of a handle and a circle (the magnifying lens above it disappears once the finger is lifted off the screen); (c) Finger clicks on the circle to fix one end of selection; (d) Widget B appears when finger clicks on the opposite end of selection (note we do not show the magnifying lens above Widget B for a clearer illustration of Widget B); (e) The user clicks on the circle of Widget B and (f) repeats step (a-d) to select the next non-adjacent text.

Figure . :

 . Figure .: Some Gedit editing gestures on the smartphone keypad in one-handed use []. All gestures start from the right edge: (A) a flick left to select a word; (B) a clockwise ring gesture selects characters to the right of the text cursor; (C) the copy gesture 'C'; (D) the paste gesture 'V'; (E) clockwise and counterclockwise ring gesture for cursor control; (F) the left thumb swipes from the left edge to trigger editing mode, and then editing gestures can be performed by the right thumb; the ring gesture in editing mode performs text selection; the user simply lifts the left thumb to stop editing.

  ) []. The latter was more efficient than the first one and the default technique. With BezelCopy [], a gesture on the bezel of the phone allow for a first rough selection that can be refined after. Finally, other solutions used a non-traditional smartphone. Le et al. used a fully touch-sensitive device to allow users to perform gestures on the back of the device [] to select text. Gaze N'Touch [] used gaze to define the start and end of the selection (see Figure .). Goguey et al. explored the use of a force-sensitive screen to control the selection (see Figure .) [], and Eady and Girouard used a deformable screen to explore the use of the bending of the screen [].

Figure . :

 . Figure .: Usage of Press & Tilt technique [].

Figure . :

 . Figure .: Usage of Press & Slide technique [].

Figure . :

 . Figure .: Force-sensitive text selection on touch devices []. (A) 'mode gauge' -using force for different selection modes (B)Example of text selection using force on a touchscreen. The user performs a long-press that displays the callout magnifier. Keeping the force in the character level, the user adjusts its position by moving her finger. She then presses harder to start the manipulation of the second cursor. If she releases her finger while the force is at the sentence level of the 'mode gauge', she will select the whole paragraph. If she releases her finger while the force is at the character level, she will only select what is between the two cursors.

Figure . :

 . Figure .: Illustration of the Gaze'N'Touch concept []. (A) Look at the starting character; (B) Touch down; (C) Look at the end character; (D) Release touch.

Figure . :

 . Figure .: Illustrations of our proposed interaction techniques: (a) continuous touch; (b) discrete touch; (c) spatial movement; (d) raycasting.

  Figure .(b)) that is highlighted in the instruction panel (right panel in Figure .(b)). The user controls a cursor (i.e., a small circular dot in red color as shown in Figure .(b)) using one of the techniques on the smartphone. Its position is always bounded by the window size. The text content was generated by Random Text Generator † † and was displayed using the Liberation Sans font with a font-size of pt (to allow a comfortable viewing from a few meters).

Figure . :

 . Figure .: (a) The overall experimental setup consisted of an HoloLens, a smartphone, and an optitrack system. (b) In the HoloLens view, a user sees two text windows. The right one is the 'instruction panel' where the subject sees the text to select. The left is the 'action panel' where the subject performs the actual selection. The cursor is shown inside a green dotted box (for illustration purpose only) on the action panel. For each text selection task, the cursor position always starts from the center of the window.

Figure . :

 . Figure .: Text selection tasks used the experiments: () word () sub-word () word to a character () four words () one sentence () paragraph to three sentences () one paragraph () two paragraphs () three paragraphs () whole text.

  .

  ). Post hoc tests showed that Continuous Touch [M = ., SD = .], Spatial Movement [M = ., SD = .], and Raycasting [M = ., SD = .] were faster than Discrete Touch [M = ., SD = .].

Figure . :

 . Figure .: Mean task completion time for our proposed four interaction techniques. Lower scores are better. Error bars show % confidence interval. Statistical significances are marked with stars (**: p < . and *: p < .).

Figure . :

 . Figure .: Mean error rate of interaction techniques. Lower scores are better. Error bars show % confidence interval. Statistical significances are marked with stars (**: p < . and *: p < .).

Figure . :

 . Figure .: Mean scores for the ranking questionnaire which are in point likert scale. Higher marks are better. Error bars show % confidence interval. Statistical significances are marked with stars (**: p < . and *: p < .).

Figure . :

 . Figure .: Mean scores for the NASA-TLX task load questionnaire which are in range of to . Lower marks are better, except for performance. Error bars show % confidence interval. Statistical significances are marked with stars (**: p < . and *: p < .).

Figure

  Figure .: (A) A physical house mock-up. (B) A drone is mounted with two white paper panels.

Figure . :

 . Figure .: An example scenario of DroneSAR. (A)The house is augmented using projection, and the main menu is composed of a set of virtual tools projected on the drone panel. (B) A user selected the 'measuring tool' application using a controller. Then, the user positions the drone at the desired location in the D space (i.e., on top of the house) and draws a line shown in blue color on the augmented house to measure its width. Finally, the measured length is displayed on the drone panel.

Figure . :

 . Figure .: The fog-display system []: (A) overall concept and (B) a prototype of it.

Figure . :

 . Figure .: The water-drop display []: (A) overall concept and a prototype setup; (B) the display is showing an image and (C) text.

Figure . :

 . Figure .: Dust []: (Top) Hardware setup; (Bottom) Example images created with this setup.

Figure . :

 . Figure .: Display drone in an outdoor setting []. (B) Drone-projected in-situ navigation instructions [].

Figure . :

 . Figure .: Different example scenarios of LightAir system []: (a) LightAir for human-drone communication (b) DronePiano application (c) D point cloud processing for DroneBall.

Figure . :

 . Figure .: A working scenario of Drone.io setup []. (A) Top view of the projected radial interface as seen by the user. (B) A user is extending his arm in a push gesture to navigate through the menu with the drone flying above. (C) A user is selecting an item in the menu.

  (A)). In ARial . Specific Related Work Texture [], the used the drone propellers as a display screen (see Figure .). Zhang et al. [] proposed a hologrammatic telepresence system by projecting a remote user's head on the drone-mounted retro-reflective cylindrical surface (see Figure .(B)). Tobita et al. [] developed a blimp-type drone-based telepresence system. Intel used drones synchronously to form the US flag [].

Figure

  Figure .: (A) iSphere prototype []. (B) LightBee [] telepresence system with two local users viewing the drone-based light field display to communicate with a person in a remote capture room (inset)

Figure . :

 . Figure .: BitDrones [] hovering in a tight formation. (B) User inspecting a remote facility with telepresence functionality provided by the drone. (C) User resizing a compound object using a bi-manual pinch gesture by moving drones.

Figure . :

 . Figure .: (Left) In SAR, the projection space is limited by the size of the physical object. (Right) DroneSAR extends this projection space (shown in yellow color) with a flying panel that can be positioned in the surround of the physical scene.

  (B)). Last but not least, DroneSAR also displays D video within the scene (see Figure .(C)).

Figure

  Figure .: (A) The drone is hovering next to the chimney to display its corresponding label. (B) A flying cursor allows participants to point at a specific location in the scene. The dotted circle in the image represents the particular location in mid-air. (C) A video explaining the history is displayed near the medieval house.

  (C)).

Figure

  Figure .: -B) The light source of our scene is at the drone hovering position. By moving the light source, the user is casting shadows on the scene. (C) An interactive D model of the mock-up displayed next to the physical one allows the user to observe the scene from another viewpoint.

Figure . :

 . Figure .: Overall architecture of DroneSAR system.

Figure . :

 . Figure .: A HTC vive with retro reflective markers.

Figure . :

 . Figure .: Drone flight control to reach the goal position.

Figure

  Figure The referenced scene (A) is decomposed into solid cells (in red) (B), then 'safe' cells (in yellow) (C). Example of way-point cells (in light green) (D).

Figure

  Figure Positioning the drone at different target locations with respect to each of the four surfaces. Here, d represents distance to the target from the surface.

Figure . :

 . Figure .: The drone positioning error to the target locations for all four surfaces. Overall error is . ± .cm.

  Hybrid condition combined a central physical monitor with two peripheral virtual monitors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (A) activity: doodling, cognitive load: low; (B) activity: brainstorming, cognitive load: medium; (C) activity: reading scientific paper, cognitive load: high. Based on the current context, Mixed Reality interfaces adapt automatically []. For example, the interface shows more elements in more detail in the case of a low

	. Comparing physical monitors, virtual AR monitors, and a hybrid combination
	of both for productivity tasks []: (a) Physical condition had three moni-
	tors side-by-side; (b) Virtual condition had three monitors rendered through
	HoloLens; (c)

cognitive load task (A). Whereas increased cognitive load leads to a minimal UI with fewer elements at lower levels of detail (B to C). . . . . . . . . . . . . . . (A) In an AR workspace, virtual windows are embedded into the user's nearby surfaces []; (B & C) AR interfaces adapt dynamically based on the user's walking and physical environment []. . . . . . . . . . . . . . . . . . . . . . . .

drone at different target locations with respect to each of the

  Radi-Eye [] in a smart home environment for control of appliances. A: The user turns on the lamp via a toggle selection with minimal effort using only gaze (orange) and head (red) movements. B: Selection can be expanded to subsequent head-controlled continuous interaction to adjust the light colour via a slider. C: Gaze-triggered nested levels support a large number of widgets and easy selection of one of the multiple preset lighting modes. The widgets enabled via Radi-Eye allow a high level of hands-free and at-a-distance control of objects from any position. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Extending large interactive display visualizations with Augmented Reality []. (A) Two analysts are working on data visualization tasks. (B) Displaying AR Brushing and Linking, Embedded AR Visualizations, and Extended Axis Views. (C) Hinged Visualizations to improve the perception of remote content. (D) Curved AR Screen is providing an overview of the entire display. . . . . . . ShARe [] is a modified AR-HMD consisting of a projector and a servo motor attached to its top (a). This allows people in the surrounding to perceive the digital content through projection on a table (b, f) or on a wall (e) and interact via finger-based gestures (c) or marker-based touch (d). . . . . . . . . . . . . . . Dynamic shader lamps for applying virtual paint and textures to real objects simply by direct physical manipulation of the object and a "paint brush" stylus. []. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Motivating surface interaction examples of the "Build, Map, Play" process proposed by Jones et al. []: (A) a virtual miniature golf game, (B) a twoplayer tank game, (C) a photo viewer. (D) The user selects a menu item with a stylus from a surface adaptive radial menu that adapts to the surface that it is displayed on. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Augmented handheld tool is providing virtual widgets for a paint application []. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Examples of ad hoc controls, in which the wooden cube is the handle for sliderinteraction []. (A) Video-editing controls on their own. (B) The same controls supplementing the existing standard controls. . . . . . . . . . . . . . . . . . . < . and *: p < .). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mean scores for the ranking questionnaire which are in point likert scale. Higher marks are better. Error bars show % confidence interval. Statistical significances are marked with stars (**: p < . and *: p < .). . . . . . . . . . . . Mean scores for the NASA-TLX task load questionnaire which are in range of to . Lower marks are better, except for performance. Error bars show % confidence interval. Statistical significances are marked with stars (**: p < . and *: p < .). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Overall architecture of DroneSAR system. . . . . . . . . . . . . . . . . . . . . . . A HTC vive tracker with retro reflective markers. . . . . . . . . . . . . . . . . . . Drone flight control to reach the goal position. . . . . . . . . . . . . . . . . . . four surfaces. Here, d represents distance to the target from the surface. . . . . . The drone positioning error to the target locations for all four surfaces. Overall error is . ± .cm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . planar windows in space, Lee et al. [] proposed Projective Windows technique which strategically uses the absolute and apparent sizes of the window at various stages of the interaction to enable grabbing, moving, scaling, and releasing of the window in one continuous hand gesture. Instead of manually managing these application windows, Lindlbauer et al. [] tried to automatically adapt which applications are displayed, how much information they show, and where they are placed based on users' current cognitive load, knowledge about their task, and environment (see Figure

	. Force-assisted text selection technique []: (a) the evaluation setup; (b-c) a . (a) The overall experimental setup consisted of an HoloLens, a smartphone, and . Different example scenarios of LightAir system []: (a) LightAir for human-
	. (A) The concept of floor projected UI in a collaborative SAR environment []. (B) Experimental setup with the extended floor UI. (C) Participants discussing a virtual scene. (D) The virtual scene from the master's (the user who controls the perspective) point of view. . . . . . . . . . . . . . . . . . . . . . . . . . . . . MirageTable is a curved projection-based augmented reality system (A), which digitizes any object on the surface (B), presenting correct perspective views accounting for real objects (C) and supporting freehand physics-based interactions (D). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mobile phone pointing techniques in SAR []: (A) viewport, where targets are captured by a camera-like view; (B) raycasting, where targets are pointed at; (C) tangible, where targets are directly contacted. . . . . . . . . . . . . . . . . (A) The user is moving a cursor (represented in blue) to a target (represented in red) on an augmented object using a standard mouse []. (B) In DesignAR system, a user is manipulating projected content for interior design using a tablet []. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example scenes to describe One Reality framework []: volcano mock-up made out of sand (top), D printed Toyota engine (bottom). Each scene can be interacted with different display technologies: spatial augmentation (left), force-assisted button interface on an iPhone to select the textual contents on a distal display. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . EYEditor interactions []: User sees the text on a smart glass, sentence-by-sentence. In the Re-speaking mode, correction is achieved by re-speaking over the text and a hand-controller is used to navigate between sentences. Users can enter the Select-to-Edit mode to make fine-grained selections on the text and then speak to modify the selected text. . . . . . . . . . . . . . . . . . . . . . . . . Distant freehand pointing and clicking on a very large, high resolution displays []: (A) raycasting (B) relative pointing with clutching (C) hybrid ray-to-relative pointing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The workflow of using Text Pin to select text []. (a) Widget A appears at the point of touch with a magnifying lens displaying above it; (b) Widget A consists of a handle and a circle (the magnifying lens above it disappears once the finger is lifted off the screen); (c) Finger clicks on the circle to fix one end of selection; (d) Widget B appears when finger clicks on the opposite end of selection (note we do not show the magnifying lens above Widget B for a clearer illustration of Widget B); (e) The user clicks on the circle of Widget B and (f) repeats step (a-d) to select the next non-adjacent text. . . . . . . . . . . . . . . . . . . . . . . . Some Gedit editing gestures on the smartphone keypad in one-handed use []. All gestures start from the right edge: (A) a flick left to select a word; (B) a clockwise ring gesture selects characters to the right of the text cursor; (C) the copy gesture 'C'; (D) the paste gesture 'V'; (E) clockwise and counterclockwise ring gesture for cursor control; (F) the left thumb swipes from the left edge to trigger editing mode, and then editing gestures can be performed by the right thumb; the ring gesture in editing mode performs text selection; the user simply lifts the left thumb to stop editing. . . . . . . . . . . . . . . . . . . . . . . Usage of Press & Tilt technique []. . . . . . . . . . . . . . . . . . . . . . . . . Usage of Press & Slide technique []. . . . . . . . . . . . . . . . . . . . . . . . . Force-sensitive text selection on touch devices []. (A) 'mode gauge' -using force for different selection modes (B) Example of text selection using force on a touchscreen. The user performs a long-press that displays the callout magnifier. Keeping the force in the character level, the user adjusts its position by moving her finger. She then presses harder to start the manipulation of the second cursor. If she releases her finger while the force is at the sentence level of the 'mode gauge', she will select the whole paragraph. If she releases her finger while the force is at the character level, she will only select what is between the two cursors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Illustration of the Gaze'N'Touch concept []. (A) Look at the starting character; (B) Touch down; (C) Look at the end character; (D) Release touch. . . . . . . . an optitrack system. (b) In the HoloLens view, a user sees two text windows. The right one is the 'instruction panel' where the subject sees the text to select. The left is the 'action panel' where the subject performs the actual selection. The cursor is shown inside a green dotted box (for illustration purpose only) on the action panel. For each text selection task, the cursor position always starts from the center of the window. . . . . . . . . . . . . . . . . . . . . . . . . . Text selection tasks used the experiments: () word () sub-word () word to a character () four words () one sentence () paragraph to three sentences () one paragraph () two paragraphs () three paragraphs () whole text. . . . . . Mean task completion time for our proposed four interaction techniques. Lower scores are better. Error bars show % confidence interval. Statistical significances are marked with stars (**: p < . and *: p < .). . . . . . . . . . . . Mean error rate of interaction techniques. Lower scores are better. Error bars show % confidence interval. Statistical significances are marked with stars (**: p . (A) A physical house mock-up. (B) A drone is mounted with two white paper panels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . An example scenario of DroneSAR. (A) The house is augmented using pro-jection, and the main menu is composed of a set of virtual tools projected on the drone panel. (B) A user selected the 'measuring tool' application using a controller. Then, the user positions the drone at the desired location in the D space (i.e., on top of the house) and draws a line shown in blue color on the augmented house to measure its width. Finally, the measured length is drone communication (b) DronePiano application (c) D point cloud processing for DroneBall. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A working scenario of Drone.io setup []. (A) Top view of the projected radial interface as seen by the user. (B) A user is extending his arm in a push gesture to navigate through the menu with the drone flying above. (C) A user is selecting an item in the menu. . . . . . . . . . . . . . . . . . . . . . . . . . . . (A) iSphere prototype []. (B) LightBee [] telepresence system with two local users viewing the drone-based light field display to communicate with a person in a remote capture room (inset) . . . . . . . . . . . . . . . . . . . . . . . ARial Texture tracks the position and orientation of a drone and projects a texture on the drone's propellers accurately []. . . . . . . . . . . . . . . . . . (A) BitDrones [] hovering in a tight formation. (B) User inspecting a remote facility with telepresence functionality provided by the drone. (C) User resizing a compound object using a bi-manual pinch gesture by moving drones. . . . . . (Left) In SAR, the projection space is limited by the size of the physical object. (Right) DroneSAR extends this projection space (shown in yellow color) with a flying panel that can be positioned in the surround of the physical scene. . . . . (A) The drone is hovering next to the chimney to display its corresponding label. (B) A flying cursor allows participants to point at a specific location in the scene. The dotted circle in the image represents the particular location in mid-air. (C) A video explaining the history is displayed near the medieval house. . (A -B) The light source of our scene is at the drone hovering position. By moving the light source, the user is casting shadows on the scene. (C) An interactive D model of the mock-up displayed next to the physical one allows the user to observe the scene from another viewpoint. . . . . . . . . . . . . . . . . The referenced scene (A) is decomposed into solid cells (in red) (B), then 'safe' cells (in yellow) (C). Example of way-point cells (in light green) (D). . . . . . . In traditional desktop computing, we are limited by a D flat window to reach the digital world. The idea of getting out of this standard WIMP (windows, icons, menus, and pointer) interface and augmenting our physical space with digital information has started in , when Ivan Sutherland de-scribed his vision of the ultimate display [] -"The ultimate display would, of course, be a room within which the computer can control the existence of matter. [...] With appropriate programming, such a display could literally be the Wonderland into which Alice walked". Three years later, he presented the first-ever head-mounted augmented reality display (AR-HMD) as a very early step towards such an ultimate display []. Since Suther-land's Sword of Damocles, we have progressed significantly in terms of display, tracking, and interaction techniques in the last five decades. Recently, optical see-through AR-HMDs (like HoloLens , Magic Leap , Nreal Light) are emerging from research labs into the mainstream. In the past, AR devices were mostly used for performing very specific tasks (which mainly involve selecting and ma-nipulating D objects, immersive data visualization, etc.) in specific application areas such as repairing and mainte-nance, training, gaming, warehouse operations, healthcare, etc. However, it is expected that we will use these displays for general-purpose computing in the future []. Keeping this in mind, lately, researchers have started exploring AR-HMDs . Positioning the Introduction for knowledge work. For example, Pavanatto et al. [] com-displayed on the drone panel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . pared physical monitors, virtual AR monitors, and a hybrid . The fog-display system []: (A) overall concept and (B) a prototype of it. . . combination of both for performing serious productivity . The water-drop display []: (A) overall concept and a prototype setup; (B) work and found that virtual monitors are a feasible option, the display is showing an image and (C) text. . . . . . . . . . . . . . . . . . . . though currently constrained by lower resolution and field . Pixie Dust []: (Top) Hardware setup; (Bottom) Example images created with of view (see Figure .). Such virtual AR workspace allows this setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . users to have an unlimited number of screens, screens of . (A) Display drone in an outdoor setting []. (B) Drone-projected in-situ any size, and visual privacy. Users can also embed these navigation instructions []. . . . . . . . . . . . . . . . . . . . . . . . . . . . . virtual application windows in their surroundings (e.g., cal-
	see-through displays (middle), and opaque head mounted displays (right). . . . Illustrations of our proposed interaction techniques: (a) continuous touch; (b) endar and map on the wall, notepad on the table, email
	discrete touch; (c) spatial movement; (d) raycasting. . . . . . . . . . . . . . . . client at the right-hand side) to locate apps quickly and to
	prevent app windows from occluding important objects in
	the environment (see Figure .(A)) [][]. To arrange D

Users correct errors by selecting word alternatives proposed by the speech recognizer or by typing on the virtual keyboard. . . . . . . . . . . . . Gaze + Pinch interactions unify a user's eye gaze and hand input: look at the target, and manipulate it (a); virtual reality users can utilise free hand direct manipulation (b) to virtual objects at a distance in intuitive and fluid ways (c). .

  Touch' problem [] of involuntary selection. To address this issue, researchers investigated different solutions mostly based on dwell time [], smooth pursuits (where eye gaze follows a target continuously) [], closing one eye [], gaze gestures [], and also by using a second modality for confirming selections (e.g., hand gesture or a button press as in HoloLens). Among all these target confirmation methods, dwell time is the most popular one as it is simple to understand and easy to implement. Considering its importance, Fernandez et al. [] presented GazeWheel, a visual feedback technique to improve its usability and performance. In GazeWheel, a visual feedback widget in the shape of a wheel is filled as the user is temporally closer to the selection event. When completely filled, a selection is made where the user is gazing.

	RotPlane (it makes use of orthogonal planes to achieve per-
	axis angular rotations), and RotBall (it combines a traditional
	arcball with an external ring to handle user-perspective roll
	manipulations). Figure . illustrates these three techniques.
	Their results showed that RotBar and RotPlane were faster
	and more accurate in performing single-axis rotations, but
	that RotBall greatly outperformed the other two methods for
	multi-axis rotations.

Recently, Mutasim et al. [] compared three target confirmation techniques (i.e., pinch hand gesture, dwell time, a button click) while selecting the target with eye-gaze pointing. Their Fitts' law task experiment did not find any significant differences in terms of execution time, error rate, and throughput among all these techniques. Still, participants preferred button click and dwell over pinch because pinch was sometimes frustrating due to recognition errors. Prior work also looked into eye-gaze based D object selection and manipulation. For instance, Sidenmark et al. [] proposed a novel technique called Outline Pursuits to support object selection in occluded environments with eye-gaze. This technique combines three concepts -cone-casting, outlining, and motion generation for object selection by smooth pursuit. Figure . illustrates how Outline Pursuits works. Liu et al. [] implemented three methods to manipulate D object rotation using only eye-gaze -RotBar (it enables per-axis rotation; it maps a 360 > rotation to a bar for each axis),

  this type of ad hoc interaction as "opportunistic controls" []. With inexpensive sensing technology like Microsoft Kinect § , users can create such low-

§ https://azure.microsoft.com/en-us/services/kinect-dk/

Table . :

 . Logistic function parameter values for continuous touch and spatial movement interaction. The unit of ⇠⇡ "0G and ⇠⇡ "8= is in mm/mm, whereas ⌫ is in sec/mm and + 8= 5 is in mm/sec.

	Techniques ⇠⇡ "0G ⇠⇡ "8=	⌫	+ 8= 5
	Continuous Touch	.	. . .
	Spatial Movement	.	. . .
	Mixed Reality Toolkit ¶ .

  al. for pointing tasks []. Continuous Touch was overall preferred, faster, and less demanding than Discrete Touch, which goes against results from the work by Jain et al. for shape selection [].

Table .

 . represents the summary of all four interaction techniques.

	. Limitations

Table . :

 . Properties, advantages, and limitations of each input interaction.

  https://v20.wiki.optitrack.com/index.php?title=Sync _ Configuration _ with _ an _ HTC _ Vive _ System Extending Physical Spaces in SAR using Projection on a Drone

https://www.onshape.com/ † https://optitrack.com/products/motive/ ‡

Nous avons exploré de nouvelles techniques d'interaction pour relever les défis mentionnés ci-dessus. Globalement, cette recherche comporte deux contributions principales décrites ci-dessous.

† † http://randomtextgenerator.com/

‡ ‡ https://developers.google.com/ar

Acknowledgements

First, I would like to sincerely thank my two advisors, Martin Hachet and Pascal Guitton, for supervising this research and encouraging me throughout this journey.