Adriano Scoditti

Wolfgang Stuerzlinger

A New Layout Method for Graphical User Interfaces

The layout mechanisms for many GUI toolkits are hard to understand, the associated tools and API's often difficult to use. This work investigates new, easy-to-understand layout mechanisms and evaluates its implementation. We will analyze the requirements for the definition of layouts of a graphical user interface. Part of the issue is that several aspects need to be considered simultaneously while laying-out a component: the alignment with other components as well as its own behaviour while resizing its container. Moreover, the used tools should isolate the designer/drawer from the implementation details of the framework.

We present the details of our new GUI layout system, discuss the choices we made for our new layout algorithm and detail implementation issues. Moreover, we present also the user interface for our new GUI builder system that contains several innovations, such as a preview window to show the effects of layout configuration choices in real-time. We present an evaluation of our new system by attacking the complex GUI layout problem mentioned above.

I. INTRODUCTION

Building GUIs involves several technical issues such as the definition of each components' layout, the minimum (and maximum) sizes of each component, the containment hierarchy, the interaction behavior of each component, the integration with the core software, and the portability of the interface itself both among different projects and different systems. As this work focuses only on the layout problem, the interaction between the GUI and the software is beyond the scope of this document. One central problem in GUIs is to define how components are placed within a window and how the components size changes when the window is resized. This is commonly refereed to as defining the layout of a GUI. Various solutions for the definition of layouts have been proposed, [START_REF] Chatty | Revisiting visual interface programming: creating gui tools for designers and programmers[END_REF][2][3] [START_REF] Dragicevic | Artistic resizing: a technique for rich scale-sensitive vector graphics[END_REF]. Most commercial toolkits make one or more of these solutions available to programmers. Using the Java/Swing terminology, some simple examples are the fixed-position layout, grid layout, flow layout, row and column layout, but all of these can only deal with restricted problems. A very flexible and powerful mechanism is the Struts and Springs layout [START_REF] Cardelli | Building user interfaces by direct manipulation[END_REF] that uses constraints to define the positions of components. This is implemented in the SpringLayout included in the latest Java Virtual Machine. The downside of the Struts and Springs method is that computing the layout requires quite a bit of computational resources. All of the above layout methods aim to make the process transparent for the graphic designer.

However, as we will see later, none of them have reached a satisfying abstraction level that permits the graphical designer to ignore implementation problems. Furthermore, the problem of defining user interfaces independently of the implementation in a portable way is addressed by a combination of the MVC (Model-View-Controller) programming pattern and storing the layout in external files, typically using some form of structured file format.

We will analyze the above-mentioned problems in depth both from the point of view of a graphical designer and a programmer. The first one wants to concentrate on the design of the GUI without having to worry about the limitations of the underlying GUI toolkit or the used GUI builder. In other words, wants to define a user interface focusing on the quality of the interface itself without having to address implementation issues. The programmer, on the other hand, strives to achieve a modular software architecture, wants to reuse his code, and aims to provide powerful tools to the interface designer.

II. RESIZING BEHAVIOR

While building a GUI, the most important step is to define the space occupied by each component of the interface itself in the right container . This means that, first of all, we must be able to define the position and the size of each component. However, this raises issue when the container is resized and still it is not sufficient to define all desired visual effectsmost importantly various forms of alignment. Alignment of components helps with visual grouping and is consequently considered necessary in graphical design. For example, when positioning a text label next to a text field, normally the baseline of the text label is aligned with the baseline of the text inside it (or it's first line). Or the center of the label is aligned with the center of a larger text box.

A. Evaluation of Previous Struts and Springs Implementations

Two different implementations of the Struts and Springs algorithm exist. The first one, the Swing SpringLayout, uses the classical approach of using constraints to define the layout of each component and then solving the whole set of constraints simultaneously. The second alternative, the Cocoa layout system, is based on an interpretation of the Struts and Springs concept, which utilizes the springs only to determine a viable partition of available space when resizing a component.

Both of the presented implementations have their pros and cons. The Cocoa implementation, for example, is extremely fast although it is not able to define and to lay out complex examples, as demonstrated in the previous chapter. On the other hand the Java SpringLayout is powerful, but solving the constraint-equation system is not very fast. Another important difference between the two implementations concerns the technical definition of which entities struts and springs can constrain to. There are two alternatives here, and the struts and springs can:

1) Connect each component of the GUI only to the edges of the window that contains the component itself. 2) Connect each component to other components present in the window. Although the first option is simple and fast to implement, it is limited as one cannot define inter-component relationships. The second alternative, as implemented in the Swing Sprint-Layout, is to permit each component to attach to arbitrary points in the window or to the sides of other components. This guarantees that no overlay can happen, as struts can be defined between two components or even among several components. The downside of this is that many constraints may be needed for complex windows.

The goal of our work is to introduce a fast and simple to use layout algorithm. Our intent is to maintain the advantages of all the algorithms presented here, while avoiding their disadvantages as far as possible. For this, we improved Cocoa's algoritm and added a new interface component.

III. INTUI STRUTS AND SPRINGS

It is an extension of the method used in the Cocoa framework. First, we introduce the concept of references, discuss the fundamental algorithm, and then introduce a new interface component, a Spacer, to increase the power of the approach.

A. References

To avoid overlap, each component in a GUI needs to keep track of other components around it. At a minimum, each component needs to consider at least one component on each side of it. For this, we introduce the so called references. Each component has four references, one for each side, that link it to it's neighbors. If there is no neighbor, it can also link a component to the window itself. Component frequently have multiple neighbors on the same side. To resolve this problem, let's analyze the scenario represented in Figure 1.

In Figure 1 on the left, each box represents a graphical user interface component. The bigger frame represents the window.

In this figure, we analyze how references for component A are defined. As mentioned above, the method searchers for the closest component relative to the center of each side. The bottom reference of A points to D, as it is the only component below. The top and left references are pointing to the window frame. On the right side of A, C will be chosen as the reference, as it is closer to the center of A. In this example, there is no reference to B, but depending on the references for the remainder of the layout it may not be necessary to have one.

IV. INTUI STRUTS AND SPRINGS ALGORITHM

In the Intui Struts and Springs layout method, each component is aware of the components around it due to the references. If a spring is associated with a reference, the algorithm can then better compute proportions relative to other components, which leads to an overall more powerful algorithm. Let's analyze the behavior of the widget A in Figure 1, by associating struts and springs to the scenario, as shown in the right part of the Figure. Furthermore, each component has also internal struts and springs, to allow widgets to vary in size depending on their content. While resizing the external frame, a widget A can count the number of springs around itself by invoking a recursive method that stops when the container frame is reached. Once the number of struts and springs in a direction is known, the widget can compute accurately how it should resize. One fairly obvious optimization of the algorithm shown above is to count the number of springs in advance for each component, whenever the layout has been finalized, as this removes the recursive search for references. Then the amount of work to be done for each resize is effectively constant, which makes the algorithm a lot faster. However, we use the above version of the algorithm in the real-time preview window of our interface builder (discussed later).

V. MINIMUM SIZE

One of the most important attributes for defining a GUI is the definition of the minimum size of the window. To explain minimum size management in our algorithm, we first state our underlying assumption: In a resizable GUI, at least one component will be resizable or have to change its origin when resizing a window. If the GUI is not resizable, this clearly does not apply. Within our framework, we define a default minimum size for each component depending on the component itself and it's normal use. For example, the minimum size of a text label is smaller than the minimum size of a text area. The management of horizontal and vertical resizing is handled independently, so that a component that has reached its vertical minimum size can still change in the horizontal direction. When resizing the window, each component first computes its future size and then compares it with this predefined minimum size to detect a resize that would make it too small. Similarly, each component also checks if the resize would move (even part of it) outside of the window frame and signals that it cannot resize accordingly in this case. Checking for overlap between components is not done, as this would require too much work on specifying all constraints by the designer. In the next section we will introduce a solution for this problem via spacers. Our solution to this problem is to introduce a new GUI component, that exists solely for the purposes of layout: a spacer. This is basically an invisible component, which has all the normal properties such as a position and a size. This size can either be constant or variable in each dimension. This spacer can assume different roles in the definition of a layout: it couples the layout of components, can be used to align components, or controls the minimum size of a window.

VI. THE SPACER -A MECHANISM

To couple components, a spacer is placed between them. Then the references for the components on either side reference the spacer, which effectively groups their influence and couples their relative layout. Figure 2 illustrates this, by placing a spacer S between components A, B, and C. The midpoint reference for A now references the spacer S, and both B and C reference it as well. This effectively couples their layout so that the three components never overlap. In the figure the horizontal extent of the spacer has been exaggerated for better illustration, typically this spacer would have a horizontal extent of a single pixel (and a variable vertical extent).

We illustrate in figure 2 how spacers can be used to align components. Here the two spacers S and T. Spacer S aligns the left sides of the three components relative to A, wherease T aligns the right sides of the two (resizable) components B and E. This example is an extension of the coupling property explained above.

As presented in section V, our algorithm automatically manages the minimum size of a window, based on the properties of the components. However, sometimes a GUI designer may need to define a minimum size for a window, where this size is larger than the sum of the components at their smallest possible size.

Figure 3 illustrates how spacers can be used to achieve this, namely by adding a spacer S between the other two components. The vertical size of S is then set to have either a constant size or to never go below a certain minimum vertical extent. If it is constant, the window is effectively non-resizable, if the other two components have fixed size, too. If it has a minimum, this minimum will always guarantee a proportional spatial separation between components A and B.

A. Relation With Previous Work

When configured as a simple non-resizable rectangle, the spacer as introduced here is similar to the concept of a (onedimensional) Gap in the Swing BorderLayout. However, a spacer can also occupy space in both dimensions simultaneously. The concept of Docking in the Windows Presentation Framework is also similar. We can simply compare the WPF anchor property to the strut concept. Furthermore, the docking property can be simply seen as a null size strut able to keep constant the distance of an edge of a component from another edge of its container.

The spacer also fulfills the role of an aligner, similar to the concept of alignment in the Windows Presentation Framework, but while WPF resolves all the alignment problem by using the Grid Layout in the appropriate way, the Intui aligner can be seen as a real object added to the interface to align other component. In this way, there is no difference between what the user wants to do and how the algorithm implements it.

In summary, the novel idea of a spacer subsumes a variety of layout concepts that have been introduced in other systems. However, spacers also extend these concepts in several ways.

VII. HIERARCHICAL INTUI LAYOUT

As presented above, our new variant of the struts and springs algorithm can cover many common cases. However, in some cases it may be unable to defined complex interdependencies correctly. As an example, consider the vertical resize behavior of the layout shown in figure 4. In this scenario, the intent of the designer is to preserve the "cross" shape in both dimensions. However, the Intui algorithm simply counts the number of springs along each dimension and distributes space proportionally. Hence, all components will be uniformly resized by a third of the dimension change and the "cross" shape will not be preserved at different window sizes. Although modifications of the Intui algorithm may guarantee a more intelligent distribution of the available space among the components, it also means introducing additional parameters, which would make the framework more complex and less intuitive. To address this situation in a simpler manner, we introduce another component, the container, which is identical to the hierarchical containers available in many other GUI toolkits. Then one can simply wrap each pair of smaller frames together into a higher level component. Then the bigger frames and the containers will share the same space, which will preserve the alignment of the overall layout. See figure 4 for an illustration. Although containers are often seen as special purpose components, they also serve to define a decomposition of space in a layout. While one can use Spacers for similar purposes in some situations, containers are more powerful. Containers can "encapsulate" several components into a new entity, which effectively allows to synthesize complex resize behaviors in a bottom-up fashion. In other words, the designer can specify different resizing behaviors at different levels of the layout hierarchy to achieve his/her design goal. For simplicity, we implemented hierarchical layout by designating a spacer as a container, i.e. allowing the placement of components inside a spacer. In other words, the spacer introduced above is also our mechanism for creating hierarchical layouts. In terms of efficiency containers can be considered a trade-off. Hierarchical containers offer a way to modularize a layout, but take a bit more time as the traversal of the hierarchy will be slower than the constant-time resize algorithm of the non-hiearchical Intui layout algorithm.

VIII. REAL TIME PREVIEW

Almost all current GUI builders permit the user test the constructed GUI at the press of a button. Although this feature lets the user identify any problems with the layout in this mode, they can fix these problems only after they have exited this test mode. In other words, the test view is not visible when the construction mode is active.

While this idea has the advantage that it allows the user to focus on the final look exclusively, it hinders the debugging and correction process for any problem found as the user can't simultaneously see the definition of the interface and the "live" version that can be resized interactively. Hence, the user is often obliged to switch repeatedly between the building environment and the testing environment. Expert users may need less iterations here, but could still benefit from an improved solution.

Our answer to this problem is a real time preview that is visible during the construction process and can be resized at any time. To illustrate the impact of a simultaneous preview in an interface builder, we refer the reader to figure 5, which depicts a screenshot of our interface builder. The Intui Interface builder features a real-time preview of the constructed interface, shown in a different size compared to the construction window.

In the top right is a small panel with all the components available for selection and/or drag and drop. The bottom left shows the construction window and the right side shows the preview. The GUI construction window contains abstract widgets that permit modification of the resize behavior, e.g. see the selected widget in the top-center. The preview window is a fully functional GUI, and the user can enter text, click on buttons, etc.

The two views of the GUI let the user directly compare the positions and the sizes of each component at different window sizes. This enhances the user experience with direct feedback for any change in the construction window 1 . Another way to express this is to say that we provide a "What You See Is What You Get" (WYSIWYG) interface.

IX. DEFINING THE RESIZE BEHAVIOR

The graphical metaphor for the resizing bit-mask is represented by solid and dashed lines in the construction window. To ensure adequate visibility, the lines illustrating the resizing behavior are only shown for the currently focused component, struts are visualized by black solid lines and springs are visualized by red dashed lines, both of them are drawn on a grey background. When a new component is dropped or otherwise instantiated in the construction window, the initial resize mask is calculated by a heuristic algorithm that uses the following information:

1) Its relative position within it's container. 2) Its size,independently width and height.

3) The resize masks of all components it references (i.e.

the masks of all components around it).

The relative position within the container is used as follows: if the center of the component is within the first 25% or the last 25% of each dimension, struts are placed towards the outside.

Otherwise, the component is in the center part of the window and springs are used. The mask to define the intrinsic size of the component is determined to be fixed depending on the type of the component (buttons are fixed, text-areas resize). The behavior of adjacent components influences the initial value for the resize mask as the following example: if a component is placed in the center of the window but it is surrounded by component with only external struts, it will be configured with external struts as well. The same algorithm also automatically determines the references between the components by shooting a ray starting from the midpoint of each side and identifying the closest component along the ray (unless the window frame is hit). To avoid potentially unstable situations, the algorithm checks that two components that reference each other has the same value from both side of the ray. If there is a disagreement, the configuration of the component where the algorithm is running (i.e. the focused component) will change the configuration to create the needed coherence (in this way, the choices of the user are always respected because as we will see later, it is possible that on the other component the user manually modified the configuration). The above algorithm is also used to determine the new resize mask every time the user moves a component (unless the resize mask is manually overridden, see below). The net result is that the construction process is greatly accelerated, as the above algorithm frequently sets the desired resize mask for a component correctly -at least to our experience. To aid the user in understanding the behavior of the algorithm, the background of each container shows a grid of the decision boundaries used for the position mask calculation in medium grey. Although this heuristic algorithm can forecast the intention of the user in most cases for simple user interfaces, it clearly cannot deal with arbitrary layouts. Hence, the user interface of the GUI builder must allow the user to change the default configuration for the resize behavior.

The visualization of the struts and springs in the construction window is fully interactive. In other words, the user can modify the resize mask by clicking on solid and dashed lines depicting the struts and springs. Clicking on a strut changes it to a spring and vice versa. Every time the resize mask is modified in the construction window, the preview window updates the layout of all components to immediately visualize the consequences of user actions.

To avoid potentially unstable situations due to conflicting relative resize masks, the GUI builder checks that two components that reference each other set the resize mask to the same value from both sides. An unstable situation can never happen because, as we saw before, the needed coherence is always created by the automatic algorithm while adding the component. This means that while manually modifying an external strut for a component, for example, this modification is automatically propagated towards the other component that this strut references.

X. EVALUATION

As an evaluation of the Intui GUI builder, we defined a complex user interface. There are several ways to achieve this, but we choose the alternative that emulates the behavior of an irregular grid. Figure 6 depicts the Intui builder with the final result and a bigger preview that illustrates the correct resize behaviour. The construction process for the test user interface is fairly simple. Once all five text boxes are dropped into the correct positions in the construction window, the following steps need to be carried out:

1) The heuristic resize mask algorithm has already set the behavior for all four corner boxes. Hence, no action is needed on them.

2) The resizing behavior for the central box needs to be changed by adjusting the internal behavior. 3) To avoid potential overlap, four spacers need to be added and their internal resize behavior adjusted by making them resizable along their shorter dimension only, because the longer one is well forecasted.

XI. HIERARCHICAL GUI CONSTRUCTION

In the last chapter we discussed the concept of a container hierarchy for GUI widgets. Here we mention how we allow the user to place components inside spacers to use them as containers. An explanatory example is shown in figure 7, where we show how the layout discussed in section X can be constructed with a hierarchy. As visible in the figure, each pair of smaller text boxes is placed within a container. Then the designer can define the layout of the four top level components to preserve the overall layout with the cross in the the middle. The smaller text boxes resize within each container and are internally resizable, but have fixed distances externally. When resizing, the space will first be distributed among the four highlevel components, and then each pair of smaller text boxes split the space within their container evenly.

XII. CONCLUSION AND FUTURE WORK

We introduced a new way to define the layout of graphical user interfaces. The new Intui layout algorithm implements a conceptually simple variant of the struts and springs model, that is powerful enough to address many common layout problems. For this, we introduced the novel concept of a Spacer, which serves multiple purposes: It couples components, aligns them, can be used to intuitively define the minimum size of a window, and can be used as a container. Furthermore, the computational cost for resizing Intui layouts is far less costly than with a full implementation of a constraint-based solver. We also introduced a novel construction interface for graphical user interface builders, which makes the construction of common user interfaces easier to perform. This is achieved in part by a heuristic algorithm that automates common component placement decisions. Furthermore, we introduced the concept of a real-time preview window, which is sized to immediately visualize how the current layout looks at a different size. Some of our design choices are arguably non-optimal and may be improved. One such issue is an improved discovery method for references. The current approach automatically finds references based on the midpoint of each side of a widget. This permitted us to ensure coherency and generality in the builder. Our intent was to enable the user to immediately see the dependencies when looking at a widget in the construction window. Furthermore, this design decision kept the file format simple. On the other hand, a user may not immediately understand the way this definition of references works. Furthermore, the references for a widget change whenever it is moved in the construction window, sometimes with results that are not immediately predictable.

Another issue is that the GUI builder does not snap to common positions when a component is moved or resized. Examples are the center of the window or positions aligned to another widget. This functionality may be added in the future. Furthermore, common guidelines, such as a default margin, should also be introduced as snapping points.

Fig. 1 .

 1 Fig. 1. Each object automatically chooses references by detecting the the nearest component in the four directions from the center of each side. On the right image, the overall configuration of the widget A in the main container is highlighted.

  Figure 2 depicts the common scenario of multiple components inside a window. The figure illustrates multiple issues, namely that the simple idea of having references always use the center of a component can easily fail and the importance of symmetry. In this case there is a vertical symmetry of components B and C relative to the vertical midpoint of component A. The Intui algorithm as described above cannot correctly deal with this example.

Fig. 2 .

 2 Fig. 2.On the left: Illustration of a failure case for the simple Intui algorithm. Due to it's use of midpoints for references, it cannot identify the two components to it's right and hence incorrectly links component A to the window frame. On the right: Spacers can be used to align components. Here the right sides of components B and E are aligned via spacer T. The horizontal extent of the spacer is exaggerated for better illustration.

Fig. 3 .

 3 Fig. 3. A spacer can also be used to control the minimum size of a layout, see text.

Fig. 4 .

 4 Fig. 4. With layers the algorithm can preserve the distribution of the blocks.

Fig. 5 .

 5 Fig. 5.The Intui Interface builder features a real-time preview of the constructed interface, shown in a different size compared to the construction window.

Fig. 6 .

 6 Fig. 6. Screenshot of IntuiBuilder while constructing the evaulation user interface.

Fig. 7 .

 7 Fig. 7. Screenshot of IntuiBuilder for the hierarchical layout.

In an early prototype, we let the user also modify the layout in the preview window. However, we removed this feature because it introduced an ambiguity, as the references for a component might be different at different sizes due to different position of the midpoint. Hence we dropped this feature.


Thomas Vincent

Laurence Nigay
email: laurence.nigay@imag.fr

Takeshi Kurata
email: t.kurata@aist.go.jp

Precise pointing techniques for handheld Augmented Reality

Keywords: Handheld Augmented Reality, Interaction Techniques, Pointing

We propose two techniques that improve accuracy of pointing at physical objects for handheld Augmented Reality (AR). In handheld AR, pointing accuracy is limited by both touch input and camera viewpoint instability due to hand jitter. The design of our techniques is based on the relationship between the touch input space and two visual reference frames for on-screen content, namely the screen and the physical object that one is pointing at. The first technique is based on Shift, a touch-based pointing technique, and video freeze, in order to combine the two reference frames for precise pointing. Contrastingly -without freezing the video-, the second technique offers a precise mode with a cursor that is stabilized on the physical object and controlled with relative touch inputs on the screen. Our experimental results show that our techniques are more accurate than the baseline techniques, namely direct touch on the video and screen-centered crosshair pointing.

Introduction

While still an open research area, Augmented Reality (AR) in terms of superimposition of graphics is now possible on camera-equipped handheld devices. However, issues related to interaction still need to be studied. In particular, pointing at physical objects through the live video playback of a handheld device with either direct touch or a screen-centered crosshair has limited accuracy [START_REF] Boring | Touch projector: Mobile interaction through video[END_REF][START_REF] Liao | PACER: fine-grained interactive paper via camera-touch hybrid gestures on a cell phone[END_REF]. Nevertheless accurate pointing at physical objects would benefit several handheld AR applications including selection or in-situ positioning of digital annotations in dense physical environments such as paper maps.

Pointing accuracy in handheld AR is limited by various factors. First, interaction with handheld devices brings specific constraints [START_REF] Roudaut | TapTap and Magstick: improving one-handed target acquisition on small touch-screens[END_REF]: the screen real estate is limited and direct touch on the screen, the de-facto standard input modality on such devices, is impaired by finger occlusion and an ambiguous selection point (i.e. the "fat-finger" problem). Moreover, when considering handheld tablets which are larger but also heavier than phones, the trade-off between device handling (i.e., one or two handed hold) and touch interaction (i.e., available fingers for touch interaction and screen accessibility) need to be taken into account [START_REF] Wagner | BiTouch and BiPad: Designing bimanual interaction for hand-held tablets[END_REF]. To address these issues, pointing accuracy on handheld devices has been studied and different techniques have been proposed [START_REF] Roudaut | TapTap and Magstick: improving one-handed target acquisition on small touch-screens[END_REF][START_REF] Vogel | Shift: A technique for operating pen-based interfaces using touch[END_REF]. Yet, those techniques do not take into account the specificities of AR.

Indeed, in AR applications, when interacting through the video on handheld devices, the physical object one wants to point at is not stable on screen. As a consequence, it is also not stable within the touch input space. As the viewpoint is controlled by the device's pose, its stability is impaired by hand tremor and motion induced by the user's touch inputs. Furthermore, when using handheld tablets for AR applications, the trade-off is between viewpoint stability (i.e. a steady hold) and touch-screen accessibility. On the one hand, a steady hold with both hands (figure 1-c/d) only allows touch interaction with the thumbs in a limited region of the screen. On the other hand, when holding the tablet with one hand (figure 1-a/b), the other hand can interact with a larger area of the screen at the expense of more instability.

For handheld AR systems, pointing with a screen-centered crosshair has been studied [START_REF] Rohs | Target acquisition with camera phones when used as magic lens[END_REF][START_REF] Rohs | Interaction with magic lens: Real-world validation of a Fitt's law model[END_REF]. This technique is impaired by viewpoint instability. Freeze-frame techniques have been used to improve direct touch interaction by pausing the video [START_REF] Boring | Touch projector: Mobile interaction through video[END_REF][START_REF] Lee | Freeze-set-go interaction method for handheld mobile augmented reality environments[END_REF]. Nevertheless, one drawback of this approach is to prevent an up-to-date view of the physical scene.

To address the limitations of pointing for touch-based handheld AR systems, we propose two techniques:

• Shift&Freeze (figure 1-b) that addresses both direct touch accuracy limitations and viewpoint instability by combining Shift [START_REF] Vogel | Shift: A technique for operating pen-based interfaces using touch[END_REF] with freeze-frame. Shift is a technique that extends Direct Touch pointing (figure 1-a) with a precise quasi-mode.

We complemented this precise quasi-mode with freeze-frame to adapt Shift to handheld AR. So Shift&Freeze improves accuracy while still allowing direct touch for coarse but fast pointing. • Relative Pointing (figure 1-d), which improves pointing accuracy without pausing the live video playback. To improve accuracy, it uses a cursor that is stabilized in the physical object's frame of reference. The cursor is controlled by indirect relative touch inputs. As such, relative pointing in handheld AR does not share direct touch limitations and the cursor position is not impaired by viewpoint instability. Moreover this technique extends the screen-centered Crosshair pointing technique (figure 1-c) with a precise mode. This allows both coarse but fast pointing and accurate pointing when needed.

In this paper, we first review related work and then present the design rationale of our two techniques, Shift&Freeze and Relative Pointing in handheld AR. We then report two experiments comparing our techniques and the baseline techniques, namely Direct Touch on the video and screen-centered Crosshair. We conclude with a discussion of our results and directions for future work.

Related Work

We build on previous work on pointing techniques for touch-based handheld devices, as well as on pointing techniques for handheld AR and spatially aware interfaces.

Pointing techniques for touch-based handheld devices

Much prior work has addressed how to improve pointing accuracy on touch-screen. Within the scope of our work, we examine pointing techniques on touch-based handheld devices that do not require prior knowledge of the targets, excluding for instance target expansion techniques as in Starburst [START_REF] Baudisch | Starburst: a target expansion algorithm for non-uniform target distributions[END_REF]. Indeed for the case of in-situ positioning of annotations in handheld AR (e.g., annotations at any position on paper maps), there is no available knowledge of possible targets. A first approach is zooming to enlarge the information space to a scale appropriate for accurate pointing [START_REF] Albinsson | High precision touch screen interaction[END_REF]. When using zooming, the user is facing the classical tradeoff between the level of zoom (for accurate pointing) and the visible context (for finding the area of interest). The interaction process can be quite tedious on handheld devices with limited screen real estate: zoom in to focus and zoom out for context. Based on zooming, TapTap [START_REF] Roudaut | TapTap and Magstick: improving one-handed target acquisition on small touch-screens[END_REF] increases pointing accuracy. Two taps on screen are performed for pointing. The first tap selects a coarse area on the screen that is displayed zoomed in a pop-up view centered on the screen. The second tap performs the precise selection in the zoomed area and closes the pop-up view.

A second approach is to display a cursor to address both finger occlusion and selection point ambiguity. Potter et al. [START_REF] Potter | Improving the accuracy of touch screens: An experimental evaluation of three strategies[END_REF] proposed Take-Off that enables pointing adjustment and avoids finger occlusion by showing a cursor slightly above the finger position. One drawback of this technique is that the user does not know the position of the cursor until her/his finger touches the screen. Building on this technique, Shift [START_REF] Vogel | Shift: A technique for operating pen-based interfaces using touch[END_REF] extends direct touch pointing with a precise quasi-mode. While in this mode, Shift displays a circular callout showing a copy of the screen area occluded by the finger and places it in a non-occluded location. The callout also shows a cursor representing the selection point of the finger, whose position can be adjusted by moving the finger. Validation is performed on finger lift. MagStick [START_REF] Roudaut | TapTap and Magstick: improving one-handed target acquisition on small touch-screens[END_REF] also extends direct touch pointing by using a telescopic stick metaphor to enable further adjustment. When the finger touches the screen, it defines a reference point; then, by dragging the finger away from the reference point, the user can extend a telescopic stick centered on the reference point with the finger at one end and the cursor at the other end.

Pointing techniques for handheld AR and spatially-aware interfaces

Seminal works on handheld AR like NaviCam [START_REF] Rekimoto | The world through the computer: Computer augmented interaction with real world environments[END_REF] have paved the way for an active research area. For example, TouchProjector [START_REF] Boring | Touch projector: Mobile interaction through video[END_REF] allows users to move pictures on remote screens through direct touch on the live video of a handheld device. Handheld AR systems augmenting different kind of objects such as sights [START_REF] Alessandro | Zooming interfaces for augmented reality browsers[END_REF], printed conference proceedings [START_REF] Liao | PACER: fine-grained interactive paper via camera-touch hybrid gestures on a cell phone[END_REF], photo books [START_REF] Henze | Who's that girl? Handheld augmented reality for printed photo books[END_REF] or paper maps [START_REF] Rohs | Impact of item density on the utility of visual context in magic lens interactions[END_REF] have been developed. In handheld AR settings, the viewpoint in the augmented scene is usually controlled by the absolute device's pose in space (controlling the back-face camera viewpoint). As a handheld device is not self-stabilized (as opposed to the mouse for example), its pose is subject to hand jitter as for other freehand interaction techniques like laser pointers or handheld projectors [START_REF] Forlines | Zoom-and-pick: facilitating visual zooming and precision pointing with interactive handheld projectors[END_REF]. As a consequence, the augmented scene the user interacts with is not stable on the screen. This is different from typical GUI situation on either desktop or handheld devices (see previous section) where the objects the user wants to interact with usually remain still on the screen during the interaction.

With such settings, pointing is usually performed with either a screen-center crosshair [START_REF] Henze | Who's that girl? Handheld augmented reality for printed photo books[END_REF][START_REF] Liao | PACER: fine-grained interactive paper via camera-touch hybrid gestures on a cell phone[END_REF][START_REF] Rohs | Target acquisition with camera phones when used as magic lens[END_REF][START_REF] Rohs | Interaction with magic lens: Real-world validation of a Fitt's law model[END_REF][START_REF] Rohs | Impact of item density on the utility of visual context in magic lens interactions[END_REF] or by direct input on the screen, using a pen or bare fingers [START_REF] Boring | Touch projector: Mobile interaction through video[END_REF][START_REF] Henze | Who's that girl? Handheld augmented reality for printed photo books[END_REF][START_REF] Lee | Snap-to-feature interface for annotation in mobile augmented reality[END_REF][START_REF] Lee | Freeze-set-go interaction method for handheld mobile augmented reality environments[END_REF][START_REF] Liao | PACER: fine-grained interactive paper via camera-touch hybrid gestures on a cell phone[END_REF][START_REF] Vincent | Classifying handheld augmented reality: Three categories linked by spatial mappings[END_REF]. Rohs et al. [START_REF] Rohs | Target acquisition with camera phones when used as magic lens[END_REF][START_REF] Rohs | Interaction with magic lens: Real-world validation of a Fitt's law model[END_REF] studied pointing with a screen-centered crosshair on a phone. They showed that the performance of this technique could be modeled with a two parts Fitt's law: physical pointing (i.e. moving the device in space) and virtual pointing (i.e. when the target is visible on screen). Hand jitter impairs accuracy of those interactions and different strategies have been proposed to improve interaction with handheld AR settings.

A first strategy is to increase target size on the screen by coming closer to the physical object or by zooming the live video. Zooming is compatible with both screencentered cursor and direct input as well as with other strategies for improving interaction. TouchProjector user study shows that automatic zooming was overall the best performing technique: While zooming improves interaction, it does not render the image steady. The study also highlights that for precise manipulation a freeze-frame mode (which also performs automatic zooming) outperforms automatic zooming alone.

The freeze-frame technique belongs to the second strategy that overcomes viewpoint instability due to small hand motions. Indeed when pausing the live video, the viewpoint becomes steady. Freezing the frame also allows moving to a comfortable position for interaction. This approach is not compatible with a screen-centered cur-sor, but it has been proven useful to improve pen and touch interaction. In a user study, Lee et al. [START_REF] Lee | Freeze-set-go interaction method for handheld mobile augmented reality environments[END_REF] showed that a video freeze mode improves accuracy for a drawing task with a pen through the handheld device video frame. They also noted that some users become lost when the live video is resumed as the viewpoint has changed. Another issue of freezing the frame is that the scene is no longer updated. TouchProjector overcomes this issue by updating the video snapshot with a digital copy of the remote screen one is interacting with. Unfortunately, a digital copy of the object of interest is not available for all AR scenarios. Freeze and zoom can be combined as previously explained in the case of TouchProjector and as in TapTap combined with video freeze for handheld AR [START_REF] Vincent | Classifying handheld augmented reality: Three categories linked by spatial mappings[END_REF]. In the latter, the combination of video freeze and zoom is a 'once' mode rather than a truly persistent mode. Nevertheless, any selection then requires two taps. Another way to stabilize the viewpoint is to use 'loose registration' as in PACER [START_REF] Liao | PACER: fine-grained interactive paper via camera-touch hybrid gestures on a cell phone[END_REF]. To interact with paper documents, they propose to display a digital copy of the document on the handheld device screen instead of the live video. This relaxes tracking requirements and allows for a coarse and filtered viewpoint to be used. Again, this requires a digital copy of the object of interest.

Finally, a third strategy consists of stabilizing inputs in the frame of reference of the physical object (or of its projection on the screen) rather than stabilizing the object of interest on the screen. Snap-to-feature [START_REF] Lee | Snap-to-feature interface for annotation in mobile augmented reality[END_REF] proposes to snap touch input on features of physical objects detected in the live video. This allows for better drawing of contours of physical objects on the screen without relying on freeze-frame or a digital copy of the scene. Our Relative Pointing technique is based on a similar strategy but does not rely on detecting features of the physical objects in the live video.

As opposed to on-screen content stabilization techniques that sever the live relation with the surrounding or use a digital copy of the scene, input stabilization offers the opportunity to improve accuracy without loosing the live relation with the surrounding. Both strategies can be complemented with zooming. Those approaches address limitations specific to the handheld AR context but not necessarily limitations of touch inputs. This is the challenge we addressed by designing Shift&Freeze and Relative Pointing, two pointing techniques that we introduce in the next sections.

3

Handheld AR Pointing

Design rationale

To systematically analyze the issue of accurate pointing for handheld AR, we base our study on the relationships between the touch input space and two frames of reference for on-screen content: that of the screen and that of the physical object of interest.

With video freeze, the physical object's frame of reference is fixed within the image plane and thus it is stable on the screen (figure 2-b). This case is similar to GUI interfaces: Existing pointing techniques for handheld devices can be combined with video freeze. Shift&Freeze combines the existing pointing technique Shift [START_REF] Vogel | Shift: A technique for operating pen-based interfaces using touch[END_REF] with video freeze.

When interacting through live video, the physical object is jittery on the screen (figure 2-a). In this case, we consider (see figure 3): (1) whether pointing is performed with or without an instrument (i.e. a cursor), and (2) in which frame of reference pointing is performed -either the frame of the screen or the frame of the physical object.

Direct Touch is the case of pointing in the screen frame without a cursor. Screencentered crosshair makes use of a cursor and points in the screen frame. Those two techniques (with and without a cursor) are impaired by hand jitter as pointing occurs in the screen frame where the physical object of interest is not stable (figure 2-a). If pointing was performed in the frame of reference of the physical object rather than in the screen frame, pointing accuracy would not be impaired by hand jitter. Pointing in the physical object's frame of reference without a cursor instrument implies interaction directly on the physical object. We did not consider this case and focused on interaction with the touch-screen of the handheld device. Moreover such physical interaction seems cumbersome while holding a handheld tablet. Our Relative Pointing technique is the case where pointing is performed with a cursor in the physical object's frame of reference. In this case, we use an indirect relative mapping of touch inputs to cursor motions in the frame of reference of the physical object. As such, the cursor position is not impaired by hand jitter. This analytical framework based on the spatial relationships between the input space and two visual output frames of reference guided the design of our two tech-niques Shift&Freeze and Relative Pointing. Their respective designs result from a twofold strategy: Shift&Freeze is conceived as an improvement of Direct Touch and solves its accuracy problem by freezing the video and Relative Pointing is an improvement of Crosshair by adding a relative cursor stabilized on the remote physical object. Figure 4 shows a walkthrough of the two modes of our Shift&Freeze technique. Scenario 1: (a) The user points the handheld device camera at the target so that the target appears on the screen. (b-c) After a short dwell time after finger contact, Shift&Freeze enters a precise quasi-mode and the video is frozen. A callout is placed above the finger and shows the area under the finger and a cursor at the current selection point position. (d) While in this mode, the video remains frozen and the user can adjust the position by moving its finger. (e) On finger lift, the target is selected and the live video playback is resumed.

Shift&Freeze

Scenario 2: (f-g) For large enough targets where hand tremor and finger occlusion are not a problem, selection can be performed with a tap on the screen at the position of the target.

To cope with touch input limited accuracy, we chose to use Shift [START_REF] Vogel | Shift: A technique for operating pen-based interfaces using touch[END_REF] for the following reasons. First, Shift does not require knowledge of existing targets to improve accuracy. Also, Shift extends Direct Touch, thus fast but imprecise pointing is still possible. Finally, similar techniques have been used to precisely place the cursor in text entry in commercial products and to improve accuracy when using 'loose registration' [START_REF] Liao | PACER: fine-grained interactive paper via camera-touch hybrid gestures on a cell phone[END_REF].

To cope with viewpoint instability in handheld AR settings, we combined Shift with freeze-frame. Touch-based pointing techniques in general and Shift in particular are designed for pointing at static targets on the screen. Instead of implementing freeze-frame as a mode, we complemented Shift's precise quasi-mode with video freeze. Compared to the original Shift technique, no extra user action is necessary to control video freeze/unfreeze. Nevertheless, as noted in [START_REF] Lee | Freeze-set-go interaction method for handheld mobile augmented reality environments[END_REF], resuming the live video leads to a discontinuity of viewpoint that might disorient the user.

As such, the Shift&Freeze technique has the following properties: (1) By extending Direct Touch, this technique requires interaction overhead only when accuracy is required; (2) It allows precise pointing using Shift's callout on a frozen frame. Scenario 1: (a) The user points the handheld device camera at the target so that the target appears on the screen. (b) In order to mitigate the instability due to hand tremor, when the user touches the screen and starts moving its finger on the screen, a relative pointing mode is triggered. (c) While in this mode, the cursor is no longer bound to the screen center. Instead, it is stabilized on the remote physical object at its current position. The user fine-tunes the cursor position by controlling the cursor indirectly with finger gestures on the screen. (d) On finger lift, no special action is performed. (e) The validation of a position is performed with a tap on the screen. Upon validation, a short animation moves the cursor back to the screen center, thus leaving the relative pointing mode.

Relative Pointing

Scenario 2: (f-g) When acquiring large enough targets, hand tremor is not a problem. In this case, the user does not need to use the relative pointing mode and can trigger a target acquisition at the position of the screen-centered cursor with a tap on the screen. This is similar to the screen-centered Crosshair technique.

To make relative pointing effective for handheld AR context, the following issues have been addressed.

Combining Absolute Physical Pointing and Touch-Based Relative Pointing.

As the device's pose controls the camera viewpoint, the target in the physical world needs to be placed in the camera field of view before interaction with it can start. So, cursor-based relative pointing needs to be combined with this absolute direct pointing in space. That is why we chose to extend the screen-centered crosshair pointing technique as it already uses a cursor and only relies on the device's pose for both viewpoint control and pointing. We extended this technique with a relative pointing 'once' mode where the cursor is no longer fixed at the center of the screen. Instead, the finger indirectly controls the cursor's position. This mode is triggered by finger motion on the screen. Lifting the finger from the screen does not deactivate this mode. This allows both finger clutching and checking the current cursor position before validation. A tap on the screen triggers the pointing validation. It is possible to cancel this relative pointing mode by pressing a button. Also, when tracking is lost, the relative pointing mode is cancelled. Finally, the cursor is bound to the screen. In case a change in the camera's viewpoint or a finger motion would otherwise make the cursor invisible on the screen (i.e., outside the screen), the cursor is automatically moved so that it remains visible on the screen.

Transfer Function. When dealing with indirect relative input, the transfer function (figure 6) that maps input motions to cursor displacements in the visual output space is of particular interest. First, a transfer function that maps touch motions in the screen frame directly to cursor displacements in the physical object's frame is not appropriate. In this case, the relative rotation between the device and the physical object's frame, the distance between the device and the physical object, and the zoom factor would affect the displacements of the cursor on the screen. Yet the user is looking at the physical object through the live video on the screen. Therefore the control loop is between the finger motions and the cursor displacements on the screen (and not in the physical object's frame).

Instead, for Relative Pointing, the transfer function is applied in the screen frame. When a finger motion input is received, the cursor position is projected from the physical object's frame onto the screen; the cursor displacement is applied on the screen; and the new cursor position is projected back onto the physical object (figure 6-right). This guarantees that the behavior of the cursor on the screen is consistent when the device is rotated (figure 6-left) and when the viewpoint or the zoom factor changes. In short, we use the physical object's frame of reference to stabilize the cursor and the screen frame of reference to apply cursor displacements.

A second question is which transfer function should be used. Transfer function is the place for interaction improvements such as adjusting the control-to-display ratio dynamically according to the input device speed. Dynamic transfer function is a default feature for the mouse and touchpad inputs in common desktop environments.

While dynamic transfer function has been studied for desktop environment, we are not informed of thorough evaluation of indirect mapping on handheld device's touchscreens [START_REF] Casiez | No more bricolage!: methods and tools to characterize, replicate and compare pointing transfer functions[END_REF]. A dynamic transfer function can be used with the Relative Pointing technique. For our developed technique and experiments, we used the transfer function of touchpad inputs on Mac OS X: osx:touchpad?setting=0.875 [START_REF] Casiez | No more bricolage!: methods and tools to characterize, replicate and compare pointing transfer functions[END_REF]. With this configuration, the transfer function allows both reaching of most of the screen with fast movement and accurate positioning at lower speed.

To sum up, the Relative Pointing technique has the following properties: (1) By extending Crosshair, this technique requires interaction overhead only when accuracy is required. ( 2) By stabilizing the cursor on the remote physical object, it offers accuracy assistance without relying on video freeze.

Experiments

We ran two experiments to evaluate four handheld AR pointing techniques (the two baseline techniques and the two proposed techniques):

• Direct Touch: Pointing with selection at the finger press position;

• Crosshair: Screen-centered crosshair pointing where validation is triggered on finger press with a tap anywhere on the screen; • Shift&Freeze with a 300 ms delay for escalation, a 44mm wide callout initially placed 22 mm above the initial touch position, and no zoom in the callout; and • Relative Pointing as described above but without a cancel button.

All cursor-based techniques (i.e. Crosshair, Shift&Freeze's callout and Relative Pointing) are using the same red square cross cursor, which is 7.7mm in size with a stroke width of 0.2mm.

The goal of the first experiment was to collect both user feedbacks and quantitative data on those techniques while performing a rather realistic task: placing marks on a wall map. Building on this experiment, we ran a second experiment to further evaluate those four techniques in a controlled experiment while acquiring small targets.

Experiment 1: User experience

For this experiment, we formulated the following hypothesis:

• H1: Relative Pointing and Shift&Freeze are preferred over Crosshair and Direct Touch. This is motivated by the extra precise mode offered by our two techniques. Moreover, this precise mode does not prevent the use of Crosshair or Direct Touch as a basic mode. • H2: On tablet form factor, indirect cursor-based techniques are preferred over direct pointing techniques. So Relative Pointing is preferred over Shift&Freeze and Crosshair is preferred over Direct Touch. This is based on both the finger occlusion issue for direct touch input and the trade-off between tablet hold and screen accessibility.

Procedure, apparatus and participants. For each of the four techniques, we first explained the technique and participants had a chance to freely try it. Then, participants performed 5 different pointing tasks to place AR marks on a physical wall map with a handheld tablet. Each task was repeated three times. For each trial, participants started at 2.5m from the wall map. They were instructed to move freely in the room and to hold the tablet in portrait mode. Finally, a debriefing questionnaire and interview concluded each technique's experiment. Before starting the experiment with the four techniques, participants started to perform each of the 5 tasks once with no interaction by only finding the required targets through the video on the tablet screen. This was introduced to help participants to become acquainted with the tasks and the experimental setting (in particular form factor and video quality). After this initial training, all participants started with the Direct Touch technique. The presentation ordering of the other three techniques was then counter-balanced across participants using a Latin-square. We used this design so that all participants share Direct Touch, the de facto standard interaction, as a common baseline. Experiments lasted about one hour including a debriefing discussion. We use a map of our campus site in A1 format in landscape orientation (841mm x 594mm). It was placed vertically on a wall with the middle of the map 1.5m above the floor (figure 7-left). The targets of the 5 tasks are shown on the right of figure 7. Tasks 1 and 2 consist of placing a mark on a single target. Tasks 3, 4 and 5 consist of placing marks on two different targets. The target sizes range from 2mm to 4cm.

The experiment was conducted on an iPad2 (weight: 601g, screen resolution 1024x768 pixels (132 dpi)). The system provides touch input with the same resolution as the screen. We developed an ad hoc application for the experiment using OpenGL|ES 1.11 , Vuforia SDK 1.5.92 and libpointing 3 [6]. This application runs at about 30 frames per second. The size of the camera images is 480x640 pixels and images are displayed full-screen. Statistical analysis was performed with R 4 .

Twelve unpaid volunteers (4 female, 8 male; 1 left-handed and 1 ambidextrous), ranging in age from 22 to 45 years (mean 27 years), were recruited from our institution. All participants had previous experience with touch-based handheld devices (seven on a daily basis) amongst whom nine had used a handheld tablet before.

Results.

User preference. The questionnaire was composed of 7 questions taken from the System Usability Scale questionnaire [START_REF] Brooke | SUS-a quick and dirty usability scale[END_REF] (except questions 4, 5 and 6 that are applied to more complex systems). Responses were on four point Likert scale and gathered as a global usability score ranging from 0 (poor) to 21 (high). The overall median score is 17/21, Crosshair has the lowest median score (14/21), followed by Relative Pointing (16.5/21), then Shift&Freeze (17.5/21) and finally Direct Touch (18/21) (figure 8left). Score differences are not statistically significant (Kruskal-Wallis rank sum test of score by technique: X 2 =6.651, p>0.05). Participants also ranked each technique in terms of overall satisfaction, speed and accuracy on four point Likert scale (figure 8-center). Kruskal-Wallis rank sum tests found significant differences between techniques for overall satisfaction (X 2 =14.3897, p<.01) and accuracy (X 2 =24.2827, p<.0001) rankings by technique. A post-hoc pairwise comparison of overall satisfaction and accuracy rankings shows significant difference (with p<0.05 for overall satisfaction and p<0.01 for accuracy) for all pairwise comparisons except for the comparisons between Shift&Freeze and Relative Pointing and between Crosshair and Direct Touch. Table 1 gives satisfaction and accuracy ranking means. Finally, during the experiment debriefing, we asked participants for the techniques they found to be the fastest and the more precise and for the techniques they preferred overall (multiple answers were allowed). Six participants said Relative Pointing seems to be the fastest one, four said Direct Touch, three said Crosshair and one said Shift&Freeze. All but one participant said Shift&Freeze seems to be the most precise one and five said it was Relative Pointing. Eight participants preferred Relative Pointing and six preferred Shift&Freeze. Two more participants would have also preferred Shift&Freeze given that it provided zoom and a cancel option.

Crosshair

Selection accuracy. We looked at the spread of selection points around the small targets of Task 4 (2mm wide). From 288 target selections, we removed 7 outliers noted during the experiment. The overall median of distances to the targets on the map is 1.7cm. The Direct Touch median (2.4cm) is more than twice that of Relative Pointing (0.9cm) and Shift&Freeze (1.0cm). The Crosshair median (1.6cm) lies in between (figure 8-right).

Discussion. These results support hypothesis H1. Indeed, Shift&Freeze and Relative Pointing are preferred over Direct Touch and Crosshair. Participants gave the best ranks in terms of accuracy and overall satisfaction to Shift&Freeze and Relative Pointing. Moreover participants have never mentioned either Direct Touch or Crosshair when asked for their preferred technique or for the most precise technique. However, these results do not support hypothesis H2. Crosshair received the lowest SUS scores, and Shift&Freeze was almost unanimously said to be the most precise technique. Moreover Shift&Freeze and Relative Pointing were almost equally preferred. So, indirect pointing techniques were not preferred over direct touch-based ones (i.e. Direct Touch and Shift&Freeze) even if the tablet form factor was presumably less convenient for direct touch-based techniques. Actually, some of our participants were used to direct touch input up to the point that they were tempted to tap on the cursor for the two indirect pointing techniques (Crosshair and Relative Pointing). The trend given by the measurable results is consistent with feedback gathered during the interviews.

Some participants complained about the handheld tablet form factor. A first reason is that due to its size and weight it is best held with both hands, but, as already explained, this impairs access to the screen with the Direct Touch and Shift&Freeze techniques. A second reason is that holding the tablet for AR application is different from other applications. The user needs to maintain the camera focus while interacting with the screen. Some participants felt that they held the tablet unsafely as they found it to be slippery and proposed to add some grips to the device. Also, the screen borders are not broad enough to allow all users to hold the tablet with their thumb on the side of the screen. This results in accidental touch inputs and an uncomfortable tablet hold when trying to hold the tablet with one hand in order to interact with the other one.

As for the distance from the wall map, most of the participants walked about the same distance for all tasks and all techniques. This is a surprising result since we expected the participants to adapt the distance to the map according to the difficulty of the task. Only one participant clearly adapted his distance from the map according to both target size and ease of manipulation of the technique. He did so up to the point that he did not walk at all for large targets with Relative Pointing (as he felt more comfortable with this technique).

The spread of selection points around small targets suggests that Relative Pointing and Shift&Freeze have higher accuracy than the two baseline techniques. It also suggests that Direct Touch is the least precise technique and that Crosshair has an intermediate accuracy. In the next controlled experiment, we further study small target acquisition.

Experiment 2: Performance

For the second experiment, we made the following hypothesis:

• H1: Relative Pointing and Shift&Freeze are more accurate than Direct Touch and Crosshair but they take longer to operate for small targets. • H2: Crosshair is more accurate than Direct Touch. While both techniques are impaired by hand jitter, Crosshair does not suffer from finger occlusion. • H3: Relative Pointing and Shift&Freeze offers similar accuracy. Both techniques overcome limitations inherent to touch input and hand jitter.

Procedure, apparatus and participants. This experiment was carried out utilizing the cyclical multi-direction pointing task paradigm. We used thirteen targets arranged in a circle on a remote screen. As the handheld tablet application uses computer vision to track the device's pose, the targets were overlaid on a background image. One target at a time was highlighted in black on the remote screen: this target must be selected by pointing at it on the tablet through the live video. In order to ensure a good visibility of the highlighted item regardless of its width, it was surrounded by a 3 cm wide white square with a green cross. Targets always appear in the same order: starting from the top item, the next item is always opposite and slightly clockwise from the selected one. One block thus consists of thirteen target selections plus the selection of the first target. The subjects were instructed to hold the tablet in portrait mode, to select the highlighted target as quickly and accurately as possible and to rest between blocks.

We used a single movement amplitude of 30 cm and 3 target widths (0.5 cm, 1 cm and 2 cm). We wished to have a consistent distance between the remote screen and the handheld tablet across participants and blocks. To do so, before each block, participants had to place the handheld tablet 1 meter (+/-5cm) away from the remote screen by following indications displayed on the tablet screen. Those indications were hid-den as soon as subjects started the block to avoid disturbing them during the experiment.

Presentation ordering of the four techniques and the three widths were counterbalanced using Latin squares. Each condition was presented three times including one time for training. The experimental design is: For the handheld tablet, we used a similar apparatus as for the previous experiment. In addition, the targets were displayed on a 27" Apple Thunderbold display with 2560x1440 pixel resolution (109 dpi). The screen was placed vertically so that its center was 1.5m high from the ground. An ad-hoc application was developed to control target widths and to highlight the target on the remote screen.

Twelve unpaid volunteers (4 female, 8 male; 1 left-handed), ranging in age from 22 to 41 years (mean 30 years), were recruited from our institution. All participants had previous experience with touch-based handheld device (nine on a daily basis) amongst whom ten had used a touch-based tablet before.

Results.

From 3744 selections, we removed 33 obvious outliers. Distance from the screen when selections were performed is on average 1.02m (1 st quartile: 0.99m, 3 rd quartile: 1.05m, range: 0.90m to 1.18m). This indicates that our experimental set-up that constrains participants to placing the handheld tablet at 1m (+/-5cm) from the screen before starting each block was sufficient to confine the distance between the handheld tablet and the remote screen to a small range.

Errors. A Pearson's Chi-squared independence test between success of target acquisition and the 4 Techniques shows a significant dependence (X 2 = 616.0356, p < .0001). The overall error rate is 44.6%. This high error rate is explained by the choice of rather small target widths. The lowest error rate over the 3 target Widths is for Relative Pointing (20.1%), then Shift&Freeze (34.6%), Crosshair (49.4%) and the highest error rate is observed for Direct Touch (75.0%) (Figure 9-left).

We performed a 4 x 3 (Technique x Width) within subject analysis of variance on error rate by user. The Technique (F 3,143 =50.835; p<.0001) and Width (F 2,143 =57.286; p<.0001) main effects as well as the Technique:Width interaction (F 6,143 =3.397; p<.01) were found significant. A post-hoc Tukey multiple means comparison found significant difference for all comparisons. Differences between Relative Pointing and Shift&Freeze and between Shift&Freeze and Crosshair were found significant with p<.012. All other differences were found significant with p<.0001.

We also performed a 4 x 3 (Technique x Width) within subject analysis of variance on median by user of distance between target center and selection point. Significant main effects were found for Technique (F 3,143 = 42.605; p < .0001) and Width with p<.015 (F 2,143 = 4.389). Technique:Width interaction was not significant. A post-hoc Tukey multiple means comparison found significant difference for all comparisons (with p<.01 for Relative Pointing-Crosshair comparison and p<.0001 for the others) except between Relative Pointing and Shift&Freeze and between Shift&Freeze and Crosshair. Duration. The overall median of selection durations is 2.1 seconds, and medians of all selections for each technique are 2.7 seconds for Relative Pointing, 2.5 seconds for Shift&Freeze, 2.2 seconds for Crosshair and 1.0 second for Direct Touch (figure 8right). We performed a 4 x 3 (Technique x Width) within subject analysis of variance on median of selection durations by user. Significant main effects were found for Technique (F 3,143 = 67.781, p<.0001) and Width (F 2,143 = 17.478, p<.0001). Effect of the Technique:Width interaction was also significant though with p=.0127 (F 6,143 = 2.827). A post-hoc Tukey multiple means comparison found significant difference for all (with at least p<.01) except for two comparisons. Again, the differences between Relative Pointing and Shift&Freeze and between Shift&Freeze and Crosshair were not significant.

Discussion. The chosen tasks were quite hard to perform, which results in high error rates for all the techniques. Still we can observe that the different techniques offer different trade-offs between speed and accuracy or better said here, between duration and error rate (figure 9). The results partly support hypothesis H1. Relative Pointing is significantly more accurate and longer to operate than the two baseline techniques (i.e. Direct Touch and Crosshair), but this is not the case for Shift&Freeze. Indeed, Shift&Freeze does not show significant difference with Crosshair. Crosshair is significantly more accurate than Direct Touch, which supports hypothesis H2. Actually, Direct Touch is not adapted for those small target widths as indicated by both high error rates and no difference of duration across target widths. Relative Pointing and Shift&Freeze offers similar performance as indicated by non-significant difference of both duration and error distance. This supports hypothesis H3.

While participants held the tablet with both hands with Relative Pointing and Crosshair, they adopted different strategies with Direct Touch and Shift&Freeze. For Direct Touch and Shift&Freeze, participants used two different strategies: (1) holding the tablet with one hand and interacting with the other hand's finger (9/12 for Direct Touch and 6/12 for Shift&Freeze) and (2) holding the tablet with both hands and interacting with their thumb (3/12 for Touch and 6/12 for Shift&Freeze). This highlights the trade-off between holding the tablet and interacting on the screen for direct touch based techniques.

One drawback of this experiment is that the choice of the selection mode for Shift&Freeze and Relative Pointing was left to the participants. This results in different strategies as some users always used the precise mode while others adapted the mode according to the difficulty of the task. Nevertheless our goal was to evaluate our two techniques that include two modes.

Conclusion and Future Work

This paper provides a comprehensive study on precise pointing techniques for handheld Augmented Reality (AR). Our contributions are twofold. First we have presented an analytical framework for the design of interaction techniques for handheld AR that is based on the relationship between the touch input space and two visual output frames, namely the screen and the physical object. The usefulness of the framework is demonstrated by the classification of existing techniques and the design of two pointing techniques. Second we have presented two pointing techniques, Shift&Freeze and Relative Pointing. Their respective designs result from a twofold strategy: Shift&Freeze is conceived as an improvement of Direct Touch and solves its accuracy problem by freezing the video and using Shift's callout. Relative Pointing improves on the screen-centered Crosshair technique by stabilizing the cursor on the remote physical object. The two experiments revealed that those two techniques are preferred to the two commonly used techniques (Direct Touch and screen-centered Crosshair) and are more accurate than these baseline techniques. Further controlled studies must be performed to compare the two techniques, firstly, with less difficult tasks and then with a phone as the tablet form factor probably favors Relative Pointing. We also plan to run experiments with 3D physical objects (e.g., production machines). Several extensions to the two proposed techniques are envisioned including zooming for Shift&Freeze and testing different dynamic transfer functions (with or without known targets) for Relating Pointing. Finally, since our studies formulate the hypothesis of a perfect tracking of the device's pose, one further research avenue we must explore is the design of handheld AR pointing techniques that takes into account the imperfection of the underlying tracking system. We expect that precise pointing techniques in any context of use will become more and more crucial in the future, as a large range of richer and more complex handheld AR applications are designed.

Fig. 1 .

 1 Fig. 1. Four techniques for pointing at physical objects through video on handheld devices: (a) Direct Touch on the live video; (b) Shift&Freeze: Shift [19] combined with freeze-frame; (c) Screen-centered Crosshair; (d) Relative Pointing with cursor stabilized on the physical object.

Fig. 2 .

 2 Fig. 2. Relationships between the frames of reference of touch input, of the screen and of the physical object on the screen: (a) with live video playback; (b) while the video is frozen.

Fig. 3 .

 3 Fig. 3. Pointing through live video: four cases.

Fig. 4 .

 4 Fig. 4. Shift&Freeze walkthrough. (a-e) Small target acquisition with Shift and frozen video;(f-g) Large target acquisition with the Direct Touch technique (one tap on the screen).

Fig. 5 .

 5 Fig. 5. Relative Pointing walkthrough. (a-e) Small target acquisition with a relative cursor stabilized on the physical object; (f-g) Large target acquisition with the crosshair technique.

Figure 5

 5 Figure5shows a walkthrough of the two modes of our Relative Pointing technique.Scenario 1: (a) The user points the handheld device camera at the target so that the target appears on the screen. (b) In order to mitigate the instability due to hand tremor, when the user touches the screen and starts moving its finger on the screen, a relative pointing mode is triggered. (c) While in this mode, the cursor is no longer bound to the screen center. Instead, it is stabilized on the remote physical object at its current position. The user fine-tunes the cursor position by controlling the cursor indirectly with finger gestures on the screen. (d) On finger lift, no special action is performed. (e) The validation of a position is performed with a tap on the screen. Upon validation, a short animation moves the cursor back to the screen center, thus leaving the relative pointing mode.Scenario 2: (f-g) When acquiring large enough targets, hand tremor is not a problem. In this case, the user does not need to use the relative pointing mode and can trigger a target acquisition at the position of the screen-centered cursor with a tap on the screen. This is similar to the screen-centered Crosshair technique.To make relative pointing effective for handheld AR context, the following issues have been addressed.

Fig. 6 .

 6 Fig. 6. With Relative Pointing: (left) Effect of screen rotation on the cursor's position; (right) The cursor is stabilized in the physical object's frame of reference and relative touch motions are applied on the screen.

Fig. 7 .

 7 Fig. 7. Experimental set-up: (left) Participants started 2.5m away from the physical wall map and then could move freely to perform the pointing tasks; (right) the targets used for each task. Tasks 3, 4 and 5 consist of placing AR marks on 2 targets.

Fig. 8 .

 8 Fig. 8. For each technique: (left) Questionnaire scores; (center) Histograms of overall satisfaction, speed and accuracy rankings; (right) Boxplot of distances between the selection point and the target's center for Task 4 (2mm targets).

4 Techniques x 3

 43 Widths x 2 Blocks x 13 Selections = 312 acquisitions per subject, and 4 Techniques x 3 Widths x 13 Training Selections =156 training acquisitions per subject.

Fig. 9 .

 9 Fig. 9. (left) Error rates (%) and (right) selection durations (sec.) by Technique by Width.

Table 1 .

 1 Means of satisfaction and accuracy ranking by techniques.

	Rank Means Crosshair	Relative Pointing	Shift&Freeze	Direct Touch
	Satisfaction 1.75	2.75	2.67	2
	Accuracy	1.5	2.75	2.83	1.67

http://www.khronos.org/opengles/1_X/

https://www.vuforia.com/

http://www.libpointing.org/

http://www.R-project.org

Acknowledgements. This work has been supported by the French-Japanese ANR/JST AMIE project (http://amie.imag.fr). Special thanks to G. Serghiou for reviewing the article and to N. Mandran (LIG) for helping with the experiments.