Romain Terrier
email: romain.terrier@b-com.com

Ferran Argelaguet

Jean-Marie Normand
email: jean-marie.normand@ec-nantes.fr

Maud Marchal

Evaluation of AR Inconsistencies on AR Placement Tasks: A VR Simulation Study

Keywords: Registration Errors, Augmented Reality, VR Simulation, Interaction

One of the major challenges of Augmented Reality (AR) is the registration of virtual and real contents. When errors occur during the registration process, inconsistencies between real and virtual contents arise and can alter user interaction. In this paper, we assess the impact of registration errors on the user performance and behaviour during an AR pick-and-place task in a Virtual Reality (VR) simulation. The VR simulation ensured the repeatability and control over experimental conditions. The paper describes the VR simulation framework used and three experiments studying how registration errors (e.g., rotational errors, positional errors, shaking) and visualization modalities (e.g., transparency, occlusion) modify the user behaviour while performing a pick-and-place task. Our results show that users kept a constant behavior during the task, i.e., the interaction was driven either by the VR or the AR content, except if the registration errors did not enable to efficiently perform the task. Furthermore, users showed preference towards an half-transparent AR in which correct depth sorting is provided between AR and VR contents. Taken together, our results open perspectives for the design and evaluation of AR applications through VR simulation frameworks.

Introduction

Current advances in Augmented Reality (AR) technology (e.g., Microsoft HoloLens or Meta 2 glasses) as well as in tracking capabilities [START_REF] Billinghurst | A Survey of Augmented Reality[END_REF] (with e.g., the release of Apple's ARKit and Google's ARCode SDKs) are showing the potential of AR applications in consumer grade applications (e.g., entertainment, education or maintenance). However, despite these recent advances, AR is still confronted to a number of challenges such as occlusion management between virtual and real objects, limitations in the field-of-view of AR devices or, as it is the main focus in this paper, registration errors. Registration is generally referred to as the process of finding in real-time the position and orientation of virtual objects so that they can be integrated in a plausible way in the real world. When errors occur during this process, inconsistencies between real and virtual objects can arise. Such inconsistencies can be constant (e.g., fixed errors on position and orientation) or irregular (e.g., shakiness of the virtual content), and can potentially hinder user interaction or alter users' behavior. Furthermore, as mentioned by Azuma and Bishop [START_REF] Azuma | Improving static and dynamic registration in an optical seethrough hmd[END_REF] "The human visual system is very good at detecting even small misregistrations [...]. Errors of just a few pixels are noticeable". As of today, most AR applications or SDKs still face many of these inconsistencies, especially when the user modifies the scene through direct interaction with real or virtual objects.

Studying how registration issues affect users when interacting with AR is of great importance but remains difficult to achieve. Indeed, since AR is typically presented on hand-held devices (tablets, phones) or Head-Mounted Displays (HMDs), it generally remains complex to propose repeatable and controlled user studies in AR environments. To that purpose, Virtual Reality (VR) provides a promising tool to evaluate, not only AR interfaces, but also to explore how current limitations of AR systems influence users' behaviors and interaction capabilities [START_REF] Lee | The role of latency in the validity of AR simulation[END_REF]. Carrying out controlled VR experiments enables to explore a particular subset of limitations while perfectly simulating other AR features (e.g., ensuring a perfect tracking).

In this paper, we explore, through a VR simulation of AR, how user behavior and performance is altered when registration errors occur. Indeed, while it has been shown that the perception of the co-existence of virtual and real objects can be altered by registration accuracy (e.g., misregistration distorts spatial relationships [START_REF] Nabiyouni | Relative Effects of Real-world and Virtual-World Latency on an Augmented Reality Training Task: An AR Simulation Experiment[END_REF]), little is known of their impact on the interaction process. Through three different experiments participants were confronted with different degrees of registration errors and AR visualizations in a VR simulation environment while performing a pick-and-place task (a common manipulation task). The main goal of the experiments were [START_REF] Azimi | Robust optical see-through head-mounted display calibration: Taking anisotropic nature of user interaction errors into account[END_REF] to explore how the intensity of registration errors alters the behavior of users, in particular, whether real or augmented content drive their interactions, and (2) to measure how irregular registration errors impact users' accuracy.

The remainder of the paper is structured as follows: Section 2 provides an overview on registration errors in AR as well as on how VR simulation is used to study AR systems. Then, Section 3 presents the VR simulation platform we used in our experiments. Sections 4, 5 and 6 detail three different experiments aiming to analyze the impact of registration errors on users' behaviors and performance. Finally, Section 7 presents a global discussion and Section 8 concludes the paper.

Related Work

Categorization of Registration Errors in AR

A critical issue in AR applications is when virtual information is misaligned with the real environment [START_REF] Azuma | A survey of augmented reality[END_REF]. This misalignment is also called registration errors [START_REF] Holloway | Registration errors in augmented reality systems[END_REF]. From the users' viewpoint, registration errors can be seen as if the AR information floats and can break the illusion that real and virtual objects co-exist [START_REF] Nabiyouni | Relative Effects of Real-world and Virtual-World Latency on an Augmented Reality Training Task: An AR Simulation Experiment[END_REF].

Registration should be stable both spatially (virtual objects must be collocated in rotation and position with real objects) and temporally (motion of virtual and real objects should be synchronous). In the literature, several classifications of registration errors have been presented [START_REF] Holloway | Registration error analysis for augmented reality[END_REF][START_REF] Azuma | A survey of augmented reality[END_REF]. Holloway [START_REF] Holloway | Registration error analysis for augmented reality[END_REF] gives a precise definition of registration errors by decomposing them into four main metrics: linear, lateral, angular and depth registration errors. In contrast, the classification proposed by Azuma [START_REF] Azuma | A survey of augmented reality[END_REF] focused on whether or not AR objects or the user's viewpoint were static (static) or in motion (dynamic). The remainder of this section is structured following the taxonomy of Azuma [START_REF] Azuma | A survey of augmented reality[END_REF] where static errors represent a spatial incoherence between AR and real content while dynamic errors represent a temporal incoherence.

Static Errors Static errors are visible even if the user does not move his/her viewpoint (or the hand-held device) or when the real environment remains immobile. From the user's viewpoint, the AR content seems to be floating near its real position. Static registration errors are due to either optical distortions, tracking errors, mechanical misalignments or incorrect viewing parameters (i.e., field of view, tracker-to-eye position and orientation, inter-pupillary distance) [START_REF] Azuma | A survey of augmented reality[END_REF]. Most of the time users perceive static errors as a constant gap between the desired and the actual position/orientation of the AR information even if the AR content has a perfect shape (see Fig. 3

for examples of static errors).

There are several ways to reduce static errors: improve calibration [START_REF] Azimi | Robust optical see-through head-mounted display calibration: Taking anisotropic nature of user interaction errors into account[END_REF] or improve tracking techniques [START_REF] Klein | Parallel tracking and mapping for small ar workspaces[END_REF]. Nevertheless, the huge variability of environments (e.g., indoor, outdoor) and behaviors of tracked objects (e.g., static, in motion, slow, fast) makes it complex to provide error-prone solutions [START_REF] Zhou | Trends in augmented reality tracking, interaction and display: A review of ten years of ismar[END_REF].

Dynamic Errors On the other hand, dynamic errors are only visible when the user's viewpoint or when objects are moving. Dynamic errors are mainly due to the latency of the system when there is a motion [START_REF] Azuma | A survey of augmented reality[END_REF]. This delay (or latency) is the time between the moment when the tracking system computes the new position and rotation of the viewpoint and the moment when the virtual information is rendered at this position. This delay implies that virtual objects are not displayed with the right position and orientation at the right time. In order to reduce dynamic errors, four methods exist [START_REF] Azuma | A survey of augmented reality[END_REF]: reduce system lag, reduce apparent lag, match temporal streams and predictive methods. Another characteristic of registration errors is that they are hard to predict. Such uncertainty introduces a strong bias when evaluating AR systems as replicability is compromised. In order to overcome such limitations, VR has been proposed to simulate AR systems and thus evaluating them.

Simulating AR in VR

Virtual reality is a powerful tool to evaluate AR systems as it allows to simulate in a controlled way many AR features, enabling repeatable user studies. Several studies conducted in VR environments simulate AR systems in order to evaluate features such as latency [START_REF] Ragan | Simulation of augmented reality systems in purely virtual environments[END_REF][START_REF] Nabiyouni | Relative Effects of Real-world and Virtual-World Latency on an Augmented Reality Training Task: An AR Simulation Experiment[END_REF], field of view [START_REF] Ren | Evaluating wide-field-of-view augmented reality with mixed reality simulation[END_REF][START_REF] Renner | Evaluation of attention guiding techniques for augmented reality-based assistance in picking and assembly tasks[END_REF] or visual realism [START_REF] Lee | The Effects of Visual Realism on Search Tasks in Mixed Reality Simulation[END_REF].

Regarding latency, Lee et al. [START_REF] Lee | A replication study testing the validity of AR simulation in VR for controlled experiments[END_REF] replicated an AR study [START_REF] Ellis | Factors influencing operator interaction with virtual objects viewed via headmounted see-through displays: viewing conditions and rendering latency[END_REF] in a VR simulation context. Their results showed that users' performance when moving a virtual ring along a virtual path was comparable between the simulated AR and the real AR studies. The VR simulator provided the feeling that the AR content was real thanks to a restricted transparent window in which the AR simulated content was displayed. Moreover, on top of the internal latency of the simulator due to tracker latency, computation time, render time and display time, Lee et al. [START_REF] Lee | The role of latency in the validity of AR simulation[END_REF] proposed to include "artificial latencies" between virtual objects and the real world. This enabled to add controlled latency to the simulated AR content and to analyze its effect. Other studies also linked latency with interaction performance showing a degradation of performance as latency increases [START_REF] Nabiyouni | Relative Effects of Real-world and Virtual-World Latency on an Augmented Reality Training Task: An AR Simulation Experiment[END_REF]. Finally, Ragan et al. [START_REF] Ragan | Simulation of augmented reality systems in purely virtual environments[END_REF] studied the impact of jitter (visual shakiness of AR content) and showed that it is predominant over visual latency.

The Field of View (FOV) is a distinctive feature of HMDs that has also been evaluated in VR simulations. While comparing different AR HMDs with different FOV is prone to bias due to confounding factors (e.g., other HMDs characteristics) VR simulations enable to only alter the FOV parameter while minimizing confounding factors. For example, Ren et al. [START_REF] Ren | Evaluating wide-field-of-view augmented reality with mixed reality simulation[END_REF] showed that a wider FOV is better because it allows to display more information (2D annotations in their case) and to explore it more quickly. Moreover, a wider FOV also reduces users' head movements in the search task. Another important factor related to FOV corresponds to where guidance information regarding objects of interest is displayed (e.g., within the FOV, outside of the FOV, on the object, etc.). Users better focus during a manipulation task if AR guidance is drawn as a line connecting the hand of the user to the center of the searched object [START_REF] Renner | Evaluation of attention guiding techniques for augmented reality-based assistance in picking and assembly tasks[END_REF]. In another study, Baričević et al. [4] proposed to use VR to simulate user perspective rendering in AR (the view is rendered according to the user's point of view) in order to study its benefits over the classical device perspective (the view is rendered according to the point of view of the device's camera) for AR applications.

Furthermore, visual realism is also an important factor that simulated AR has to deal with. Lee conducted a study [START_REF] Lee | The Effects of Visual Realism on Search Tasks in Mixed Reality Simulation[END_REF] about the impact of the realism of the VR environment (photo-realistic, etc.) on a user task. Results did not show relevant effects and the necessity to design a realistic VR environment has not been proved in this case. Finally, AR content presentation (transparency of the virtual content or occlusion management) has also been shown to impact interaction with AR systems [START_REF] Ellis | Localization of virtual objects in the near visual field[END_REF][START_REF] Furmanski | Augmented-reality visualizations guided by cognition: Perceptual heuristics for combining visible and obscured information[END_REF]. In order to study its impact, we integrated those conditions into our experiments.

As a conclusion, although registration is a major challenge of AR, there is still a lack of studies on its impact in the interaction process. Such studies are complex to perform using existing AR systems since it is nearly impossible to ensure experimental conditions comparable across participants (e.g., registration errors will be hardly reproducible). To this end, inspired by existing works, we propose to study the impact of registration errors in a VR simulation of an AR interaction task.

Experimental Platform: VR Simulation of AR

As mentioned above, in order to be able to carry out repeatable and fully controlled user experiments, we chose to simulate our AR environments in VR. Our VR environment was designed to replicate a pick-and-place task of an AR application where the user has to position a cube (10x10x10cm) precisely onto a target. The cube and the target were lying on a table and both could be augmented (see Fig. 1, bottom).

In our VR simulation, some virtual objects played the role of real objects in while others represented AR objects (i.e. virtual objects inserted into a real scene). As a consequence the virtual environment consisted of a set of "real objects", or simulated Real (sReal) objects, and AR objects, or simulated AR (sAR) objects. The sReal objects could also be augmented by sAR objects.

Additionally, we could also manipulate the virtual AR FOV in the simulation. This allowed the simulation of both a device-based and of an HMD-based AR environment by adapting the size of a virtual window with a slightly different color. In our experiments, the virtual AR FOV covered ∼ 90% of the VR HMD's FOV. We chose this value so that users would not be disturbed in their interaction with the sAR content. Fig. 1 (bottom) shows two first person views of the virtual environment were sAR and sReal objects co-exist.

The following subsections detail the different simulated conditions in terms of registration errors and AR content presentation. Only sAR objects were affected by registration errors (see Fig. 2) and sAR visualization conditions (see Fig. 3).

Registration Errors

In order to study how registration errors would affect user interaction and performance in an AR pick-and-place task, we propose to simulate two different kinds of registration errors (see Fig. 2): constant and irregular. For ours experiments, we make the choice to focus the analysis on errors with only a rotation of the sAR content with respect of the sReal content around the Y-axis. Those errors are called rotational errors. The choice is motivated by the simplicity of the error that has only one degree of freedom. Translation errors are not in the scope of our study. Constant registration errors introduce a constant misalignment between sAR and sReal objects (either in position or orientation). In this condition, no matter how the user manipulates the sReal cube, its sAR counterpart is always misaligned by the same amount either in position or rotation. We do not use the term "Static" registration errors proposed by Azuma [START_REF] Azuma | A survey of augmented reality[END_REF] (that only happen when the user's viewpoint does not move) since in VR we can simulate a constant error even with a dynamic environment (objects in motion or user's changing his/her viewpoint). For simplicity, and due to the nature of the pick-and-place task, in order to avoid inter-penetrations we only considered rotation errors on the vertical axis (Y-axis). sAR content is rotated on its own Y-axis with respect to the sReal object.

Irregular registration errors are dynamic and variable over time. There are two kinds of such errors that we chose to name jitter and intermittent vibrations. Here again we chose not to use the term "Dynamic" [START_REF] Azuma | A survey of augmented reality[END_REF] since the registration errors we simulate are not due to latency or delay but we rather wanted to study the effect on a non-constant registration error in a dynamic environment.

Intermittent vibrations are defined as discrete vibrations of the sAR object. More specifically, a pulse is generated on the sAR object's Y-axis for a certain amount of time (e.g., for 0.1s every 1s). The pulse follows a noise function f (see Equation 1) with x 1 , x 2 being two real values ∈ [0, 1], and k a real positive constant which determines the maximum absolute rotation error (in radians).

f (x 1 , x 2 ) = sin(2πx 2 ) * -2 log (x 1 ) 3k (1)
On the other hand, "jitter " refers to a continuous vibration of the sAR object around it's Y-axis. The "jitter " is generated continuously for a certain amount of time and follows the same function f as intermittent vibrations. We refer the reader to the supplementary material which illustrates the intermittent vibrations and jitter conditions.

AR Content Presentation

The second aspect of the simulation we wanted to evaluate was the presentation of the AR content. Although numerous rendering styles could have been considered, we decided to focus on transparency of the AR content and occlusion management (whether or not AR objects were correctly occluded by real objects) of sAR objects, as illustrated in Fig. 3. In particular the two considered conditions were:

1. "Transparency" of the sAR content. A transparency of 100% makes the sAR object invisible while a transparency of 0% makes it completely opaque. 2. "Occlusion" management. This allows us to control whether the sAR content was properly (Occlusion ON) or not (Occlusion OFF) occluded by the sReal content.

Experiment 1: Analyzing the Effect of Constant Registration Errors in User Behaviour

The goal of the first experiment was to explore how registration errors influence the perception of co-existing sAR and sReal objects. In particular, the experiment aimed at determining which object is used as a reference (sAR or sReal) or in other words, which content drives the users' actions. Furthermore, as different AR visualizations might play an important role on users' behaviour, different sAR visualizations were evaluated.

Apparatus and Participants

Experiments were performed using an HMD (HTC Vive) and users were able to interact with the environment through a controller (an HTC Controller).

During the experiment, the users were comfortably seated in an office chair.

The experiment was conducted using the platform described in Sec. 3 which was implemented in Unity (5.6).

Twelve right-handed users (1 female, 11 males) participated in this experiment (age: M = 22.83; SD = 2.66). All participants had previous experience with AR and/or VR.

Experimental Protocol

Upon arrival participants read and signed a consent form which briefly described the experiment and their rights. The consent form did not provide any information that could bias the users during the experiment. At the end of the experiment additional information was provided regarding the real purpose of the experiment. Users were told that they should consider the VR environment as their reality (i.e. with sReal objects) and that the AR content (i.e. sAR objects) which is displayed in the brown window is the AR. In addition, users had to fill out a pre-experiment questionnaire to gather background information (e.g. age, VR and AR experience, headset experience, laterality, visual impairment). Once users fully understood the experimental task, the experimenter equipped them with the HMD and they were immersed in the virtual experimentation room.

The experimental task was a pick-and-place. Users had to pick a sReal cube augmented with an sAR cube (see Fig. 4) and place it at the center of a colored target (also a sReal object). Users had to orient the cube to match the color code, (e.g. the blue face of the sAR cube should face the blue marker on the target, etc., see Fig. 4). The field of view of the AR window was enough to ensure that all cubes fit the AR display. Participants were asked to be as precise as possible and no indication was given regarding which cube (sAR or sReal) had to be aligned on the target. Users performed several trials grouped in four blocks (see Sec. 4.3).

The users' virtual hand was represented as a green sphere to let them focus on the manipulation and not on their appearance. The green sphere was controlled with the HTC Vive controller (the trigger enabled to grab the sReal cube). Once users were pleased with the location of the cube, they had to signal the end of the trial by pressing the touchpad of the Vive controller. Basic physical simulation capabilities were enabled (gravity and collision detection with the virtual table). At the end of each block, participants had to fill out a questionnaire to gather their subjective impressions.

Experimental Design

In order to assess the effect of registration errors, we artificially considered a different range of constant registration errors. In particular, we considered seven rotational mismatches {15, 10, 5, 0, -5, -10, -15} • . In addition, we also considered four different visualizations for sAR objects. The visualization was by two independent variables (transparency and occlusion) with two levels each (see Figure 3) in order to analyze whether they biased the relationship between sAR and sReal content. The transparency of the sAR content was set to either 0% or 50%. Occlusion of the sAR content also had two options: ON (the sAR content is always visible) or OFF (the sAR content is correctly occluded by sReal objects depending on which object is closer to the user's viewpoint). To sum up, the experiment had a 7x2x2 factorial design with 4 repetitions for each condition, resulting in a total of 112 trials. The four combinations of transparency and occlusion were split into four blocks and counterbalanced using a Latin-Square design. For each block (28 trials), the order of the registration mismatch was randomized.

The dependent variables were the task-completion time (s), the position accuracy (cm) and the rotational accuracy (degrees). The task-completion time was measured from the moment the user grabs the cube until he/she validates the placement. The position accuracy was computed as the distance between the center of the sReal cube (sReal and sAR cubes shared the same center position) with respect to the center of the perfect placement position. The rotation accuracy was computed as the minimum angle between the sReal and sAR cubes with respect to the perfect placement rotation (see Figure 5). Participants had no explicit instruction whether hey have to align the sReal or the sAR cube, considering the minimum provides a more insightful result. Positional and rotational accuracies are measured for each trial upon user validation of the placement.

Regarding subjective information, participants had to fill out a questionnaire after each block and at the end of the experiment. The aim of the after-block Fig. 5. Depiction of how rotational error θe is calculated. The blue, orange and red cross is the target. The textured square is the sReal cube. The green dotted line square is the sAR cube. In this example, the error is the angle between the sReal cube and the sReal target and not between the sAR cube and the sReal target because the angle of the error between the two sReal objects is the lowest.

questionnaires was to observe the progression of the state of the participants (i.e., tiredness, visual discomfort and task difficulty). A 7-point Likert scale was used where 1 meant "not at all" and 7 "meant completely". The final questionnaire was designed to understand the behaviour of the participants and their impression on their performance. Therefore, the participants answered which reference they chose and if they placed the cube in relation to the sAR content or in relation to the sReal content. Then participants quantified the variability of the choice about the picked reference. Additionally, the participants evaluated on a scale of one (never) to seven (always) how often their choice changed. Finally, participants ranked the 4 conditions from their most preferred to their least preferred one. In the final questionnaire, we use the term "AR cube" to refer to the sAR cube and the term "real cube" to refer to the sReal one. Given this experimental design, we hypothesized that:

H1.1 Smaller registration errors will result in lower rotation accuracy. The bigger is the mismatch the lower will be the ambiguity. H1.2 When the sAR content is semitransparent (i.e. Transparency 50%) the task completion time will be higher. Seeing sAR and sReal contents at the same time can be distracting. H1.3 When the sAR content is displayed on top (i.e. Occlusion ON) it will result in lower task completion times due to the same reasoning as H1.2. H1.4 As only rotational accuracy is considered and the sReal and sAR cubes share the same center position, no significant difference in positional accuracy is expected.

Results

Regarding the objective measures, the analysis of the data (ANOVA analysis with post-hoc Tukey tests (α > 0.05)) showed that the different independent variables had no impact on the dependent variables. All comparisons were notsignificant. For the sake of simplicity we only report the two-way ANOVA anal-ysis for the factors Transparency and Occlusion, pooling the data from the different levels of rotation mismatch. Anderson Darling tests were performed to ensure the normal distribution of the data. Rotation Accuracy. No significant main effect was found for Transparency (F1,11 = 0.77, p = 0.399, η 2 p = 0.06) nor for Occlusion (F1,11 = 0.83, p = 0.382, η 2 p = 0.07).

In overall, participants were extremely precise M = -0.12 deg; SD = 0.93 deg no matter the reference and the condition. In addition, as previously stated, the amount of registration mismatch did not influenced rotation accuracy. Thus, results do not support H1.1. Task-Completion Time. No significant main effect was found for Transparency (F1,11 = 1.27, p = 0.284, η 2 p = 0.10) nor for Occlusion (F1,11 = 2.83, p = 0.121,

η 2 p = 0.20).
In overall, participants required M = 10.086s; SD = 5.927s to complete the task. Although the mean task completion time was considerably high considering the nature of the task, we have to note that participants were requested to perform the task as accurately as possible. Thus, we cannot accept neither H1.2 nor H1.3. Position Accuracy. No significant main effect was found for Transparency (F1,11 = 2.31, p = 0.156, η 2 p = 0.17) nor for Occlusion (F1,11 = 0.36, p = 0.560, η 2 p = 0.03).

In overall, participants were extremely accurate M = 0.15cm; SD = 0.07cm no matter the visual condition. These results support H1.4. Yet, if any difference exists, considering the level of accuracy, it would be non-relevant.

Reference Cube

After analyzing users' behaviors, we observed that the majority of participants had a different yet consistent behaviour during the entire experiment. In particular, we computed the object (sAR or sReal cube) that was considered as the reference by the user. The reference cube is the cube (sAR or sReal) that minimizes the error to the target orientation at the moment the user validates the trial. Fig. 6 shows the amount of time the sReal cube was chosen as reference for each user and for each condition. Cases in which the rotation mismatch was 0 were not considered as they do not provide any information. Six out of twelve users considered the sReal cube as the main reference (users 2, 3, 5, 7, 8, and 12), three users considered the sAR cube as the main reference (users 1, 4 and 6), one user (user 10) did the task considering that both cubes were the reference (minimizing the error between both cubes) and two users (9 and 11) changed reference in the middle of the experiment.

Subjective Questionnaires

After-Block Questionnaires Concerning the task difficulty, users considered the task relatively easy M = 2.54; SD = 1.18 no matter the visual condition. Interestingly, small misalignments (e.g., ±5 deg) were reported to increase the difficulty of the task. Three users reported that "With small misalignments it was harder". Moreover, some users felt confused about the task: "I had the impression of making wrong choices" or "I was confused and a little bit disturbed because of misalignments". Concerning visual discomfort, it remained constant during the entire experiment M = 2.90; SD = 1.43, which shows that the task and the set-up do not cause strong visual discomforts. Finally regarding tiredness, the mean value is almost constant and below 3 M = 2.85; SD = 0.92.

Post-Experiment Questionnaire Nine users (75%) stated that they picked a reference and kept it during all the experiment, whereas three users (25%) reported to change their reference. Six users out of twelve (50%) chose the sReal cube, three chose the sAR cube and the others changed during the manipulation. Only one user picked all the time an imaginary cube placed at the mid-position between the sAR cube and the sReal cube. These results confirm results from the previous section. Globally users pick a reference and keep it during all the task. According to users, the choice of the reference was made because the task seems easier for them with their choice of reference: "With AR it was more practical". Finally, regarding user preferences on sAR visualization, the condition in which the sAR content was always visible and opaque was the less preferred one. Seven out of twelve participants (58.33%) answered that it was their less preferred choice. Yet, results do not show a clear user preference.

Discussion

The main outcome of this experiment is that constant registration errors did not alter the way users performed the task nor their performance. The analysis of users' behaviors as well as questionnaires showed that participants have a strong preference when choosing the main reference (either sReal or sAR) no matter the registration mismatch nor the visualization. Interestingly, while we hypothesized that the occlusion would have an influence on users' behavior, it was not the case, nor any other visualization condition. One explanation could be that users did not need to change their reference (there was no significant effect on accuracy) since in all cases they were able to accurately place the chosen cube.

Another interesting finding is that there was no significant difference in taskcompletion time for the different sAR visualization conditions. As users were requested to be as accurate as possible, there was a moderate user variability. This could have decreased the power of the analysis. Nevertheless, we were expecting strong differences and it was not the case. Finally, the condition that we hypothesized to be the less optimal (no transparency and no occlusion) was the condition users preferred less.

Experiment 2: Analyzing the Effect of Irregular Registration Errors in User Behaviour

The results of the previous experiment shown that when constant registration errors occur, they do not have a strong influence on users' behavior, as the reference chosen by users rarely changed during the experiment. In order to explore in depth users' behavior, we performed a follow up experiment in which we explored the effects of irregular registration errors. The goal of this second experiment was to analyze the tolerance to irregular registration errors and measure if there was a threshold that triggers a shift in the selected reference. The experimental protocol and the apparatus of the experiment were the same as the first experiment. Upon their arrival, participants read and signed a consent form which briefly described the experiment and their rights. After users fully understand the experimental task, the experimenter equipped them with the HMD.

Participants

Twelve right-handed users (2 females, 10 males) participated in this follow up experiment (M = 23.25; SD = 2.22). Seven never had any experience with VR, five never had any experience with AR and six never used a HMD before.

Experimental Design

Participants had to perform the same pick-and-place as in the first experiment task while varying the intensity of the registration error. We considered five different levels of intensity, defined by the amplitude of the vibration (the k parameter of Eq.1) and the time (discrete vs. continuous). The registration errors were presented by increasing intensity:

L1 No Error: Perfect registration between sAR and sReal objects. L2 Low intensity intermittent vibration: a pulse of 0.11s is generated every 1s with k = 0.3rad. L3 High intensity intermittent vibration: a pulse of 0.11s is generated every 0.5s with k = 0.3rad. L4 Low intensity jitter: a continuous vibration is generated with k = 0.05rad. L5 High intensity jitter: a continuous vibration is generated with k = 0.3rad.

Only one sAR visualization was considered which corresponded with the most preferred condition of the first experiment (50% transparency and Occlusion OFF). For each intensity level, participants performed 5 repetitions of the pick-and-place task. To better account for users' choice of reference, constant registration errors were also introduced for each intensity level (four repetitions randomly chosen between {-10, 10} • and one repetition had no constant registration error). The target is a sReal object once again. We were only interested in measuring the reference chosen for each trial, which we determined following the same approach as in the first experiment, choosing the object (sReal or sAR) which minimized the placement accuracy. We hypothesized that: H2.1 As the intensity of the irregular registration error increases the users will have the tendency to shift their reference towards the sReal object.

Results

Similar to the first experiment, the ANOVA analysis did not show any significant differences among, participants were extremely precise in the placement task (position accuracy: M = 1.65mm; SD = 0.9mm and rotation accuracy: M = 0.86 deg; SD = 0.95 deg). Also, participants followed a similar behavior: once participants chose the reference (typically in the first few trials), they tend to keep it during the entire experiment (see Fig. 7 right). Six users had a clear tendency towards choosing the sAR cube as reference while the remaining six users chose the sReal cube as reference.

Regarding the strength of registration errors (see Fig. 7), we observed a change in behaviour for some users who chose the sAR reference in the L5 condition. Seven out of twelve users (58.33%) explained that they picked a reference and kept it during all the experiment. Four users (33.33%) said they changed their reference, explicitly mentioning that the change occurred at the beginning of the last condition (L5, high intensity jitter). Thus, in the presence of strong registration errors, and due to the inability to perform the task, the reference object was changed to the more reliable reference.

Discussion

Results of the second experiment tend to reinforce the fact that users make a strong reference choice at the beginning of the experience. Even with the addition of irregular registration errors, participants are reluctant to change their reference. However, if the error can discourage the completion of the task, they have the tendency to shift towards a more stable reference. In conclusion, H2.1 is validated only in the presence of high intensity of jitter. in AR applications pure virtual objects (i.e. without any relation to real objects) are commonly used. In such situations, the pure virtual object will always be considered as the reference as there is no ambiguity. This third experiment explores the impact of irregular registration errors on user performance when the task is driven just by AR content. Can users perform effectively and efficiently in presence of irregular registration errors in such context? The participants in this experiment were the same group which participated in the second experiment.

Experimental Protocol

Upon their arrival, participants read and signed a consent form which briefly described the experiment and their rights. After users fully understand the experimental task, the experimenter equipped them with the HMD. The task was a variation of the pick-and-place task of the first experiment. In this case, the sReal cube was not augmented, and the target location was a sAR object. Users had to pick the sReal cube and place it at the location indicated by the sAR target. sReal colored stickers were placed on the sReal cube to display the correspondence with the sAR target (see Fig. 8). At the end of the experiment participants had to fill out a questionnaire to gather their subjective impressions.

Experimental Design

The experiment had two independent variables which defined the intensity of the irregular registration error and the sAR visualization. Only the sAR target is affected by vibrations. Regarding registration errors, we considered (and simulated) the following five different levels of intensity (sorted by increasing intensity): L1 No Error is applied to the target. L2 Low intensity intermittent vibration: a pulse of 0.11s is generated every 2s with k = 0.33rad. L3 Medium intensity intermittent vibration: a pulse of 0.11s is generated every 1s with k = 0.33rad. L4 High intensity intermittent vibration: a pulse of 0.11s is generated every 0.5s with k = 0.33rad. L5 High intensity jitter: a continuous vibration is generated with k = 0.33rad.

Regarding the sAR visualization, we only considered the occlusion condition with two levels, either the sAR content is occluded (Occlusion ON) or not (Occlusion OFF) by the real content. For both conditions the sAR target was displayed with a 50% transparency. The experiment had a 5x2 factorial design, each combination being repeated six times (60 trials per participant). To avoid ordering effects, the experiment was divided in four blocks in which the order of the sAR visualization was counterbalanced. For each block order the intensity level was randomized. The dependent variables were the task-completion time, the position accuracy and the rotation accuracy.

According to this experimental design, our hypotheses were that:

H3.1 Occlusion OFF will have a negative impact on task-completion time. H3.2 Occlusion OFF will have a negative impact on position accuracy. H3.3 Occlusion OFF will have a negative impact onrotation accuracy. H3.4 as the intensity of the error increases task-completion time will increase. H3.5 as the intensity of the error increases position accuracy will decrease. H3.6 as the intensity of the error increases rotation accuracy will decrease.

Results

Two-way ANOVA analysis were performed to determine the significance of registration errors and Occlusion conditions for each dependent variable. When needed, post-hoc Tukey tests were performed (α > 0.05). Anderson-Darling tests were performed to ensure normal distribution of the data.

Task-Completion Time. The ANOVA analysis showed a main effect on Intensity (F4,44 = 5.34, p < 0.001, η 2 p = 0.33), but there was no effect on Visualization (F1,11 = 0.49, p = 0.498, η 2 p = 0.04) nor on interaction effect (F4,44 = 0.76, p = 0.557, η 2 p = 0.06). Post-hoc tests showed that participants significantly required more time to perform the task for L5 than for the other conditions (see Fig. 9, left). These results do not support H3.1 and only partially support H3.4. Position Accuracy. The ANOVA analysis showed a main effect on Intensity (F4,44 = 47.66, p < 0.001, η 2 p = 0.82) and in Visualization (F1,11 = 15.16, p < 0.01,

η 2 p = 0.59).
No interaction effect was found. Post-hoc tests showed that the intensity level L5 resulted in significantly lower accuracy compared to the other four intensities (see Fig. 9, center), and that when the sAR content is not occluded by the sReal content (M = 1.97mm; SD = 1.05mm) significantly (although slightly) decreased position accuracy compared to the opposite condition (M = 1.57mm; SD = 1.04mm). These results support H3.2 and only partially support H3.5.

Rotational Accuracy. The ANOVA analysis showed only a main effect for Intensity (F4,44 = 9.18, p < 0.001, η 2 p = 0.46). No main effect for Visualization (F1,11 = 0.34, p = 0.572, η 2 p = 0.03) and no interaction effects were found (F4,44 = 1.39, p = 0.300, η 2 p = 0.10). Post-hoc tests showed that for the Intensity variable, the L5 level induced significantly lower rotation accuracy than the other levels (see Fig. 9, right). These results do not support H3.3 and only partially support H3.6.

Subjective Questionnaires

Eleven users out of twelve (91.66%) preferred when the sAR content was occluded by sReal objects (i.e. the Occlusion ON condition). Most of the users felt perturbed when the occlusion was not computed correctly: "it was easier to see edges and borders of the cube when the AR target did not hide the real cube" and "when objects that should be behind everything appear in the foreground, it is not natural". Regarding the amount of jitter, as in the second experiment, users only complained when the registration error did not allowed them to perform the task efficiently (i.e., the L5 level).

Discussion

Results of this experiment show that low to moderate levels of irregular (see Sec. 3.1) registration errors (L1 to L4) do not have a noticeable impact on user performance and accuracy. Surprisingly, no significant effects were found between levels L1 to L4. Indeed, this result shows a relatively strong tolerance of users towards registration errors. Furthermore, in terms of user preferences of the sAR visualization, although the results did not present any significant differences, the visualization minimizing the perceptual conflicts (i.e., correct occlusion management) was preferred.

General Discussion

In this paper, we have presented three experiments focusing on the impact on registration errors in an AR pick-and-place task. In order to ensure controlled and repeatable experimental conditions, the AR environment was simulated in VR. Data analysis has mainly focused on user behavior (how registration errors alter the way users interact) and user performance (how users can account for registration errors).

The first two experiments shown that participants chose a reference (the sAR content or the sReal content) from the very beginning and they were reluctant to change it afterwards. This choice of reference impacted their performance since it drove their actions (e.g., they either align the sReal or the sAR cube on the target). What was more interesting is that even in the presence of irregular registration errors, participants kept the chosen reference. Only when the task was compromised (e.g., when unable to perform the task) some users shifted their reference from the sAR content to the sReal content. Nevertheless, several users still kept the sAR content as reference when the task was compromised (L5 condition). These results show that an AR interface designer should unambiguously define which objects serve as reference to ensure that all users exhibit the same behavior.

In contrast, the third experiment focused on the impact of irregular registration errors on user performance, and more importantly on how users tolerate them. Results showed that, for the tested errors, the tolerance was high, and that errors of low to medium intensity did not significantly altered users' performance. These results show that users had a strong adaptation to adverse situations.

Along the different experiments, a secondary goal was to assess the impact of how AR content is presented. Simulating AR in VR enabled us to test a wide range of visualizations which were difficult or almost impossible to achieve with current AR systems. The preference results are not ground-breaking: users prefer a perceptually-correct integration (e.g., correct occlusion management) as they provide a non-ambiguous layout. Yet, when occlusion management cannot be achieved (e.g., due to technical limitations) semi-transparent visualizations should be considered as they minimize perceptual conflicts. Regarding the impact on performance, interestingly, we did not find any significant difference between visualizations (i.e., transparency and occlusion). This result shows for all conditions users were able to recover enough information from the sAR and sReal contents to successfully perform the task (e.g., a fully visible edge or corner of the cube).

Conclusion

Although registration algorithms are continuously being perfected, inconsistencies between sReal and sAR contents are still a major issue in AR applications. Results obtained our three user studies show that visual inconsistencies generated by registration errors can alter users' behavior as they require a subjective interpretation. According to their interpretation, the outcome of the interaction task might vary. This effect is clearly visible in the first and second experiments as the reference chosen (the sReal cube or sAR cube) determines the outcome of the task. In the same direction, visualization strategies should minimize inconsistencies between the AR and the real contents. However, if this is not possible, the interface designer must ensure that the visualization provides enough information to perform the task effectively.

The presented studies highlight the potential of evaluating registration errors in controlled VR simulations. Nevertheless, this paper has only focused on a particular subset of registration errors. Further studies should be conducted to evaluate other types of registration errors such as depth or lateral errors. This knowledge is paramount to better design AR applications and ensure they enable users to interact effectively and efficiently.

Fig. 1 .

 1 Fig. 1. Depiction of the different simulated objects (Top -cube; Bottom -target). The depiction is either sReal (Left) or sAR (Right). When the cube is sReal and the target is sAR, colored dots are painted on the sReal cube to indicate the orientation of the cube. When there is a sReal cube with its sAR cube and a sReal target, the sReal cube has no colored dot. The orientation is given by the sAR cube. The sAR objects have different conditions of visualization (transparency condition and opacity condition).

Fig. 2 .

 2 Fig. 2. Simulation of AR rotational errors. A black solid square (resp. red dotted) represents a sReal (resp. sAR) object. Top-Left: no rotational error. sAR and sReal objects are perfectly aligned. Bottom-Left: constant error. The angle between sReal and sAR objects is constant over time. Right: two states, intermittent vibrations (rotational vibration that occurs from time to time, Top-Right) and jitter (rotational vibration around a position that constantly occurs, Bottom-Right).

Fig. 3 .

 3 Fig. 3. AR presentation conditions. Each row corresponds to a transparency condition (50% and 0%). Each column corresponds to an occlusion condition (precise depth sorting vs. sAR always on top).

Fig. 4 .

 4 Fig. 4. Depiction of the first experiment's task. The user, using a virtual hand metaphor (see Sec.4.2) had to pick and place the sAR/sReal cube on a target. Only the sAR cube was displayed in the AR overlay.

  Fig. 4. Depiction of the first experiment's task. The user, using a virtual hand metaphor (see Sec.4.2) had to pick and place the sAR/sReal cube on a target. Only the sAR cube was displayed in the AR overlay.

Fig. 6 .

 6 Fig.6. Bar plot showing the number of trials in which for each participant the sReal cube was calculated as the reference object.

6 Experiment 3 :Fig. 7 .

 37 Fig. 7. Choice of reference during the second experiment. (Left) Bar plot showing the choice of reference for each participant. (Right) Bar plot showing the choice for each condition.

Fig. 8 .

 8 Fig. 8. Task description of the third experiment. The user had to place the sReal cube onto the sAR target by aligning the colored stickers placed on the sReal cube with the sAR target's colors.

Fig. 9 .

 9 Fig. 9. Confidence Intervals (95%) of the mean for each dependent variable. Left: taskcompletion time. Center: position accuracy. Right: rotation accuracy.


Antoine Costes
email: antoine.costes@technicolor.com

Fabien Danieau

Argelaguet Ferran

Anatole Sanz

Philippe Lécuyer

Guillotel

Ferran Argelaguet-Sanz

Anatole Lécuyer

Philippe Guillotel

KinesTouch: 3D Force-Feedback Rendering for Tactile Surfaces

Keywords: Touchscreen, Surface haptics, Sliding, Force feedback

come

Introduction

Touchscreens have become ubiquitous in human-computer interaction. They enable freehand direct interaction with 2D and 3D content and they are effectively used in numerous applications. They can be found everywhere, from public ticket machines to mobile phones and laptops.

Despite their intrinsic qualities, as for today, touchscreens still often lack tactile sensations. Irrespective of the visual content, they feel flat, rigid, smooth and static under the finger. Although touchscreens take advantage of finger dexterity, they do not exploit finger sensitivity.

The haptic enhancement of touchscreens is a relatively young and active research field known as "surface haptics" [START_REF] Chubb | Shiverpad: A glass haptic surface that produces shear force on a bare finger[END_REF]. An impressive amount of work has already been done to conceive and develop such technologies in the last decade [START_REF] Winfield | T-pad: Tactile pattern display through variable friction reduction[END_REF] [3] [START_REF] Levesque | Enhancing physicality in touch interaction with programmable friction[END_REF] [START_REF] Mullenbach | Tpad fire: Surface haptic tablet[END_REF] [START_REF] Wiertlewski | A highfidelity surface-haptic device for texture rendering on bare finger[END_REF]. Most efforts have been concentrated on generating various types of vibration that can alter the physics of the finger sliding on the screen, providing friction and even small relief sensations [START_REF] Winfield | T-pad: Tactile pattern display through variable friction reduction[END_REF] [START_REF] Kim | Tactile rendering of 3d features on touch surfaces[END_REF]. However, such approaches do not allow to display other haptic properties such as stiffness or large-scale shapes.

A few solutions have proposed a touchscreen with kinesthetic feedback, i.e., able to move in space rather than vibrate, in order to involve spatial proprioception. Some approaches used parallel platforms for co-localized inclination rendering [START_REF] Kim | Haptic interaction with virtual geometry on robotic touch surface[END_REF] [START_REF] Maiero | Forcetab: Visuo-haptic interaction with a force-sensitive actuated tablet[END_REF], eventually combined with variable friction [START_REF] Hausberger | Surfticskinesthetic and tactile feedback on a touchscreen device[END_REF], but they kept a focus on rendering geometric features rather than material properties like stiffness, slipperiness or roughness. Sinclair et al. have proposed a remarkable solution combining 1-DoF kinesthetic and force feedback [START_REF] Sinclair | Touchmover: actuated 3d touchscreen with haptic feedback[END_REF] [START_REF] Sinclair | Touchmover 2.0-3d touchscreen with force feedback and haptic texture[END_REF], showcasing many interesting perceptual and interaction possibilities. Yet, besides its limitation to one axis, their device remains cumbersome and complex to spread out. The work of Takanaka et al. [START_REF] Takanaka | Multitouch haptic interface with movable touch screen[END_REF] is the only one, to our knowledge, to provide a touchscreen with lateral motion to evoke haptic properties. Interestingly, they chose to keep a non-slipping contact with the screen and simulated inertia and stiffness rather than sliding the screen against the finger to simulate friction or slipperiness. Although many innovative technologies have been developed to provide co-localized friction effects, the potential of the lateral motion of the screen under the finger has not been investigated yet.

(a) The KinesTouch concept.

(b) The six dimensions of surface haptics, adapted from Okamoto et al. [START_REF] Okamoto | Psychophysical dimensions of tactile perception of textures[END_REF].

Fig. 1: The KinesTouch approach provides four different types of haptic feedback to a touchscreen.

In this paper, we propose to use a single force-feedback device to provide four different types of haptic feedback to a tactile tablet: Shape, Stiffness, Roughness, and Sliding (see Figure 1). In particular, our Sliding effect alters the sliding velocity of the finger on the screen through large lateral movements, which constitute a novel approach in friction rendering.

In the remainder of this paper, we first present related work on surface haptics in the light of haptic perception of surfaces. Then, the KinesTouch concept is introduced together with our set of haptic effects. The technical feasibility of our approach is then demonstrated with a proof-of-concept prototype. The results of a user study focused on the Sliding effect are presented and discussed. Finally, several use cases of our system are exposed.

Related work

Dimensionality of surface haptics

The dimensionality of real and artificial textures perception has been investigated in manifold studies. In a survey paper synthesizing over forty years of research, Okamoto et al. proposed five psychophysical dimensions that synthesize state of the art results: 1) compliance (hardness), 2) friction, 3) fine roughness, 4) macro roughness, and 5) warmness [START_REF] Okamoto | Psychophysical dimensions of tactile perception of textures[END_REF] (see Figure 1b).

Yem and Kajimoto [START_REF] Yem | Wearable tactile device using mechanical and electrical stimulation for fingertip interaction with virtual world[END_REF] suggested a correspondence between these five dimensions and the different types of tactile receptors in the skin. The four types of mechanoreceptors are known to be especially receptive to a specific stimuli: static pressure for SA-1, local deformation of the skin for SA-II, rapid lateral skin stretch for FA-I and high frequency vibrations for FA-II [START_REF] Johnson | The roles and functions of cutaneous mechanoreceptors[END_REF]. In addition, thermoreceptors are responsible for temperature gradient sensing [START_REF] Johnson | Thermal sensibility and thermoreceptors[END_REF]. Like Okamoto et al., they did not consider the kinesthetic sense, that is the perception of one's body movements, which is crucial to perceive large scale shapes.

The division between fine and macro roughness, was confirmed by several studies [11] [12] [4] many decades after it was hypothesized by Katz [START_REF] Katz | The world of touch (le krueger, trans.)[END_REF] under the famous name of "duplex theory". This theory states that fine and coarse asperities are mediated by two distinct perceptual mechanisms, the first one relying on contact vibrations and the second one involving pressure spatial distribution. It was notably found that contact vibrations are necessary to perceive asperities under 0.1mm, indicating a perceptual shift around this scale [START_REF] Hollins | Evidence for the duplex theory of tactile texture perception[END_REF]. It is noticeable that these two properties are spontaneously explored with two distinct strategies, namely lateral motion and static contact. These two "exploratory movements", identified by Lederman and Klatsky decades ago [START_REF] Lederman | Hand movements: A window into haptic object recognition[END_REF], are appropriate ways to elicit the most relevant stimulus, namely static pressure distribution or rubbing vibrations.

Another exploratory movement named "contour following" [START_REF] Lederman | Hand movements: A window into haptic object recognition[END_REF], aims at inspecting the global shape or volume of an object with large movements. In this case the kinesthesia (or proprioception) is likely to be predominant in the perceptual process. Therefore, there should be a perceptual shift from macro roughness to shape similar to the one from fine to macro roughness. The location of this shift is obviously in the vicinity of a finger width, although it would be reasonable to expect some overlap, similarly to fine and macro roughness. In the remainder of this paper, we will call this sixth dimension the "shape" dimension.

Surface haptics systems

Vibrators and vibrotactile feedback have been early embedded in commercial touchscreen products and can be used for fine roughness simulation. But many researchers have proposed original ways to enrich touchscreens with an additional vibrator placed either on the nail [START_REF] Ando | Nail-mounted tactile display for boundary/texture augmentation[END_REF], between several fingers and the screen [START_REF] Burch | Using multiple contacts with texture-enhanced graphics[END_REF], on the device [START_REF] Brewster | Tactile feedback for mobile interactions[END_REF] [38] [START_REF] Zhao | Intermanual apparent tactile motion on handheld tablets[END_REF] or both on the device and on haptic gloves [START_REF] Israr | Po2: augmented haptics for interactive gameplay[END_REF]. In particular, Romano and Kuchenbecker used a high-quality one-dimensional vibration to display compelling texture details through an actuated stylus, according to normal contact force and lateral speed [START_REF] Romano | Creating realistic virtual textures from contact acceleration data[END_REF].

Several variable friction devices have been developed, either using ultrasonic vibration [19] [23] or electrovibration [3] [21]. In both approaches, friction can be modulated to produce texture effects and even 3D pattern features [START_REF] Winfield | T-pad: Tactile pattern display through variable friction reduction[END_REF] [START_REF] Kim | Tactile rendering of 3d features on touch surfaces[END_REF]. Some researchers also chose to instrument the finger, either with vibrators [START_REF] Burch | Using multiple contacts with texture-enhanced graphics[END_REF] [START_REF] Ando | Nail-mounted tactile display for boundary/texture augmentation[END_REF] or with a lateral-force proxy [START_REF] Yang | Fingviewer: A new multi-touch force feedback touch screen[END_REF] [26] [START_REF] Saga | Simultaneous geometry and texture display based on lateral force for touchscreen[END_REF]. Several solutions producing mechanical planar vibrations [START_REF] Wang | Haptic overlay device for flat panel touch displays[END_REF] or short-range movements [START_REF] Mullenbach | Activepad surface haptic device[END_REF] were also proposed. However all these approaches had no force or motion abilities in the normal direction, and thus could not provide compliance and large shapes sensations.

Parallel platforms were used for co-localized curvature feedback [START_REF] Kim | Haptic interaction with virtual geometry on robotic touch surface[END_REF], notably the "SurfTics" [START_REF] Hausberger | Surfticskinesthetic and tactile feedback on a touchscreen device[END_REF] and the "ForceTab" [START_REF] Maiero | Forcetab: Visuo-haptic interaction with a force-sensitive actuated tablet[END_REF] devices. Another approach presented in [START_REF] Takanaka | Multitouch haptic interface with movable touch screen[END_REF] consists in a touchscreen with planar force feedback and large translation and rotation abilities. These approaches were focused on shape rendering and did not address other dimensions of haptic perception.

The "TouchMover" device [START_REF] Sinclair | Touchmover: actuated 3d touchscreen with haptic feedback[END_REF] is a touchscreen actuated and moved using force feedback in the normal direction, showcasing interesting applications notably in volumetric data manipulation. The second version [START_REF] Sinclair | Touchmover 2.0-3d touchscreen with force feedback and haptic texture[END_REF] includes vibrators that render fine shape details at contact point. The normal force-feedback allows for stiffness or inertia simulation and shape rendering, but no lateral friction sensations. Besides its limitation to one axis and two psychophysical dimensions, the TouchMover remains rather cumbersome and complex to spread out, involving custom and expensive mechanics and electronics.

There are actually rather few systems aiming at simulating a wider range of haptic sensations. The device designed by Yem and Kajimoto [START_REF] Yem | Wearable tactile device using mechanical and electrical stimulation for fingertip interaction with virtual world[END_REF] is able to simulate up to four psychophysical dimensions of texture perception: compliance, friction, fine and macro roughness. But this system is finger-mounted and not touchscreen-based, and it does not co-localize visual and haptic displays. Culbertson and Kuchenbecker [START_REF] Culbertson | Importance of matching physical friction, hardness, and texture in creating realistic haptic virtual surfaces[END_REF] combined a pen-shaped force-feedback rendering stickiness through tangential forces with an high-quality vibrator rendering hardness through tapping transients and fine roughness through vibrations.

Table 1 provides an overview of previous contributions in surface haptics, with the type of haptic sensation they have addressed. Interestingly enough, these previous systems are able to simulate only one or two psychophysical dimensions. Temperature and macro roughness were not taken into account here, as we could not find representative examples of touchscreen enhancement involving one of them combined with another dimension.

The KinesTouch approach

The KinesTouch approach enriches touchscreen interactions with a set of tactile and kinesthetic effects in both normal and lateral directions. When the user touches an object or an image displayed on the touchscreen, the screen is given forces or motion simulating various haptic properties: it can resist more or less to pressure to render material stiffness, move up and down according to object's shape, vibrate during a stroke to evoke texture roughness, or slide laterally to change the slipperiness sensations.

In the following sections, we present our set of four co-localized haptic effects. We will focus on the case of using a 3-DOF impedance device for the control law. But the KinesTouch approach is scalable and could be used with higher end 6-DoF haptic interfaces that could allow for even more effects than what we propose hereafter.

Notations

In the remainder of this paper, vectors and matrices will be expressed in the fixed reference frame with positive z upwards. The screen is considered to be horizontal, parallel to the xy plan. Also: X 0 will refer to the 3D center position of the workspace, X t will refer to the 3D screen position with respect to X 0 , f will refer to the 2D finger position on the screen, I 3 will refer to the identity matrix, e z will refer to the vertical unit vector, K max will refer to a high stiffness value,depending on hardware performance, used for position control 3 (1 N/mm in our setup).

Stiffness effect

The Stiffness effect allows the user to feel a resistance to deformation when they push an object on the screen. It simulates the elasticity of a material, and address the compliance perceptual dimension. The effect consists in a normal opposing force that increases with penalty, as shown in Figure 2. The two other directions of the touchscreen are locked in position. Using an elastic deformation model, the control law of our Stiffness effect is:

F stiffness =     K max 0 0 0 K max 0 0 0 k mat     (X 0 -X t ) (1)
with k mat the simulated stiffness.

Shape effect

The Shape effect allows the user to feel the 3D shape of an object. It reproduces reliefs that are larger than a finger and need active exploration to be perceived. The effect consists in a normal displacement corresponding to the change in vertical projection of the 2D finger position on the object's 3D shape, as shown in Figure 3. The two other directions of the touchscreen are locked in position (i.e., there is no lateral motion). The control law of our Shape effect is:

F shape = K max I 3 (X 0 + h(f ) e z -X t ) (2)
with h(x, y) the vertical projection of the finger position onto the 3D shape. The shape is accessed "from the top": only its visible upper part, relatively to the horizontal plane, can be explored. However, a simple rotation of the shape in the virtual space allows to access its bottom part.

Roughness effect

The Roughness effect allows the user to feel vibrations evoking a periodic grating when they stroke an object on the screen. It renders the fine roughness property, modeled by a small spatial period. The effect consists in an oscillating force taking into account both the simulated spatial period and the finger exploration velocity, as shown in Figure 4a. The touchscreen is otherwise locked in position.

F roughness = δ sin(2πλ|| ḟ ||) e z + K max I 3 (X 0 -X t ) (3)
with δ the grating depth, λ the grating spatial period.

Sliding effect

The Sliding effect provides various sliding sensations to the user when they stroke an object on the screen. As it modifies the sliding phenomenon between the screen and the finger, it addresses the friction perceptual dimension. It consists in a tangential movement of the screen meant to increase or diminish the relative sliding, that is the velocity difference, with the finger. We expect two different sensations corresponding to the two possible sliding directions: a "Follow effect" and a "Reverse effect" which are described hereafter. The touchscreen motion is locked here in position in the normal direction.

The "Follow effect", illustrated in Figure 4b, consists in moving the screen the same way the finger moves on the screen, so that relative sliding is decreased or even kept close to zero. In this case, while the finger moves in the reference frame, its position on the screen remains almost static.

The "Reverse effect", illustrated in Figure 4c, consists in moving the screen in the opposite direction to finger's movement, so that relative sliding is increased.

The Sliding effect is achieved with the combination of two forces: a "moving force" proportional to finger's tangential velocity, and a damping force in the binormal direction:

F slipperiness = α ḟ -ν ḟ ∧ e z + K max (X 0 • e z -X t • e z ) (4)
with α ∈ [-1, 1] the slipperiness coefficient and ν the damping coefficient.

Idle behavior

When the screen is not touched, it should stay still or move back to the center of the workspace, so that the force-feedback device remains close to its neutral position. This is done by applying a simple centering force instead of one of the previous effects:

F idle = K max I 3 (X 0 -X t ) (5)
4 The KinesTouch prototype

In this section, we describe the design and implementation of our prototype using a standard tablet and Novint Falcon haptic device. We designed a custom end-effector in order to be able to attach the tablet on the haptic device handle, and a prediction-correction algorithm to compensate the touch tracking latency.

We also present the handling of synchronization between visual and haptic loops, and the control law for the haptic rendering.

Hardware

The Falcon is a standard 3-DoF impedance haptic device, initially designed for the gaming industry. We combined it with a Galaxy Tab SM-T810, which exhibits rather high resolution (2048x1536), comfortable size (9.7") and an acceptable weight (389g).

Assembly of tablet and force-feedback device The Falcon's grip has several buttons and is removable, but a security mechanism deactivates the device when the grip is removed, detecting the electrical contact with the grip. This problem was overcome by unmounting the default grip and keeping only the coupling part and electronic circuit. A tablet adapter, shown in Figure 5a, that reproduced the interlock while offering a flat shape to affix the tablet, was 3Dprinted. As the precise relative positioning of the tablet was not of importance for the haptic effects presented in this paper, it was affixed to the adapter with a simple velcro grip. The Falcon was then rotated by 90 degrees and positioned sideways so that it "pushed forward" the tablet vertically, as shown in Figure 5b.

Software

Handling latency issues Besides the visual display, the tablet application is also responsible for touch tracking and filtering. In practice, the built-in touch tracking of the Galaxy Tab SM-T810 has a latency of a few dozens of ms, and the Unity application has a refresh rate of 60 Hz. This results in a delay in the position measurement up to 2 cm in usual slide movements, which is problematic for real-time haptic rendering. Furthermore, despite the high resolution of the screen, instantaneous touch velocity estimation suffers from spikes due to pixel quantization. For these reasons, touch position and velocity were computed and filtered before being sent and used in the haptic rendering loop, according to the following prediction algorithm, inspired from [START_REF] Ushirobira | A forecasting algorithm for latency compensation in indirect human-computer interactions[END_REF]. First, measured touch position f mes is converted in real-world meter coordinates. Then, a simple linear prediction is applied to measured touched position:

f pred = f mes + k pred * (f mes -f prev mes ) (6)
where f prev mes is the previous measured touch position and k pred the filter parameter.

Finally, an exponential smoothing filter is applied to get the corrected position:

f = α * f pred (1 -α) * f prev pred (7)
where f prev pred is the previous predicted position and α the filter parameter. The parameters were set after testings to: k pred = 8 and α = 0.15. Instantaneous touch velocity is smoothed with an exponential smoothing filter with α = 0.45.

Visual and haptic loops synchronization

The haptic rendering is computed by a dedicated application running on a laptop and using the CHAI3D framework 4 . On the tablet, a Unity application is used for the visual rendering and the touch tracking. The two applications communicate with each other using the Open Sound Control (OSC) protocol 5 . As applications run at different rates, this communication is asynchronous. On both sides, incoming messages are treated in a specific thread and update global variable values which are then used in the main thread. A network connection is emulated through the USB cable connecting the tablet and the laptop, so that OSC communication latency is kept under 1ms.

The haptic rendering is mostly located in a haptic thread running at about 1000 Hz inside the CHAI3D application. An additional 60 Hz thread is meant to send the Falcon position to the tablet application. The synchronization of Fig. 6: Software architecture the two loops is illustrated in Figure 6. In the Unity application, a main loop updates touch information, sends them to the CHAI3D application, and updates the visual display. This visual display compensates the Falcon movements so that when the tablet is moving, displayed objects remain immobile in the user's reference frame.

Transparency In the previous descriptions of our haptic effects, the system is supposed to be perfectly transparent, with no inertia. However the weight of the touchscreen and effector are not negligible compared to the other involved forces, and have to be compensated by adding a constant opposite force in the control law.

Control law

The final haptic rendering was obtained using a single control law that merged all our haptic effects:

F total = (mg + δ sin(2πλ|| ḟ ||))e z + α ḟ -ν ḟ ∧ e z +K (X 0 + he z -X t ) (8)
with K the stabilization matrix, given in Table 2. The Falcon was found to produce forces proportional, but not equal, to the forces requested through the CHAI3D API. This problem was overcome by applying a gain factor that was empirically found to of about 4.5 on two different Falcon devices to get the right forces. This is consistent with another study, although they found the gain to be equal to 3 [START_REF] Vanacken | The phantom versus the falcon: Force feedback magnitude effects on users performance during target acquisition[END_REF]. This difference of value might be explained by the difference of CHAI3D version.

K Kmax I3     Kmax 0 0 0 Kmax 0 0 0 kmat         0 0 0 0 0 0 0 0 Kmax    
5 User study

Introduction

We conducted a user study to evaluate the sensations produced by the KinesTouch prototype. Due to the large variety of our haptic effects, we have focused on our most innovative effect: the Sliding effect. Our choice was motivated by the fact that equivalents of Stiffness, Shape and Roughness effects have already been largely studied in the haptic literature. In contrast, the Sliding effect had never been explored in the literature and there are no clear assumptions on what the user's perception will be. Thus, we conducted a user study to answer the following question: are users able to consistently and efficiently discriminate different Sliding effects?

We compared three sliding sensations: the Reverse effect (REVERSE, see Figure 4), the Follow effect (FOLLOW, see Figure 4), and a control stimulus in which the tablet stays static (STATIC). Three hypotheses were tested:

-H1: different stimuli would produce different sensations -H2: seeing the moving screen contributes to distinguish between stimuli, i.e., visual cues increase the discrimination accuracy. -H3: the smoothness of the screen diminishes the sensations produced, i.e., a tactile cues increase the discrimination accuracy.

Materials and methods

Procedure 18 volunteer unpaid subjects (16 male, age 31.2 ± 12.1) took part in the experiment which consisted in two sessions of about 45mn on different days. All of them were right-handed or ambidextrous.

After reading and signing a consent form, subjects were asked to seat with the right arm resting besides the tablet screen. For each trial, a narrow white area was displayed on the screen, and the subject was invited to slide their finger inside this area.

Each trial was composed of the active exploration of two stimuli, followed by a forced-choice question to designate on which one the subject felt the more sliding.Each stimuli lasted 3.5 seconds from the moment the screen was touched, then the screen turned to black and waited for the touch release to pass to the second stimulus or the question. The subject provided the answer to the question directly on the screen.

At the beginning of each session, two practice trials were first performed to ensure that subject understood the procedure. During these introductory trials, a moving target was displayed to suggest a back and forth movement at 0.5 Hz. Subjects were informed that the stimuli would be optimally felt within this range of velocities but were left free in their inspection otherwise.

Experimental Design The experiment had three independent variables: the stimulus, the visual cues (i.e., seeing the tablet moving) and the tactile cues (i.e., screen roughness). Three pairwise comparisons were considered: REVERSE vs. FOLLOW, REVERSE vs. STATIC and FOLLOW vs. STATIC. To avoid order effects, the inverse comparisons were also considered.

In order to evaluate the importance of visual cues, half of the trials were performed with the whole mechanism being visible (V1, see Figure 7a), and half with a black cover hiding the mechanism and its movements (V0, see Figure 7b). In order to evaluate the importance of tactile cues, half of the trials were performed with a window privacy film applied on the screen (F1) and half without (F0). This transparent and electro-statically adhesive film had small but clearly perceptible reliefs that produced quite strong vibrations under the finger when being stroked. Affixed to the screen, there was no decrease in brightness but a tiny pixel diffraction on each relief. Trials were split in four condition blocks corresponding to the visual and tactile crossed conditions: V0F0, V0F1, V1F0, V1F1. In order to avoid order effects, the order of the blocks was given by the Latin-square method. In each of the two sessions, two condition blocks of 60 trials (10 repetitions for each of the 6 pairwise combinations) were performed. Collected data and scoring For each trial, the answer as well as the response time were recorded. In addition, a discrimination score for each subject was computed for each combination and factor (3 comparisons x 2 visual conditions x 2 tactile conditions). The discrimination score was computed as follows. First, each trial was counted as +1 or -1 according to stimulus chosen as the "more sliding" (the pair order being taken into account). For example, in a REVERSE vs. STATIC comparison, +1 will mean that REVERSE is considered to be more sliding that STATIC and vice-versa. Second, the data for each combination was normalized between [-1,1], showing the preference between the two stimuli. Finally, as we observed that subjects had different interpretations of the question, but were consistent in the stimulus they chose as "more sliding", we considered the absolute value of the discrimination score [0,1]. Thus, as indicated in Table 3, a discrimination score of 0 indicated that the subject had no preference between the two stimuli and answered randomly (with a 50% accuracy), whereas a discrimination score of 1 indicated that the subject consistently chose one stimulus over the other (with a 100% accuracy).

Table 3: Correspondence between preference rate and discrimination score.

Preference rate 50% 60% 75% 80% 90% 95% 100%

Discrimination score 0 0.2 0.5 0.6 0.8 0.9 1

Results

Figure 8 show the distributions of the discrimination scores grouped according to the independent variables. On each figure, the red dot indicates the mean value, in addition to the median value and quartiles indicated by the box. An Anderson Darling normality test revealed that the data distribution were not normal, so we performed an aligned rank transform in order to enable a full factorial analysis using ANOVA. The three-way ANOVA comparison, visual and tactile cues vs. the discrimination score revealed a significant main effect on the visual condition (F 1,17 = 9.56, p < 0.01). Post-hoc tests showed that this effect was significant (p<0.05), V1 had a higher discrimination score (M = 0.71; SD = 0.3) compared with V0 (M = 0.59; SD = 0.33). These results support H2. In contrast, no main effect was found on the tactile condition (F 1,17 = 3.64, p = 0.073). Yet, the results seems to suggest that there is an impact of the screen roughness: F0 (M = 0.61; SD = 0.34) compared to F1 (M = 0.69; SD = 0.30). Nevertheless the results do not support H3. Regarding the different comparisons, the ANOVA did not show a significant effect (F 2,17 = 3.00, p = 0.063). Again, the results are close to the significance threshold. Post-hoc tests seems to suggest that subjects were less accurate for the REVERSE vs. STATIC comparison (p=0.053). Finally, the ANOVA did not show any interaction effect. Figure 8a shows the score distributions according the visual condition. Scores were significantly higher in the V1 condition, that is with the mechanism visible, than in the V0 condition, that is with a cover hiding it. Scores were also higher, but not significantly, in the F1 condition than in the F0 condition, i.e., with the textured film on the tablet rather than without. The distributions of the crossed visuo-tactile conditions are consistent with the results of the non-crossed conditions (Figure 8a): scores were significantly higher with the mechanism visible, and not significantly higher with the textured film on the tablet rather than without. The highest average score is achieved, as expected, in the V1F1 condition, with half of the subjects having a score above 0.9.

Figure 8b shows that the scores were different regarding which stimuli were compared. When the Reverse and the Follow effects were compared, the scores are distributed quite uniformly between 0 and 1. In contrast, for the comparison between the Follow effect and the control condition, half of the subjects have a discrimination score above 0.8 and a few have a score close to zero.

Discussion

Our results suggest that our Sliding effect is well and consistently discriminated by a great majority of subjects. Indeed, even in the least favorable condition, V0F0, half of the subjects had a score above 0.6, which means they were consistent in at least 80% of their answers. In the most favorable condition, V1F1, half of the subjects had a score of 0.9 or higher, indicating 95% of their answers were consistent. It is noticeable that in most conditions, score distributions were very large, ranging from 0 to 1, meaning that some subjects answered randomly and some subjects answered with a perfect consistency. The mean values, however, are above 0.5 in all conditions, which means that in average, whatever the condition, the subjects were consistent in their classification on at least 75% of the trials. Moreover, in almost all conditions this mean value is slightly lower than the median value, which indicates that it is worn down by a few values close to 0.

These results demonstrate that the subjects' ability to discriminate between the three stimuli were generally well above the random threshold with or without visual and/or tactile cues. As expected, visual cues had significant positive impact on discrimination. More surprisingly, the rough textured film on the screen had only a minor effect. We were expecting it to make the difference between stimuli very clear, as it produces strong vibrations according to sliding speed, in contrast with the very smooth screen that does not provide much sliding sensations.

However, an unexpected side effect was that the textured film was much less sticky than the screen, so that although the tactile sensations were stronger, it was much easier to stroke it fast. We think that this could have biased the answer about the "sliding" sensation, and could explain why subjects had different strategies to rank the stimuli. During the experiment, we noticed that most users had a clear ranking for a given visuo-tactile condition, but it was not necessary the same when the visual or tactile condition changed.

While the subjects were clearly able to discriminate the three stimuli, their ranking in terms of sliding was different among subjects and conditions. This might simply reflect the polysemy of the "sliding" term, and the very blurred vocabulary we have when it comes to describe tactile experiences. Further studies could disambiguate the sensations produced by the lateral sliding of the screen during stroke. For instance, asking the subjects about both roughness and sliding sensation could help to identify the dependence or independence of these two parameters. Also, a comparison with real material samples rather than between haptic effects might help avoiding misinterpretations and keep a low inter-subject variability.

Use cases

The KinesTouch approach allows for various haptic effects based on force feedback and movements of the touchscreen in the 3D space. In this section we provide several illustrative use cases, illustrated Figure 9 and in the accompany video, that we have been developed in order to show the potential of our approach.

Interacting with 3D objects: In our first use case, the user can explore and interact with virtual 3D objects. This use case relies mainly on the Shape effect. In our implementation, the user can feel the shape of several objects such as a vase or rocks.

Perceiving the texture of 2D images and pictures: In our second use case, KinesTouch is used to interact with a 2D image in order to feel its texture. This use case relies mainly on the Stiffness, Sliding, and Roughness Thanks to these effects, the user can feel the changes in: local elasticity, friction, and relief in the picture. In our implementation, a picture of a plant landscape is used, associated with several "haptic maps", similarly to the normal maps used for textures in 3D engines (here: "stiffness map", "friction map" and "roughness map").

Augmenting graphical user interface and haptic widgets: In our third use case, KinesTouch is used to enhance interaction with a Graphical User Interface made of several buttons. This simple use case relies on the Stiffness effect. In our implementation, the buttons need to be pushed at a certain depth, but have different levels of stiffness, which makes them easy or hard to validate.

Exploring Interactive maps: In our fourth use case, the user can explore the interactive map of a building. This use case relies on the Shape, Sliding, and Roughness effects. In our implementation, the 2D map (in top-view) of a big mall with three floors is used. The user can explore the layout of the shops using the finger. When stroking over stairs the user can move up or down to a different floor. The user can be attracted or repulsed from specific points/areas of interest. A vibration can also be added in presence of a targeted item.

Conclusion and Future Work

In this paper we have presented KinesTouch: a novel approach to enhance touchscreen interactions using kinesthetic and force feedback. With a single device, the KinesTouch provides four different types of haptics sensations: compliance, shape, fine roughness and friction. Moreover, we address a novel way of dealing with friction rendering: lateral kinesthetic feedback. We designed a proof-of-concept prototype based on the combination of a standard tablet and a consumer-grade impedance haptic device, in order to illustrate several use cases including: interacting with 3D objects or haptic widgets, exploring an interactive map, or perceiving the texture of a 2D image. We also conducted a user study on the Sliding effect to confirm that it could well induce different sliding sensations.

Creating rich haptic effects that combine the different psychophysical dimensions is probably the most promising, but also challenging following of this work. Except for vibrations that can be easily "added" to a force or kinesthetic rendering without much interferences, the compliance, shape, and friction dimensions are not trivial to associate, at least with a Falcon device that is limited in terms of dynamics and workspace. Beyond the device technical limitations, there are also conceptual limitations to combine force and position control. The proper algorithms and hardware able to tackle this issue should be explored in future.

Fig. 2 :

 2 Fig. 2: Stiffness effect: the screen provides an elastic force under pressure.

Fig. 3 :

 3 Fig. 3: Shape effect: the vertical displacement during stroke reproduces reliefs.

  (a) Roughness effect. (b) Follow effect. (c) Reverse effect.

Fig. 4 :

 4 Fig. 4: Roughness effect: the screen vibrates during stroke to simulate roughness (a). Sliding effect: the screen moves laterally to cancel (b) or increase (c) relative sliding.

  (a) 3D printed adapter. (b) Global setup.

Fig. 5 :

 5 Fig. 5: KinesTouch prototype.

  (a) The V1F0 condition (without cover). (b) The V0F0 condition (with cover).

Fig. 7 :

 7 Fig. 7: General setup without and with cover.

  Visual condition (H2).

Fig. 8 :

 8 Fig. 8: Score distributions across visuo-tactile conditions and stimuli pairs.

  (a) Interaction with a 3D object. (b) Texture of a 2D image. (c) GUI and haptic buttons. (d) Interactive map.

Fig. 9 :

 9 Fig. 9: Illustrations of our four use cases

Table 1 :

 1 Main previous approaches in surface haptics. Most of them address only one or two psychophysical dimensions.

	Addressed psychophysical dimensions

Table 2 :

 2 Stabilization matrix values for the different effects.

	Effect	Idle, Shape,	Stiffness	Sliding
		Roughness

Impedance force-feedback devices provide forces to their end-effector, while measuring its position. Although they can't act directly on position, they can still be used for pseudo position control with a high stiffness force linking the measured position to the desired one.

http://chai3d.org/download/license

http://opensoundcontrol.org/introduction-osc