Henrique G Debarba

Jerônimo G Grandi

Anderson Maciel

Luciana Nedel
email: nedel@inf.ufrgs.br

Ronan Boulic
email: ronan.boulic@epfl.ch

Disambiguation Canvas: a precise selection technique for virtual environments

Keywords: Selection techniques, 3D interaction, usability evaluation, progressive refinement

We present the disambiguation canvas, a technique developed for easy, accurate and fast selection of small objects and objects inside cluttered virtual environments. Disambiguation canvas rely on selection by progressive refinement, it uses a mobile device and consists of two steps. During the first, the user defines a subset of objects by means of the orientation sensors of the device and a volume casting pointing technique. The subsequent step consists of the disambiguation of the desired target among the previously defined subset of objects, and is accomplished using the mobile device touchscreen. By relying on the touchscreen for the last step, the user can disambiguate among hundreds of objects at once. User tests show that our technique performs faster than raycasting for targets with approximately 0.53 degrees of angular size, and is also much more accurate for all the tested target sizes.

Introduction

Selection is one of the four fundamental forms of interaction in a virtual world [START_REF] Bowman | 3D User Interfaces: Theory and Practice[END_REF][START_REF] Mine | Virtual environment interaction techniques[END_REF]. It is the ability of the user to specify objects in the virtual environment for subsequent actions [START_REF] Steed | Towards a general model for selection in virtual environments[END_REF]. The literature is rich in immediate selection techniques; however, this class of technique is exposed to problems of accuracy, ambiguity and complexity [START_REF] Haan | Intenselect: Using dynamic object rating for assisting 3d object selection[END_REF]. Many applications rely more on correctness of selection than on time of selection, but ordinary selection techniques in use tend to favor performance over accuracy. We intend to provide a precise yet fast alternative for selection in virtual environments. Therefore, we rely on selection by progressive refinement, as proposed by Kopper et al. [START_REF] Kopper | Rapid and accurate 3d selection by progressive refinement[END_REF].

Selection by progressive refinement proposes the breakdown of a selection task into "effortless" subtasks. It aims to avoid the attention and precision usually required by traditional selection techniques, so-called immediate selection techniques [START_REF] Kopper | Rapid and accurate 3d selection by progressive refinement[END_REF][START_REF] Bacim | Design and evaluation of 3d selection techniques based on progressive refinement[END_REF]. However, there is an inevitable tradeoff between immediate and progressive refine-ment selection techniques. To complete a selection, the latter requires a process which usually consists of more than one quick subtask, generally resulting in higher accuracy and longer selection time. On the other hand, immediate selection techniques consist in performing the selection in only one step, being generally faster but less accurate.

We propose the disambiguation canvas, which is a technique for quick disambiguation of selection. We use the observed high precision of control provided by the touchscreen [START_REF] Baudisch | Back-of-device interaction allows creating very small touch devices[END_REF][START_REF] Debarba | Lop-cursor: Fast and precise interaction with tiled displays using one hand and levels of precision[END_REF] to allow the disambiguation of the desired object among a subset of hundreds of other objects in only one step of refinement. Previous progressive refinement techniques do not scale as well as ours. Available techniques that disambiguate only in one step are limited to a small subset of objects, while those that refine among large subsets require multiple steps of disambiguation. Fig. 1. Disambiguation canvas walkthrough: (a) the user points to the region where the desired object is located; (b) starting a touch rearrange the subset of objects pre-selected by the volume casting technique over a selection canvas; (c) the canvas has an absolute mapping to the mobile device touchscreen, the user slides his thumb in order to point out the desired object. Disambiguation canvas was designed to be compatible with immersive displays, such as the depicted head-mounted display. In this figure, the hand inserts illustrate the user gestures for each step, and are not displayed by our technique.

The disambiguation canvas is based on a two steps process. In the first step, the user employs a volume casting technique to point in the direction of the desired target object (see Figure 1a). When the target object is inside or intersecting the volume of selection, the user may start a touch on the mobile device touchscreen to enter the second step of the selection. A rectangle aligned parallel with the image planeor with the mid orientation of the two image planes when stereoscopic rendering is in useappears in front of the user; all the subset of objects pre-selected by the volume moves in an animation to form a matrix inside this rectangle (Figure 1b). The rectangle has a 1:1 mapping with the mobile device touchscreen, sliding the thumb on the touchscreen allows the superposition of the desired object by the arrow (Figure 1c). Selection is performed by a take-off gesture. If the user wants to leave the disambig-uation phase without selecting any object, they simply perform a take-off gesture with the arrow over an empty region of the rectangle.

The remainder of this paper is organized as follows. In order to provide the reader with a context, Section 2 summarizes the main selection techniques that consider progressive refinement; Section 3 presents the proposed technique design decisions, prototype hardware technology and software implementation. In Section 4 we present the evaluation of the disambiguation canvas, comparing it with the ray-casting and SQUAD [START_REF] Kopper | Rapid and accurate 3d selection by progressive refinement[END_REF], and present the analysis of results. A deeper discussion covering qualitative results as well as suggestions for design changes are presented in Section 5. Finally, in Section 6 we highlight our findings and suggest future developments.

Related Work

The design space of selection by progressive refinement was presented by Kopper et al. [START_REF] Kopper | Rapid and accurate 3d selection by progressive refinement[END_REF]. It is defined as an approach to progressively reduce the group of selectable objects and hence reduce required precision of pointing. From the current literature, we could identify three major groups of progressive refinement selection techniques: menu disambiguation, zoom and persistence of pointing. These are discussed below, as well as some techniques that do not fit these main categories.

Menu disambiguation: generally uses a volume of selection on the initial phase to reduce the effort of pointing into the desired object. Objects that fall inside or intersect that volume are then presented as a subset of objects using some sort of menu for disambiguation.

Dang et al. presented the transparent sphere and transparent cylinder [START_REF] Dang | Visualization and interaction on flight trajectory in a 3d stereoscopic environment[END_REF]. On the transparent sphere technique, a positional cursor similar to the virtual hand metaphor [START_REF] Mine | Virtual environment interaction techniques[END_REF] is used to place a spherical volume of selection in space. Objects inside or intersecting this sphere have their names shown on a menu. Disambiguation is performed by selecting the desired object name. On the other hand, the transparent cylinder uses a ray-casting based approach where a cylindrical volume is attached along the cast ray in order to define the subset of objects. Transparent sphere and cylinder present only the name of the target for disambiguation. Thus their original design is unsuitable for a series of applications.

Grossman and Balakrishnan [START_REF] Grossman | The design and evaluation of selection techniques for 3d volumetric displays[END_REF] proposed the flower ray for interaction with a volumetric display. The flower ray uses ray-casting, and disambiguates using a marking menu. When entering the menu disambiguation step, intersected objects animate towards the user viewport and spread as a marking menu. However, this technique still requires precision of pointing as it relies on ray-casting for both phases, and would have problems disambiguating among a large subset of objects.

When proposing the taxonomy for the progressive refinement selection techniques, Kopper et al. [START_REF] Kopper | Rapid and accurate 3d selection by progressive refinement[END_REF] also presented the SQUAD technique (sphere-casting refined by QUAD-menu). SQUAD consists of defining a subset of objects through their intersection with a sphere volume, and further refining the subset through QUAD menus until only one object remains. As SQUAD relies on several steps of disambiguation, we believe the major drawback of this approach is that the visual search is repeated in each step. If the desired object is similar to others, the visual search can be even more time consuming than the pointing task itself. This question was not addressed in the original study.

Zoom: Bacim et al. [START_REF] Bacim | Design and evaluation of 3d selection techniques based on progressive refinement[END_REF] propose two techniques for progressive refinement based on zoom, discrete zoom and continuous zoom. In the discrete zoom, the user defines a quadrant of the screen they want to see in more detail, the frustum changes so that the specific quadrant covers all the field of view (FOV). In the continuous zoom technique, the zoom happens continuously towards the pointing direction. Zoom based techniques have the advantage of showing the objects in their original context. However, manually controlled zoom tends to be more time consuming. Indeed, the evaluation presented by Bacim et al. showed lower performance when compared to the SQUAD technique.

Score accumulation: although not originally classified as progressive refinement selection techniques [START_REF] Kopper | Rapid and accurate 3d selection by progressive refinement[END_REF], we advocate that score accumulation techniques present the expected behavior described by the authors. These generally rely on the consistency of pointing, where objects that were targeted for a larger duration accumulate higher scores, becoming more likely to be the intended target of a selection.

Haan et al. [START_REF] Haan | Intenselect: Using dynamic object rating for assisting 3d object selection[END_REF] use an approach similar to lightspot (cone-casting) [START_REF] Liang | Jdcad: A highly interactive 3D modeling system[END_REF] for the intenselect technique. However, on intenselect an alternative disambiguation function is applied and expanded to the dimension of time. Objects that fall inside the cast cone accumulate scores over time. Their scores increase proportionally to their proximity to the center of the cone and its casting origin. If an object stops intersecting the cone, its score is gradually lowered. Visual feedback of pointing is given by a bended ray connecting the origin of the ray to the object with the higher score.

Grossman and Balakrishnan have implemented and evaluated the smart raytechnique first proposed by Steed [START_REF] Steed | Towards a general model for selection in virtual environments[END_REF] on a volumetric display [START_REF] Grossman | The design and evaluation of selection techniques for 3d volumetric displays[END_REF]. All objects intersected by the ray accumulate scores; to disambiguate, the user moves the origin and direction of the ray so that it always intersects the intended object. As long as the user succeeds maintaining the ray over the desired object for more time than any other, she is able to select it. Smart ray gradually decreases the score of the objects that have lost intersection with the ray. Therefore, it still maintains most of the score of objects that unintentionally lost contact with the ray for a short period of time.

Other approaches: Steed and Parker have proposed shadow cone-casting [START_REF] Steed | 3D selection strategies for head tracked and non-head tracked operation of spatially immersive displays[END_REF], which uses cone casting persistence of pointing along time to define a selection. When the user starts a selection, all objects inside the cone are selectable. The user must disambiguate among these objects by moving the origin of the cone while trying to always maintain the desired object(s) inside the cone. If an object falls outside the cone, it is cut out of this selection process. This technique allows the selection of multiple targets. However, it relies on the proximity of objects for this. Additionally, this technique is likely to be very time consuming in a cluttered environment, where high precision is required.

Grossman and Balakrishnan have proposed the lock ray [START_REF] Grossman | The design and evaluation of selection techniques for 3d volumetric displays[END_REF], which expands their own technique of depth ray. The depth ray uses forward and backward hand movements to disambiguate which of the objects intersected by ray-casting will be selected. In the lock ray, these steps are performed in sequence, assuring higher precision con-trol as the ray-casting becomes locked, while in the depth ray they are performed simultaneously. Both techniques still require high precision of pointing in order to hit a target with the ray-casting metaphor.

Stellmach and Dachselt presented Look & Touch, which has more similarities with our approach [START_REF] Stellmach | Look & touch: gaze-supported target acquisition[END_REF], although being designed for 2D selection. With Look & Touch the user uses his gaze to control the direction of the lower precision phase of pointing; objects intersecting the circular area of their cursor are pre-selected, and the user can disambiguate cycling through the objects on an iPod touch touchscreen. Besides this disambiguation technique, they also propose a relative and an absolute control of the cursor using the touchscreen, similar to [START_REF] Nancel | Mid-air pan-and-zoom on wall-sized displays[END_REF][START_REF] Debarba | Lop-cursor: Fast and precise interaction with tiled displays using one hand and levels of precision[END_REF].

Technique Design

This section exposes our design decisions while developing the disambiguation canvas, its particularities (such as objects distribution over the plane) and its implementation details.

Volume Casting Techniques

The most common approaches for volume casting are the cone-casting and the sphere-casting techniques. Which one of these is best fit for our technique may depend on the application. Thus, we decided to support both volume-casting techniques for the disambiguation canvas.

Using the cast of a sphere, it is likely that the amount of objects intersecting the volume is smaller, as the sphere has a limited depth. However, it is also harder to control the first step of the selection as the depth must be somehow provided; there are two common approaches to determine the sphere depth. The first uses the near intersection of the sphere, and sets its distance to the intersection position. The second casts a ray through the center of the sphere; the distance of the first intersection of this ray is used to set the sphere depth. In SQUAD for instance, Kopper et al. [START_REF] Kopper | Rapid and accurate 3d selection by progressive refinement[END_REF] favored the sphere-casting with the depth of the sphere determined by ray-casting, but this may be due to the type of environment for which they developed the technique, a virtual supermarket application. In the supermarket environment, objects were very cluttered and organized as stacks in many shelves. Such organization facilitates the task of pointing out a cluster of objects with sphere-casting.

On the other hand, cone-casting allows reaching objects even if an intersection occludes them from the casting origin point of view, which sphere-casting is unable to do. Cone-casting can also reduce the necessary precision during the first stage, as no depth input is required. However, cone-casting may intersect too many objects if the scene is very cluttered. In the supermarket case, cone-casting would require constraints in order not to select objects behind the shelves; otherwise a huge amount of objects may fall inside the conic volume.

Based on the report by [START_REF] Argelaguet | Visual feedback techniques for virtual pointing on stereoscopic displays[END_REF] that users achieved up to 4 ∘ of error during the coarse precision phase of distal pointing, we suggest the angular size of 12 ∘ for the sphere/cone casting technique. Six degrees from the center of the ray to the sphere/cone borders. The sphere/cone always rescales to achieve the angular size of 12 ∘ from the casting position point of view.

Graphic Representation

We use an arrow shape for the cursor, and a semi-transparent rectangle to represent the canvas. Based on its widespread use, we concluded that the arrow would be the most natural and intuitive representation for the cursor pointing position, while the rectangle helps to easily match and associate the mobile device touchscreen with an area on the virtual environment. These shapes are only visible during the second step, which is the disambiguation step. The mobile device pointing direction is represented by a semi-transparent sphere or a cone, depending on the volume-casting technique used. The volume-casting shape in use is always visible, so the user can always have feedback on his/her pointing direction, even when performing the second step of a selection (disambiguation step).

The rectangle uses an absolute mapping with the mobile device touchscreen. It is drawn to use 30 ∘ of the total 45 ∘ standard vertical FOV of the camera. We positioned it 70cm away from the camera on our immersive display implementation, thus we obtained the size of ≈38cm. However, in order to make better use of the FOV, this size should be decided according to the available display, allowing the user to inspect the objects more efficiently, and therefore reducing the visual search time. The distance on which it is drawn may also vary in order to avoid occlusion with other objects in the scene. In our specific implementation, we adopted the distance of 70cm to maintain a pleasant stereoscopic rendering when switching from background to canvas.

Mapping Objects to the Canvas

When entering the second step of the disambiguation canvas, the subset of objects must be reorganized side by side over the canvas plane. As most users might have trouble reaching the whole touchscreen with the thumb, we propose two standard layouts on which the objects are reorganized so the user can easily reach them. The first consists of ≈53.4% of the total area, and is oriented to user handedness (Figure 2a). This layout takes 5% from the right, the left and the top, and 25% from the bottom out of the useful area, as well as 1/8 and 1/16 of the remaining that is too close to the palm and far from the thumb reach respectively. The second layout consists of ≈42.4% of the total touch screen area. This layout takes 5% from the right and the left, 10% from the top, and 30% from the bottom out of the useful area. The final layout consists of a circle inside the remaining area, as illustrated in Figure 2b. For a preliminary (not presented) and the first evaluation (Section 4.1), the layout presented in Figure 2a was used. Although in general it has worked properly for most subjects, two users from the preliminary evaluation had difficulty to reach a large portion of the layout, and in consequence obtained significantly higher error rates. Thus, we have also decided to approach a layout calibration method. To calibrate, the user performs circular movements with the thumb on the touchscreen within their range of comfortable motion (Figure 3a). A flood fill algorithm identifies the outermost bounds for that user's specific layout. This approach was used in the second evaluation, presented in Section 4.2. When switching to the disambiguation phase, a matrix fitting every pre-selected object inside the layout is computed, and each object is designated to a valid slot of the matrix (Figure 3b). A slot is considered valid if its center is located inside the usable area defined by the layout. In order to fit every object inside their designed slot, the objects are rescaled so their bounding box does not pass beyond that space. However, this may make some visual attributes less apparent or even impossible to be perceived, such as when the user wants to select a target of specific size within a group of similar objects. To overcome this issue the rescale factor may also be proportional to the largest and smallest target, being linearly remapped between a minimum and maximum threshold. That is, if the smallest object is ten times smaller than the largest one, this proportion factor (1:10) will be lowered to a maximum of half of the size (1:2) so both objects remain visible and distinguishable. Objects with intermediate sizes are proportionally rescaled in between these thresholds. By default, our current implementation uses this approach.

In our current implementation, which was used for the second user evaluation (Section 4.2), the position where the user starts a touch is not superposed by objects. This design facilitates leaving a selection procedure if the user does not want to select any of the objects pre-selected by the first phase (Figure 3b). In addition, it also avoids accidental selection in the case of an unintentional touch by the user. A regular immediate selection technique usually has its difficulty of pointing increased by reducing the target object size. However, when using the disambiguation canvas the difficulty increases according to how many objects have been pre-selected by the volume-casting during the first phase. This is an expected behavior of progressive refinement selection techniques, and might make the refinement slower and/or harder. However, as the disambiguation canvas relies on the mobile device touchscreen for disambiguation, we are able to align hundreds of objects within a single disambiguation step while still ensuring high precision. During the first technique evaluation (Section 4.1), we have used three distinct object densities; they are shown in Figure 4. For the worst case depicted in Figure 4c, 224 objects went to the disambiguation phase. Still, our technique offered the very convenient sensing area of ≈230 pixels of the touchscreen (the orthogonal projection of the spherical object over an area of 19×19 pixels), which has a total input area of 320×480.

Prototype Implementation

We used an Intel Core i7 computer, equipped with two AMD Radeon HD 5870 Eyefinity6. The immersive display is a Sensics zSight Integrated SXGA Head Mounted Display (HMD) (Figure 5a). It provides stereoscopic vision using two 1280 x 1024 displays, and has a FOV of 60 ∘ . This HMD also provides the orientation of the head.

For the mobile device, we have chosen the Apple iPhone 4/4S and iPod touch 4.

The software displayed by the zSight HMD is implemented in C++, using Ogre3D for graphics [START_REF]Ogre 3D[END_REF] (Figure 5b). We support stereoscopy in our application. The mobile device software is an app implemented in Objective-C. It acquires the sensor readings and communicates them over a Wi-Fi infrastructure through UDP. To obtain the orientation of the iPhone 4/4S, which has a magnetometerthus providing the recalibration of drift on the yawwe have used the strategy proposed by Madgwick [START_REF] Madgwick | Estimation of IMU and MARG orientation using a gradient descent algorithm[END_REF] with the adaptations presented in a previous work [START_REF] Debarba | Lop-cursor: Fast and precise interaction with tiled displays using one hand and levels of precision[END_REF]. In [START_REF] Debarba | Lop-cursor: Fast and precise interaction with tiled displays using one hand and levels of precision[END_REF], a complete description on the acquisition and processing of sensor data to provide orientation is presented. To obtain the orientation of the iPod touch 4, which does not contain a magnetometer, we have used the standard orientation provided by the iOS SDK.

Disambiguation Canvas Evaluation

We conducted two sets of user tests for the disambiguation canvas technique. The main design decisions are common to both evaluations and were based on those used by Kopper et al. for the SQUAD technique evaluation [START_REF] Kopper | Rapid and accurate 3d selection by progressive refinement[END_REF]. Both were comparative evaluations and used a within-subject design. In the first we compare disambiguation canvas with ray-casting, while in the second we compare it with SQUAD. The implemented ray-casting relies only on the orientation of the deviceassumption of a constant casting positionand is therefore referred to as ORayCasting (orientation ray-casting); disambiguation canvas and SQUAD also use only the orientation of the device for the volume-casting step. The disambiguation canvas is referred to as DCanvas. For both evaluations the objects pre-selected by the first phase of DCanvas were animated from their original position to the canvas in a 250ms animation. An iPod touch 4 was used in both evaluations. As mentioned earlier, it is not equipped with a magnetometer, and thus is subject to drift on yaw, losing its correct orientation. Therefore we used blocks with no more than 11 trials, and applied an offset in order to correct any accumulated error between blocks. On the other hand, it produces an orientation less noisy than using the magnetometer to correct yaw, necessary for reliability of pointing while using ray-casting. Mobile device orientation was filtered using a dynamic low-pass filter, interpolating between cutoffs of 0.2Hz and 50Hz, with 60Hz sampling rate. Cutoff is defined according to the angular change speed in degrees: when < 1 ∘ /sec, the lowest cutoff is used (0.2Hz); when > 50 ∘ /sec, the highest cutoff is used (50Hz). For any speed between those, a linearly interpolated cutoff value is used.

General goal: the goal was to select a yellow sphere among several distractors of same size represented as blue spheres. These objects were arranged as a matrix. To position them, we use a main sphere of 2.155m radius. All the selectable spheres were positioned with their centers intersecting the borders of this main sphere. The origin of the ray/sphere-casting was set to the center of the main sphere. This guarantees the same angular pointing size for all the objects. The virtual camera was positioned 50cm above the ray/sphere-casting origin. This configuration is shown in Figure 5.

Evaluation procedure: The same procedure and similar questionnaires were used for both of our evaluations. The procedure was as follows:

1. The subject was asked for any health issue or impairment that could prevent them from participating (such as a history of epilepsy and color blindness) 2. The subject filled in a characterization questionnaire 3. The subject was presented to the first technique on an ordinary screen display (so the experimenter and the subject could share the view while explaining how the technique works) 4. The experimenter presented the HMD and how to adjust it to the head 5. The subject performed practice blocks with the first technique 6. The subject performed evaluation blocks with the first technique 7. The subject answered a questionnaire about the first technique 8. The subject repeated steps 3, 5, 6 and 7 for the second technique 9. The subject filled in a post-experiment questionnaire comparing both techniques A block consisted of a collection of trials. A trial consisted of a selection task, ending with an activation of selection, which could be successful or not. The number of blocks and trials is different for each evaluation. Subjects were allowed to remove the HMD and rest between the blocks if desired.

Comparison with ORayCasting

Design: sphere-casting was used for the volume-casting step of selection on DCanvas. Instead of using the suggested standard size of ≈12 ∘ for the casted sphere, we used the angular size of ≈26 ∘ , so more objects would be pre-selected for the subsequent disambiguation phase. This sphere is represented by a semitransparent sphere. The ORayCasting casted ray is represented by a cylinder with 1cm of diameter.

The independent variables are angular space between objects (resulting in different distractors density): 5 ∘ , 2.5 ∘ and ≈1.67 ∘ ; and angular size: ≈0.53 ∘ , ≈1.06 ∘ and ≈1.6 ∘ (2cm, 4cm and 6cm respectively). We used blocks of 10 trials, 9 of them representative of the combination of density × size, and an initial target which was used to start the block. Training consisted of 5 blocks, while the evaluation consisted of 10. The 9 valid targets within each block were randomly presented, while technique presentation order was counterbalanced.

The target was randomly chosen among the objects with only one constraint, this object should have its center within a range between 52cm and 77cm from the center of the matrix of objects. We did so in order to keep the possible targets within the user's field of view, thus reducing visual search bias. We have also colored objects farther than 77cm to green, so the user knows they are not target candidates. For this evaluation, the collision checking on the disambiguation step was performed using the superposition of the arrow over the projection of the sphere on the canvas (a circular area).

The comparison with ORayCasting was also intended to verify design decisions and to evaluate whether the technique was comprehensive and easy to use. The design choices presented in Figure 3 were not used in this evaluation. In fact, they were implemented after the feedback from this experiment and were used for the following SQUAD comparison (Section 4.2).

Subjects: six graduate students in Computer Science from our university participated in this experiment (mean age of 29, four right handed). All of them were very experienced in managing mobile device touchscreens, and had at least some experience using natural pointing devices. In a 7 points scale, only two reported experience with virtual reality equipment of 3 or more points. Each test took from 25 to 40 minutes to be performed. We have obtained a total of 1,080 valid trials: 2 techniques × 6 subjects × 10 blocks × 9 trials.

Results: overall mean selection times with DCanvas and ORayCasting were respectively: 2.37 and 2.29 seconds. One-way ANOVA showed that DCanvas was slower with statistical significance when compared to ORayCasting (F(1.1078)=4.43, p<0.036). See Figure 6 for detailed mean time for each combination of target size and density. Error rate with ORayCasting was significantly higher than with DCanvas (F(1.1078)=70.34, p<0.0001). Error rates for each combination of size and density are presented on Figure 6.

Figure 7 presents the time and error rate per user. For this experiment, DCanvas obtained a lower mean time for two users. Four users have not made any selection error while using our technique. Subject 6 presented an exceptionally low error rate for ORayCasting, but still higher than with DCanvas.

The intermediate questionnaire asked users to rate each technique concerning: ease of learning and ease of use; how well it performs for small, medium and large targets; and how much fatigue was felt on their wrist, hand, fingers, back and legs. Results are presented in Figure 7. Both were considered very easy to learn, while our technique was considered easier to use. DCanvas was preferred over ORayCasting for small and medium targets, while large targets received equivalent ratings for both techniques.

Overall fatigue was lower for DCanvas; its mean of the 5 related questions was 2.1, against 2.5 of ORayCasting. Regarding the comparative questionnaire, DCanvas presented higher scores for all the questions. It was considered more accurate (6.7 against 1.6), faster (5.9 against 2.7), less tiring (4.8 against 2.6) and easier to use (5.6 against 2.5). Curiously, users felt that DCanvas was faster, which is true for ≈.53 ∘ angular size targets, but false for the overall evaluation. DCanvas was also preferred by all the users.

Comparison with SQUAD

Design: for the comparison between DCanvas and SQUAD we have eliminated the target size from the set of independent variables. Our previous evaluation, as well as the one performed by Kopper et al. to test SQUAD, showed that the target size is not significant for time of selection or error rate. Thus we used the constant angular size of ≈1.06 ∘ (4cm) for all the objects. For the independent variable of angular space between objects (distractors density) we used: 6 ∘ , 3 ∘ , 2 ∘ , 1.5 ∘ and 1.2 ∘ . We used blocks of 11 trials, 2 trials of each density, which were randomly presented, and an additional initial target used to mark the start of the block. Training consisted of 3 blocks, while the evaluation consisted of 4. Technique presentation order was counterbalanced. The target object was randomly chosen with the same constraint as before, but accepting a range between 45cm and 55cm from the center of the matrix of objects. Objects beyond 60cm from the center were colored in dark blue, which is less distractive than the green used for the previous evaluation. We used sphere-casting for the first step of selection with DCanvas and SQUAD. Instead of adopting the suggested standard size of ≈12 ∘ for the casted sphere, we used the angular size of ≈17 ∘ for both techniques, so more objects would be pre-selected for the subsequent disambiguation phase. Combined with the possible angular space between objects for this evaluation, the sphere-casting phase could pre-select ≈6, ≈25, ≈55, ≈100 or ≈160 objects for the disambiguation phase. The sphere-casting step of the DCanvas and SQUAD was represented by a semitransparent sphere.

The SQUAD casted raydisambiguation phasewas represented by a cylinder with 1cm of diameter. Figure 8a-c presents the walkthrough of our SQUAD implementation. The quad menu is drawn at a distance of 100cm from the camera, and is oriented to face the camera. The quad menu is composed of four triangles that assemble a square of 4×4 meters (Figure 8d). To recover from a mistaken pre-selection, the user can point outside of the quad menu and perform a tap gesture, or select an empty quadrant. An error only occurs when the user selects a wrong quadrant that contains only one object inside.

For this evaluation, the collision checking on the disambiguation step of DCanvas was performed using the superposition of the arrow over the designated objects slot (instead of the object projection) in the canvas. This approach increases the effective selection size of the object, and also allows a simpler collision test when dealing with objects with a mesh more complex than those tested (spheres). Additionally, the improvements presented in Figure 3 were also used. Subjects: eleven subjects participated on this evaluation, all of them students or professors in computer science or electrical engineering (mean age of 31). On a seven points scale, only one reported little experience with mobile device touchscreens (below 5), and three reported experience with pointing devices equal or above 4 points. Only one subject reported high frequency of use of virtual reality devices.

Results: as demonstrated by Figure 9, DCanvas performed significantly faster than SQUAD for all conditions. We also highlight that the increase in selection time for DCanvas was less steep than SQUAD. For the error rate, both techniques achieved good marks. DCanvas had an error rate of 0.018 errors per trial, while SQUAD achieved 0.009 errors per trial. Figure 10 show the individual performance of mean time and error rate for each user.

Concerning the comparative questionnaire, 6 subjects preferred the DCanvas, while 5 liked SQUAD most. The mean scores of comparative questions were very similar between techniques. DCanvas was considered less precise by a difference of ≈1.26 points on the 7 points Likert scale, and more difficult to learn and use by a difference of ≈0.45 and ≈0.39. DCanvas was regarded as faster by a difference of ≈0.19.

However, on the questionnaire specific for each technique, when no direct comparison was required, the DCanvas obtained higher scores concerning its ease of learn and use (Figure 10). On the other hand, SQUAD received higher absolute scores concerning the level of cluttering of the environment. Concerning the fatigue, answers were generally very similar, except by the fatigue on the fingers, which was higher for DCanvas. The mean of these scores are reported in Figure 10.

5

Discussion and Final Remarks

Transition to the Canvas

The most recurrent feedback left by the users regards the transition of the subset of objects from its original context to the control canvas. Users frequently had the conviction that positioning the intended target near the center of the sphere during the sphere-casting step would take that target near the center of the canvas when switching to disambiguation. This intuition may arise from the arrangement of objects as a matrix, which would be easily fitted inside the layout. However, on a more complex scenario, with targets spread in depth, such organization is not so obvious. We are currently working on this issue, as it could reduce user effort of reaching objects mapped to distant regions of the touchscreen and reduce visual search time.

Keeping the context

As for being a progressive refinement technique based in menu disambiguation, objects that go from the first to the second phase lose their original context. This could make it difficult to distinguish the intended object in real applications if they are very similar in shape or if the selection depends on their original topology. We propose three possible solutions for such limitation of the menu disambiguation approach. The first solution is to control the instant of interpolation that animates the objects while bringing them over to the disambiguation menu. To cast a ray or a volume for pointing, only 2 degrees of freedom (DOF) among the 3 provided by the device orientation are required. Our proposal is to use the 3rd DOF to dynamically control the instant of the interpolation. The mapping from orientation into instant of interpolation can be achieved with an absolute relation, where a certain orientation always results on the same instant, or with a relative relation, where after a threshold the orientation controls acceleration forward or backward on the interpolation instant. This strategy was implemented, and showed to be functional; however it was not yet evaluated. The second technique consists in duplicating the original object into the canvasinstead of moving the originaland using the copy superposed by the arrow cursor to highlight the original object. It can indicate whether the user is pointing to the desired object when there is more than one object with the same or similar shapes.

The third approach is to draw a trajectory curve to connect the original position of an object with its final position on the canvas. This approach allows the simultaneous observation of all connections between original and final positions at the same time. However, this can result in cluttering and may overwhelm the user with information when too many objects are taken for disambiguation.

Notice that these suggestions are not exclusive and can be combined among them. Given that this is a general problem of menu based progressive refinement selection techniques, we intend to investigate these approaches further in future works.

Immersive tool

The tests we led, as well as the prototype implementation described in Section 3.4 considered the use of a head-mounted display (HMD) and 3D stereo visualization to enhance the realism and immersion. Subjects were comfortable with this set up. However, even if the camera is driven by the movement of the user's head, we are not taking much advantage of this, since the objects were concentrated in a relatively small area, in front of the user. Our objective so far was to compare the disambiguation canvas with other techniquesas ray-casting and SQUADaccordingly. Then, we tried to avoid any other independent variable. Informally, we also tested the disambiguation canvas with a regular display. Our intuition is that its use with a regular display depends strongly on the layout of the objects. On the other hand, using the HMD provides easy control of the camera, which comfortably overcomes this limitation.

We are also aware that, to verify the robustness of the technique, more tests should be done with other layouts for the objects in the scene. Currently, they are all disposed on the surface of the sphere that surrounds the user.

Conclusions and Future Work

In this paper we have presented the disambiguation canvas, a technique for fast and high accuracy selection of objects in a 3D space. It relies on progressive refinement to address the lack of accuracy common to immediate selection techniques, and uses distinct input hardware to optimize the control over its two steps and overall time performance on a selection procedure.

By using the touchscreen on the second step of the disambiguation canvas, users were able to select objects represented in a motor area of ≈7.9mm 2 of the touchscreen surface very efficiently during evaluation, allowing consistent disambiguation among a group of 150≈250 objects. However, the limits for efficient disambiguation with our technique are still unknown. Perhaps the simultaneous exhibition of so many objects to the user may be more limiting than the precision of input of the mobile device touchscreen. In [START_REF] Debarba | Lop-cursor: Fast and precise interaction with tiled displays using one hand and levels of precision[END_REF], precision above 60% was obtained on a touchscreen area as small as ≈0.6mm 2 . If we transfer this parameter to the disambiguation canvas, the whole device touchscreen surface would allow the disambiguation of up to 6,144 objects in one step. Are we able to display meaningful objects in the order of thousands to the user? Is the user able to search for a specific object within such a large group? If this is the case, we also intend to adapt our technique to ensure a reliable selection with one additional step, such as an area selector that points a group of objects within a radius from the thumb position. With this strategy, and taking advantage of the touchscreen precision, we expect to reduce the selectable objects by a factor of at least 10, instead of the factor of 4 used by SQUAD.

Nevertheless, we emphasize that the user's subjective rating assumes that our technique was faster than ray-casting, while it had in fact performed slightly slower. This might be a clue to how unpleasant it is to perform a difficult selection with full attention. We have observed that even the breath had to be controlled for some users.

Fig. 2 .

 2 Fig. 2. Standard layouts of useful touchscreen area proposed for the disambiguation canvas techniques. The (a) standard layout was used for the first presented evaluation.

Fig. 3 .

 3 Fig. 3. Proposed layout calibration process, (a) contour of the reachable area defined by a user. (b) Arrangement of objects inside the reachable area, notice that the starting position of the arrow is kept empty. This procedure was used for the second evaluation (Section 4.2).

Fig. 4 .

 4 Fig. 4. Difficulty of selection is proportional to the amount of objects pre-selected by the first step; in (a) there are 25 objects on the canvas, in (b) 97 objects, and in (c) 224 objects.

Fig. 5 .

 5 Fig. 5. Merged images of the prototype overview (a), and the test application used for evaluation (b).

Fig. 6 .

 6 Fig. 6. Mean trial completion time and error rate for each combination of target angular size and density for the comparison with ORayCasting.

Fig. 7 .

 7 Fig. 7. Mean time and error rate per user and subjective questionnaire scores for DCanvas and ORayCasting.

Fig. 8 .

 8 Fig. 8. SQUAD implementation used for comparison: (a) the group of objects inside the spherecasting volume is selected by a tap gesture; (b) these objects are rearranged into quadrants in a 250ms animation, to select a quadrant the user perform a tap gesture while intersecting it with ray-casting; (c) the subgroup is rearranged in new quadrants with a 200ms animation, a new tap gesture while intersecting the target quadrant results in a successful selection. Note that although the animations impose some time constraints, it also avoids the need for visual search at each new step, and the user can start the repositioning of the ray during the animation. (d) Overview of the quadrants area of selection, a tap gesture while not intersecting any quadrantor intersecting an empty quadrantmay be performed to leave the selection procedure.

Fig. 9 .

 9 Fig. 9. Mean trial completion time for each angular space (density) of objects distribution for the comparison with SQUAD.

Fig. 10 .

 10 Fig. 10. Mean time and error rate per user and subjective questionnaire scores for the comparison with SQUAD.

  Figure 11 shows four frames of an animation, moving the objects from their original to new position in the disambiguation canvas.

Fig. 11 .

 11 Fig. 11. Four frames illustrating the animation of the objects from their original positions to the new ones on the disambiguation canvas. The animation can be reproduced in both directions by twisting the mobile device in the corresponding direction.

Acknowledgments

We would like to thank CNPq-Brazil for the financial support through projects 311547/2011-7, 485820/2012-9, 302679/2009-0 and 305071/2012-2. Thanks are also due to Microsoft Interop Lab at UFRGS. We finally thank the volunteers who kindly agreed to participate in the experiments.


Andrea Kohlhase

Michael Kohlhase

Alexandru Toader {a.kohlhase, m.kohlhase, c.jucovschi Constantin Jucovschi

Full Semantic Transparency: Overcoming Boundaries of Applications

Keywords: Full semantic transparency, multi-application SEMANTIC ALLI-ANCE, frame shifts, spreadsheets, CAD systems, semantic services

Complex workflows require intelligent interactions. In this paper we attack the problem of combining user interfaces of specialized applications that support different aspects of objects in scientific/technical workflows with semantic technologies. We analyze the problem in terms of the (new) notion of full semantic transparency, i.e., the property of user interfaces to give full access to an underlying semantic object even beyond application lines. In a multiapplication case full semantic transparency is difficult, but can be achieved by representing the semantic objects in a structured ontology and actively supporting the application-specific framings of an object in a semantic interface manager. We evaluate the proposed framework in a situation where aspects of technical constructions are distributed across a CAD system, a spreadsheet application, and a knowledge base.

Introduction

As the objects we create and interact with get more and more complex, the corresponding workflows increase in complexity as well. Part of this complexity is mitigated by specialized support systems for specific sub-goals. But this only shifts the underlying problem from mastering complexity to mastering sequences of autonomous steps to smoothly perform a given task. Even though sub-goals might get simpler to reach, individual subtasks often require distinct support systems. In particular, users need to use multiple applications offering support to address specific rather unpredictable aspects of a task. Therefore, this user-driven workflow, we call it "useflow", has become a topic of research interest in recent years.

To clarify the issues, let us look at a concrete multi-application useflow, which will serve as a running example for this paper and for which we implemented a fully semantically transparent interface.

Running Example

Charles A. is a design engineer in a small company that specializes in the production of small and fast luxury yachts. As it is a small company Charles needs to analyze customer needs in interviews, has to create design solutions with a CAD system like Autodesk Inventor, and also decides on production issues.

The design tasks range from the yacht's shape down to pipe fittings (see Fig. 1 here a blind flange fitting, a particular pipe end that allows allows access to the piping system for revision and cleaning; we will refer to it as a "pipe end"1 in the following).

As customers often set tight budget restrictions, Charles uses several spreadsheets to inform himself about vendors' pricing schemes for frequently needed items as for example pipe ends. In Fig. 2 we see such a spreadsheet for pricing details of pipe ends (e.g. in MS Excel). Charles can 'play with the numbers' by entering distinct quantities or specifying different attributes of the components of a pipe end to compare prices between vendors. The price conditions for components like flanges and bolts are stored in other worksheets in the same workbook (see example in Fig. 3). In this industrial sector it is customary that vendors offer percentual discounts for sale items depending on the quantity of units to be ordered. Charles' useflow forces him to switch between the CAD program and the spreadsheet system quite frequently.

Contribution and Structure of the Paper

We note that application switches in a useflow become necessary to access different aspects of an object currently in focus (e.g. pricing for the currently active CAD assembly). In particular, Charles would like to have the application he switches to already focused on the target object: that object in the target application that represents the respective aspect of the source object (e.g. the price of the type of bolt focused in the CAD system). Normal application switchers (e.g. Alt-Tab) do not provide this quality.

In this paper, we use sense-making theory to better understand this "aspect switching" resulting in a model of "frame shifts" and an interface requirement we call "full semantic transparency", i.e., a UI property that provides users access to the full meaning of information objects. Moreover, we present a fully semantically transparent interface. In particular, we strive to support frame shifts via a semantic "mash-up" of user interfaces on a fine-grained object level. Concretely, we make use of an existing semantic interaction framework, which links the semantic objects in applications to concepts in a structured background ontology. These concepts act as a "pivot table" that allow us to predict and focus target objects in frame shifts.

Before we present the interface solution in Section 4, we survey related work (Section 2) and establish a model for frame shifts and develop the concept of "full semantic transparency" (Section 3). Section 5 concludes the paper.

Fig. 3. A Spreadsheet as Database

Related Work

Generally there are three ways to address useflow smoothing and application interoperability: U1 Merge applications into a single large system. U2 Make applications interoperable on the data level. U3 Make applications interoperable on the user interface level.

Option U1 is only feasible for very established and high-volume workflows. A good example for this option are the office suites that combine word processors, spreadsheets, slide presenters, and drawing applications under one joint interface. Other examples consist in integrated trade-specific solutions that cover the main workflows of a specific industry sector. The workflows covered by such verticallyintegrated systems are usually quite rigid, since any change in workflow needs to be reproduced in the system and thus needs a system extension. Thus, such systems cannot cater to the needs of single users; moreover, Charles' small company could probably not afford one of these large-scale integrated systems.

Option U2 is facilitated by the trend towards offering open APIs that allow access to its underlying domain and event models. In [START_REF] Díaz | Irastorza: Improving Portlet Interoperability through Deep Annotation[END_REF] e.g., Diaz et al. describe a method that uses deep annotation, i.e., an ontology representation, to define the data flow from one portlet to another. Therefore, objects are related semantically on a data level. Most existing approaches though don't pay attention to the resulting user interface challenges, that is a users' access to their full meaning. In [START_REF] Iturrioz | Toward the Semantic Desktop: The seMouse Approach[END_REF] this approach is taken up on a conceptual level, stating for example, that a document " will no longer be a mere file, but rather a knowledge base resource", but a closer reading still reveals a similar bias towards machine-understanding. Another approach is the Gnowsis Semantic Desktop framework [START_REF] Sauermann | Semantic Desktop 2.0: The Gnowsis Experience[END_REF], "a tool for personal information management" [ibd., p. 887]. This collects data from various desktop-based applications in an RDF-based "personal information management ontology" (PIMO), which services can tap e.g. to mash-up data by distinct applications. Even though the data modeling component is similar to what we will present below, the focus of the Semantic Desktop is on information retrieval and semantic information management services (e.g. cataloging or sorting images according to the PIMO) and -to the best of our knowledge -not on UI-level support for useflows, which is what we concentrate on.

Option U3 has gotten less general attention, except for a call for uniform look-andfeel across applications driven by usability concerns. The underlying reason consists of the idiosyncrasies of user interfaces based on the particular technologies used and tailored task support. User interface mash-ups only seem simple to realize if the applications share the same base technology, e.g. for web applications. But there are many non-web applications supporting subtask achievement that are designed as insular systems that excel in their specific domain.

The coordination of multiple visualizations or mash-up of visualizations of information is often used when the "information is sufficiently complex to require different types of visualizations for different aspects or layers" [15, p. 715]. North & Shneiderman note that such visualization requirements tend to be rather unforeseeable. Thus, they devised "Snap-Together Visualization" [ibid.], an approach to dynamically support multiple coordinated visualizations based on a relational database model that generates views, which in turn are linked together through different events. In contrast, Bull models the coordination of visualizations via software engineering modeling approaches [START_REF] Bull | Integrating Dynamic Views Using Model Driven Development[END_REF]. Tudorache et al. note in [START_REF] Tudorache | A Knowledge Base Driven User Interface for Collaborative Ontology Development[END_REF] that such approaches are essentially based on dynamic communication between independent components, but envisioned beforehand by a software designer and not necessarily employable by an application user involved with situated actions [START_REF] Suchman | Plans and Situated Actions: The Problem of Human Machine Communication[END_REF][START_REF] Nardi | Studying Context: A Comparison of Activity Theory, Situated Action Models, and Distributed Cognition[END_REF]. In particular, they are mainly targeting the workflow rather than the useflow.

The unpredictable flexibility requirements for the useflow may be explained by the difference between a user's task environment and his information environment, the background environment that allows users to acquire more information to take better actions (see [16, p. 20]). Pirolli discusses multiple information foraging models of humans, one of which is the "patch model" [ibd., 31ff]. Here, information is assumed to be distributed in a 'patchy' manner. In our running example, Charles finds information needed in his useflow at different locations in distinct applications. Therefore, we can apply the patch model to Charles' useflow. Obviously, an information forager needs to optimize his use of information patches. In which cases or at which steps in the useflow, for instance, should Charles switch applications in order to get price information about pipe ends? If, for example, Charles wants to minimize the patch distance, he makes use of keyboard shortcuts like "Alt Tab" to switch applications. In contrast, if he wants to maximize information gain on pricing issues for pipe ends, he authors a spreadsheet to support him. To keep customers within their applications, companies draw on this trade-off by enriching environments with added-value services like providing piece-lists of assemblies in grids from within a CAD system (with all the disadvantages listed in Option U1).

Pirolli shows in [16, p. 38] that reducing between-patch costs improves the overall average rate of information gain and moreover that the optimal gain is achieved by spending less time within a patch. Thus, information gain can be improved by mashing up patchy interfaces to reduce between-patch distance and at the same time supporting application-switching. Therefore, in this paper we concentrate on Option U3 departing from a concept called "semantic transparency" [START_REF] Kohlhase | Semantic Transparency in User Assistance Systems[END_REF], that assumes a user's need for semantic alignments of applications in his useflow, towards "full semantic transparency" based on sense-making theory. In particular, we will apply the notion of "frames" [START_REF] Weick | Sense-Making in Organizations[END_REF] to semantic transparency and propose to address useflow-centered mashing up of user interfaces by semantic technologies.

Full Semantic Transparency

If we want to optimize between-patch costs in Charles' information environment, then we have to better understand how humans conceive information. This is a central question in Sense-Making Theory (see e.g. [START_REF] Sharma | Role of Available and Provided Resources in Sense-Making[END_REF], [START_REF] Dervin | Sense-Making Theory and Practice: An Overview of User Interests in Knowledge Seeking and Use[END_REF] for an overview). In [START_REF] Weick | Sense-Making in Organizations[END_REF] Weick introduces the terms "frame", "cue", and "connection":

Frames and cues can be thought of as vocabularies in which words that are more abstract (frames) include and point to other less abstract words (cues) that become sensible in the context created by the more inclusive words. Meaning within vocabularies is relational. A cue in a frame is what makes sense, not the cue alone or the frame alone. Said differently, the substance of sense-making starts with three elements: a frame, a cue, and a connection.

To anticipate trajectories of information (and act accordingly) Charles needs to make sense of the information in his focus. In a nutshell, this means that he continuously strives to understand the potential connections of the information cues he is perceiving with respect to possibly different frames via the UI of applications. Kolko notes the long-term importance of frames in sense-making as a frame can be viewed as "an active perspective that both describes and perceptually changes a given situation" [START_REF] Kolko | Sensemaking and Framing: A Theoretical Reflection on Perspective in Design Synthesis[END_REF].

In the following we suggest a model of sense-making in the process of application switching in UI terms.

Cues via Information Objects

Information is carried in a user interface via "information objects". Each application supports the presentation of this information in a different way, particularly via different means. In a CAD system, the assemblies and the parts they are made of are information objects. Consider for instance a hex bolt such as the one on the left of Fig. 1, which is made of a cylindrical shaft and a hexagonal head: while the former is a geometric object the latter two are not. In contrast, in a spreadsheet document cells and tables are information objects: they are mere data containers on a technical level, but their associated properties like the type give their content (e.g. a specific number) a meaning. This meaning of a cell might be implied by other information objects. In particular, the position of a cell within a table consisting of row and column specification carries specific meaning itself. Therefore, we can view information objects with their contents as cues.

Frames via Applications Frames conceptualize knowledge in a way that enables information system stakeholders to make use of the elements of information systems (see [START_REF] Le Roux | People Frames: The Social Construction of Information Systems[END_REF]). Usually, the terms "frame" and "cue" are employed with respect to people's cognitive abilities, but they are also expressed in the design of their tools. In particular, we can consider applications as tools that frame information objects. They are manifestations of frames, thus, we can treat them as frames themselves.

Connections via Conventions

The information objects in the applications can be seen as constituting a symbolic, domain-specific language for application authors and readers based on common conventions. These can be general conventions like the interpretation of a table as a grid having specific properties or application-dependent conventions like cell content being of a specific type. For making sense of given information the connection between information object and application is established with these conventions.

Sense-Making across Frames via Semantic Objects

Frame shifts in a certain useflow assume the existence of a meaningful object that aligns these frames. Charles, for example, looks at the object "bolt" from two viewpoints: under the frame of geometric construction from within a CAD system and under the frame of pricing of a product from within a spreadsheet system (see Fig. 4). To stress that an object carries several meanings depending on the perspective it is looked at, this is called "semantic object". A semantic object functions as a pivot for application shifting, thus, if we establish a semantic object as a node in an according ontology, this node can serve as a pivot for frame shifts.

For each frame the representation of a semantic object is stored separately in a document. Each application in turn is a "player" of certain document-types (i.e., it plays special documents), it presents the representations of semantic objects. The "pipe end" e.g. is represented as *.iam file and as *.xlsm file. Note that such representations also include representations of other semantic objects like "bolts" or "flanges", that e.g. in the CAD system Autodesk Inventor are stored as independent part-files (*.ips). In the following we restrict applications to be document players as only those enable application switching as described above. [START_REF] Kohlhase | Semantic Transparency in User Assistance Systems[END_REF] introduces the term "semantic transparency": "a user interface [is called] semantically transparent, if it enables a user to access its semantic objects and their relations via the corresponding UI objects". But what does "access its semantic objects" mean? In [START_REF] Kohlhase | Semantic Transparency in User Assistance Systems[END_REF] it means access to more information about the semantic object. But in our running example we have observed that respective information objects in the CAD system and the spreadsheet application only handle their very own specific frame of the semantic objects Charles has to deal with in his daily work. In particular, access to a semantic object should be extended to mean access to all available frames of a semantic object -enabling a "full semantic transparency". Even though -as our example above shows -this is exactly what Charles would need, neither of the user interfaces MS Excel nor Autodesk Inventor involved in our running example allow access to both frames of the object under consideration, that is, neither UI is fully semantically transparent.

Sense-Making of Frame Shifts via Full Semantic Transparency

Full semantic transparency hinges on the sense-making process during frame shifts for a semantic object. As we mentioned above, sense-making has three components: a frame, a cue and a connection. In this paper, we assume that frames are given by particular documents and their players. Moreover, we can view information objects with their contents as cues. But who or what establishes the connection across frames? Charles aligns the applications, that is, his cognition establishes the connection between a frame and a cue, so that a specific aspect of a semantic object can be communicated. This relation can also be called an interpretation, i.e., "the assignment of Fig. 4. A Semantic Object as Pivot For Sense-Making meaning to data" [START_REF] Beyer | Contextual Design[END_REF], which in turn can be interpreted as subjective extension or enhancement of the given data [START_REF] Kolko | Sensemaking and Framing: A Theoretical Reflection on Perspective in Design Synthesis[END_REF]. In the following we formalize this connection across frames so that we obtain computer support for frame shifts.

Realizing Full Semantic Transparency

The very basic idea to reach full semantic transparency consists in the fact that semantic objects can be modeled in structured ontologies. So instead of trying to connect e.g. the information object "bolt" in the CAD system directly with the information object "bolt" in the spreadsheet system, we formally align both by the detour through a structured ontology (see Fig. 5) which acts as a pivot structure. Semantic technologies handling these ontologies can provide intelligent services and user interfaces.

We choose the open source "SEMANTIC ALLIANCE" framework [START_REF] David | SEMANTIC ALLIANCE: A Framework for Semantic Allies[END_REF], which allows to superimpose semantic services over an existing (and possibly proprietary) application provided that it gives open-API access to user events as a basis for a joining, fully semantically transparent user interface. The central advantage for our purposes is that the interaction intelligence is encapsulated in an autonomous, application-independent semantic ally called "Sally", which remains invisible to the user, but which orchestrates the joint user interface. In particular, information objects of applications are Fig. 5. Mashup of Interfaces linked to concept specifications of the semantic object in a structured background ontology via an "interpretation mapping" which can serve as a model for the sensemaking component "connection" between a frame and a cue. In turn we can model frame shifts by aligning information objects via their interpretation mapping to a presentation of a semantic object in an ontology. This way, we can support useflows by extending the SEMANTIC ALLIANCE framework towards integration of applications of different media types.

Mashing Up User Interfaces

We have implemented the "MA SEMANTIC ALLIANCE" ("MA" stands for "multiapplication") framework, an extension of SEMANTIC ALLIANCE which allows to integrate multiple applications. It enables mediating frame switches between applications of different media type in complex useflows.

It has been stressed according to [18, p. 1815] that "sense-makers benefit not just from good sources of information but also from good sources of structure". Moreover, information objects are the UI objects that carry meaning for the user. In particular, clicking those objects is a natural entry-point for access to the semantic object. Therefore, to "unleash […] sense-making potential" [6, p. 36], the Sally-based UI adds interaction elements at two levels: Sally Frames is a frame selection menu (Fig. 6) which allows users to cognitively shift frames very comfortably without necessarily yet shifting applications. The idea is that the user focuses on an information object in one application, which triggers awareness about other meanings of the underlying semantic object. If the interest in another meaning is large enough, he can choose a frame shift via the menu "Sally Frames" and all available services for an information object under the new frame will be shown on demand via Sally Services. Sally draws on an ontology of frames (including an empty one), which services can register for. Therefore, Sally can gather services with respect to a certain frame.

Sally Services is a service selection menu at the information-object level (Fig. 7 2 ). An information object is assumed to be in the user's focus when he left-clicked it, for example the "bolt" in the assembly in Fig. 1. Once he has opted for a frame, the respective "Sally Services", a menu for services currently registered with Sally and available for the semantic object "bolt" in the current frame, replaces the Sally Frames menu. These services provide intelligent user assistance at a very finegranular level. They can range from a Google search with keyword "bolt" over a definition look-up service in a company-wide knowledge base up to a service in a spreadsheet to calculate prices for the selected bolt in the CAD system (see [START_REF] David | SEMANTIC ALLIANCE: A Framework for Semantic Allies[END_REF]). Additionally, the respective services can add their own custom interactions, either through custom menus or via popup windows provided by the SEMANTIC ALLIANCE framework.

To realize the MA SEMANTIC ALLIANCE, we modeled the abstract document models (ADM; called "abstract document types" in [START_REF] David | SEMANTIC ALLIANCE: A Framework for Semantic Allies[END_REF], for a concrete example see [START_REF] Ch | Modeling the Structure of Spreadsheets[END_REF]) for spreadsheets, CAD assemblies, etc. in abstract document ontologies, so that multiple documents and their interpretation mappings can be jointly represented in the form of RDF triples in an in-memory triple store in Sally. Whenever a document is opened in the MA SEMANTIC ALLIANCE, salient aspects of its contents are exported to the triple store together with information about their interpretations. Instead of direct ADM API calls, MA SEMANTIC ALLIANCE services now use SPARQL [START_REF] Prud | Seaborne: SPARQL Query Language for RDF, W3C Recommendation World Wide Web Consortium[END_REF] queries to access the respective information objects. This redesign allows queries across information objects and their interpretations from all open documents. In particular, services can use queries to determine whether they are applicable to a chosen information object (and frame); this information can be used to populate the menus Sally Frames (Fig. 6) and Sally Services (Fig. 7).

Our experiments show that the query-based approach is expressive enough to realize the smooth task-switching capabilities in all studied cases, including our running example which we will now revisit.

A Use Case with a Fully Semantically Transparent UI

To fortify our intuition about these interface elements on the one hand and to evaluate the expediency of full semantic transparency on the other, let us see how an instance of the MA SEMANTIC ALLIANCE combining a CAD system UI, a spreadsheet system UI, and a knowledge base UI transforms Charles' application-switching useflow from our running example to an interaction model allowing full semantic transparency via frame shifts.

We will go through Charles' new useflow step by step (italicized text) and discuss the UI repercussions (normal text). fulfilling some geometric side conditions like strength and pipe diameter. Satisfied, he wants to verify his design in terms of the customer requirements and senses that his pipe end design may violate financial constraints. He has a closer look at the kind of bolt he used (by clicking on one bolt) and verifies that it is a rather expensive carriage bolt (Fig. 6). Normally, Charles would have had to switch to his spreadsheet application, load the worksheet in Fig. 2, enter part quantities, and evaluate the result.

a. Frame selection Now, Charles looks at Sally Frames for the "Pricing" frame. In this context menu Sally offers the user access to all current frames, here e.g. "Pricing", "Knowledge Base", or "Customer Contracts".

Under the frame "Knowledge Base" he expects for instance services that deepen his understanding for the concept "pipe end", whereas under "Customer" he wants to get information about exact customer requirements given in the contract for pipe ends. But for now Charles selects the "Pricing" frame. His choice replaces the Sally Frames with the Sally Services menu (containing currently available services for the semantic object "bolt") yielding Fig. 7.

b. Service Selection

Charles studies the Sally Services menu (as seen in Fig. 7). To determine the available pricing services, Sally needs to know the pertinent information object; as Charles has chosen the "bolt" component, Sally gathers all pricing services available in an extra window overlaying the CAD application. If those services depend on cue x like "Give me the definition for x", then these services are offered for the content of the selected information object, here for "bolt".

Concretely, Charles has the choice between a company-wide, official spreadsheet for pipe end prices, a private one, that he authored himself as a calculation template, the price conditions for the component "bolt" pricing" worksheet with the resp. properties of the selected bolt, which in turn becomes the top most UI, that is, Charles switched the application (Fig. 8).

Here, Sally makes use of the interpretation mapping for the Services menu in two ways: it can focus on the pertinent worksheet, since the respective table is mapped to the pipe end via the interpretation mapping, and in particular, the quantity cell to the concept of multiplicity of parts in assemblies. Charles glances over the values and decides that his suspicion was warranted: the design is over the price range set by the customer.

a. Frame Selection He tries to reduce the overall price by looking for alternatives for the pricey carriage bolts. Therefore, Charles now selects the frame "Knowledge Base", since he needs information on homologous bolts, and clicks cell [A7].Sally in turn opens a window next to cell [A7] offering all available knowledge base services for "bolts".

b. Service Selection Here, Charles chooses the "variants" service.

Variants Service Menu

The variants service uses a complex user interface of its own inside the Sally-originated UI window to determine the full specification of the bolts. In particular, Sally replaces the menu in the window by a list of Fig. 8. Pipe End Pricing with Sally Frames Menu bolt properties that can be altered without affecting functionality. This list tells Charles that he can use arbitrarily colored bolts with arbitrary head types as long as he stays with the M15 thread.

Direct Interaction

This variants menu stays on top even though Charles now selects [D7] and modifies the head type to "machine", which triggers the spreadsheet system to recalculate the numbers.

Variants Service Menu

The price is already better, but he isn't yet satisfied, so he selects the color entry in the variants menu. In consequence a temporary worksheet is created with a number of tables that display prices for the different color options for M15 machine bolts as in Fig. 9. Alternatively, Charles could have chosen the "Best price option", then the input data from the variant with the best price had replaced the original values.

We see that the interactions in the Sally-based interface are triggered by the interpretation mapping and the target concepts in the ontology. The advantage of this approach is that some aspects of the user interface can be declaratively specified in the ontology, but the MA SEMANTIC ALLIANCE-based approach also incurs requirements on the ontology and costs, which we will discuss next.

Requirements for the Ontology

The ontology has to include conceptualizations of the data models of the (information objects in the) applications themselves, e.g. concepts of CAD assemblies that Fig. 9. Variants for Bolts in Terms of Color are made of "parts" -which may be geometric objects like nuts or sub-assemblies themselves. Fig. 10 shows the taxonomic part of the ontology used in our example enriched with the "has part" relation (for the structure of assemblies). In our case, the ontology also contains the fact that a bolt is characterized by its head type, thread, and color.

Note that in the development of the ontology, we chose to only represent the object types, not the object instances themselves. For instance, the ontology only has the concept of a machine bolt (and its characteristic dimensions), but not the six individual instances in the pipe end assembly. Similarly, the ontology only specifies the notion of a volume pricing function of an object, but not the actual prices for the objects in the respective assemblies. The instance data stays in the applications themselves: the CAD system for the parts of the assemblies and the spreadsheet for the pricing information. This seems like a generally applicable heuristic for the development of engineering domain ontologies, as this information still justifies data import/export between frames: For the frame switch from the CAD frame to the pricing frame, we only need to know that the six bolts in the pipe end assembly co-reference the M15 machine bolt to export the data into the spreadsheet, which also references it in a functional block that takes the bolt characteristics as inputs.

Cost-Benefit Evaluation

The MA SEMANTIC ALLIANCE architecture consists of a network of semantic services that provide value, but need an interpretation mapping and background ontology to operate. Like any network-based approach, the potential benefits grow in proportion to the size of the network while the costs -often called network joining costs -grow with the size of the joining node. Our approach incurs three kinds of joining costs: a) the applications have to be equipped with a SEMANTIC ALLIANCE API, b) the domain has to be represented in a structured background ontology that specifies the semantic objects concerned, and c) each document must be equipped with an interpretation mapping into the ontology.

The costs for a) are almost negligible: [START_REF] David | SEMANTIC ALLIANCE: A Framework for Semantic Allies[END_REF] report that the effort required is in the order of developer weeks, which we can confirm for the API for Autodesk Inventor that was necessary for our example (the spreadsheet APIs already existed). Moreover, Fig. 10. A Simple Pipe End Ontology the API costs are only incurred once per application and are therefore domainindependent.

The costs for b) are non-trivial, and are incurred by every useflow as long as it involves new semantic objects. But note that the costs for establishing an ontology are network costs, since we can assume the ontology to be shared, either globally or in the specific institution (in our example Charles' company). In our experience, large parts of the ontology concern very general topics (engineering parts, measurements, and legal transactions in our example), and can therefore be shared widely. In the future, they might even be outsourced to existing ontologies like DBpedia [5]. But of course, the useflow-specific parts of the ontology have to be hand-coded.

Finally, the costs for c) are document-specific, i.e., they are incurred by every document and service to be added to the useflow. Even though these costs can be somewhat mitigated by the interpretation mapping editors under development in the SE-MANTIC ALLIANCE context, they have to be taken into account in the evaluation of the framework.

Another kind of costs involved in our approach is the provision and maintenance of services. The current implementation of the MA SEMANTIC ALLIANCE framework restricts the content of the RDF triple store to the contents of already open documents, which enables smooth application switching for a semantic object that is already in the user's focus. A persistent RDF triple store not only could offer more services, it also might allow an integration of MA SEMANTIC ALLIANCE and already existing Semantic Desktop services offered, for example, by the Gnowsis Semantic Desktop framework [START_REF] Sauermann | Semantic Desktop 2.0: The Gnowsis Experience[END_REF]. The synergies could drive down service development and maintenance costs quite heavily.

Conclusion

In this paper, we have addressed the problem of integrating complex useflows spanning multiple applications at the interface level. We have concentrated our investigation on applications that can be described as document players, where the objects addressed are represented in "documents" and the applications give the user access to these objects -"playing the documents", where each application is responsible for different aspects of the objects under consideration. We analyze the underlying problem as one of (lacking) full semantic transparency: Particular useflows are sensemaking processes in which an information becomes sensible via frame (here: application), cue (here: information object) and connection (here: interpretation mapping), and switching between distinct applications force frame shifts with respect to the semantic object in the user's focus. In traditional useflows, the connection between the cues remains in the mind of the beholder and cannot be called upon to integrate the respective user interfaces.

We propose to solve that with (more) semantic technologies and materialize the connection -but not as a direct connection between information objects. Rather we make use of an existing technology that establishes a connection between the application's information objects and a semantic object represented in a structured back-ground ontology and uses this mapping to mash semantic services into existing user applications, effectively turning them into user frames. We show that this architecture can be used for the task-switching and information integration requirements of user interface integration on a real-world engineering use-flow involving a CAD system for construction tasks and spreadsheets for calculations.

The point of this paper is not that user interfaces can be integrated at all, but that the integration can be achieved modularly, where new applications (and semantic services) can be integrated flexibly as the useflows warrant. Indeed most business applications opt for integration in a single system (see Option U1 in the introduction) with the well-known lock-in problem for useflows. Our CAD system (Autodesk Inventor) already supports the integrations of spreadsheets, but this integration is at the data integration level only (Option U2).

In our semantic integration we can modify useflows easily by adding frames and services or replacing existing frames and services by compatible ones. For instance the spreadsheet-based pricing service in our example workflow could be replaced by a more scalable OLAP-based one if volumes and resources warrant.

Future Work

The most significant shortcoming of the work presented here is that, even though our use cases and experiences with the system make it plausible that smooth application switching will enhance productivity in knowledge-intensive useflows, we have not subjected the system to evaluation with real users. We plan to do this once the system has stabilized further. Then we will also see whether our current assumption that semantic objects in the ontology are sufficient for pivoting between applications, or whether we need to extend pivoting to a two-step operation, which can utilize closely aligned semantic objects in the ontology. Our ontology representation already has candidates for such alignments, but we will have to see which ones can be used for pivoting -clearly not all ontology relations would be adequate.

We can improve the reach of our method and the implementation: For instance, one of the key ingredients of the solution proposed in this paper is the materialization of the connection between information objects across frames via interpretation mappings. This is also one of the largest cost factors in deploying it -assuming that the background ontologies will be shared and thus stabilize for a given domain. To reduce the costs we want to experiment with the powerful background ontology framework the MA SEMANTIC ALLIANCE employs. This provides a notion of "views" (mappings between semantic object representations) that may be used to materialize the semantically interesting parts of interpretation mappings into the ontology by representing information objects in the ontology and establishing ontology views to the semantic objects. This would have the consequence, that the interpretation mapping, i.e., the mapping between an information object in an application to its ontology representation, would be extended by an intra-ontology view which can be shared by the resp. Community of Practice or the institution. In our experience, the necessary views are often already implicitly present in the ontology. Take for instance the bolts in our running example: as they are physical objects, they have a time-space extent (their geometry, which is the aspect modeled in the CAD system) and they are possible goods in financial transactions (the aspect modeled in the pricing spreadsheet). Rely-ing on such views made explicit in the ontology would simplify the interpretation mappings sufficiently that they may be amenable to heuristic methods for automating interpretation mapping creation.

So far, the user's focus is assumed to be on all relevant frames for a semantic object, as the documents have to be open to become available to Sally. With a persistent RDF triple store, not only this limitation would disappear, but new, e.g. collaborative services can be offered. Collected abstract document ontologies can be used for discovery of relevant data sources (e.g. other pricing resources) and smooth application switching to them. Thus, if the synergies with the Gnowsis framework are made use of, then the MA SEMANTIC ALLIANCE framework could even be employed for realizing full semantic transparency for information objects on the Semantic Desktop.

Fig. 2 .Fig. 1 .

 21 Fig. 2. A Spreadsheet for the Calculation of Pipe End Prices

Fig. 6 .

 6 Fig. 6. The Sally Frames Menu for a Pipe End

Fig. 7 .

 7 Fig. 7. Sally Services for Selected Bolt under Pricing Frame

This assembly is chosen for purely expository reasons, nothing in this paper hinges on this choice.

Note that this layout is still a preliminary proof-of-concept realization that will evolve, for instance, Sally Frames could be realized as a dropdown menu that doesn't occupy as much screen space.

Acknowledgements We want to thank the anonymous reviewers for their constructive suggestions. This work has been funded by the German Research Council under grants KO-2484-10-1 and KO-2484-12-1.