Keywords: H.5.1 [Information Interfaces and Presentation (e.g. HCI)]: Multimedia Information Systems-Artificial, augmented Computer Graphics]: Methodology and Techniques-Interaction techniques 3D Interaction, 2D Pointer, 3D Ray, Ray-Casting, Collaborative Virtual Environments

Abstract

In this paper we present a new metaphor for interaction within Collaborative Virtual Environments (CVE). This metaphor is dedicated to non-immersive or semi-immersive 3D interactions, for which users cannot afford to buy expensive devices neither for 3D visualization of their virtual environment nor for interaction. With these low-cost restrictions, we think that it is more effective to use basic 2D metaphors rather than to try to adapt 3D virtual metaphors which would be more difficult to use because of the poor immersion level offered by such systems.

The problem that will arise within a CVE is that it is difficult to make a user aware of the 2D metaphors used by another user, because they are not associated with a 3D virtual object of the shared universe. So our idea is to provide to a user a 3D virtual ray (using ray-casting for object selection) that would act like a 2D pointer on the screen, allowing the user to only control the 2D position of the closest ray end, and calculating the orientation of the ray so that its projection on the screen would always be a point. This way, since the user is controlling a 3D virtual ray, the other users can be made aware of his activity.

To test the efficiency of this 2D Pointer / 3D Ray, we have made some experiments making users compare different devices to realize some simple selection and manipulation tasks. The results show that this kind of 2D solution is efficient and allows 3D interaction within Virtual Environments by people who cannot afford expensive immersive hardware. This new metaphor allows more users to collaborate within CVE.

Introduction

What is the best technical solution for easy and natural 3D interaction within Virtual Environments (VE)? Most people will answer that it is immersion, but to obtain high quality immersion you need stereovision for the visualization, linked to a 3D tracking device in order to track the position of tools of the user and of her head. Indeed, such technical solutions allow the images to be generated such that virtual tools can be colocated with parts of the user's body or with the real tools she is using, so a user feels like her arms, hands, or tools were really embedded within the virtual environments. Furthermore, interaction metaphors that are usually used in this context, such as virtual hands [START_REF] Poupyrev | The Go-Go Interaction Technique: Non-Linear Mapping for Direct Manipulation in VR[END_REF], virtual rays [START_REF] Bowman | An evaluation of techniques for grabbing and manipulating remote objects in immersive virtual environments[END_REF] or virtual 3D cursors [START_REF] Zhai | The "Silk Cursor": Investigating Transparency for 3D Target Acquisition[END_REF], are interesting for Collaborative Virtual Environments (CVE) because they provide a natural 3D representation that is perceptible for the other users of the CVE. Due to this 3D visualization of the interaction tools, a user can be easily aware of the activity of the other users of the CVE. Nevertheless, 2D metaphors and input devices have also to be considered for 3D interactions because they are sometimes easier to use than 3D metaphors, as stated in [START_REF] Bowman | 3D User Interfaces: New Directions and Perspectives[END_REF]].

However, a good immersion cannot be obtained without expensive hardware such as high-frequency video-projectors (for active stereovision) or double projectors (for passive polarized stereovision). Providing only stereovision is not enough to obtain a good immersion, because it cannot ensure a good colocation between the virtual tools driven by the users and the physical objects or body parts that the user uses to control the virtual tools. We need wireless tracking systems (optical, ultrasonic or magnetic) for head tracking, tools tracking and body parts tracking.

Without colocation, we consider that it would be difficult for somebody to use efficiently the classical 3D interaction metaphors, and that these metaphors will not be user-friendly. So perhaps basic 2D interaction tools such as a 2D pointer driven with a classical 2D mouse could be as efficient as the usual 3D metaphors for simple tasks such as object selection and manipulation (3D positioning, for instance).

Two problems arise when using such basic 2D interaction metaphors. First, when several users share a CVE, it will be difficult to make a user aware of the interactions of other users, because their 2D interaction tools will not be associated with any 3D virtual objects. Second, using a classical mouse will not fit semiimmersive environments when a user stands in front of a big image produced by a videoprojector, generally without any keyboard or 2D mouse. This is the reason why we propose a new 2D pointer that will be associated with a 3D geometry in order to appear visually within the Virtual Environment. This 2D pointer will be easy to use and will be driven by any device that can control a 2D position: for example a classical 2D mouse, a gamepad or a Nintendo wiimote remote gaming controller. The 3D geometry of this pointer will be a virtual ray, so other users can be easily made aware of the movement of this 3D ray, in the same way they can be made aware of the evolution of classical 3D interaction metaphors. This 2D Pointer / 3D Ray will use the classical ray-casting technique for object selection and manipulation. In this way, its behavior is similar to the aperture based selection technique [START_REF] Forsberg | Aperture based selection for immersive virtual environments[END_REF]] and to the technique developped in [START_REF] Ware | Selection Using a One-eyed Cursor in a Fish Tank VR Environment[END_REF].

In order to show that our 2D Pointer / 3D Ray can be useful for selection and basic interaction tasks, we have made some experiments comparing four interaction techniques. We will describe the conditions of the experiments, then we will present the results of these experiments and we will discuss them in order to show that our new interaction metaphor is efficient enough to be used for interaction within CVE when some of the users do not have access to expensive immersive hardware devices.

Related Work

3D interaction metaphors

The "3D Interaction Techniques for 3D Manipulation" chapter of [START_REF] Bowman | 3D User Interfaces: Theory and Practice[END_REF]] presents many metaphors dedicated to 3D manipulation. The ray-casting [START_REF] Poupyrev | Egocentric Object Manipulation in Virtual Environments: Empirical Evaluation of Interaction Techniques[END_REF]] is very interesting because it is very simple and efficient, especially when used at close range. This metaphor is difficult to use at long range because it requires high angular accuracy, but some approaches allow to minimize jitter, such as adaptive control display gains (e.g. pointer acceleration) [START_REF] Frees | PRISM interaction for enhancing control in immersive virtual environments[END_REF]]. Driving such a metaphor through a 2D input device allows the reduction of jitter around the orientation of the virtual ray, especially when the 2D device is a mouse [START_REF] Balakrishnan | The Rockin'Mouse: Integral 3D Manipulation on a Plane[END_REF]]. The remaining problem is that rotating a 3D ray using a 2D input device is not user-friendly. This is why we propose to adapt this metaphor so that it can be easily driven with a device that only provides a 2D position.

Awareness within CVE

It is important for people sharing a CVE to be aware of the activity of other users, as explained in [START_REF] Fraser | Supporting Awareness and Interaction through Collaborative Virtual Interfaces[END_REF], in order to help them to understand the evolution of the CVE and to collaborate more efficiently with the other users. Showing the activity of a user to the other users with whom he may collaborate is a central point for an efficient collaboration, a lot of work has been realized in this area [Fraser, M. et al. 2000][Gutwin, C. and[START_REF] Gutwin | Design for Individuals, Design for Groups: Tradeoffs Between Power and Workspace Awareness[END_REF]]. Many egocentric metaphors, such as the virtual ray casting, are well suited for interaction within CVE, thanks to their graphical visualization that can be shown to the other users.

The asymmetric 2D Pointer / 3D Ray

Our idea is to use a 3D virtual ray that would be as easier to drive than the classical 2D mouse pointer. The result looks like a classical 2D pointer moving on the surface of the screen. In fact it is a quite thin and long 3D virtual ray, moving near the viewpoint of the user, staying always at the same depth, which orientation is calculated in a way that its projection on the screen is always a small spot. As shown on figure 2, the 2D device used to control the pointer will provide the Xc and the Y c values, and the Zc value is a chosen one, so the rho and theta values can be calculated this way, if the rho angle (the heading) is first applied around the Y axis and then the theta angle (the elevation) is applied around the X ′ axis :

Xc

• rho = atan(-Xc/Zc) • theta = atan(Y c/sqrt(Xc * Xc + Zc * Zc))
This way, the user of the 2D Pointer / 3D Ray will always feel that she is using a 2D pointer (figure 3), while other users will see a 3D virtual ray moving thanks to the action of the first user (figure 4). So it is quite easy to use by the first user, and quite easy to understand by the other users. This 2D Pointer / 3D Ray is completely independent from the hardware device that will be used to drive it: either a classical 2D mouse, or a game pad, or any device able to provide 2D coordinates, or even a graphical 2D user interface.

As the 2D Pointer / 3D Ray is turning around the closest extremity of the virtual ray, the movements of a manipulated object can also be affected by a small rotation and will not stay at the same Z coordinate within the user's coordinate system, except if we force it to preserve its relative orientation and Z coordinate. This metaphor can be easily extended to 3D movements within the user's coordinate system: the X and the Y coordinates are directly provided by the physical device used to drive the 2D pointer, and the Z coordinate can be changed by moving the manipulated object along the 3D ray. To achieve such a translation along the virtual ray, the device used to drive the 2D pointer must also provide the information needed to calculate the Z coordinate, or it can be associated to another device providing this value. For example, this Z coordinate can be obtained thanks to the wheel of a 2D mouse, or some buttons of a gamepad.

A rotation of the manipulated object within the user's coordinate system can also be calculated with additional devices, for example the keyboard or some buttons or joysticks of a gamepad. We consider our technique as an egocentric interaction metaphor using a pointer as described in [START_REF] Poupyrev | Egocentric Object Manipulation in Virtual Environments: Empirical Evaluation of Interaction Techniques[END_REF]]. As our 2D Pointer / 3D Ray is a tool associated to the user's viewpoint, the user carries this interaction tool with her when she navigates within the VE, in the same manner as 3DM [START_REF] Butterworth | 3DM: a Three Dimensional Modeler using a Head-Mounted Display[END_REF]]. So as the 2D Pointer / 3D Ray moves with the viewpoint when the user navigates, the object that has been grabbed by the moving tool navigates also within the VE, which is another complementary way to provide a new position and orientation to this manipulated object.

Last, the 2D Pointer / 3D Ray can simply be used as a classical 2D pointer to trigger somme elements of a 3D GUI that could be carried by the user, in order to control the state of the application.

So, according to Hand [START_REF] Hand | A Survey of 3D Interaction Techniques[END_REF]] who separates virtual interactions into 3 categories: 3D interaction (selection and manipulation), navigation and application control; we see that our 2D Pointer / 3D Ray, carried by the user, is well suited for these three kinds of interactions.

Hypotheses

We think that our 2D Pointer / 3D Ray can be quite efficient for 3D manipulation, at least for simple tasks such as positioning objects within a VE, especially when we cannot offer a good immersion to the user. So we will make some hypotheses about the usability of our interaction metaphor, then we will make some experiments to verify these hypotheses.

H1: the best solution for 3D interaction is immersion with head-tracking

This solution will be used as a reference for our evaluation. We will compare the time spent during the manipulation and the accuracy of the other evaluated techniques relative to this one. We will also ask the users which technical solution they prefer: we think they will prefer immersion with head-tracking and colocation of the virtual ray with the interaction device they use.

H2: our 2D Pointer / 3D Ray can be as efficient than immersion with head-tracking

We hope that this hypothesis will be true, at least for basic tasks such as positioning 3D objects, to be able to propose it to a user when we cannot offer him any immersion. This would also offer to new VR users one efficient interaction tool which is as easy to use as the classical mouse and its associated 2D pointer.

H3: immersion without head-tracking is not a good solution for 3D interaction

We think that incomplete immersion is not a good solution, because users can feel deceived by interactive solutions that would work much better if virtual tools could be colocated with real tools or with body parts of the user. This solution should not be neither as fast nor as accurate as a solution offering total immersion. This solution could even be the worst because it denatures the 3D metaphors it uses. Indeed, in this case the user is nearly placed in an exocentric situation, as she cannot really use the egocentric metaphors in the way they should be used, as it is not possible here to colocate a virtual ray and its associated interaction device.

H4: in a semi-immersive context, our 2D Pointer / 3D Ray can be as efficient as 3D interaction without head-tracking

This semi-immersive situation appears quite often, and can easily be obtained thanks to a simple videoprojector, which is quite affordable today. In this context, the user is generally standing in front of a big projection screen, so she cannot use neither the mouse nor the keyboard for 3D interaction within the VE. So our idea is to provide to this user our 2D Pointer / 3D Ray, driven with another kind of device: a wireless one in order to allow the user to move easily.

Here we propose to use a Nintendo wiimote to drive our interaction metaphor. We hope that this low-cost solution (which costs around 50 US dollars) can be nearly as efficient as an optical 3D tracking technology (which costs roughly fifty thousand dollars).

In order to verify our hypotheses, we have set up a simple experiment that consists in positioning four 3D objects (colored spheres) within their four associated supports (semi-transparent colored cylinders). We ask the users to realize the task as fast as possible, with the best accuracy possible.

The tasks to complete

We make the user aware of the selection of a 3D object by overlaying a semi-transparent upscaled geometry, and we also give them information about the accuracy of the manipulation task: a flag, associated to each support, changes its color from red towards green when the position of its associated object is accurate enough to consider that the positioning task is completed.

All the objects are at the same depth relative to the user

For the first manipulations, in order to make it easy for the user, all the 3D objects (the spheres and their associated cylindric supports) are at the same depth relative to the user. The positioning task is only a 2D task, and we do not allow the user to change the position of his viewpoint by navigating.

The user must complete this task three times: first with big objects, second with medium objects, and third with small objects. The experiment set-up is shown in figures 5, 6 and 7.

All the objects are at different depths relatively to the user

Second, we place the 3D objects at different depths relative to the user: there is not any 3D object to manipulate that is located at the same depth as its associated support. So the user must grab the 3D objects and move them (also along the front/back axis) to place each object into its support. Here again, the user must complete this task three times: first with big objects, second with medium objects, and third with small objects. The experiment set-up for medium objects is shown on figure 8.

If the navigation is not accurate enough, as we use the same navigation step for the three sizes configurations, users will have to adjust the depth in a way proposed by the current interaction technique they are testing.

The four interaction techniques to compare

Each user has to complete these six positioning tasks with four hardware configurations, from non-immersive context with a simple mouse in front of a workstation, to fully immersive context with stereovision, 3D tool tracking and head-tracking.

For these four configurations, we will use the same device for the front/back navigation: the joystick of the Nintendo nunchuk extension of the Nintendo wiimote, and only the front/back information will be used to constrain the user to stay on this front / back axis.

So the user will not be allowed to navigate in order to come near every object: he will have to be quite far from the objects placed on the right and on the left, otherwise there would not have been significant differences due to the objects sizes between these three experimental setups.

Technique 1: the 2D mouse used as a 2D pointer in front of a simple screen

First, the classical 2D mouse is used to drive the 2D Pointer / 3D Ray and the user sits in front of a 20" screen. The 2D Pointer / 3D Ray is perceived here as a simple 2D pointer associated with the usual mouse pointer. We use the left button press of the mouse to grab an object located "under" the pointer, and the release of this button releases the grabbed object. An object which is grabbed by the pointer can have its Z coordinate (its depth relative to the user viewpoint) adjusted by using the wheel of the mouse.

Technique 2: the Nintendo wiimote used as a 2D pointer in front of a big videoprojection

Second, a Nintendo wiimote is used to drive the 2D Pointer / 3D Ray and the user stands in front of a 2.5 meters high -3 meters wide projection. The 2D coordinates are acquired thanks to the infrared camera of the Nintendo wiimote, pointing at an infrared emitter placed between the user and the projection. Here again, the 2D Pointer / 3D Ray is perceived as a simple 2D pointer. We use the press of the "A" or "B" button of the Nintendo wiimote to grab an object located "under" the pointer, and the release of one of these buttons releases the grabbed object. An object which is grabbed by the pointer can have its Z coordinate (its depth relative to the user's viewpoint) adjusted with successive presses of the "Up" and "Down" buttons of the Nintendo wiimote.

Technique 3: optical tracking for the 3D Ray in front of a wide videoprojection with stereovision

Third, we use an optical 3D tracking system to acquire the 3D position and orientation of a Nintendo wiimote that is used as a tool in the hand of the user. This 3D position, relative to the viewpoint, is used to control a 3D virtual ray. The user stands in front of a 2.5 meters high and 9 meters wide semi-cylindric projection screen, offering stereovision. As for the second technique, we still use the press of the "A" or "B" button of the Nintendo wiimote to grab an object traversed by the 3D virtual ray, and the release of one of these releases the grabbed object. An object which is grabbed by the 3D virtual ray can have its 3 position coordinates (including its depth relative to the user's viewpoint) affected by the position and the orientation of the 3D virtual ray, which can be much more important than in the case of the manipulation of the 2D Pointer / 3D Ray. Here the user will adjust the Z relative position of a grabbed object by moving the Nintendo wiimote front or back. With this third technique, the infrared camera of the Nintendo wiimote is not used any more, but we still use the Nintendo wiimote for its "A" and "B" buttons.

Technique 4: optical tracking for the 3D Ray and the head of the user in front of a wide videoprojection with stereovision

Fourth and last, we extend Technique 3 to use the optical 3D tracking system not only to acquire the 3D position and orientation of a Nintendo wiimote that is used as a tool in the hand of the user, but also to acquire the 3D position and orientation of the head of the user, to be able to compute the best image for him. These 3D positions enable the position and orientation of a 3D virtual ray to be computed and to produce the most appropriate image in order to make the user believe that the virtual ray is placed at the exact end of her Nintendo wiimote.

Completing the tasks

Each user had to complete the 6 tasks in the same order :

1. big objects at the same depth, 2. medium objects at the same depth, 3. small objects at the same depth, 4. big objects at different depths, 5. medium objects at different depths, 6. small objects at different depths.

These 6 tasks had to be completed for each of the 4 techniques, always in the same order : Technique 1, Technique 2, Technique 3, Technique 4.

We did so because we were thinking that the users could then encounter the 4 techniques beginning with the simplest one and finishing with the most complicated one, so the first two techniques could serve as a practice for the last two techniques.

Most of the results show that we were right to make this assumption because the best manipulation times are those obtained with the first technique.

Before doing the real tasks, the users had a few minutes to practice each of the 4 techniques with a simple task: only 2 big objects with their supports, a object with the same depth than the supports, and the second object with a different depth, in order to make the user try the navigation with the nunchuk.

Test user demograph

34 people completed our experiments. 94% were men. Their average age was 26.5 years old. Almost everybody had experience with computers and 2D interaction with the mouse. Around 50% of the users had already used 3D interaction. 20% of the users had already played 3D video games. Most of the users were computer science students, software engineers or computer science researchers or teachers.

Results

Raw results

For each task (placement of 4 objects), we measured the time needed to complete the task, the accuracy of the positioning of the objects, and the average of the selections needed to grab an object and to release it at the correct position.

Time spent

Figure 12 shows the average time spent (in seconds) to complete each task with each technique. The second ANOVA (2D tasks) indicated that the difference in participants' time was significant for Technique 1 (F (1, 404) = 65.93, p < 0.0001)), Technique 2 (F (1, 404) = 13.61, p = 0.0003)), and for Technique 3 (F (1, 404) = 4.58, p = 0.0328)). The third (3D tasks) indicated that the difference in participants' time was significant only for Technique 2 (F (1, 404) = 29.70, p < 0.0001)), and not for neither Technique 1 (F (1, 404) = 1.16, p = 0.28)) nor Technique 3 (F (1, 404) = 1.32, p = 0.25)). Nevertheless, a Student test performed between Technique 3 and Technique 4 indicated that their difference was significant enough (t = 2.2782, p = 0.02325).

The preliminary conclusions are that:

1. For 2D positioning tasks:

• Technique 1 is the fastest technique (H2 partially verified for Technique 1).

• Technique 3 is the worst one (H3 partially verified).

• 2 is better than Techniques 3 and 4 (H4 partially verified, H2 partially verified for Technique 2).

For 3D positioning tasks:

• Technique 1 is better than Technique 4 (H2 partially verified for Technique 1 and H1 partially verified), but not very significantly.

• Technique 4 is the second most efficient one, but the difference with the best Technique is not very signicant (H1 partially verified).

• Technique 3 is slower than technique 4 (H3 partially verified), but not very significantly.

Accuracy

Figure 13 shows the average relative error for each task with each technique. single factor ANOVA on the participants' precision was also performed for the six tasks with Technique 4 as a reference. It indicated that the difference in participants' precision was significant for Technique 1 (F (1, 814) = 109.83, p < 0.0001)), Technique 2 (F (1, 814) = 4.84, p = 0.0283)), and Technique 3 (F (1, 814) = 4.20, p = 0.0407)). A Student test performed between Technique 3 and Technique 4 indicated also that their difference was significant (t = 2.22, p = 0.02697).

The preliminary conclusions about accuracy are nearly the same than about speed :

1. Technique 1 is always the most accurate technique (H0 verified, H2 verified for Technique 1).

2. For 3D positioning tasks, Technique 4 is second (H1 partially verified).

3. Technique 3 is always less accurate than technique 4 (H1 and H3 partially verified).

4. For 2D positioning tasks, Technique 3 is the worst (H3 partially verified).

5. For 2D positioning tasks, Technique 2 is better than Techniques 3 and 4 (H4 partially verified, H2 partially verified also for Technique 2).

6. For 3D positioning tasks, Technique 2 is quite as good as Technique 3 (H4 partially verified).

Average number of selections per object

Figure 14 shows the average number of selection per object needed to complete each task with each technique. The preliminary conclusions are only that:

1. Technique 1 is always the most direct technique (H2 verified for Technique 1).

2. Techniques 2, 3 and 4 are equivalent except for small objects with different depths where Technique 2 is not good.

Subjective results

Last, we asked each user to give a relative ordering, then an absolute scoring, of this 4 techniques according to :

1. How pleasant was it ?

2. How easy was it to use ? 3. How fast was it to use ? 4. How accurate was it to use ? 5. How efficient was it to use ? 6. How tiring was it to use ?

For the relative ordering, users had to give 1 to the best technique and 4 to the worst.

For the absolute scoring, users had to give 1 if the technique was very good, 2 if it was good, 3 if it was acceptable, and 4 if it was not good.

Relative ordering of the 4 techniques

Figure 15 shows how the users ordered the 4 techniques we wanted to compare. The preliminary conclusions are that:

1. Technique 1 is the easiest, the fastest, the most accurate, the most efficient and the least tiring of the fourth techniques (H2 verified for Technique 1).

2. Technique 4 is the preferred one and the second one concerning performances (H1 partially verified).

3. Technique 3 is always worse than technique 4 (H1 and H3 partially verified).

Techniques 2 and 3 have very similar results.

We notice that people have quite a bad opinion of Technique 2, even for speed and accuracy, although the time and accuracy effectively measured for this technique were not so bad, except for the last task to complete, with small objects at different depths. No doubt that this task influenced the votes. We will discuss this point later.

Absolute scoring of the 4 techniques

Figure 16 shows how the users scored the 4 techniques we wanted to evaluate.

Discussion

Now let us examine our hypotheses to see if they were verified.

H1: the best solution for 3D interaction is immersion with head-tracking

Yes. Technique 4 was the second best for speed and accuracy, according to the measures, and the preferred technique according to the votes of the users. Most of the users really enjoyed this technique, and succeeded in using it quite efficiently even if it was the first time they had used stereovision and head-tracking. Nevertheless, some users signaled that this technique can be quickly tiring.

H2: our 2D Pointer / 3D Ray can be as efficient than immersion with head-tracking

Yes. It is even more efficient, especially when it is driven with a simple 2D mouse in front of a workstation. The Technique 1 is the fastest and the more accurate one, according to the measures, and also according to the votes of the users. It will allow a user, without immersion capabilities, to interact in a CVE with our 2D Pointer / 3D Ray that will be perceived as a 3D virtual ray by the other users.

H3: immersion without head-tracking is not a good solution for 3D interaction

Nearly yes. Technique 3 is always worse than Techniques 1 and 4, and many users told us that Technique 3 was lacking "something", which was "head-tracking" once they had experienced Technique 4. Technique 2 is better than Technique 3 for 2D positioning, and nearly as good for 3D positioning except for small objects with different depths.

H4: in a semi-immersive context, our 2D Pointer / 3D Ray can be as efficient as 3D interaction without head-tracking

Nearly yes. Considering the raw results, Technique 2 is the second best for 2D manipulations, but the slowest technique as soon as the user must make depth adjustments. Considering the subjective results, Technique 2 is the technique users do not like, but it does not make such a big difference with technique 3 for the absolute scoring. We think that these results can be explained because we did not choose the best way to make the depth adjustments for 2D positioning: it was quite difficult for the users to press the "Up" and "Down" buttons whilst simultaneously keeping the "A" or "B" button. The Nintendo wiimote suffered additional difficulties because of the optical targets that were attached for optical tracking, making it difficult to reach the wiimote buttons (the same Nintendo wiimote with the optical targets was used for the experiments of Techniques 2, 3 and 4), as shown figure 18. Last, Z movements were discrete in this setup whereas the Z movements were continuous in the three other setups.

Conclusion

We have proposed a new metaphor for 3D interaction within Collaborative Virtual Environments: the 2D Pointer / 3D Ray, which associates a 3D representation with a 2D pointing device (for example a 2D mouse). This metaphor allows an asymmetric collaboration between users immersed within a CVE (via stereovision and head-tracking) and users simply sitting in front of the screen of their workstation. The user without immersion will interact as easily as if he had a simple 2D pointer, as the associated 3D ray (a 3D virtual ray) will be continuously moved and oriented in a way that its projection on the screen of the user will always be a small spot. The other users of the CVE will be made aware of the action of this user thanks to the movements of his associated 3D virtual.

We have made some experiments to confirm the usability of our interaction metaphor, comparing it to usual 3D interaction with a virtual ray with stereovision and head-tracking for colocation. As we were assuming that our metaphor is as well adapted to collaboration as the classical 3D ray-casting technique, we have only made experiments about its stand-alone usability. The results show that this metaphor is very easy to use with a 2D mouse for 3D positioning tasks. It is more difficult to drive the metaphor with a Nintendo wiimote especially for depth adjustments of small objects. However, we feel that this kind of device can be very helpful to provide interaction tools in semi-immersive environments as described in our experiments. This device has been used to provide interaction facilities in the context of 3D exploration of scientific data [START_REF] Duval | Collaborative Exploration of 3D Scientific Data[END_REF] as illustrated figure 17. A first solution to improve the depth adjustment is to adapt the size of the tool to the size of the objects we have to manipulate, by changing the size of the user relatively to the size of the world as proposed in [START_REF] Mine | Moving Objects in Space: Exploiting Proprioception in Virtual-Environment Interaction[END_REF]].

We could even determine the ideal size automatically [START_REF] Kopper | Design and Evaluation of Navigation Techniques for Multiscale Virtual Environments[END_REF]] by placing the correct information within the object we want to manipulate.

Future work

There are several other ways to improve the depth adjustment with a device such as the Nintendo wiimote. The immediate solution is to use its infrared camera, which is able to provide the position of several targets. This allows to calculate the distance between the 2 targets of our sensor-bar acquired by the camera, and use the variation of this distance to propose a depth adjustment (as it is used in the snoocker game of the "Wiiplay" pack). This solution is already operational but has yet to be experimented. Another immediate solution is to change the gain of the joystick of the nunchunk used for navigation, it would work in the same way as changing the size of the interaction tool relatively to the size of the world. Other solutions can also be proposed by using the accelerometers of the wiimote.

We think that this kind of device will be easier to use if it can also control the orientation of a selected object, although full 3D rotations are not always necessary as explained in [START_REF] Bowman | 3D User Interfaces: New Directions and Perspectives[END_REF]. So, we should realize some experiments in order to test if our metaphor can be extended to 3D orientation of 3D objects. A first solution is to associate the 2D mouse and keyboard modifiers to switch from a "depth adjustement" mode towards several "axis rotation" modes. Another solution is to take a similar approach with gamepads or with the Nintendo wiimote, combining buttons and 2D pointing or depth adjustment to provide rotation around a chosen axis.

Figure 1 :

 1 Figure 1: Asymmetric use of the 2D Pointer / 3D Ray within a Collaborative Virtual Environment

Figure 2 :

 2 Figure 2: Projection of the 3D Ray as a small spot on the screen.

Figure 3 :

 3 Figure 3: User 1 moves a 3D slider with her red 2D pointer and she sees the green 3D virtual ray of user 2 ready to select another slider.

Figure 4 :

 4 Figure 4: User 2 is ready to select a slider with her green 2D pointer while she is looking at user 1 moving a slider with her red 3D virtual ray.

Figure 5 :

 5 Figure 5: Big objects at the same distance from the user.

Figure 6 :

 6 Figure 6: Medium objects at the same distance from the user.

Figure 7 :

 7 Figure 7: Small objects at the same distance from the user.

Figure 8 :

 8 Figure 8: Medium objects at different distances from the user.

Figure 9 :

 9 Figure 9: Technique 1: the 2D mouse used as a 2D Pointer.

Figure 10 :

 10 Figure 10: Technique 2: the Nintendo wiimote used as a 2D Pointer.

Figure 11 :

 11 Figure 11: Technique 3: optical tracking for the 3D Ray.

Figure 12 :

 12 Figure 12: Time needed to complete the tasks.

Figure 13 :

 13 Figure 13: Error percentages.

Figure 14 :

 14 Figure 14: Average selections per object to complete a task.

Figure 15 :

 15 Figure 15: Relative ordering of the techniques.

Figure 16 :

 16 Figure 16: Absolute scoring of the techniques.

Figure 17 :

 17 Figure 17: 3 users interacting within a CVE: left and center users use a 2D Pointer / 3D Ray while right user uses a 3D virtual ray

Figure 18 :

 18 Figure 18: Optically tracked Nintendo wiimote

Acknowledgements

This work has been initiated in a joint of the ANR N • 06 TLOG 031 Part@ge project and of the ANR N • 06 TLOG 029 SCOS project. It is now going on in the ANR-08-COSI-003-01 CollaViz project.


Isabelle Guaïtella

Serge Santi

Christian Cavé

Links between Eyebrow Movements and Voice Variations : an Experimental Investigation

Introduction

Taking a multimodal approach to communication, we experimentally test some of our hypotheses regarding the links between rapid eyebrow movements and vocalization during speech production in dialogue situations. In particular, we examine the relationship between eyebrow movements and variations in the fundamental frequency of the voice. We have already found event simultaneity and resemblance of form in the eyebrow-movement and fundamental-frequency curves [START_REF] Cavé | Relations entre geste et voix: Le cas des sourcils et de la fréquence fondamentale[END_REF]Cavé, Guaïtella, Bertrand, Santi, Harlay, & Espesser,1996). We use an experimental device that supplies objective gestural data via an automatic movement acquisition and analysis system (Elite). Once the eyebrow data is plotted on curves, it can be related to and compared with the fundamental frequency curves obtained using sound signal processing software.

1. Theoretical background [START_REF] Darwin | L'expression des émotions chez l'homme et les animaux, French translation[END_REF] described the gestures that contribute to the expression of emotions as resulting from the interaction between physiological constraints and functional needs. For example, an activity like raising the eyebrows to enlarge the visual field and see as far away as possible will be preserved --through the influence of the so-called association principle --as a means of expressing surprise and astonishment, and will thus become a mark of special attention given to what is happening. Scientists like [START_REF] Ekman | Unmasking the face. A guide to recognizing emotions from facial clues[END_REF] have studied facial cues to gain insight into the emotional aspects of communication, and [START_REF] Darwin | L'expression des émotions chez l'homme et les animaux, French translation[END_REF] in particular attempted to characterize rapid movements, of which eyebrow movements are a part. Such rapid movements are thought to be mainly linked to discourse production, where they serve as a rhythmic demarcation device (see also [START_REF] Birdwhistell | Kinesics and context[END_REF]. For many authors, the difference between rapid and slow movements is a categorical one, for rapid movements are said to be rhythmic, while slow ones are viewed as semiotic. We have criticized this point of view [START_REF] Guaïtella | Rythme et parole: comparaison critique du rythme de la lecture oralisée et de la parole spontanée[END_REF][START_REF] Guaïtella | Mélodie du geste, mimique vocale?[END_REF], and contend that both rapid and slow movements contribute not only to rhythm, but to the semiotic production as well. Rapid eyebrow movements thus have a semiotic component that is rooted --probably by association --in the emotional expressions of the face. However, the function of these rapid movements is essentially discoursebased, for their use in the expression of emotions has come to take on a linguistic function (an eyebrow movement occurring on an unpredictable word, for example, can be interpreted as "Pay special attention to the word I'm pronouncing" ...). For [START_REF] Bolinger | Intonation in American English[END_REF],the eyebrows even more than other gestural parameters are related in a privileged way to speech, and benefit from a status equivalent to vocality.

Experimental procedure

Corpus

Ten unpaid subjects volunteered to participate in the experiment. They were questioned about their work by means of interviews with a highly conversational style. Thus, the corpus consisted of ten dialogues between an interviewer and an interviewee, each lasting approximately five minutes. For technical reasons, we were only able to process seven of the ten dialogues.

Data acquisition

For each subject, three types of recordings were obtained: (1)the speech signal, (2) the video image, and (3) head and face movement data. Subjects were filmed on videotape for the full duration of the experimental session. The subject's speech was recorded using a DAT recorder and a Sennheiser microphone (type ME 40). The techniques used to record the head and face movements are described below.

Kinematic data acquisition and processing

Head and eyebrow movements were recorded using a motion analysis system with a 100-Hz automatic optical TV-image processor (Elite system) enabling computer reconstruction of three-dimensional trajectories of small infrared retroreflective markers [START_REF] Ferrigno | ELITE: A digital dedicated hardware system for movement analysis via real-time TV signal processing[END_REF].The kinematic properties of five key points were captured by means of hemispherical markers (5 mm in diameter) attached to the subject's skin. The marker for recording head movements was placed on the forehead at the upper extremity of the frontal suture. For scowl detection, a marker was placed at each of the inner ends of the left and right supra-orbital arches. Markers for analyzing eyebrow raising and lowering were placed at the outer ends of the left and right supra-orbital arches. For the last two markers, a lateral position was chosen to avoid confusion between the inner and outer markers during head rotation. An external reference marker was placed on a vertical rod behind the subject.

Two video cameras equipped with wide-angle lenses were positioned in front of the subject parallel to his/her frontal plane at a distance of 1.5 m. The sight angles of the two cameras in the horizontal plane were -35°and +35°, respectively .

The Elite system detected the markers in real-time by means of a hardware shape-recognition procedure. It then calculated their centroid x-y coordinates and generated a 3-D reconstruction of the marker trajectories. The powerful algorithms used for the centroid calculation and the stereometric procedure allowed us to attained a measurement accuracy of about 1/3000, i.e., 0.5 mm for the linear displacements in our experimental conditions. After digital filtering based on the preliminary automatic selection of the appropriate bandwidth for each signal (D'Amico and [START_REF] D'amico | Technique for the evaluation of derivatives from noisy biomechanical displacement data using a model-based bandwidthselection procedure[END_REF], the first and second derivatives of the markers' linear displacements were computed.

Data analysis

The kinematic data output by the Elite system (x, y, and z coordinates of the eyebrow movements) and the speech signal were transferred to a Sparc SUN station in a Unix environment. An automatic procedure was used to display the spectrogram of the speech signal, the corresponding Fo curve, and the eyebrow movement curve. In order to separate the real movements of the eyebrows from of those resulting from the movements of the head (which also sent data to the eyebrow sensors), the head movement component was factored out by an automatic procedure. The entire output data set was verified a posteriori by viewing of the video documents.

Hypotheses

Our first hypothesis regarding the role of the eyebrows was related to the part they might play in managing the conversational interaction, i.e., in turn-taking. It was assumed that speakers would use this device to signal their desire to say something and thus, to be given a speaking turn. It appeared valid to test this hypothesis by measuring the interval between eyebrow movements and the beginning and ending of the speaking turns. Eyebrow movements that fell closer to the onset of a speaking turn than to its offset would lend support this hypothesis.

Our second hypothesis concerned the role of the eyebrows in rhythmic demarcation. During speech production, a speaker may accentuate a syllable(of an important word, for example) by means of an intonation variation (by using a fundamental frequency contour with an "accentuating" value, i.e., one likely to lead to the perception of a stressed unit) in conjunction with an eyebrow movement. This hypothesis was tested in a previous analysis of video documents [START_REF] Cavé | Relations entre geste et voix: Le cas des sourcils et de la fréquence fondamentale[END_REF][START_REF] Cavé | About the relationship between eyebrow movements and Fo variations[END_REF] which revealed a systematic link between rapid upward-downward eyebrow movements and the fundamental frequency. The Fo curve had the same rising-falling contour (and thus, an accentuating value) as the eyebrow movement curve, and the two curves were synchronized. The question raised by this study was whether this link is truly systematic, or even automatic, and whether it is rooted in the effects of muscular synergy and/or is the outcome of a communicational strategy on the part of the speaker.

Data reduction

Selection of eyebrow movements

We limited the present study to rapid upward-downward eyebrow movements (eliminating slow movements, considered to be "postural"). Only those movements where both eyebrows moved simultaneously were retained for analysis. The cutoff point for inclusion of a movement was set at a displacement of at least 3 mm by at least one eyebrow.

Relationship to vocalization

To relate the occurrence of eyebrow movements to the vocal production, various intervals were measured in the following manner: a) If the eyebrows moved when the speaker was talking, two intervals were measured: the time between the onset of the vocal signal and the eyebrow movement, and the time between the eyebrow movement and the end of the vocal signal. beg spkr voc eyebrow mvmt end spkr voc interval (ms) interval (ms)

These cases --where the eyebrow movement occurred during vocalization -were classified in category 1 (C1).

b) If the movement took place while the speaker was silent (even if the interviewer was talking), the intervals measured were between the eyebrow movement and the end of the speaker's last speaking turn, and between the eyebrow movement and the onset of the speaker's next speaking turn.

end spkr voc eyebrow mvmt beg spkr voc interval (ms) interval (ms)

Whenever the eyebrow movement and the end of the vocal production were synchronized (difference of 30 ms or less), the interval between the movement and the beginning of the next vocalization was measured. Similarly, whenever the movement was synchronized with the beginning of the vocalization (difference of 30 ms or less), the time since the end of the preceding vocalization was measured.

end spkr voc + eyebrow mvmt beg spkr voc interval (ms) end spkr voc beg spkr voc + eyebrow mvmt interval (ms)

These cases --where the eyebrow movement took place during speaker silence or at a vocalization boundary --were grouped into category2 (C2).

Coding fundamental frequency

For the speaker's vocal production, the fundamental frequency contours were classified into seven categories: rising-falling (RF),falling-rising (FR), plateau (P), rising (R), falling (F),rising-falling-rising (RFR), and falling-rising-falling (FRF). In the passages where the speaker's and the interviewer's speech overlapped, the fundamental frequency could not be analyzed, so they were discarded. An intonation contour was considered to have an accentuating value when it contained points where there was a change in direction ("target points" in D.J. Hirst's terminology) [START_REF] Guaïtella | Rythme et parole: comparaison critique du rythme de la lecture oralisée et de la parole spontanée[END_REF]. Simple contours R and F were checked at the beginning and end to determine the presence of target points within the observed sequence. Only flat and falling contours had no target points, and thus no accentuating value.

Results

Eyebrows and vocalization

For category C1 (movement during vocalization), the eyebrow-movement/endof-vocalization interval averaged 2100 ms and the beginning-ofvocalization/eyebrow-movement interval averaged 1500 ms. For category C2, the mean end-of-vocalization/movement interval was 1200 ms and the mean movement/beginning-of-vocalization was 490 ms. These means take into account the data for a speaker whose interactive strategy was the opposite of the others. (C1: beginning = 543 ms, end = 780 ms; C2:beginning = 1300 ms, end = 1300 ms). This speaker's data contributed to weakening the overall tendency, nevertheless exhibited by the majority of the subjects.

For the two categories pooled, the interval data as a whole showed that eyebrow movements occurred closer to the beginning of a vocal production than to the end. This effect was statistically significant (p < .05).

Eyebrows, accentuation, and fundamental frequency

Only C1 data was analyzed here. This category contained 43.5%of the observed eyebrow movements (excluding the 16 cases of overlapping out of 96). RF intonation contours --which are similar in shape to eyebrow movement curves -were found for 67.5% of the movements. Contours RF, FR,RFR, FRF, and R, all of which are accentuating, accompanied 93.75% of the movements. This left a mere 6.25% for contours with a non-accentuating value (F: 5%, P: 1.25%).

Discussion

The observed proximity of vocalization onset and eyebrow movements supports the idea that eyebrow movements act as marks of a new speaking turn.

Concerning the relationship between eyebrow movements and fundamental frequency, the data obtained here refine the results of previous studies [START_REF] Cavé | Relations entre geste et voix: Le cas des sourcils et de la fréquence fondamentale[END_REF][START_REF] Cavé | About the relationship between eyebrow movements and Fo variations[END_REF]. Although 93.75% of the eyebrow movements were associated with accentuating intonation contours(mostly RFs), the presence of eyebrow movements during non-accentuating productions (in particular, during periods of hesitation) seems to argue for the idea that the link between eyebrow movements and fundamental frequency variations is not an automatic one. In other words, eyebrow movements are not solely dictated by Fo variations, but may be speaker-controlled, being the consequence of linguistic and communicational choices. However, hand-voice coordination during the production of accentuating markers is known to be highly prevalent. The muscular synergy hypothesis cannot be rejected, then, even if it does not apply in all cases. In the contour distribution obtained here, we can see a distinction between what might be called "rhythmic" movements, which are more or less expressive and associated with accentuating variations in the fundamental frequency, and movements we could call "dynamogenic", which contribute to discursive planning and are associated with flatter contours (Guaïtella,1995). Thus, rapid raising/lowering of the eyebrows does indeed appear to act as an attention-getting device, whether during the speaker's silent periods ("Look, I would like to speak"), or during speech production("Listen, what I'm saying is important"). Accordingly, we can contend that this gesture in fact derives from the eyebrow-raising movement whose primary function was to manifest enhanced visual attention.

In the future, we plan to look at what goes on for the interviewer, from the vocal and visual standpoints, when the speaker's eyebrows move, in hopes of finding out whether these attention signals may also function in a more interactive way, and, at least in certain cases, reflect either focusing on the interviewer's speech or between-speaker rhythmic harmonization, as proposed in [START_REF] Condon | An analysis of behavioral organisation[END_REF] intersynchrony hypothesis.


Philippe Blache
email: blache@lpl-aix.fr

Roxane Bertrand
email: roxane.bertrand@lpl-aix.fr

Gaëlle Ferré
email: gaelle.ferre@univ-nantes.fr

Creating and exploiting multimodal annotated corpora: the ToMA project

come

Introduction

In recent years, linguists have become aware that a theory of communication describing real interactions should involve the different domains fo verbal and nonverbal description. This is the reason why linguistics and natural language processing have turned to multimodal data where human communication is represented in its entire complexity. By multimodal data, we mean a complex annotation in the auditory-visual domains, not only visual information, Each domain is itself composed of a set of parameters and must be related to the other dimensions of speech. However, annotating such inputs remains problematic both for theoretical and technical reasons. First, we still need a linguistic theory taking into account all the different aspects of multimodality, explaining in particular how the different linguistic domains interact. At the same time, we need to specify a standardized way of representing multimodal information in order to give access to large multimodal corpora, as richly annotated as possible. What is meant by large corpora is however quite a relative notion since in some linguistic fields such as syntax for instance, corpora of several million words are used whereas in prosody where most of the annotations are made manually, a few hours of speech are considered as a large corpus.

This paper describes the first results of the ToMA project1 which aims at answering these different issues. In this project we propose to specify the different requisites and needs in the perspective of multimodal annotation. Different from many other projects, ToMA does not focus on a specific problem such as information structure, gesture studies or prosody-syntax interaction. Our goal is the development of generic and reusable annotated resources, providing precise annotations in all possible domains, from prosody to gesture. We propose transcription conventions and information encoding as well as tools helping in the annotation process and access to information.

In the first section, we specify a coding scheme adapted for multimodal transcription and annotations. In the second part, we describe the automation of the production of multimodal resources by means of a platform integrating different annotation tools. This platform consists of a sequence of tools leading from raw data to enriched annotations in each linguistic domain. We illustrate the application of this environment by the description of a large multimodal annotated corpus for French. Finally, we present some first results obtained thanks to this resource.

Multimodal resources and coding schemes

Several projects address the question of multimodal resources and their annotation. For example, the LUNA project (cf. [Rodriguez07]) focuses on spoken language understanding. The corpus is made of human-machine and human-human dialogues. It proposes, on top of the transcription, different levels of annotation, from morphosyntax to semantics and discourse analysis. Annotations have been done by means of different tools producing different formats that become interoperable thanks to the use of an interchange format called PAULA (cf. [Dipper05]). SAMMIE ) is another project aiming at building multimodal resources in the context of human-machine interaction. Annotations are done using the Nite XML Toolkit (cf. [Carletta03]); they concern syntactic and discourse-level information, plus indication about the specific computer modality used in the experiment. A comparable resource, also acquired following a Wizard-of-Oz technique, has been built by the DIME project (cf. [Pineda02]) for Spanish. In comparison with previous ones, this resource mainly focuses on first-level prosodic information as well as dialog acts.

These three examples are quite typical of multimodal resources development. The main differences with ToMA are first the nature of the source (in our case humanhuman natural interaction) and second the richness of the annotation (much more exhaustive and precise for ToMA).

Annotating corpora first requires to specify what kind of information it is necessary to represent and how it is organized. This problem consists in defining a coding scheme. Several of them have been developed in different projects such as MATE, NIMM, EMMA, XCES, TUSNELDA, etc. What comes out is that they are very precise in one or two modalities. However, they usually do not cover the entire multimodal domain nor the very fine-grained level of annotation required in every modality. We propose to combine several existing schemes and to extend them so as to obtain a coding scheme that would be as complete as possible.

• Corpus metadata: we use a TUSNELDA-like coding scheme ([Tusnelda05]) in which all the information such as speaker name, sex, region, etc. is noted. Using the framework of conversation analysis, we also annotate conversational units (turn-constructional units, [Selting00]). We follow the MUMIN coding scheme again to annotate backchannels phenomena.

On top of these schemes, we also take into account different proposals in our encodinglike the one elaborated in Potsdam (cf. [Dipper07]) which covers many of the annotation domains used in ToMA. The following descriptions illustrate, in the manner of the TEI formalization, some annotation conventions at different levels: Our coding scheme, still under development, proposes then a general synthesis taking into account all the different levels of annotation for multimodal corpora such as phonetics, prosody, syntax, or gestures, as well as annotations at the discourse level (humor, backchannels, narrative units, conversational turns, etc.).

A general coding scheme is of deep importance not only in terms of standardization and knowledge representation, but also for practical matters: it constitutes the basis for a pivot language, making it possible for the different tools (Praat, Anvil, etc.) to exchange formats. This is one of the goals of the PAULA format (cf. [Dipper05]). From the same perspective, starting from an adaptation of this format, we are developing tools implementing such interoperability, relying on a translation between the source format of the tool and this language.

The ToMA annotation process

Until now, corpus annotation has been essentially based on written corpora, the annotation of oral corpora being very limited. Some transcribed oral corpora exist, but they rarely contain precise phonetic and prosodic information on top of transcription. The Switchboard corpus has been recently annotated in such perspective (see [Carletta04]) and constitutes an exception. As for multimodality, only few initiative try to build large broad coverage annotated corpora, including such level of precision in each domain. The AMI project is one of them (see [Carletta06]), even though the annotations does not seem to be at the same precision level in the different domains.

Our project aims at building such large resource, trying to answer to the needs of researches in each domain (in other words being as precise as possible in the annotations) and at the same time making possible the analysis of domain interaction (annotating as many domains as possible). The problem first comes from the lack of annotation tools and second, the difficulty in integrating annotations into a common format.

The ToMA project's aim is to develop a platform providing help at each step of the process, from raw data to high-level annotations. ToMA specifies conventions for manual annotation steps and is based on freely available tools from different sources for the automatic ones. Most of the tools have been developed by the project partners and are adapted for the specific needs of spoken language processing. They will be distributed under the auspices of ToMA, as well as the annotated corpora. The experiment described in this paper has been used for the annotation of a corpus (Corpus of Interactional Data -CID) which is already freely available from the CRDO2 . Figure 1 describes the state of the general process in which the status of each step, automatic (auto) or manual (manual) is specified. We briefly sketch in what follows the main steps of the process:

• Segmentation in Interpausal-Units: Transcriptions are made starting from an automatic pre-segmentation of the speech signal into interpausal-units (IPU) that are blocks of speech bounded by silent pauses of at least 200 ms. IPU segmentation makes transcription, phonetization and alignment with the signal easier. Moreover, speech overlap phases are extracted from IPUs. • Transcription: conventions are derived from [Blanche-Benveniste87] on top of which other information is added (such as elisions, particular phonetic realizations, etc.). From this initial enriched orthographic transcription (EOT), two transcriptions are derived: one is phonological, the other is phonetic. The following example illustrates this step:

-EOT: • Phonetization: This step produces a list of phonemes. After a tokenization, the symbolic phonetizer (see [DiCristo01]) provides a list of tokens and their phonetization labeled in SAMPA. The EOT may sometimes be difficult to use, and a direct phonetic transcription can be, in some cases, simpler for the transcriber; the phonetizer therefore accepts mixed orthographic and SAMPA symbols as an input.

• Alignment: The aligner (cf. ). Because of the lack of consensus on the phonological system in French, we use the MOMEL-INTSINT system which precisely does not suppose any a priori knowledge of the phonological system of the language. The interest to have both manual annotations and automatic INTSINT annotations is to improve INTSINT itself, but also the knowledge, which is still very fragmentary, of the prosodic domains in French. • Morphosyntax: Morphosyntactic annotation is done automatically, using a POStagger (LPLsuite,cf. [VanRullen05]) which has been adapted to spoken language.

The system has been trained with appropriate data, and custom correcting code heuristics has been developed. It is then checked and corrected manually. • Syntax: We have developed an original statistical parser, adapted to the treatment of spoken data. This is done in two different phases. The first consists in parsing a spoken language corpus by means of a symbolic parser (cf. [Blache05]). In a second stage, the output is corrected manually, the result being a treebank for spoken language. Finally, the statistical parser is trained on these data. The tool we obtain is used in order to generate the trees of the corpora to be annotated automatically. This output also has to be checked manually. • Gesture: The annotation of the gestures made by the participants is being done manually using ANVIL as shown in Figure 3

The CID: a first multimodal annotated corpus in French

The Corpus of Interactional Data is an audio-video recording of spontaneous spoken French (8 hours, 8 pairs of speakers). It features data recorded in an anechoic room and containing 110.000 words. Each speaker of the dialogue is equipped with a headset microphone enabling the recording of the two speakers' voices on two different sound tracks. This enables the study of speech at a phonemic and prosodic level. It also enables the study of overlapping speech which is frequent in spontaneous interactions but seldom analyzed because of the difficulty to separate the intertwined voices of the two speakers a posteriori. Yet, overlapping speech plays a major role in conversation and requires experimental investigations.

In this corpus, we aimed at collecting useful data for the investigation of all the levels under study: the audio quality is optimum and they have been videotaped with a high quality digital camera. The corpus, described in [Bertrand08], has been annotated following the different steps described above.

We then aligned the orthographic and phonetic transcription with the signal and added information from different linguistic fields (prosodic units, intonative contours, morphosyntactic categories, syntactic phrases, etc.). These annotations have been done separately on the audio files and constitute the basis for our present project which consists in the annotation and processing of the corpus from a multimodal / multi-dimensional perspective.

The annotation of the gestures made by the participants is being done manually using ANVIL as shown in Figure 2. The tiers bear different information, from the f0 curve and pitch targets coded with INTSINT (tier 3) to the conversational units (tier 5). Tier 4 encodes the function of the rising contour (RMC: major continuation rise). The following tiers encode the gestural information and the last tier refers to the interlocutor which produces a gestural backchannel signal (nod).

Exploiting multimodal annotations

In this section we present several examples to illustrate the kind of information and results that can be obtained thanks to such multimodal resources. In the first subsection we propose some observations which can be done from these data concerning the relations between different levels, in particular prosody and syntax. After this subsection, three studies will be presented: the first study on backchannelling has led to the analysis of some general extenders in French and to the question of reinforcing gestures.

Data synchronisation

Before entering into data description, it is necessary to tell how data alignment (or synchronization) is done. In our approach, all information is aligned with the signal. This means that identifying interaction between objects requires a temporal comparison. In most cases, an intersection of the temporal segments of the objects to be considered is the sign of an interaction. For example, an intersection between a deictic gesture and a pronoun specifies a referential relation.

Of course, we have to take into consideration that objects have been aligned on the signal by means of different techniques. In some cases, the alignment is done automatically (e.g. syntactic units aligned with words and phonemes, then with the signal). All the annotations aligned automatically from the phonemes can be strictly aligned in the sense that they share exactly the same boundaries. For example, we have developed a tool segmenting the input on the basis of morpho-syntacitc information. Segments correspond to what we call "pseudo-sentences" that show a certain syntactic coherence. Pseudo-sentences' right boundaries are strictly aligned with that of syntactic groups.

The situation is different when at least one of the annotations has been created manually. In this case, boundaries can be less precise and some flexibility has to be taken into account when identifying the synchronisation between the different annotations. For example, contours and syntactic units usually do not have the same boundaries in our annotations, so that when looking for synchronisation this parameter has to be taken into account.

Prosody/syntax interaction

Different kinds of prosodic information are available in the CID. At a first level, relations between prosodic units and contours can be underlined. Here are the different categories that have been used: • Units : accentual phrase (AP), intonational phrase (IP),

• Contours: mr (minor rising), m0 (other minor contours), RMC (major continuation), RL (list rising), fl (flat), F (falling), R (rising), RF1 (rising-falling), RF2 (falling from the penultimate), RQ (interrogative rising), RT (terminal rising). Chunks are non recursive units, defined by the PEAS formalism (see [Paroubek06]) used for the parsing evaluation campaign regularly organized for French parsers. Concretely, we have shown that chunks correspond to a certain kind of supertags (see [Blache08]) or, in other words, identify left boundaries together with the nucleus of the corresponding phrases. The evaluation campaign shows good results for our parser as for spoken language parsing (F-score 80%).

The results of the alignment indicate a strong correlation between /mr/ contours and /NV/ (nucleus VP). This effect can be explained by the fact that these chunks do not contain verbal complements. These complements (in particular direct objects) have a strong syntactic (and semantic) relation with verbs, which explains the fact that no strong prosodic boundary or contour occurs in this place. Reciprocally, chunks corresponding to constituents (such as NP or PP) that usually end main phrases show an important proportion of cooccurrences with intonational phrases. Beside general annotations, the CID also contains more specific ones, added for the study of precise phenomena. For example, detachment constructions have been identified and located (this is an ongoing study, lead by Lisa Brunetti). Among different detachment types, figure 5 focuses on lexical and pronominal dislocated subjects. It shows a relative low level of cooccurrence with intonational phrases and a high proportion of minor rising contours. These data tend to illustrate the syntactic cohesion between the dislocated element and the matrix sentence. This figure also gives indications concerning canonical lexicalized subjects (that are relatively rare in spoken language corpora). What is interesting is that these subjects seem to have the same prosodic characteristics as dislocated ones, including the ones which occur in intonational phrases. This observation should be refined, but it seems that it could illustrate the fact that the detachment construction could become marked in spoken language.

Backchannels

Backchannel signals (BCs) provide information both on the partner's listening and on the speaker's discourse processes: they are used by recipients to express manifest attention to speakers in preserving the relation between the participants by regulating exchanges. They also function as acknowledgement, support or attitude statement, and interactional signals in marking specific points or steps in the elaboration of discourse. Until recently, they were still considered as fortuitous, but recent works showed that they have a real impact on the speaker's discourse (see [FoxTree99]).

Although they can be verbal like "ouais" (yeah), "ok", etc, vocal ("mh") or gestural (nods, smiles, etc), most of the studies on BCs only concern one modality. Our aim is to integrate the different nature of BCs in order to draw up a formal and functional typology. Such helps in automatically labelling BCs, as well as understanding more accurately the communication strategies (see [Allwood05]). Moreover, we also try to have a better understanding of the BC context which can also inform on its function and contribute to the study of the turn-taking system.

The following example, taken from the CID, illustrates the interest of a multimodal study of BCs. Verbal BCs are represented in italics, gestural ones in frames.

A ah ouais nous on est rentré à (...) dix heures dix heures et demi je crois du soir (...) B nod A et elle a accouché à six heures je crois (...) B ah quand même ouais B head tilt / eyebrow raising A donc c'était ouais c'était quand même assez long quoi (...) B head tilt

[A] oh yeah we were admitted at 10, 10.30 I think pm [A] and she had the baby at 6 I think

[B] [oh yeah right?]

[A] so it was yeah it was quite long indeed Several questions can be raised: in what context do backchannels appear, do verbal and gestural BCs behave similarly, etc. Such problems require the study of the different levels of information. Among them, the prosodic and discourse layers seem to play an important role for backchannels. Figure 6 shows the relations between these prosodic-unit levels, prosodic contours and conversational turns. By conversational turns, we mean the different units of turn (the turn-constructional units) defined as points of completeness from a syntactic, prosodic and pragmatic point of view [Selting00]. A TCU may be labelled as final (complete from the three criteria), as non final (incomplete from the pragmatic point of view for instance) or as cont(inuation) to refer to cases of adds-on or completion of turn (after a final TCU). As for prosody, the figure shows that BCs are realized preferentially after IPs and /fl/ contours. Concerning discourse, they are realized after final turns, in other words when the speaker reaches a certain level of completion in discourse. In [Bertrand07], we have shown that vocal and gestural BCs have similar behavior, in particular concerning the morphological and discursive production context. They appear after nouns, verbs and adverbs, but not after connectors or linking words between two conversational units. As for prosody, gestural BCs can occur after accentual phrases (AP) and intonational phrases (IP) whereas verbal BCs only occur after IPs. Both BCs seem to be favoured by rising and flat contours. However, rising contours (R, which brings together the whole rising contours RMC + RT) exhibit a specific behavior according to the nature of BC. Proportion tests with a z-score to measure the significant deviation between the two proportions confirm that significant relevant typical contours at points where BC occur are: RT (z-score = 3.23 for vocal BC and 2.18 for gestural BC); RMC (z-score = 2.9 for vocal BC and 4 for gestural BC); and fl (z-score = 2.8 for vocal BC and 3.9 for gestural BC). By producing preferentially a gestural BC after a RMC contour, the recipient shows that not only does he understand that the speaker has not finished yet but he does not want interrupt him. On the other hand, by producing more frequently a vocal BC after a terminal rising contour, the recipient displays a minimal but sufficient contribution. But thanks to it the recipient also shows his willingness to stay as recipient at a potential transition relevance place. These first results show that different cues at different levels of analysis are relevant for BCs occurring. More generally they confirm the relevance of a multimodal approach for conversational data and corpusbased investigations.

Adjunctive general extenders

The study on BCs has led to an analysis of specific French locutions on 3 hours of the corpus. These locutions are a set of adjunctive general extenders (cf.

[Overstreet00]) such as "(et) tout (ça)" ("and all that") and "et cetera", which are favorable contexts for the production of BCs by the hearer. Two issues are at stake concerning them: (1) whether they should or not count as a category of discourse markers (DM), and ( 2) what their function is. Our aim is to refine existing work mainly based on syntax and discourse analysis, adding prosodic and gestural descriptions of the extenders.

There is yet no consensus concerning the classification of general extenders as DMs (also sometimes called pragmatic markers or pragmatic particles). Whereas some do consider they are, they do not meet all the criteria defined by [Schiffrin87] and [Fraser99] to enter the category of DMs: for instance, they cannot be inserted at the beginning of an utterance, and their meaning in context is not always different from the core meaning of the locution.

Yet, they fully meet other criteria such as the fact that they cannot stand alone as subject or object of a VP, they show a range of prosodic contours, etc. An intermediate standpoint consists in considering some instances of general extenders as DMs, but not all of them. This is the point of view we adopted in this study, one coder determining the status of DM whenever the locution showed prosodic dependence with the intonation unit which preceded or followed it. This first annotation would of course have to be cross-examined by other coders as well but our preliminary results are quite interesting to mention here.

They show that DMs are almost systematically de-accented (they do not carry nuclear stress and a number of items are phonetically reduced although this is not systematic: for instance "tout ça" [tu sa] is often pronounced [tsa] in this context) and usually follow a rising contour, which is not the case of locutions. They are also significantly accompanied with reinforcing gestures, either head movements or hand gestures. As will be shown in the next section, reinforcing gestures reveal discourse structure and this is also the role of "tout ça" in the following example. It is written in capital letters, is accompanied by a metaphoric hand gesture and is produced just after another DM "et tout" with which it forms a single prosodic unit. The example however should not be considered as a case of reduplication of the locution for emphasis.

Example:

tu sais tout ce qui était Provence et tout TOUT CA "you know all the stuff made in Provence and all that TOUT CA" When it comes to the second issue concerning general extenders, i.e. their linguistic function, we adopted the typology proposed by [Overstreet00] who suggested that the items have three main values:

1. List extenders (extending a list without naming all the items): "ceux qui font les courses ceux qui font la vaisselle et cetera" ("the people who do the shopping, the ones who wash the dishes et cetera") 2. Illustration (giving an example): "c'est comme les marrons qu'on bouffe tout ça c'est des châtaignes aussi" ("it's like the horse chestnuts that we eat and stuff they are indeed chestnuts") 3. Intersubjectivity (relationships between the participants to the dialogue): "il avait perdu ses parents tout ça" ("he had lost his parents and stuff")

Each general extender was assigned a function on the basis of semantics only (on the written script) with Praat by one coder. Out of 104 occurrences of general extenders, only 4 instances could not be decided on. Our aim was to see if the intuitive annotation of the functions of general extenders would meet any pattern in prosody and gesture.

In prosody, we expected to find congruence between the LIST function and the enumerative contour for instance, although the results concerning the correspondence between contours and values need to be developed. But as far as gestures are concerned, we do have preliminary results which are very encouraging. The gestures, which accompany 40 % of the adjunctive general extenders in this corpus, are only head movements (head shakes and head tilts) and hand gestures (metaphorics and iconics).

We never met any eyebrow rising for instance or any smile. We will have to think about such a gestural specificity on general extenders since in other contexts in the corpus, movements and gestures are much more varied. What is more, although head movements were equally distributed among the different functions, we found a much higher proportion of hand gestures reinforcing extenders with an intersubjective value, especially metaphorics.

At last, to loop the loop, since DMs are used to express the pragmatic relationships in dialogue, we expected a higher proportion of DMs than locutions to have an intersubjective value, since this value is the one which is the farthest from the core meaning of general extenders, and this is exactly what our results show.

Reinforcing gestures

As we have seen in the previous subsection, some gestures can be interpreted as discourse reinforcement devices (cf. [Ferré07], [Ferré09]). To illustrate our point, let us say that there is a difference, for instance, between a head nod produced by the audience as backchannel, and a nod produced by the speaker without any prompt, when this nod doesn't stand for the emblem of "yes". This is the case of the example provided below, where the nod slightly anticipates "super strict", and can be understood as reinforcement of the degree adverb "super". We started with annotating what we intuitively felt were reinforcing gestures, in order to adopt a more gestural perspective rather than a discursive or a prosodic one. Here, we wanted to know what exactly would be reinforced, e.g. instead of focusing on semantic and morphological criteria for intensification, we wanted to find out if there were other possible production contexts for reinforcing gestures. We also wanted to know if gestural and prosodic reinforcement would be simultaneous.

The study showed that intensive gestures are more liable to accompany degree adverbs and negation particles, as well as connectors (DMs which show the discursive or pragmatic links between speech turns). Considering this, we concluded that the gestures we looked at -which were head and eyebrow movements as well as gaze direction -rather played a discursive role of intensification, especially since none of these gestures were associated with any specific stress type. The study also showed that intensive gestures are not redundant in their expression of emphasis: the segments they highlight do not fall under intonational focalization, for instance, with which they are in complementary distribution. This does not mean that reinforcing gestures are never used at the same time as prosodic focalization: in the example above, for instance, there actually is a focal accent on the first syllable of "super" which is also reinforced by the nod. What it means however, is that there are many other contexts, especially negation particles, which are unstressed but yet reinforced by a gesture. Yet, when looking at the gestures themselves and their distribution, one wouldn't speak of emphasis. Indeed, whereas a large gesture could be considered as emphatic (or giving the accompanying speech some emphasis), these movements are not necessarily large at all. Most of the time, they are even very slight. What counts here rather seems to be a question of gesture density, pretty much in the same way as S. Norris [Norris04] speaks of modal density, e.g. the accumulation of body movements on certain parts of speech, listener-oriented.

This study was actually a pilot experiment which must be extended and this will be done in two ways: (a) the rest of the corpus should be annotated in the same way and (b) at the time of the study, the syntactic annotations were not ready to allow their being taken into account, and as they have been done since, they would certainly refine enormously the analysis of the co-occurrences of reinforcement gestures with adverbs and connectors.

Conclusion

Annotated multimodal corpora now constitute an essential resource in linguistics. The understanding of language mechanisms (both in production and perception) needs to take into account very precisely the interaction between all the different domains or modalities (phonetics, prosody, lexicon, syntax, pragmatics, gestures, discourse, etc.). Producing such resources represents however a huge amount of work. It is then necessary to specify a precise framework, identifying the different tasks, the kind of information they have to produce and to what extend they can be automatized.

We have presented in this paper the main characteristics of the ToMA project, providing a general framework for building richly annotated multimodal corpora. The main characteristic of ToMA is that it aims at the description of natural human-human interaction. In order to do this it is necessary to exploit a set of precise and high-level annotations in each linguistic domain. This annotation process has been made possible thanks to a precise definition of different steps, each coming with a set of recommendations and tools.

We have shown in this paper that new results can be obtained from such resources: several examples have been presented here illustrating the importance of a description which brings together information from different levels. It now becomes possible to specify linguistic information in a new perspective, in which phenomena are described in terms of interaction between objects from different domains. This we hope to become an open door for multimodal grammars.

-

  et c(e) qu(i) était encore plus le choc c'est que en [fait, faiteu] (what was even a greater shock was that…) Phonologic version: et ce qui était encore plus le choc c'est que en fait -Pseudo-phonetic version: et c' qu était encore plus le choc c'est que en faiteu

  [Brun04]) takes as input the list of phonemes and the audio signal. It then localizes each phoneme in the signal.• Prosody: Prosodic annotations essentially encode the prosodic categories (intonational and accentual phrases [Jun02]) and the intonation contours associated to them. Such annotations are exclusively done by experts. The intonative level is also encoded with the Momel-Intsint algorithm([Hirst00]) in an automatic way: from a phonetic representation of the fundamental frequency curve, INTSINT provides a level of surface phonological representation where the melody is represented by a sequence of discrete symbols([Hirst05]

Figure 2 :

 2 Figure 2: CID annotation board

Figure 3 :

 3 Figure 3: Relations between contours and prosodic units Figure 3 illustrates the distribution of the different contours in one of the dialogue of the CID. This distribution shows that flat and major rising contours are the most frequently used ones at the right boundary of an IP. Conversely, minor rising contours are by far the most frequent type in association with APs.

Figure 4 :

 4 Figure 4: Relations between syntactic and prosodic contours Figure 4 includes syntactic information on top of these prosodic relations. More precisely, it shows the distribution of syntactic units in relation with the different contours. Syntactic annotation of our corpus has been done by means of a stochastic chunker adapted from the techniques developed by the LPL (see [Vanrullen06]).Chunks are non recursive units, defined by the PEAS formalism (see[Paroubek06]) used for the parsing evaluation campaign regularly organized for French parsers. Concretely, we have shown that chunks correspond to a certain kind of supertags (see[Blache08]) or, in other words, identify left boundaries together with the nucleus of the corresponding phrases. The evaluation campaign shows good results for our parser as for spoken language parsing (F-score 80%).The results of the alignment indicate a strong correlation between /mr/ contours and /NV/ (nucleus VP). This effect can be explained by the fact that these chunks do not contain verbal complements. These complements (in particular direct objects) have a strong syntactic (and semantic) relation with verbs, which explains the fact that no strong prosodic boundary or contour occurs in this place. Reciprocally, chunks corresponding to constituents (such as NP or PP) that usually end main phrases show an important proportion of cooccurrences with intonational phrases.

Figure 5 :

 5 Figure 5: Relations between detachments and prosodic contours

Figure 6 :

 6 Figure 6: Relations between backchannels, prosody and discourse

•

  Morphology and Syntax: we propose to adapt the Maptask coding scheme for French in the morphological dimension, completed with syntactic relations and properties.• Phonetics: some annotations are a completed version of MATE([Carletta99]). The phonetic representation is coded in SAMPA.• Phonology and Prosody: we adopt the coding scheme proposed in[DiCristo04] in which prosodic information is annotated both in an automatic (MOmel-Instsint algorithm, [Hirst00]) and manually.

• Gesture analysis: we adapt the MUMIN coding scheme

([Allwood05]

), which provides an extensive description of gesture forms, but we propose to code gestures and discourse tags separately. The gesture typology is encoded following the scheme proposed in

[McNeill05]

. A gesture lexicon is compiled from the existing descriptions found in the literature ([Kendon04], [Kipp04], [Krenn04]).

• Discourse and conversation analysis: we use the Maptask

([Isard01]

) and DAMSL coding schemes, extended to other discourse types such as narration, description, etc.

  Discourse events (narrative units, speech particles, etc.) are annotated manually in distinct tiers either in Praat or in Anvil depending on the need for video information (for instance, verbal and vocal backchannels were annotated in Praat whereas gestural ones were annotated in Anvil. After all the annotations are made, they were grouped into a single file for the queries to be made). Annotations are created from the aligned orthographic transcription.

	head), gaze direction and facial expressions (smiles, etc) are encoded as well as
	hand gestures. For the latter, McNeill's typology [McNeill05] was used
	(metaphorics, iconics, deictics, beats) and completed with emblems and adaptors. It
	has also been decided to annotate gesture phases (preparation, stroke, hold,
	retraction), as well as gesture apex as proposed by [Loehr04], although this
	annotation will come in a second step.
	• Discourse:
	below. Facial movements (eyebrow,

ToMA stands for "Tools for Multimodal Annotation" (the French acronym is "OTIM"). Project supported by the ANR French agency, involving different partners (LPL, LSIS, LIMSI, LIA, RFC and LLING).

http://www.crdo.fr/