Baptiste Caramiaux
email: caramiau@ircam.fr

Sarah Fdili Alaoui

Tifanie Bouchara

Gaétan Parseihian

Marc Rébillat

S Fdili Alaoui

Gestural auditory and visual interactive platform

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

This paper introduces GAVIP (Gestural Auditory and Visual Interactive Platform), an interactive and immersive platform allowing for audio-graphical virtual objects to be controlled in real-time by physical gestures. GAVIP is based on a control unit driven by gesture processing monitoring the behaviour of the virtual objects. A global control of the virtual world ensures the inter-modal coherency of the proposed environment. The SMART-I 2 audiographical rendering engine achieves the spatialization of the virtual objects in a 3D audio-graphical scene [START_REF] Rébillat | Smart-i 2 : "spatial multi-user audio-visual real time interactive interface", a broadcast application context[END_REF]. The spatialization increases the coherence and thus the user's feeling to be «present» in the virtual scene [START_REF] Bormann | Presence and the utility of audio spatialization[END_REF]. Gesture processing allows for a wide range of natural interaction enhanced by the presence sensation [START_REF] Schuemie | Research on presence in virtual reality: A survey[END_REF].

GAVIP can be either considered as a sound installation, a platform of development for experimental protocol or an extended musical virtual instrument. By «extended», we mean that the created virtual musical instruments do not only allow an interaction with the audio modality, but an interaction with three modalities: audio, graphic and spatial. Declinations of GAVIP mainly depend on scenarios that are implemented. In this paper, focus is particularly put on two scenarios exploring the interaction between a user and the audio, graphic, and spatial synthesis of a virtual world where the interaction is thought in the sense of virtual musical instrument design. Previous works have dealt with the interaction between gesture and sound for the design of virtual musical instruments [START_REF] Hunt | The importance of parameter mapping in electronic instrument design[END_REF]. Early works have led to low interaction expressivity caused by a direct mapping between gesture parameters and sound synthesis engine parameters. Extensions have led to take into account more expressive interactions by considering higher level descriptors or mapping based on physical models [START_REF] Wanderley | Gestural control of sound synthesis[END_REF]. Progressively, virtual musical instrument became more linked to the sound perception and immersion. This has led to consider the spatial diffusion of sounds produced by such instruments [START_REF] Leslie | Grainstick: A collaborative, interactive sound installation[END_REF], thus changing the terminology from virtual musical instrument to sound installation in which interactions are multimodal [START_REF] Camurri | Multimodal interfaces for expressive sound control[END_REF]. In this paper, our contribution is to propose a platform that combines sound spatialization and audiographical systems. To enhance interaction, our platform aims to guarantee inter-modal coherency (see Sec. 2.2).

The paper is structured as follows. In the section 2 we present the concept of GAVIP (the general platform for designing new extended musical instruments). Section 3 describes a first scenario based on a simple interaction approach. To increase the degree of inter-modal coherency, a second scenario was developed based on a physical model. It is described in Section 4.

THE GAVIP CONCEPT

Concept

The goal of GAVIP is the conception and the exploitation of an immersive, interactive and multimodal platform. Such a platform is built as a support for different scenarios. These scenarios will typically plunge the participants into a virtual environment populated of several audio-graphical entities among which they can freely evolve. Participants can interact in real-time with these entities through their gestures. Gesture analysis is used as a natural way to control the behaviour of audio-graphical entities and their spatialization in a 3D scene. Thus, a great effort is made to design a general architecture that allows for wide interaction strategies.

Interaction and Inter-modal coherency

In this paper, interaction means the bidirectional relationships between human gestures and virtual elements. Our hypothesis is that perception of interaction is enhanced by the coherency between audio and graphical objects (the so-called inter-modal coherency). Without being stated, this idea is shared across complementary research fields like Human Computer Interactions (HCI) [START_REF] Vertegaal | Looking for sound?: selling perceptual space in hierarchically nested boxes[END_REF], New Interfaces for Musical Expression (NIME) [START_REF] Rodet | Study of haptic and visual interaction for sound and music control in the phase project[END_REF], or Virtual Reality (VR) [START_REF] Bormann | Presence and the utility of audio spatialization[END_REF]. By audio-graphical object we mean a virtual object with consistent audio and graphical components. Inter-modal coherency of the audio and graphical components of the virtual object (semantic and spatial) is required to integrate the audio and graphical streams into one unique percept [START_REF] Spence | Audiovisual multisensory integration[END_REF]. However the classical approach is to synthesize separately the audio and graphical streams. Mismatches between these two information streams typically exist and are thus perceived. This can significantly degrade the quality of the audio-graphical rendering and the resulting interaction. To enhance presence and interaction, Inter-modal coherency has to be ensured.

Architecture

In order to achieve interaction with consistent audio-graphical objects in a virtual 3D scene, we designed a general virtual space where the audio and graphical parts of each entities are governed by the same control unit. Concretely, one unique control model is mapped with both modalities (the sound synthesis and the graphical synthesis) and the virtual space (spatialization) according to three consistent intra-modal mapping strategies (see Fig. 1). This allows the immersed user to influence through his/her gesture the control model itself and, consequently, the audio-graphical objects behaviour and their position in the virtual space. The consequence of an action of the user is by nature multimodal (audio/graphical/spatial) and a careful design of the mapping strategies ensures, by construction, the preservation of a strong intermodal coherency.

Audio-graphical spatial rendering

The SMART-I 2 (Spatial Multi-user Audio-graphical Real-Time Interactive Interface) [START_REF] Rébillat | Smart-i 2 : "spatial multi-user audio-visual real time interactive interface", a broadcast application context[END_REF] provides an immersive, real-time and consistent audio-graphical spatial rendering. The spatial sound rendering is achieved by means of Wave Field Synthesis (WFS) [START_REF] Berkhout | Acoustic control by wave field synthesis[END_REF]. The WFS technology physically synthesizes the sound field corresponding to one or several virtual audio sources (up to 16 sources) in a large rendering area. The 3D graphical rendering is made using passive tracked stereoscopy. In the SMART-I 2 system, frontprojection screens and loudspeakers are integrated together to form large multi-channel loudspeakers also called Large Multi-Actuator Panels (LaMAPs). The rendering screens consist of two LaMAPs (2 m × 2.6 m with each supporting 12 loudpeakers) forming a corner (see Fig. 2). Overall, the SMART-I 2 allows to generate 16 independent virtual objects. Each virtual object is a sound source and a graphical object. See [START_REF] Vries | Wave Field Synthesis[END_REF] for more informations regarding audio-visual immersive systems using WFS.

Figure 2: Front (left) and back (right) views of the SMART-I 2 audio-graphical rendering system. The left picture shows a participant using the early approach of interaction.

EARLY APPROACH

This approach was presented as a demo during the european meeting European VR-EVE 2010 that took place at the LIMSI-CNRS, Orsay, France.

Overview

In this first approach, each graphical object was a bubble and each sound source was a single sound sample taken from a given database (see Fig. 2). The challenging task was to control the set of 16 independent sound sources generated by the SMART-I 2 . We chose to synthesize sounds of water drops. Two modes of interaction correspond to either a sequential or a single triggering of sound samples. Since a simple one-to-one mapping was impossible, we chose to have automatic controls of some of the sound parameters. We separated the sound controls (sample volume, sample playback speed, period between two successive sounds) from the spatial controls (voice number, distance, azimuth). Each of these parameters was controlled by a specific probability density function from which each parameter's value was drawn. This allowed to control the parameters' variations and to have less predictable behavior.

Gesture Control

The gesture control process is twofold. First, an Optitrack motion capture system composed of six circular cameras is used to track the user motion and adapts the graphic feedback to his/her spatial position. The calibration of the cameras is performed with the help of the Optitrack Rigid Bodies software. This software locates the markers dispatched on the user stereoscopic glasses and sends their 3D positions and orientation via a VRPN communication protocol.

Second the sound control is achieved by means of a WiiMote controller. Hence, the gyroscopes are used for driving azimuth and distances: the user can choose to be either closer or further from the sound sources and put the sources either on the right side or on the left side. Accelerometers are used for triggering control. We refer the reader to the previous section (Sec. 3.1) for the presentation of the two triggering modes. In the first mode (triggering mode), the user can use the controller to trigger percussive sound of water drop. In the second mode (sequential triggering), regular hits with the controller at a certain frequency trigger sequentially played sounds with the same frequency The frequency range allowed for synthesizing sound textures from separate water drops to heavy rains.

Limitations

The gesture control of audio-graphical objects proposed by this approach is straightforward. This enables a direct understanding of the general behavior by the user that can experience the scenario in a ludic way. However, this approach features several limitations. First, the global behavior is mostly controlled by probability density functions and the user's degree of control is limited. Hence the user can not really consider the interface as a virtual musical instrument as its control in sound production and positioning in space is limited. Second, the audio and graphical elements have a basic discontinuous behavior that consists in appearingdisappearing which does not simulate sound texture.

CURRENT APPROACH

To overcome the limitations of the first scenario, focus has been put on the control aspects of the platform for two main reasons. First, we aim at providing a smoother control of sound synthesis and sound spatialization. Second, in our quest for inter-modal coherency, we were looking for a more coherent relationship between audio and graphical parts of the virtual entities.

Overview

We refer the reader to the general architecture of GAVIP depicted in Fig. 1. The control process in the early approach was based on accelerometer-gyroscope sensor data analysis and simply linked triggering and orientation messages between gesture and audio modalities (also called direct mapping). Here we propose to define a more complex control unit based on a physical model simulating the interactions between elements in the scene. Audio elements are sound grains obtained by segmenting recorded environmental sounds. Since we aim to recreate sound textures we basically segment environmental textures like rains, winds, etc. Graphical sources are deformable sphere whose dynamics was inspired by magma lamp. In the following we mainly focus on audio control and rendering.

Physical Model

Here we define a physical model governing the dynamic of N abstract spherical particles in a 2D space. These particles are in a mutual interaction and in interaction with their environment. A physical model of N punctual masses linked together with springs governs their movements in this environment. Let us denote an(t) the acceleration of particle n at t, each element n has a mass mn and follows the Newton's second law of motion defined by:

mn an(t) = k =n F k→n (t) + Fcentre→n(t) + Fext→n(t)
Where at time t, k =n F k→n (t) is the sum of the forces applied from the other particles on the particle n, Fcentre→n(t) is the force applied from the centre on the particle n, and Fext→n(t) are the external forces exerted on n. Within the framework of this physical model, the forces are defined as follows:

• F k→n (t) accounts for: the elastic force (characterized by a parameter α k ), the viscous friction (characterized by µ k ), the electrostatic attraction (characterized by the product of their electrostatic load qnq k )

• Fcentre→n(t) accounts for: the elastic attraction (characterized by K) and the central viscous friction (characterized by η)

• Fext→n(t) accounts for: the global rotation (characterized by βn) and a magnetic field that is linked to the electrostatic load of the particle qn.

This physical model allows to generate the relative movements of masses from a given initial state. At each time step t, the physical model returns the particles' index together with their Cartesian coordinates.

Sound Synthesis

The sound synthesis is based on the CataRT software developed by Diemo Schwarz at Ircam [START_REF] Schwarz | Real-time corpus-based concatenative synthesis with catart[END_REF]. The software takes a set of sounds and segments them given a grain size (the default grain size is 242ms) and/or a segmentation algorithm (for instance based on onset detection). Since we work with sound textures with no onset, we choose to segment the input audio stream every 242ms. CataRT analyzes each grain by computing a set of audio descriptors. The resulting grain representation is a vector whose elements are the descriptors' mean values. CataRT then visualizes those grains in a 2D space (also called descriptor space) where the dimensions are chosen by the user among the available descriptors, for instance the loudness along the x-axis and the spectral centroid along the y-axis. Depending on an input path on this 2D space, sound grains are selected: at each time step t, the selected grain (i.e., played) is the closest to the current position xt, yt in the path in terms of a Mahalanobis distance. More precisely, this distance is computed between xt, yt and the grain descriptors mean values that correspond to the 2D space axis (e.g., loudness and spectral centroid). Note that a path in the descriptor space can be given either by using a gesture interface such as a mouse or by an input sound described in the same descriptor space. The latter case is sometimes called mosaicing.

Here, the sound grains are selected by the abstract particles' positions defined previously. To that end, we match both 2D representation used for the physical model and the sound corpus. Fig. 3 depicts the process. The sound corpus remains unchanged but the output of the physical model (which consists in each particle index and Cartesian coordinates) is scaled to fit the sound corpus dimensions. Hence we have two layers in the same Cartesian space (see the image at the middle of Fig. 3). An abstract particle position selects the closest grain (according to a certain neighborhood radius) at each time step. Concretely, our physical model is composed of 16 particles navigating in the sound corpus and selecting 16 grains (i.e., sound sources). Finally, this virtual plane is the sound field produced by the WFS.

Gesture control

The user's head is still tracked using an Optitrack motion capture system that allows to adapt the graphic feedback to his/her spatial position. Then, the gesture control of the virtual environment and objects corresponds to the possibilities of action and influence that a participant will have on them. One of the main challenges is to offer the participant an important power of expression in his control of the environment and the virtual objects.

At this step, the mapping between gesture analysis outputs and physical model parameters is not implemented but the global con- cept is as follows. Two main gesture parameters will be computed (periodicity and energy) and the mapping is one-to-many. A periodic gesture with low energy will involve a stable behavior of the system with low changes in its parameters and high viscosity, and high magnetic rotation forces. A periodic gesture with high energy will decrease the viscosity and the elastic force between particles leading to a set of particles with higher quantity of motion. Finally, a non-periodic gesture with high energy will produce a highly entropic and unstable behaviour of the physical system. Hence, energy will be computed on data coming from the 3D accelerometers and periodicity on the data coming from the 3D gyroscope (using an autocorrelation criterion for example).

Implementation

A first version is already implemented in the Max/MSP real-time programming environment. A demonstration video of the phyiscal model controlling CataRT is available online 1 . A smaller version using binaural sound spatialization and simple 3D graphical rendering will be shown during the conference.

CONCLUSION

In this paper, a platform for the conception and exploitation of immersive, interactive and multimodal scenario were presented. An effort has been made to define a modular architecture that allows for various interesting interaction designs with respect to the coherency between audio and graphical objects. Two concrete sce-1 http://imtr.ircam.fr/imtr/GAVIP narios were presented. The first one made use of simple triggering and orientation interactions between gesture and audio synthesis and spatialization. The second scenario proposed a general physical model that governs the behavior of audio-graphical objects in the scene. Gestures analysis controls the physical model. This new approach offered better coherency between audio and visuals as well as a smoother gesture control of them.

Future works will consist in (1) completing the implementation of the gestural control of the physical model and (2) designing perceptual experiments to evaluate the platform.

Figure 1 :

 1 Figure 1: Overview of the general architecture of the GAVIP

Figure 3 :

 3 Figure 3: Sound Synthesis. Physical model environment and the sound corpus environment are matched and assigned to the sound field produced by the WFS.

Proc. of the 14 th Int. Conference on Digital Audio Effects (DAFx-11), Paris, France, September 19-23, 2011

ACKNOWLEDGMENTS

We are grateful to both LIMSI-CNRS and UMR STMS Ircam-CNRS for the lending of equipment. We also would like to thank Matthieu Courgeon for his contribution and technical assistance on the 3D graphical programming.


Marc Rébillat

Xavier Boutillon

Etienne Corteel

Brian F,G Katz

Étienne Corteel

Brian F G Katz

Audio, visual, and audio-visual egocentric distance perception in virtual environments

Keywords: Audio, visual, and audio-visual egocentric distance perception in virtual environments PACS no. 43.66.Qp, 42.66.Si

HAL is

Introduction

Virtual reality aims at providing users with a virtual world where they would behave and learn as if they were in the real world [START_REF] Brooks | What's real about virtual reality[END_REF]. Correct distance perception in virtual environments is thus crucial for many virtual reality applications.

Because distance perception is a cognitive task, measurement protocols are needed to estimate the distance perceived by a participant. Existing measurements protocols for absolute egocentric distance can be divided in 3 main classes [START_REF] Klein | Measurement protocols for medium-field distance perception in large-screen immersive displays[END_REF][START_REF] Grechkin | How does presentation method and measurement protocol affect distance estimation in real and virtual environments? A C MT r a n s a c t i o n so nA p p l i e dP e rception[END_REF]: verbal estimations, perceptually directed actions, and imagined actions. In verbal estimation protocols, participants assess perceived distance in terms of familiar units, such as meters. In perceptually directed action protocols, an object is presented to the participant who then has to perform an action, such as blind-walking, without perceiving the object any more. In imagined action protocols, the action is imagined instead of being performed and response times or other measures are used to infer the results of the action. The advantage of perceptually directed actions over other protocols is that the observer's perception of distance can be directly inferred from the action [START_REF] Klein | Measurement protocols for medium-field distance perception in large-screen immersive displays[END_REF].

(c) European Acoustics Association

Using these protocols, numerous recent studies have focused on visual distance perception in virtual environments involving large screens [START_REF] Klein | Measurement protocols for medium-field distance perception in large-screen immersive displays[END_REF][START_REF] Grechkin | How does presentation method and measurement protocol affect distance estimation in real and virtual environments? A C MT r a n s a c t i o n so nA p p l i e dP e rception[END_REF][START_REF] Armbruster | Depth perception in virtual reality: Distance estimations in peri-and extrapersonal space[END_REF][START_REF] Naceri | Depth perception within virtual environments: a comparative study between wide screen stereoscopic displays and head mounted devices[END_REF][START_REF] Alexandrova | Egocentric distance judgments in a large screen display immersive virtual environment[END_REF]. These studies found that visual distances are systematically underestimated. In audio virtual environments [START_REF] Loomis | Mixed reality: Merging real and virtual worlds, chapter Auditory distance perception in real, virtual, and mixed environments[END_REF][START_REF] Kim | Control of auditory distance perception based on the auditory parallax model[END_REF][START_REF] Zahorik | Assessing auditory distance perception using virtual acoustics[END_REF][START_REF] Corteel | Caractérisation et Extensions de la Wave Field Synthesis en conditions réelles[END_REF][START_REF] Wittek | Spatial perception in wave field synthesis rendered sound fields: Distance of real and virtual nearby sources[END_REF][START_REF] Wittek | Perceptual differences between wavefield synthesis and stereophony[END_REF][START_REF] Kearney | Depth perception in interactive virtual acoustic environments using higher order ambisonic soundfields[END_REF] near distances (< 2m) are overestimated whereas far distances (>2 m) are underestimated. Little is known regarding combined audio and visual interactions on egocentric distance perception in the real world [START_REF] Gardner | Proximity image effect in sound localization[END_REF][START_REF] Mershon | Visual capture in auditory distance perception: proximity image effect reconsidered[END_REF][START_REF] Mershon | Perceived loudness and visually-determined auditory distance[END_REF][START_REF] Zahorik | Estimating sound source distance with and without vision[END_REF] and even less in virtual environments [START_REF] Bowen | Visual Localization Accuracy Determines the Bias of Auditory Targets in Azimuth and Depth[END_REF].

In this paper a study of audio (A), visual (V), and audio-visual (AV) egocentric distance perception in virtual environments is presented. AV rendering is provided via the SMART-I2 platform1 [START_REF] Rébillat | SMART-I 2 "Spatial Multi-user Audio-visual Real-Time Interactive Interface[END_REF][START_REF] Rébillat | SMART-I 2 : "Spatial Multi-user Audio-visual Real-Time Interactive Interface[END_REF] using tracked passive visual stereoscopy and acoustic wave field synthesis (WFS) [START_REF] Berkhout | Acoustic control by wave field synthesis[END_REF]. Distances are estimated using a perceptually directed action (indirect blind walking [START_REF] Klein | Measurement protocols for medium-field distance perception in large-screen immersive displays[END_REF][START_REF] Fukusima | Visual perception of egocentric distance as assessed by triangulation[END_REF][START_REF] Loomis | Assessing auditory distance per-ception using perceptually directed action[END_REF][START_REF] Loomis | Virtual and Adaptive Environments: Applications, Implications, and Human Performance Issues[END_REF][START_REF] Thompson | Does the quality of the computer graphics matter when judging distances in visually immersive environments?[END_REF]) under A, V, and AV conditions. The question to be discussed in the study is: What kind of cross-modal interaction in audio-visual stimuli exists for distance perception in virtual environments?

Method

Experimental design

The goal of the current experiment was to study distance perception under three different rendering conditions (audio, visual, audio-visual). For that purpose, 5 distances in the action space, i.e. the space where "we move quickly, talk, and if needed can throw something to a compatriot or at an animal" [ 2 6 ] .w e r e tested: 1.5 m, 2 m, 2.7 m, 3.5 m, 5 m.

A total of 15 volunteers between 24 and 45 years old participated in the experiment (12 men, 3 women). All participants reported normal vision (eventually corrected) and normal hearing. Each participant had to estimate the 5 distances 4 times under each rendering condition. They performed three sessions of 20 iterations each after a training phase of 2 iterations under each rendering condition. Participants had pauses between sessions and the whole experiment lasted approximately one hour. To limit the influence of learning effects, order of the sessions were balanced between the 6 possible permutations of the 3 conditions.

The chosen experimental design was therefore a repeated-measures design with two factors: the real distance d r (5 levels, fixed factor) and the rendering condition (3 levels, fixed factor). Tab. 2.1 summarises the chosen experimental design.

Independent variables

Experimental setup

Experiments have been conducted in the audio-visual environment produced by the SMART-I 2 platform [START_REF] Rébillat | SMART-I 2 "Spatial Multi-user Audio-visual Real-Time Interactive Interface[END_REF][START_REF] Rébillat | SMART-I 2 : "Spatial Multi-user Audio-visual Real-Time Interactive Interface[END_REF]. In this system, front-projection screens and loudspeakers are integrated together to form large flat multi-channel loudspeakers also called Large Multi-Actuator Panels (LaMAPs). The rendering screens consist of two LaMAPs (2 m × 2.6 m with each supporting 12 loudpeakers) forming a corner (see Fig. 1).

Visual rendering is produced using tracked passive stereoscopy at 80 frames per second (in the GPU) and with a resolution of 1280 × 960 pixels on each screen. At the starting position (the black «×»i nF i g s .2 and 3), the horizontal field of view is approximately 150˚and the vertical field of view approximately 70˚.

Graphical resolution [START_REF] Thompson | Does the quality of the computer graphics matter when judging distances in visually immersive environments?[END_REF][START_REF] Kunz | Revisiting the effect of quality of graphics on distance judgments in virtual environments: A comparison of verbal reports and blind walking[END_REF] and field of view [START_REF] Creem-Regehr | The influence of restricted viewing conditions on egocentric distance perception: Implications for real and virtual indoor environments[END_REF] have been shown not to have an influence on visual distance perception.

Spatial audio rendering is realised using wave field synthesis [START_REF] Berkhout | Acoustic control by wave field synthesis[END_REF]. This technology allows one to recreate physically the sound field corresponding to a virtual source at any given position in the horizontal plane, without the need for tracking. The interloudspeaker distance of 21 cm leads to an aliasing frequency f al ≃ 2 kHz, above which the sound field is not spatially correctly reconstructed [START_REF] Corteel | Caractérisation et Extensions de la Wave Field Synthesis en conditions réelles[END_REF].

Furthermore, fine temporal and spatial calibration has been performed to ensure that the audio and visual renderings are coherent.

Audio-visual stimuli

The visual environment consists of an open, grassy field with a forest in the background (see Fig. 1). The associated audio environment consists of the sound of wind in the trees accompanied by some birds songs in the distance (overall level of 36 dBA). Environmental sound level has been adjusted to be slightly above the background noise produced by the video-projectors (34 dBA).

The chosen visual target object is a footless 3D loudspeaker, approximately spherical, with a diameter of ≃ 30 cm (see Fig. 1). Its foot has been removed to avoid window violation when the object is displayed in front of the screen. The floating loudspeaker is positioned at a height of 1.6 m and shadows are displayed.

The associated audio target object is a 4 kHz lowpass filtered white noise with a 15 Hz amplitude modulation. White noise has been chosen in order to have a wide spectral content and to allow subjects to rely on numerous audio localisation cues. The 4 kHz low-pass filtering has been performed to limit energy above the aliasing frequency f al ≃ 2 kHz and thus to limit possible incoherent spatial cues. In practice, it has been shown in [START_REF] Corteel | Objective and subjective comparison of electrodynamic and map loudspeakers for wave field synthesis[END_REF] that even if not exact, spatial cues above f al were generally coherent with spatial cues below f al when using MAPs. This white noise has been modulated in amplitude to produce attack transients that are also useful in sound localisation. No room-effect (i.e. ground reflections) has been included. The sound level of the audio object corresponds to a monopole emitting 78 dB SPL when placed at 1 m and is thus well above the environmental sound level at each of the tested distances.

Audio and visual objects are always displayed coherently, i.e. at the same spatial position. In addition their visual size and audio level decrease naturally with distance. Participants are allowed to move within the rendering area. They can thus rely on all the cues naturally available in the corresponding real environment for distance estimations. These available audio-visual cues are summarised in Tab. II. The audio-visual background environment was active in all the rendering conditions. In the audio condition, the spatialised sound corresponding to the virtual object was played with no image of the virtual object. The only visual image consisted of the open, grassy field with a forest in the background. In the visual condition, the 3D image of the virtual object was displayed with no corresponding sound. The only audio signal consisted of the sound of wind in the trees accompanied by some bird songs. In the audio-visual condition, the spatialised sound of the virtual object was rendered with its corresponding 3D image and the audio-visual environment.

Experimental task

Distance estimation is performed here in two phases: an exploration phase (see Fig. 2) and a triangulation phase (see Fig. 3). Participants start each iteration at the start position, indicated by a black «×»i n Figs. 2 and3. Virtual objects are displayed on the right screen of the SMART-I 2 platform and their real distance d r is relative to the start position (see Fig. 2).

During the exploration phase, subjects can move in the exploration area which is a rectangle of 1×0.8 m 2 . During this phase, participants are instructed to move in the exploration area to acquire "a good mental representation of the object and of its environment". This exploration phase is depicted in Fig. 2. Once a "a good mental representation" has been acquired, the perceived distance d p is estimated by means of triangulated blind walking [START_REF] Klein | Measurement protocols for medium-field distance perception in large-screen immersive displays[END_REF][START_REF] Fukusima | Visual perception of egocentric distance as assessed by triangulation[END_REF][START_REF] Loomis | Assessing auditory distance per-ception using perceptually directed action[END_REF][START_REF] Loomis | Virtual and Adaptive Environments: Applications, Implications, and Human Performance Issues[END_REF][START_REF] Thompson | Does the quality of the computer graphics matter when judging distances in visually immersive environments?[END_REF]. During the triangulation phase, depicted in Fig. 3, participants close their eyes, make a 40˚right-turn and walk blindly for an imposed distance of ≃ 2 m. A handrail guide has been included to help during blindwalking. Subjects stop at the end of the guide, turn in the direction where they thought the object is, and make a step forward in that direction. Participants are told that the perceived distance will be calculated from this step. They then indicate that they have completed a trial by pressing a wiimote button. After that they can open their eyes and go back to the start position for a new iteration. The experimental protocol is fully automated.

Results

Because of large differences observed relative to other participants of the mean estimated distances, data from 2 subjects have been removed from the analysis. Results from the 13 remaining participants are shown in this section.

Analysis of perceived distances

Results for the dependant variable d p , i.e. the perceived distance, are given in Tab. III and shown in Fig. 5 for each condition. On average, participants performed the exploration task in 12.2 s ± 6.6 sa n d the triangulation task in 8.6 s ± 2.3 s. As expected, perceptual compressions are observed in the different rendering conditions.

The perceived distances obtained in the present repeated-measures design were analysed using a repeated-measures two-way analysis of variance. There were two within-subjects factors: rendered distance d r (5 levels, fixed factor) and rendering condition (3 levels, fixed factor). Rendered distance d r was significant at the 5 % level with F (4, 40) = 89 and p<10 -4 . Surprisingly, rendering condition was not significant at the 5 % level as F (2, 20) = 1.98 and p =16.5 %. After the effects of individual differences

Discussion

Modality precision

Azimuth judgements based on the visual modality are much more accurate than azimuth judgements based on the audio modality. This is not the case for distance judgements. A Kruskal-Wallis test performed on the standard deviations given in Tab. III with rendering condition as an independent factor gives χ 2 (2, 12) = 2.66 and p =26.5 %. Standard deviations for the audio, visual,a n daudio-visual perceived distances are thus not significantly different. This is coherent with results from [START_REF] Loomis | Assessing auditory distance per-ception using perceptually directed action[END_REF] where real-world distance are estimated using blind direct walking under audioonly and visual-only conditions. The audio modality is therefore as precise as the visual modality for the perception of distances in virtual environments when rendered distances are between 1.5 ma n d5 m.

Limits of visual capture theory

The visual capture theory, already demonstrated in [START_REF] Gardner | Proximity image effect in sound localization[END_REF][START_REF] Mershon | Visual capture in auditory distance perception: proximity image effect reconsidered[END_REF][START_REF] Mershon | Perceived loudness and visually-determined auditory distance[END_REF][START_REF] Zahorik | Estimating sound source distance with and without vision[END_REF][START_REF] Bowen | Visual Localization Accuracy Determines the Bias of Auditory Targets in Azimuth and Depth[END_REF] for audio-visual distance perception, suggests that the visual modality is the more reliable and dominates the audio modality for localisation tasks. In the hypothesis that the audio modality is as precise as the visual modality (see Sec. 4.1), doubt is cast on the validity of such a theory. However, the present results are not sufficient to conclude on this point. Additional studies involving differences between the audio and visual rendered distances (incoherent stimuli presentation) are needed to clearly investigate the limits of the visual capture theory.

Virtual is not real

Examining Tab. III and Fig. 5, it can be seen that the mean values in the audio, visual,andaudio-visual conditions are not significantly different. This has been assessed by a repeated-measures 2-way ANOVA which found no significative differences between the different rendering conditions. We can form the hypothesis that the perceived distance is, in the virtual environment, thesameintheaudio, visual,andaudio-visual rendering conditions (same compression curve in Fig. 5). We note that this hypothesis is not true in real environments [START_REF] Wiest | Stevens exponent for psychophysical scaling of perceived, remembered, and inferred distance[END_REF][START_REF] Zahorik | Auditory distance perception in humans: A summary of past and present research[END_REF]. We can thus wonder for what reason visual and audio cues are treated the same way in virtual environments while being treated differently in real environments?

Influence of the physical rendering setup

A final discussion point in the results presented in Fig. 5 is the point at which rendered distance equals perceived compressed audio, visual,o raudio-visual distances. This point is situated here around 2.3 m. This distance is the distance at which is effectively positioned the user from the right LaMAP, i.e. the flat array of loudspeakers used as a front projection screen (see Sec. 2.2). When a virtual object is rendered at a distance of 2.3 m, its image and sound position are thus physically and perceptual ly close to the position of the LaMAP. The physical rendering setup thus seems to act here as an anchor between the real and virtual worlds.

  Dependent variablesPerceived distance d p (m)

Figure 1 .

 1 Figure 1. A participant and an audio-visual object in the virtual world. Visual rendering is front-projected on the front faces of the two LaMAPs which form a corner.

Figure 2 .

 2 Figure 2. Exploration phase. Start position:black«×». Exploration area: grey rectangle. Virtual object:b l a c kd i s k placed at a distance dr from the start position. The dotted line indicates a hypothetical exploration trajectory performed by the subject.

Figure 3 .

 3 Figure 3. Triangulation phase. Start position:b l a c k« ×». Guide: large continuous black lines. Virtual object:b l a c k disk placed at a distance dr from the start position.T h e dotted line indicates a hypothetical trajectory performed by the subject. Perceived object:g r e yd i s kp l a c e da tt h e estimated perceived distance dp from the start position.

3. 1 .

 1 Data treatment Position of the head of the participant (middle of the eyes) and corresponding times are recorded for each iteration during both the exploration and the triangulation phases. Examples of exploration and triangulation trajectories are shown in Fig. 4. Perceived distances are estimated from the triangulation trajectory as follows. A line y = ax+b is fitted to the trajectory points during the forward step. Afterwards, the estimated perceived distance is given by d p = -b/a.

Figure 4 .

 4 Figure 4. Example of exploration and triangulation trajectories (participant 2, iteration 15, audio condition, dr =1.5 m and dp =1.83 m). Exploration area: grey rectangle. Beginning position:biggrey«×». End of the guide: small grey «×». Virtual object:b l a c kd i s k .Perceived object:g r e yd i s k .Participant trajectories: solid black lines.

  Mean and standard deviation (in m) of the perceived distances dp as a function of rendered distance dr for each rendering condition (see Fig.5).

Figure 5 .

 5 Figure 5. Perceived distances dp as a function of rendered distance dr for each rendering condition (see Tab. III). «×,•,+»: Mean under each modality. Vertical lines: Standard deviation under each modality. For the sake of readability, results corresponding to the different rendering conditions have been slightly shifted.

Table I . Independent and dependent variables used in the experimental protocol

 I

Table III .

 III

		A	V	AV
	1.5 m	1.92 ±0.42 ±0.35 ±0.40 1.60 1.65
	2 m	2.14 ±0.53 ±0.37 ±0.29 2.03 2.01
	2.7 m	2.44 ±0.57 ±0.37 ±0.35 2.30 2.27
	3.5 m	2.60 ±0.53 ±0.41 ±0.69 2.42 2.59
	5 m	2.96 ±0.83 ±0.76 ±0.78 2.79 2.91

SMART-I

: Spatial Multi-user Audio-visual Real-Time Interactive Interface

Acknowledgements

The authors thank all the volunteers who took part in the experiment. Thomas Chartier and Philippe Cuvillier, currently students at the École Polytechnique (France) are also thanked for their help in the design and performance of preliminary tests. Finally, a special thanks is given to Matthieu Courgeon for the time he spent on visual rendering.


M Rébillat
email: rebillat@lms.polytechnique.fr

X Boutillon

Identification of elastic and damping properties of sandwich structures based on high resolution modal analysis of point measurements

A method is proposed to identify the mechanical properties of the skin and core materials of sandwich structures having heterogeneous cores. All the elastic coefficients and loss-factors that matter in the dynamics of such a panel in the thick-plate approximation are identified. To this end, experimental natural modes (i.e. eigenmodes of the damped system) are compared to the numerical modes of large sandwich panels (l x,y /h ≃ 80). The chosen generic model for the visco-elastic behaviour of the materials is E(1 + jη). The numerical modes are computed by means of a Rayleigh-Ritz procedure and their dampings are predicted according to the visco-elastic model. The frequencies and dampings of the natural modes of the panel are estimated experimentally by means of a high-resolution modal analysis technique. An optimisation procedure yields the desired coefficients. A sensitivity analysis assess the reliability of the method. Identification is conducted on two very different kind of sandwich panels to illustrate the method.

Introduction

Because of their light weight and the easy adjustment of their mechanical properties, sandwich structures with an heterogeneous core are widely used nowadays. However, due to the heterogeneities, their structural mechanical properties are difficult to predict accurately on the basis of the material properties and identification procedures are often needed. Mixed numerical (Num) / experimental (XP) methods are used to identify the parameters of a model by comparing simulated and measured characteristics (for example modal dampings α Num n vs. α XP n and frequencies f Num n vs. f XP n of the first modes of the system). In order to obtain good identification results, the model parameters must be sensitive to the measured characteristics.

Several authors have addressed the problem of the identification of elastic and damping properties of homogeneous thick plates from full field measurements [START_REF] Pagnacco | Inverse strategies for the identification of elastic and viscoelastic material parameters using full-field measurements[END_REF][START_REF] Matter | Numerical-experimental identification of the elastic and damping properties in composite plates[END_REF][START_REF] Giraudeau | An alternative to modal analysis for material stiffness and damping identification from vibrating plates[END_REF] or of homogeneous thin plates using point measurements [START_REF] Devisscher | Identification of the damping properties of orthotropic composite materials using a mixed numerical experimental method[END_REF][START_REF] Talbot | The vibration damping of laminated plates[END_REF][START_REF] Mcintyre | On measuring the elastic and damping constants of orthotropic sheet materials[END_REF]. However no method has yet been presented for the identification of elastic and damping properties of thick plates from point measurements.

Sandwich panels often raise special difficulties because of their heterogeneous cores (honeycombs core for example). Equivalent elastic properties of such sandwich plates can be identified [START_REF] Bastos | Numerical and experimental approach for identifying elastic parameters in sandwich plates[END_REF][START_REF] Ichchou | Identification of effective sandwich structural properties via an inverse wave approach[END_REF], but obtaining their damping properties is still challenging. Following the work of De Visscher et al. [START_REF] Devisscher | Identification of the damping properties of orthotropic composite materials using a mixed numerical experimental method[END_REF] a point measurement method for the identification of the elastic and damping properties of sandwich panels with heterogeneous cores is presented in this paper, based on a thick plate approximation.

In order to consider the sandwich core as homogeneous in the in-plane directions, up to a given frequency f , the corresponding wavelength λ must contain at least 50 cells [START_REF] Burton | Assessment of continuum models for sandwich panel honeycomb cores[END_REF]. For a typical cell side-length s cell and height h this implies that the dimensions of a panel must be such that l x,y > λ > 50 s cell . On the other hand, the panel must appear as a thick-plate (rather than a thin-plate) if the out-of-plane elastic and damping properties are to be identified. For flexural waves this implies that high-enough frequencies are included in the processed data: λ/h ≤ 6 [START_REF] Cremer | Structure-Borne Sound: Structural Vibrations and Sound Radiation at Audio Frequencies Third Edition[END_REF]. In other words, the panel must be large enough and the observed dynamics must include high-enough modes, within the limit of a plate model. Due to the intrinsic dissipations of the materials, the modal characteristics of high modes may be difficult to measure with the Fourier transform (FT) which is limited to modal overlaps of ≃ 30 % in most implementations. The high-resolution modal analysis (HRMA) technique [START_REF] Ege | High-resolution modal analysis[END_REF] is an alternative to the FT for the estimation of modal parameters up to a modal overlap of ≃ 70 %.

In the present work, the identification of most elastic and damping properties of sandwich structures with an heterogeneous core is considered by means of the modal analysis of large panels (l x,y /h ≃ 80). The HRMA technique is used to estimate modal frequencies and dampings of the first ≃ 40 modes of the panels. An optimisation procedure, based on a numerical thick-plate model is used afterwards to identify the corresponding elastic and damping properties. The sandwich panel consists in two identical skins and a core (Fig. 1). The thicknesses of the core and the skins are h c and h s respectively. The thickness of the panel is h = h c + 2h s . In the following, "panel" designs the physical structure whereas "plate" refers to the idealized structure made out of the equivalent homogeneous material. The following hypotheses are made on the panel and plate:

A mechanical model of sandwich panels 2.1 Hypotheses

• Displacements are small so that the materials and structures behave linearly.

• Only flexural waves are considered.

• The plate is considered to follow the Reissner-Mindlin approximations (thick-plate model). For the aspect ratio l x,y /h > 80 of the panels under study, a more sophisticated theory such as "Third Order Shear Deformation Theory" would not give significantly different modal frequencies and dampings [START_REF] Soula | The effects of kinematic model approximations on natural frequencies and modal damping of laminated composite plates[END_REF].

• The wavelengths include at least 50 cells. According to Burton et al. [START_REF] Burton | Assessment of continuum models for sandwich panel honeycomb cores[END_REF], this ensures that errors on the modal frequencies of the plate (with a homogeneous equivalent core) are less than 2% when compared to those of the panel as computed by various FE-models.

The skin and core materials are each considered as homogeneous, orthotropic in the x and y directions, and viscoelastic.

The formalism chosen for describing the viscoelastic behaviour is that of complex moduli E = E( The following hypotheses are made on the sandwich panel:

• The sandwich panel is symmetric with respect to its mid-plane.

• Skins are very thin compared to the core so that shear stress in the skin can be ignored:

h s G s xz ≪ h c G c xz
(and the same in the y direction).

• The core is considered to be very soft

(E c x ≪ E s x , E c y ≪ E s y and G c xy ≪ G s xy ).
Given the generic expression of the moduli of the homogeneous equivalent material

E H = h c h 3 E c +        1 - h c h 3        E s ,
h s /h c ≃ 10 -1 E c x /E s x ≃ E c y /E s y ≃ G c xy /G s xy ≃ 10 -5 (1)

Model of the materials

The damping of plate vibrations has different origins. In the present study, it is assumed that panels vibrate below their coincidence acoustical frequencies. Consequently, damping due to acoustical radiation in surrounding air is very small compared to the structural damping. Among the different structural damping models, we have retained the standard hysteretic model (which is frequency-independent, see for example [START_REF] Muravskii | On frequency independent damping[END_REF]). The relationship between the stress ǫ γ and the strain σ γ in each γ-material (γ = s, c, or H) involves 7 complex numbers and can be written, to first order in η as:

σ γ =                      E γ x (1 + jη γ x ) ν γ yx E γ x [1 + j(η γ ν yx + η γ x )] 0 0 0 ν γ xy E γ y [1 + j(η γ ν xy + η γ y )] E γ y (1 + jη γ y ) 0 0 0 0 0 G γ xz (1 + jη γ xz ) 0 0 0 0 0 G γ yz (1 + jη γ yz ) 0 0 0 0 0 G γ xy (1 + jη γ xy )                      ǫ γ (2)
The symmetry of the strain/stress relation adds the following relationships

ν γ xy E γ y = ν γ yx E γ x and η γ ν xy + η γ y = η γ ν yx + η γ
x which leaves 12 independent real parameters to be identified for each material (24 altogether). In order to keep a formal symmetry in the mathematical treatment, one defines 1 :

ν γ = ν γ xy ν γ yx η γ ν = η γ ν xy + η γ y (3)
1 One must keep in mind that η γ ν is not the imaginary part of ν γ .

Equivalent thick plate

Under the hypothesis and for the orders of magnitude given in section 2.1, the sandwich panel behaves in the low frequency range like a homogeneous thick-plate [START_REF] Nilsson | Prediction and measurement of some dynamic properties of sandwich structures with honeycomb and foam cores[END_REF]. The thickness of the plate is chosen to be h. Its mechanical properties are given in Eq. ( 4) and ( 5) as functions of the mechanical and geometrical properties of the skins and the core.

                 E H x = E s x        1 - h c h 3        E H y = E s y        1 - h c h 3        ν H = ν s G H xy = G s xy        1 - h c h 3        G H xz = G c xz G H yz = G c yz (4)                  η H x = η c x E c x E s x h c h 3 + η s x        1 - h c h 3        η H y = η c y E c y E s y h c h 3 + η s y        1 - h c h 3        η H xy = η c xy G c xy G s xy h c h 3 + η s xy        1 - h c h 3        η H xz = η c xz η H yz = η c yz η H ν = η s ν (5)
The 12 independent real parameters

{E H x , η H x , E H y , η H y , G H xy , η H xy , G H xz , η H xz , G H yz , η H yz , ν H , η H ν }
are to be identified. Their knowledge yields the elastic and damping properties of each layer of the sandwich panel provided that the 12-equation system formed by Eqs. ( 4) and ( 5) is invertible. A sufficient condition is: ). This condition is not satisfied only if the η c -coefficients are several orders of magnitude larger than the η s -ones. This is not the case here and rarely the case in general 2 . Consequently, the identification of the E H x , etc . . . yields a measurement of the mechanical properties of the skin and core materials.

η c x E c x E s x ≪ η s x η c y E c y E s y ≪ η s y η c xy G c xy G s xy ≪ η s xy (6) since E c x E s x ≪ 1, E c

Potential, kinetic and dissipated energies in the equivalent thick-plate

Within the frame of the first order Reissner-Mindlin theory the displacements {u, v, w} in the {x, y, z}-directions respectively are [START_REF] Ayorinde | Elastic constants of thick orthotropic composite plates[END_REF]:

u(x, y, z, t) = -zΦ x (x, y, t) v(x, y, z, t) = -zΦ y (x, y, t) w(x, y, z, t) = w 0 (x, y, t) (7)
The potential energy of the plate is:

U = 1 2 V (σ H ) T ǫ H dτ = 1 2 S D 1 ∂Φ x ∂x 2 + D 2 ∂Φ x ∂x ∂Φ y ∂y + D 3 ∂Φ y ∂y 2 + D 4        Φ 2 y -2Φ y ∂w 0 ∂y + ∂w 0 ∂y 2        + ... D 5        Φ 2 x -2Φ x ∂w 0 ∂x + ∂w 0 ∂x 2        + D 6        ∂Φ x ∂y 2 + 2 ∂Φ x ∂y ∂Φ y ∂x + ∂Φ y ∂x 2        dxdy (8)
2 It can be the case when skins are made of metal and the core is made of paper honeycombs or of viscoelastic foam.

PROCEEDINGS OF ISMA2010 INCLUDING USD2010

with

D 1 = E H x h 3 12(1 -ν xy ν yx ) D 2 = ν xy E H y h 3 6(1 -ν xy ν yx ) D 3 = E H y h 3 12(1 -ν xy ν yx ) D 4 = 2κ 2 yz hG yz D 5 = 2κ 2 xz hG xz D 6 = G yz h 3 6 (9)
The shear correction factors κ 2 yz and κ 2 xz account for the fact that Eq. ( 7) is an approximation: the Φ x,y coefficients depend lightly on z and sections of the plate do not remain plane in the flexural deformation. The values κ yz = κ xz = 1 have been chosen according to the recommendations of [START_REF] Birman | On the choice of shear correction factor in sandwich structures[END_REF] for sandwich panels.

By definition, the fraction of energy lost during one cycle is:

∆U = -T V (σ H ) T ∂ǫ H ∂t dτ dt ( 10
)
Based on section 2.2, ∆U can then be expressed as:

∆U = -π S η H x D 1 ∂Φ x ∂x 2 + η H ν D 2 ∂Φ x ∂x ∂Φ y ∂y + η H y D 3 ∂Φ y ∂y 2 + η H yz D 4        Φ 2 y -2Φ y ∂w 0 ∂y + ∂w 0 ∂y 2        + ... η H xz D 5        Φ 2 x -2Φ x ∂w 0 ∂x + ∂w 0 ∂x 2        + η H xy D 6        ∂Φ x ∂y 2 + 2 ∂Φ x ∂y ∂Φ y ∂x + ∂Φ y ∂x 2        dxdy (11)
The kinetic energy T of the system is given in Eq. ( 12) as a function of Φ x , Φ y , and w 0 . In this expression, ρ H is the density of the equivalent homogeneous thick plate which is given by hρ H = h c ρ c + 2h s ρ s .

T = ρ H ω 2 2 (V) u 2 + v 2 + w 2 dτ = ρ H ω 2 2 (S) h 3 12 (Φ 2 x + Φ 2 y ) + hw 2 0 dxdy (12)

Numerical model of the thick plate

In order to compare experimental results to numerical simulations, it is necessary to evaluate the damping factors of numerical modes. The dynamics of the panel is given by the hypotheses listed in section 2.1, the Eqs. (2), and the boundary conditions. Instead of a direct time-integration of the motion, we model here the damping of the numerical modes of the associated conservative system, under the hyptothesis of light damping. The problem consists in evaluating the relationships between the α Num n damping factors and the η H loss-factors.

Modal representation

The honeycomb sandwich panel is considered here as a non conservative system P NC with N degrees of freedom q = {q n } where q is any set of generalised displacements. The damping model presented in section 2.2 corresponds to viscous damping. Under this hypothesis, the equation of the free motion of P NC can be written as:

M q + C q + Kq = 0 ( 13
)
where M, C and K are the mass, damping, and stiffness matrices. In what follows, the modes of P NC are called natural modes. We also refer to the associated conservative system P C corresponding to C = 0. The modes of P C will be noted ξ n and associated with the normal frequencies f n .

If P NC is lightly damped, it can be shown [START_REF] Geradin | Mechanical Vibrations: Theory and Applications to Structural Dynamics[END_REF] that the natural modes are ξ n and the natural frequencies are f n + jα n to first order.

Let U NC n be the potential energy associated with the n th mode of P NC . It varies in time as exp(-2α n t) so that the energy lost by this mode during one cycle ∆U NC n is:

∆U NC n = -2 α n f n U NC n ( 14
)
Since P C and P NC have the same modes, U NC n = U C n . And since U C n = T C n , one obtains:

∆U NC n = -2 α n f n T C n ( 15
)
Once the modes ξ n of P C (and of P NC ) are known, Eqs. ( 8), [START_REF] Ege | High-resolution modal analysis[END_REF], and ( 14) yield the modal dampings α n of P NC .

Rayleigh-Ritz procedure for the conservative modes

A Rayleigh-Ritz procedure has been used to derive the mode shapes ξ Num n and the modal frequencies f Num n of P C . To this end, the generalised displacements Φ x (x, y), Φ y (x, y), and w 0 (x, y) have been projected on an orthonormal polynomial basis of order Q satisfying the free-free boundary conditions [START_REF] Bhat | Natural frequencies of rectangular-plates using characteristic orthogonal polynomials in Rayleigh-Ritz method[END_REF]:

Φ x (x, y) = i, j L i j p i (x)p j (y) Φ y (x, y) = i, j M i j p i (x)p j (y) w 0 (x, y) = i, j N i j p i (x)p j (y) (16)
This procedure generates a new set of generalised displacements L i j , M i j and N i j . The kinetic and potential energies T and U, defined in section 2.4, have been expressed explicitly as functions of the new coordinates. The Hamilton principle reads as:

∀(i, j) ∈ [0, Q -1] 2 : ∂(T -U) ∂L i j = 0 ∂(T -U) ∂M i j = 0 ∂(T -U) ∂N i j = 0 (17)
The above system of 3Q 2 linear equations can be re-written as [K -4π 2 f 2 M]ξ = 0, see also Eq. ( 13), where f is the eigenfrequency and ξ is the eigenvector of unknown coefficients L i j , M i j and N i j . The resolution of this eigenvalue problem gives a straightforward access to the modal frequencies f Num n and mode shapes ξ Num n of P C .

Derivation of α Num n

By introducing the numerical modes ξ Num n and frequencies f Num n found in section 3.2 in the energies expressions of section 2.4, the relations Eqs. ( 18) are obtained. The coefficients t n and u nk depend only on the geometry and mass parameters of the plate and on the modal shape ξ Num n . For the subscripts of η, {x, ν, y, yz, xz, xy} have been replaced by {1, 2, 3, 4, 5, 6}.

∀n ∈ [1, N] : U NC n = U C n = 6 k=1 D k u nk ∆U NC n = -π 6 k=1 η k D k u nk T C n = 4π 2 f 2 n t n (18)
Using relations Eqs. [START_REF] Bhat | Natural frequencies of rectangular-plates using characteristic orthogonal polynomials in Rayleigh-Ritz method[END_REF], the expression Eq. ( 19) of the modal dampings α Num n can be deduced from Eq. ( 15). One can notice than α n is a linear combination of the η k .

α n = f n ∆U NC n 2T C n = 6 k=1 η k D k u nk 4π f n t n (19)
4 Experimental study of sandwich panels

Experimental setup

Two different sandwich panels with heterogeneous cores have been studied experimentally. The first one is a rectangular lightweight honeycomb sandwich panel (Nomex R honeycombs core and paper skins). The second one is a sandwich sheet composed of two stainless steel face sheets and two bidirectionally corrugated steel layers form its core of 20 % relative density. Known parameters for each panels are given in Table 1.

Since l x,y /h ≃ 80 for the honeycombs core panel and l x,y /h ≃ 90 for the bidirectionally corrugated core one, they both satisfy the Reissner-Mindlin approximations (see Sec. 2.1). Panels were suspended by thin wires in order to ensure free-free boundary conditions. The honeycomb core panel has been acoustically excited by an electro-dynamical loudspeaker placed in its vicinity and driven by a wide-band electrical signal. Its response was measured with a laser vibrometer pointing one corner. This ensures that all modes are present in the response. By means of a specially designed excitation signal [START_REF] Rébillat | Prediction of harmonic distortion generated by electro-dynamic loudspeakers using cascade of Hammerstein models[END_REF], the impulse response of the panel was reconstructed.

High resolution modal analysis

The impulse response of the non-conservative system can be expressed as a summation over its natural modes:

h(t) = N n=1 ξ n exp( j2π f n t -α n t + jφ n ) (20)
In order to extract the experimental modal frequencies f XP n and dampings α XP n , a recently developed modal analysis method [START_REF] Ege | High-resolution modal analysis[END_REF] has been applied to velocity impulse responses of the sandwich panels obtained in section 4.1. In the available noise conditions, the parameters of the 45 first modes could be extracted for the honeycombs core panel and of the 38 first modes for the corrugated core panel. The modal overlap of the highest modes of the honeycombs core sandwich panel was ≃ 50 %, which is out of reach of traditional implementations of the Fourier transform, hence our need of the new method.

Using several bandpass filters associated with the ESPRIT and ESTER algorithms (see reference [START_REF] Ege | High-resolution modal analysis[END_REF] for details), it is shown that this method yields a precise estimation of f n and α n in presence of moderate noise: the modal frequencies f n can be estimated with a precision of ≃ 0.01 % and the modal dampings with a precision of ≃ 1 %. Moreover, this method allows for the identification of modal parameters of modes having a modal overlap up to 70 %. and the real systems are equal to first order (section 3.1), it is valid to find separately and successively the elastic constants and the loss factors.

Optimisation procedure

Estimation method

To first order, the modal frequencies depend only on the elastic constants of the homogeneous equivalent thick-plate model

{E H x , E H y , G H xy , G H xz , G H yz , ν H }.
Since this dependence is non-linear, a cost function C E is defined (Eq. ( 21)) and an optimisation procedure based on the gradient-method has been implemented.

C E = N n=1 f XP n -f Num n f XP n 2 (21)
It has been shown in section 3.3 that the damping coefficients {α Num n } n∈ [1,N] can be expressed as linear combinations of the {η H

x , η H y , η H xy , η H xz , η H yz , η H ν } loss factors. Therefore, the latter can be obtained by a simple least mean square method, with the constraint that loss factors remain positive.

Results for the honeycombs core sandwich panel

Identification of elastic and damping constants

The optimisation is performed on the 45 first modal frequencies and dampings obtained experimentally for this panel. The numerical model used a Q = 14-order basis which proved to ensure the convergence of the highest modes values. The identified visco-elastic parameters of the equivalent homogeneous plate are given in Tab. 2. The relative errors in modal frequencies and dampings are shown in Fig. 2. It can be seen that the agreement is very good for modal frequencies (mean absolute error of 1.8 %). The predicted modal dampings fit well the mean measured ones, but the difference is more important (mean absolute error of 10.2 %). 2: Identified parameters of the homogenised model corresponding to the honeycombs core sandwich panel. The coefficients in the two last columns are mutually related by the symmetry relationships (cf. section 2.2).

Sensitivity analysis

The sensitivities of the frequency values f n to the coefficients

{E H x , E H y , G H xy , G H xz , G H yz } are defined as S f n (X) = ∂ f n ∂X f n X -1
. They reflect the information contained in a modal frequency relatively to the elastic parameter X.

Results are presented in Fig. 3. Since the modal frequencies are very little sensitive to the Poisson coefficients compared to the other in-plane parameters, their sensitivities to these parameters have not been represented. As expected, it can be seen in Fig. 3a that modes of the form (0, i) or ( j, 0) convey a lot of information relatively to E x and E y respectively. Since the thick-plate model differs from the thin-plate model for the higher frequencies, it is normal that there is almost 10 times more information relative to G xz and to G yz in the higher modes than in the lower ones (Fig. 3b). The lower sensitivity of G xz to the modal frequencies than that of G yz is simply due to the aspect ratio of the plate (l x < l y ).

Results for the corrugated core sandwich panel

Identification of elastic and damping constants

The optimisation is performed on the 38 first modal frequencies and dampings obtained experimentally for this panel. The numerical model used a Q = 13-order basis which proved to ensure the convergence of the highest modes values. The identified visco-elastic parameters of the equivalent homogeneous plate are given in Tab. 3. The relative errors in modal frequencies and dampings are shown in Fig. 4. It can be seen that the agreement is satisfying for modal frequencies (mean absolute error of 2.2 %). The predicted modal dampings fit correctly the mean measured ones, but as previously the difference is more important (mean absolute error of 36 %). In Fig. 4, it can be seen that the error on damping estimation is moderate in the mid-frequency range, but larger for the first modes and in the high-frequency range. As the identified loss factors are very low (see Tab. 3), the measured modal dampings are very sensitive to the way the panel is suspended and to the damping due to acoustical radiation. In the present case, the thin wires used to suspend the panel may be the origin of some additional damping in the low frequency range that was not included in the model. In the high-frequency range, the damping is systematically underestimated above 3 kHz. Since the equivalent elastic parameters of the panel have been identified, the coincidence frequency of this panel can be estimated around 4 kHz. This results in an additional damping due to acoustical radiation as the modal frequency comes closer to this frequency. In the same spirit, the modal frequencies seem systematically underestimated by 2 % below 3 kHz. In order of magnitude, this is consistent with air loading in the low frequency range.

Sensitivity analysis

Results of the sensitivity analysis are presented in Fig. 5. Since the modal frequencies are almost insensitive to the Poisson coefficients, their sensitivities to these parameters have not been represented. For this panel it can be seen that the sensitivity to the out-of-plane properties is very low compared to the sensitivity relative to the in-plane properties. This means that the modal frequencies and modal dampings are very little influenced by the out-of-plane complex moduli. As a consequence the estimations of these parameters are to be interpreted very carefully. In this case, even if the plate fulfills the hypothesis of the Reissner-Mindlin model (i.e. l x,y /h ≃ 90), the core material has too high out-of-plane shear moduli to allow for their precise identification.

Conclusion

An identification method that yields all the mechanical parameters of sandwich materials that matter dynamically, under only mild hypotheses has been presented. The performances of the method have been illustrated successfully on two different sandwich panels having heterogeneous cores: paper honeycombs core and steel corrugated core.

Compared to the methods proposed in Refs. [START_REF] Bastos | Numerical and experimental approach for identifying elastic parameters in sandwich plates[END_REF][START_REF] Ichchou | Identification of effective sandwich structural properties via an inverse wave approach[END_REF], this method also provides loss factors. Compared to the methods presented in Refs. [START_REF] Devisscher | Identification of the damping properties of orthotropic composite materials using a mixed numerical experimental method[END_REF][START_REF] Talbot | The vibration damping of laminated plates[END_REF][START_REF] Mcintyre | On measuring the elastic and damping constants of orthotropic sheet materials[END_REF], out-of-plane complex moduli of the materials are also extracted. Compared to the method proposed in Ref. [START_REF] Pagnacco | Inverse strategies for the identification of elastic and viscoelastic material parameters using full-field measurements[END_REF][START_REF] Matter | Numerical-experimental identification of the elastic and damping properties in composite plates[END_REF][START_REF] Giraudeau | An alternative to modal analysis for material stiffness and damping identification from vibrating plates[END_REF], this method is considerably easier and faster to implement since only one vibrating point is to be measured. It also reaches frequency domains that are usually out of reach of the modal analyses based on the Fourier transform. Incidentally, the method presented here could be used to access the frequency-dependence of the loss factors by considering only modes in a given frequency range. Compared to the static investigations on each sandwich component, this dynamical method is nondestructive and the experimental test needs very little time. Avoiding heavy lab-equipment, it is a good candidate for industrial in-line process of quality control.

Figure 1 :

 1 Figure 1: Geometry of the sandwich plate.

≪ 1 (

 1 see last point in section 2.1

Figure 2 :

 2 Figure 2: Comparisons between measured and predicted modal frequencies (left) and dampings (right) for the honeycomb sandwich panel.

Figure 3 :

 3 Figure 3: Sensitivities of the mechanical parameters to the modal frequencies for the honeycombs core sandwich panel. In the figures, each column is the sum of the sensitivities of the different involved parameters. On the top of the column the number of nodal lines in the x (top) and y (bottom) directions of this mode is specified .

Figure 4 :

 4 Figure 4: Comparisons between measured and predicted modal frequencies (left) and dampings (right) for the corrugated core panel.

Figure 5 :

 5 Figure 5: Sensitivities of the mechanical parameters to the modal frequencies for the corrugated core sandwich panel. In the figures, each column is the sum of the sensitivities of the different involved parameters. On the top of the column the number of nodal lines in the x (top) and y (bottom) directions of this mode is specified .

  skins are denoted by the s index. The properties of the homogeneous material equivalent to the whole sandwich are denoted by the H index.

	1 + jη)
	which do not depend on the frequency (see the model of materials in section 2.2). The Young's and shear
	moduli and the Poisson coefficient of the core are E c x , E c y , E c z , G c xy , G c xz , G c yz , ν c yx , ν c xz , ν c yz and ν c xy . The same
	parameters for the

  this ensures that all in-plane stress in the plate are entirely due to those in the skins.

	According to these hypotheses, there is no stress associated with E c,s,H z ν c xy , ν c yx which are ignored in what follows. These hypotheses are generally fulfilled in common sandwich , ν c,s,H xz , ν c,s,H yz , G s xz , G s yz , E c x , E c y , G c xy ,
	panels. The typical orders of magnitude for the considered parameters in this kind of sandwich panels are:

Table 1 :

 1 The corrugated core panel has been excited at different positions by means of an impact hammer. The panel responses were measured with an accelerometer located in one corner. Impulse responses were obtained after deconvolution with the force signal. Honeycomb core panel 39.15 cm 59.10 cm 0.2 mm 4.88 mm 4 mm 37.8 kg/m 3 713 kg/m 3 Corrugated core panel 17.78 cm 22.86 cm 0.2 mm 1.48 mm 1 mm 2164 kg/m 3 7800 kg/m 3 Geometry and mass of each sandwich panels. The typical length of the core-cells is denoted s cell .

	l x	l y	h s	h c	s cell	ρ c	ρ s

Table 3 :

 3 Identified parameters of the homogenised model for the other sandwich panel. The coefficients in the two last columns are mutually related by the symmetry relationships (cf. section 2.2).

		E H x	E H y	G H xy	G H xz	G H yz	ν H xy	ν H yx
	Real part	116 GPa 102 GPa 46 GPa 78 GPa 163 GPa 0.32 0.28
	Loss factor (%)	0.12	0.15	0.1	0	0	0.1 0.13

Acknowledgements

Authors would like to thank Étienne Corteel, Camille Besse and Dirk Mohr for providing the different test samples that have been characterized in this paper.


Matthieu Courgeon

Marc Rébillat

Brian Katz

Céline Clavel

Jean-Claude Martin

Life-Sized Audiovisual Spatial Social Scenes with Multiple Characters: MARC & SMART-I2

With the increasing use of virtual characters in virtual and mixed reality settings, the coordination of realism in audiovisual rendering and expressive virtual characters becomes a key issue. In this paper we introduce a new system combining two systems for tackling the issue of realism and high quality in audiovisual rendering and life-sized expressive characters. The goal of the resulting SMART-MARC platform is to investigate the impact of realism on multiple levels: spatial audiovisual rendering of a scene, appearance and expressive behaviors of virtual characters. Potential interactive applications include mediated communication in virtual worlds, therapy, game, arts and elearning. Future experimental studies will focus on 3D audio/visual coherence, social perception and ecologically valid interaction scenes.

INTRODUCTION

With the increasing use of virtual characters in virtual and mixed reality settings, the coordination of realism in audiovisual rendering and expressive virtual characters becomes a key issue.

The advancement of immersive environments has separately produced systems with improved quality for 3D stereoscopic graphical rendering [START_REF] Froehlich | Implementing Multi-Viewer Stereo Displays[END_REF] and also for spatialized audio rendering [START_REF] Gerzon | Periphony: With-Height Sound Reproduction[END_REF][START_REF] Pulkki | Virtual sound source positioning using vector base amplitude panning[END_REF][START_REF] Berkhout | Acoustic Control By Wave Field Synthesis[END_REF]. Despite these advances, few combined modality systems of high quality have been developed (see [START_REF] Kuhlen | A true spatial sound system for CAVE-like displays using four loudspeakers[END_REF][START_REF] Springer | Combining Wave Field Synthesis and Multi-Viewer Stereo Displays[END_REF] for example). This can be mainly attributed to the different and stringent technical requirements that are needed to render each modality with a high degree of quality.

Similarly, virtual characters are now able to display facial expressions of emotions and postural expressions of interpersonal attitudes. Experiments involving virtual characters in immersive environments have highlighted presence as a key feature related to a user's perception of these virtual characters. The impact of realism of expressive virtual characters are also investigated in terms of appearance or behaviors. Yet, few platforms enable one to investigate both the realism and the interactivity of expressive characters at the same time.

In this paper we introduce a new system combining two systems for tackling the issue of realism and high quality in audiovisual rendering and expressive characters. SMART-I² (Spatial Multi-user Audio-visual Real-Time Interactive Interface) is a high quality 3D audio-visual immersive interactive rendering system [START_REF] Rébillat | SMART-I² "Spatial Multiuser Audio-visual Real-Time Interactive Interface[END_REF][START_REF] Rébillat | SMART-I²: Spatial Multi-users Audio-visual Real Time Interactive Interface, a broadcast application context[END_REF]. MARC (Multimodal Affective and Reactive Character) [START_REF] Courgeon | Impact of Expressive Wrinkles on Perception of a Virtual Character's Facial Expressions of Emotions[END_REF][START_REF] Courgeon | Appraising Emotional Events during a Real-time Interactive Game[END_REF][START_REF] Courgeon | User's Gestural Exploration Of Different Virtual Agents' Expressive Profiles[END_REF] is a platform for real-time affective interaction with multiple characters. The goal of the resulting SMART-MARC platform is to investigate the impact of realism on multiple levels: spatial audiovisual rendering of a scene, appearance and expressive behaviors of virtual characters. Potential interactive applications include mediated communication in virtual worlds, therapy, game, arts and elearning.

Section 2 describes related work. Section 3 details the integration of the SMART-I² and MARC. Section 4 explains the future experimental studies and applications that will be possible with this integrated platform.

RELATED WORK

2.1

Affective Virtual Characters Embodied conversational agents are computer-generated characters that aim to demonstrate the same properties as humans in face-to-face conversation, including the ability to produce and respond to verbal and nonverbal communication [START_REF] Cassell | Embodied Conversational Agents[END_REF]. They need to be able to express emotion in multiple modalities (e.g. face, gesture and voice). Expression of basic emotions using these modalities has been extensively studied for the past thirty years, progressively increasing realism and credibility of virtual agents [START_REF] Pasquariello | Greta: A Simple Facial Animation Engine[END_REF].

More recently, technology has evolved to a point where multiple characters can be displayed and rendered simultaneously. Simulating and evaluating social interactions with multiple characters is now a key challenge for such systems leading to recent prototypes and experimental studies.

Following the social inhibition and social facilitation theory, the mere presence of a virtual character was observed to have an impact on the achievement of a task by users; the presence of the virtual character improving user performance on simple tasks and decreasing performance for complex tasks [START_REF] Hayes | That Avatar Is Looking at Me! Social Inhibition in Virtual Worlds[END_REF].

Several studies have explored the coordination requirements in a small group of avatars using desktop PC configurations. The importance of simulating territorial behavior in a small group of conversing avatars was observed by [START_REF] Pedica | Spontaneous Avatar Behavior for Human Territoriality[END_REF]. They designed the "Populous platform" which simulates spatial behavior of small groups of virtual characters. Each character takes into account interpersonal distance according to social rules. Bodily and gaze attitudes of dominance and affiliation were also simulated in conversing avatars [START_REF] Gillies | A model of Interpersonal Attitude and posture generation[END_REF]. Dialog turn-taking management was simulated using up to twelve virtual characters. The Ymir Turn Taking Model is a cognitive model of multimodal realtime turntaking that has been tested in numerous two-party dialogue systems. It was recently extended to manage multiple character scenes in real-time [START_REF] Thórisson | A Multiparty Multimodal Architecture for Realtime Turntaking[END_REF].

Several studies displayed virtual characters in immersive and mixed reality set-up. The concept of presence is important in such a setup. Presence refers to experiencing the computer-generated environment rather than the actual physical local one [START_REF] Slater | Measuring presence: A response to the Witmer and Singer presence questionnaire[END_REF], experiencing the scene, and not the technology. Bailenson et al. extended this concept to social presence, occurring when virtual characters are displayed in a virtual environment [START_REF] Bailenson | Interpersonal Distance in Immersive Virtual Environments[END_REF]. The impact of a mixed reality display configuration on user behavior with a virtual human was also investigated. The virtual character was judged as more engaging, empathetic, pleasant, and natural while displayed by mixed reality on a 20 inch plasma TV, than rendered on a standard desktop screen. Despite these findings, few systems enable users to interact with multiple characters in an immersive set-up. This paper presents a new platform, displaying semi-realistic virtual characters using immersive audiovisual spatial rendering where modality coherence is a key design goal.

Audiovisual spatial rendering

In any audio-visual application, the sensation of immersion [START_REF] Blauert | Communication Acoustics[END_REF] and the intelligibility of the scenes [START_REF] Bruijn | Application of wave field synthesis in videoconferencing[END_REF], which are closely related to presence, depend highly on the spatial quality provided by both the audio and the visual renderings. It is thus important to find ways to fulfill the requirements for both audio and visual modalities, achieving a device which provides a perceptually coherent spatial audio-visual rendering over a large rendering area. Such a rendering device thus realizes a seamless transition between the real and virtual worlds. As so, a device which ideally renders a spatial audio-visual scene can be thought of as a "large open window through which the users experience the virtual world" [START_REF] Brooks | What's real about virtual reality?[END_REF].

The key concept is therefore to create this virtual audio-visual window through which a plausible virtual world is perceived. All the spatial properties of the audio-visual scene must then be accurately conveyed to the user for any position within the rendering area, including angular and distance perception of objects which should remain accurate throughout. The audiovisual window therefore ensures that both static and also dynamic localization cues, such as the motion parallax effect, are preserved for both auditory and visual virtual objects

SYSTEM

SMART-I²

The SMART-I² system [START_REF] Rébillat | SMART-I² "Spatial Multiuser Audio-visual Real-Time Interactive Interface[END_REF][START_REF] Rébillat | SMART-I²: Spatial Multi-users Audio-visual Real Time Interactive Interface, a broadcast application context[END_REF] is able to realize a high degree of 3D audio-visual integration with almost few compromises on either the audio or the graphics rendering quality. Moreover users can progress and move about within a large rendering area while still perceiving coherent audio-visual localization cues coming from the different virtual characters and their surrounding environment.

3.1.1

Sound rendering using Wave Field Synthesis Wave Field Synthesis (WFS) is a spatialized sound rendering technology which was first really developed at Delft University [START_REF] Berkhout | Acoustic Control By Wave Field Synthesis[END_REF]. It is an audio implementation of Huygen's principle which states that: "Every sound field emerging from one primary sound source can be reproduced by summing contributions of an infinite and continuous distribution of secondary sound sources". At the theoretical level, WFS allows one to synthesize a sound source at any given position. Implementations of WFS are simplified versions of this principle, typically using a linear array of equally spaced loudspeakers.

Figure 1: Illustration of sound rendering using WFS Figure 1 illustrates the principle of WFS. The violin on the left part is the primary source producing the target natural sound field. The linear array of secondary sound sources on the right produces, through summation of the contributions of each loudspeaker driven appropriately, a synthesized sound field equivalent to the original target field. The sound field of the virtual violin is synthesized, perceived by users in the reproduction area as emanating from the precise spatial location of the violin. Additional sound sources may be simultaneously synthesized through simple linear superposition.

Using this physical basis, different types of fundamental sound sources, or sound fields, can be synthesized (Figure 2). Plane waves represent sound objects situated far away from the immersion area and are perceived as coming from a constant angle, independent of listener position. Point sources represent sound objects near the immersion area. Point sources can be synthesized at positions behind or in front of the loudspeaker array. Such focused sources (i.e. point sources in front of the array) are perceived as being physically present in the immersion area. Sound rendering using WFS, due to its approach in physically recreating the entire field within a spatially large area, is not limited to a single user at a single location. The sound perspective, including parallax, is correct for every user in the immersion area, without the need for a tracking device.

As previously stated, practical WFS implementations are limited to a linear array and hence reproduction is optimized for the horizontal plane. Auditory perception is the most precise and more stable in the horizontal plane and therefore a more pertinent choice for array orientation. With this restriction, which reduces the required calculation power and audio hardware, the digital audio processing can be done with a latency of less than 5 ms. This is more than sufficient for real-time AV applications.

Visual rendering with tracked passive stereoscopy

To produce a 3D visual rendering, each eye of the user must see the scene from a slightly different point of view. One means of realizing this is to use light polarization properties to independently address each eye of the user. The user wears special polarized glasses for visual cross-talk cancellation. The graphic rendering should also be adapted to the user's head position in order to maintain the correct point of view. Using this approach, the 3D visual rendering is coherent regardless of the user's position in the viewing area. This technique is referred as tracked passive stereoscopy (TPS).

Integration using Large Multi-actuator Panels

The seamless integration of the two different technologies (WFS and TPS) is achieved through an innovative use of multi-actuator panels (MAPs) [START_REF] Scherer | Appraisal considered as a process of multilevel sequential checking[END_REF]. MAPs are stiff lightweight panels with multiple electro-mechanical exciters attached to the backside. For this project, a novel large dimension MAP (LaMAP) has been designed, (~5 m² with a 4/3 ratio) in order to provide sufficient surface area and size to be used as a projection screen in an immersive scenario. To accommodate polarized light projection, the front face of the panel has been treated with metallic paint designed to preserve light polarization. Due to the nature of the LaMAP design, screen displacements caused by acoustic vibrations are very small and are not visible with regards to the 3D video projection on the surface of the panel. Such a structure allows one to efficiently integrate a 3D visual rendering technology (TPS) and a spatialized sound rendering technology (WFS). Currently, the SMART-I² is made of two large MAPs of 2.6 m x 2 m which form a corner of stereoscopic screens and a 24 loudspeakers array (12 ch / screen). With this configuration, users can move within an immersion area of approximately 2.5 m x 2.5 m, as can be seen in Figure 3.

MARC MARC (Multimodal Affective and Reactive Characters

) is a framework for real-time affective interaction with multiple characters. Simulation of affective behaviors is achieved using implementations of several emotional models: categorical models of emotion [START_REF] Courgeon | Impact of Expressive Wrinkles on Perception of a Virtual Character's Facial Expressions of Emotions[END_REF], P.A.D. [START_REF] Russell | Evidence for a three-factor theory of emotions[END_REF] dimensional model [START_REF] Courgeon | User's Gestural Exploration Of Different Virtual Agents' Expressive Profiles[END_REF] and CPM [START_REF] Scherer | Appraisal considered as a process of multilevel sequential checking[END_REF] cognitive evaluation models [START_REF] Courgeon | Appraising Emotional Events during a Real-time Interactive Game[END_REF].

In addition to our research, MARC was applied to several domains, such as arts, or clinical application on autistic children [START_REF] Grynszpan | A new virtual environment paradigm for high functioning autism intended to help attentional disengagement in a social context', Special Issue of the internationnal[END_REF].

MARC features three main modules: facial expressions edition, body gesture edition, and real-time interactive rendering. MARC's real-time rendering relies on GPU programming (OpenGL/GLSL) to render detailed models and realistic skin lighting (shadow casting, BSSRDF: simulation of light diffusion though skin). Our facial animation system extends the MPEG-4 model [START_REF] Pandzic | MPEG-4 Facial Animation: The Standard, Implementation and Applications[END_REF] by adding wrinkle management (Figure 4). Key expressions are predefined as a set of keypoint displacements. The real-time rendering system performs animation by blending several key expressions. This technique also enables one to perform Action Unit based animation. Action Unit expressions are then considered as MPEG-4 key expressions. Manual Emotional Control Several techniques where explored to control the character's expressivity. The recognition software FaceReader is used to decode emotions expressed by a user's face. This signal is modulated by expressive profiles (High expressivity, Low expressivity, Negative, or Positive) and reproduced on the virtual character's face. (Figure 5 Top). Another approach is to use 3D devices, such as 3D mice or joystick, to control the emotional state of the agent, represented in a 3D space (Pleasure, Arousal, Dominance). Then, we apply the modulation using expressive profiles, and the resulting emotion is display on the virtual character's face (Figure 5 Bottom). Automatic emotional control We developed emotional reaction modules based on cognitive evaluation theories [START_REF] Scherer | Appraisal considered as a process of multilevel sequential checking[END_REF]. The virtual character appraises the game's events and situation according to several cognitive criteria (novelty, expectedness, pleasantness, goal conductivity, coping ability, external/internal causation, internal norms, and external norms). Figure 6 presents the interactive architecture. These modules were evaluated during a real-time interactive game with the virtual character in a non-immersive environment. Posture edition and animation Body animation is based on skeleton rigging (Figure 7). The body gesture editor enables to edit sequences of key poses. MARC features two ways to create posture animations: 1) the animation designer directly manipulates the bone positions and rotations to edit each posture composing the animation, 2) the sequence of postures is generated using a motion capture file (BVH format).

3.2.4

Real-time interaction MARC real-time rendering is controlled by external applications using a standard protocol, (Behavior Markup Language [START_REF] Vilhjalmsson | The behavior markup language: recent developments and challenges[END_REF]). It uses predefined key animations (both face and body) to create a dynamic interactive animation.

The MARC framework enables one to integrate virtual characters in various 3D environments. These environments can be edited using 3D software such as blender, and exported in X3D standard format. MARC reconstructs the scene using an environment file describing which objects are in the scene and where they are. Environment sounds can also be added, as WAV files with a corresponding spatial position (Figures 8 and9).

MARC currently provides three models, two adult male, and one adult female (Figure 10).

SMART-MARC: Integration of SMART-I² and MARC

The MARC and the SMART-I² systems were integrated in order to enable high quality spatial rendering of virtual social scenes. Specific network protocols where defined to exchange information such as tracking information, and sound localizations in the 3D virtual space. An overview of the architecture of the integrated system is shown in Figure 11.

Max/MSP patches compute information relative to sound spatialization, handle individual source audio through files or synthesis, and control the WFS engine. The WFS engine performs the real time signal processing necessary to spatialized the received audio sources which are rendered over the 24 loudspeaker array. MARC computes information relative to the visual modality, such as frustum deformation (adapting screen aspect to user location).

Due to this architecture, synchronization between Max/MSP and MARC is a crucial aspect for audio-visual congruence. To address this issue, we designed synchronization protocols. Loading and pre-computations are performed independently on both sides, and then "bang" signals trigger synchronized audiovisual animation. We introduced a new platform for high quality spatial rendering of virtual scenes featuring multiple virtual characters. The SMART-I2+MARC system is being used for experimental studies testing the influence of coherent audiovisual rendering and immersion on the perception the user has of multimodal expressions of emotions displayed by virtual characters.

Figure 2 :

 2 Figure 2: Synthesis of different wave types. Left: Point source behind the loudspeaker array. Center: Plane wave. Right: Point in front of the loudspeaker array

Figure 3 :

 3 Figure 3: An example of a simple audio-visual scene provided by the SMART-I²

Figure 4 .

 4 Figure 4. MARC facial edition and rendering

Figure 5 .

 5 Figure 5. Interaction for emotional control 3.2.2Automatic emotional control We developed emotional reaction modules based on cognitive evaluation theories[START_REF] Scherer | Appraisal considered as a process of multilevel sequential checking[END_REF]. The virtual character appraises the game's events and situation according to several cognitive criteria (novelty, expectedness, pleasantness, goal conductivity, coping ability, external/internal causation, internal norms, and external norms). Figure6presents the interactive architecture. These modules were evaluated during a real-time interactive game with the virtual character in a non-immersive environment.

Figure 6 .

 6 Figure 6. Architecture of the autonomous emotional reactions during a real-time game 3.2.3Posture edition and animation Body animation is based on skeleton rigging (Figure7). The body gesture editor enables to edit sequences of key poses. MARC features two ways to create posture animations: 1) the animation designer directly manipulates the bone positions and rotations to

Figure 7 .Figure 8 .Figure 9 .

 789 Figure 7. MARC's body posture editor

Figure 10 :

 10 Figure 10: The Three MARC's characters.

Figure 11 .

 11 Figure 11. Integration of MARC and SMART-I2

Figure 12 .

 12 Figure 12. The SMART-I² environment rendering two MARC characters

ACKNOWLEDGEMENTS

The SMART-I2 system is a joint project between LIMSI-CNRS and sonic emotion, with funding received through the Action Initiative program for immerging research at LIMSI-CNRS.