Arnault Nagle
email: arnault.nagle@orange-ftgroup.com

Nicolas Tsingos
email: nicolas.tsingos@sophia.inria.fr

Guillaume Lemaitre

Aurelien Sollaud

ON-THE-FLY AUDITORY MASKING FOR SCALABLE VOIP BRIDGES

Endpoints or conference servers of current audio-conferencing solutions use all the audio frames they receive in order to mix them into one final aggregate stream. However, at each time-instant, some of this content may not be audible due to auditory masking. Hence, sending corresponding frames through the network leads to a loss of bandwidth, while decoding them for mixing or spatial audio processing leads to increased processor load. In this paper, we propose a solution based on an efficient on-the-fly auditory masking evaluation. Our technique allows prioritizing audio frames in order to select only those audible for each connected client. We present results of quality tests showing the transparency of the algorithm. We describe its integration in a France Telecom audio conference server. Tests in a 3D game environment with spatialized chat capabilities show a 70% average reduction in required bandwidth, demonstrating the efficiency of our method.

1

INTRODUCTION

A significant number of VoIP systems are available, such as Microsoft Live Messenger (http://get.live.com/messenger/overview), Orange Link (http://orangelink.orange.fr/), Yahoo Messenger (http://fr.messenger.yahoo.com/), Skype (http://www.skype.com/intl/fr/), Teamspeak (http://www.goteamspeak.com/), which enable to create and manage audio chat sessions between remote participants. These systems have seen explosive growth in their usage over the last few years.

Three main configurations are usually used in VoIP audio conferences:

• centralized conferences with a mixing bridge (e.g., Orange Link ) that decodes the data, generates a suitable mix for each client and streams the final result, • loosely coupled conferences which includes multicast or multi-unicast conferences, • semi centralized conferences with the use of a forwarding bridge. We recall the role of a forwarding bridge in Figure 1. Each client sends one stream to the server and receives from it as many streams as there are remote participants. It must next decode, potentially process (e.g., spatial audio processing) and mix them for final restitution.

In most cases, however, audio conferencing systems do not integrate spatial audio restitution. Spatialized audio gives the listener the feeling of being in a real environment where the voice of each participant is coupled with its location. This location can be arbitrarily set by the software or the user, or e.g. tied to the position of a participant in a game. Although developers can use the capabilities of the 'Xbox Live' (www.xbox.com/live/) for spatialized chat, only Unreal Tournament 2004 (http://www.unrealtournament.com/) seems to use it to date. With the development of massively multiplayer on-line gaming, chat servers must also face the problem of dealing with an increasingly large number of simultaneous participants. This is particularly true for video game applications where an additional constraint is to send the speech data in separate streams to each participant to allow spatial audio processing prior to restitution. In this paper, we propose a novel approach in which we stream only the audible audio frames to each participant, depending on an energy importance criteria and a small set of additional audio descriptors computed for each frame. As such, our approach is independent of the coding strategy adopted for streaming the actual audio data.

In the context of a mixing bridge, the approach does not result in any bandwidth gain, except if there is no speaker, but it optimizes the decoding of audio frames prior to mixing. For the same reasons, it is equally useful on client terminals. However, the best use of this algorithm is obviously on the forwarding bridge in order to optimize bandwidth.

This paper presents the details of our masking evaluation technique, its performance and integration into a real-time gaming application, enhanced with spatialized chat capabilities.

In section 2, we present the masking algorithm and the results of an off-line perceptual quality evaluation test.

In section 3, we describe its integration in a VoIP bridge. The evaluation of this algorithm in a chatenabled game is presented in section 4. We finally discuss our approach and outline other possible improvements of our solution before concluding.

EFFICIENT MASKING EVALUATION

Our masking algorithm can be decomposed in two steps. First, a set of audio descriptors must be computed for each frame of input audio signal. Typically this will be done on the client and the data will be sent to the server as additional side information together with the coded speech signal. Next, the server performs an onthe-fly masking calculation for each connected client. Since this calculation needs to consider N-1 streams for each of the N connected clients, it must be efficient in order to scale well to large numbers of participants.

Computing audio descriptors

The first stage of our masking approach computes audio descriptors from which the subsequent real-time operations can be efficiently performed. For each frame of the input audio signal, we first compute the short-time Fourier Transform (STFT) of the audio data. For our off-line tests scenarios with 44.1 KHz signals, we used 1024 sample Hanning-windowed frames with 50 % overlap, resulting in 512 complex values in the frequency domain. Overlap is not mandatory and can be discarded to avoid additional delays in on-line applications. From the complex STFT, we then compute a number of additional descriptors:

• RMS level, including a spread-of-masking model [START_REF] Painter | Perceptual Coding of Digital Audio[END_REF][START_REF] Zwicker | Psychoacoustics: Facts and Model[END_REF], for a predefined set of i frequency bands (e.g., typically 4 to 8 bands on an octave or Bark scale), • Tonality T calculated as a spectral flatness measure [START_REF] Painter | Perceptual Coding of Digital Audio[END_REF]; tonality is a descriptor in [0,1] encoding the tonal (when close to 1) or noisy (when close to 0) nature of the signal. The descriptors can be seen as a compact representation of the signal, typically requiring a few additional kBytes of data per second of audio signals (e.g., 3kBytes/sec. at 44.1 kHz for 1024 sample frames with 50% overlap and 8 frequency bands). In a client-server setup, this calculation is performed by the client prior to sending the audio data and requires only minimal overhead.

2.2

Masking algorithm From the descriptors thus obtained we can efficiently evaluate which of the input signals are going to be audible in the final mixture at a given time frame. Signals that have been identified as inaudible can be safely removed from the pipeline reducing both the arithmetic operations to perform and network traffic. Since the calculation must be carried out at each processing frame, it must be very efficient so that it does not result in significant overhead. The masking algorithm is similar to the one presented in [START_REF] Tsingos | Perceptual Audio Rendering of Complex Virtual Environment {Proceedings of SIGGRAPH'04[END_REF][START_REF] Tsingos | Scalable Perceptual Mixing and Filtering of Audio Signals using an Augmented Spectral Representation[END_REF] and is illustrated in Figure 2. First, all input frames are sorted according to some importance metric. In [START_REF] Tsingos | Perceptual Audio Rendering of Complex Virtual Environment {Proceedings of SIGGRAPH'04[END_REF], a loudness metric was used but some of our recent experiments seem to indicate that the RMS level would perform equally well, if not better on average, due to specificities of some loudness models [START_REF] Gallo | Prioritizing audio signals for selective processing[END_REF]. If the signals must undergo filtering or equalization operations (e.g., distance attenuation for positional audio rendering), we dynamically weight the RMS level values pre-computed for several frequency-bands to account for the influence of the filtering operations in each band. We can then compute the importance as the sum of all weighted RMS values. Then, all signals are considered in decreasing importance order for addition to the final mixture according to the following pseudo-code: This process basically adds the level RMS k of each source to an estimate of the level of the final result in each band P mix (initially set to zero). Accordingly, it subtracts it from an estimate of the remaining level in each band P toGo (initially set to the sum of all RMS levels for all signals). The process stops when the estimated remaining level in each band is below a given threshold M mix from the estimated level of the final result. The process also stops if the remaining level is below the absolute threshold of hearing ATH [START_REF] Zwicker | Psychoacoustics: Facts and Model[END_REF]. Threshold M mix is adjusted according to the estimated tonality of the final result T mix , following rules similar to the ones used in perceptual audio coding [START_REF] Painter | Perceptual Coding of Digital Audio[END_REF]. In our applications, a constant conservative threshold of -27 dB also gave satisfying results indicating that precomputing and estimating tonality values might not be mandatory. Note that all operations must be performed for each frequency band, although we simplified the given pseudo-code for the sake of clarity (accordingly, all quantities should be interpreted as vectors whose dimension is the number of used frequency bands and all arithmetic operations as vector arithmetic). In particular, the process stops when the masking threshold is reached in all frequency bands.

Evaluation of the masking procedure

To verify the transparency of our masking evaluation, we conducted off-line quality evaluation tests to assess whether listeners aware of the algorithm principles are able to detect possible artifacts (e.g., over-masking, borderline signals rapidly switching status between masked and audible, etc.).

Experimental Conditions

21 subjects (17 men and 4 women) volunteered as listeners. They were aged from 23 to 40 years old with a median of 29 years old. All reported normal hearing. They all were computer scientists. Sound reproduction was done over Sennheiser HD600 headsets connected to a laptop computer.

The test stimuli consisted of several mixtures of various sounds. The sounds were chosen in six categories: music (separated tracks of two pieces of pop music, both instrumental and vocal), speech (male and female speakers, speaking English, French, Greek, German, and Polish), environmental noises (transportation noises, animal recordings, usual office furniture noises), mixed (speech, music, noises) and elements of the reverberation of an anechoic recording of percussions computed with the image source method. This latter category is made of several delayed copies of an anechoic recording. This category was chosen because we suspected that such sounds would be difficult cases for our algorithm. In each category, we created three mixtures of sounds, made with different number of sounds. To choose which and how many sounds we mixed, we defined three levels of "masking efficiency" ("low", "medium", and "high'') corresponding roughly to cases where respectively 30% to 80% of the signals were found to be masked and could be discarded in the final mixture. Each level was defined by the amount of signals actually removed by our masking algorithm. Hence, eighteen mixtures (six categories, three levels) were created. For each mixture, we created two versions: a reference mix containing all sounds, and a mix resulting from the output of the masking algorithm (i.e., a mixture of the audible sounds only). The mixtures were roughly equalized in loudness in a prior informal session. We ran a double-blind two-alternative forced-choice (2AFC) with hidden reference procedure: subjects were presented with an interface where three buttons were displayed. The button in the middle (labeled "reference") allowed subjects to listen to the original (unprocessed) sound. The two others (labeled "A" and "B") allowed subjects to listen again to the reference (hidden reference) or to the processed mixture. The mapping between A and B and the processed/unprocessed signals was randomized at each step. Neither the listener nor the experimenter was aware of the mapping (double-blind). Subjects were instructed that one of the two signals (A or B) was different from the reference. The subject had to indicate which one they perceived as different from the reference. They could continuously switch between the three sounds, or restart each at the beginning. They were also able to define portions of the signal for looping playback. Before the test, the algorithm was explained to the subjects and they were familiarized with some clear failures of the algorithm (we had to use an older version of the algorithm to find clear impairments). This procedure was designed to get the subjects trained to identify algorithm failures and to focus on parts of the sounds where the algorithm may provide artifacts. Our hypothesis was that despite these strict conditions, subjects would be unable to correctly identify the processed sounds.

Analysis and discussion

Averaged over all stimuli, the identification rates per subject ranged from 39 % to 78 % with a median of 55%. Due to the binary nature of the test, it was not possible to post-screen the subject consistency hence we used the responses of all subjects as a source of variation. A t-test [START_REF] Howell | Statistical methods for psychology[END_REF] was not able to reject the nullhypothesis: "over all the sounds, the identification rate is 50 % in the parent population" (df=18, t obs =0.13, p(t>t obs )>0.05). This indicates that subjects were overall unable to identify the processed sounds better than chance.

To examine each sound individually, we performed 18 Pearson χ 2 tests with one degree of freedom [START_REF] Howell | Statistical methods for psychology[END_REF] over the identification rates per sound (i.e. averaged over the subjects). The χ 2 hypothesis is "the identification rate is 50 %". As this test is repeated for each sound, we have to use a Bonferroni procedure for multiple comparisons [START_REF] Howell | Statistical methods for psychology[END_REF], which amounts to decreasing the threshold of significance: p<0.05/18. The results of the test are represented in Figure 3.

Identification rates range from 42.8 % to 66.7 % with a median of 52.4 %. Once again, the null hypothesis of the χ 2 test could not be rejected for any of the stimuli. We obtain the same results if we consider only listeners with a musical background. It was not possible to find any relationship between identification rates and type of sound or efficiency of the masking. Thus, we can conclude that signal degradations introduced by applying our masking algorithm to compute mixtures of sounds are statistically unnoticeable. However, during post-experimental interviews, several subjects reported to have found differences. Their descriptions were consistent with the expected artifacts (for instance, most of them have reported that reverberations were shorter than in the reference mixture, probably due to over-masking). Furthermore, one subject was able to precisely indicate some part of a mixture where he found a difference. Examination of the sound revealed that the algorithm had actually removed some parts of the mixture at this very location. This shows that some subjects are able to hear the differences introduced by the algorithm in some isolated cases which could not be uncovered by our test procedure. Hence, we can conclude that the masking algorithm is globally transparent, even if we can not exclude that a trained listener, listening carefully, may detect some localized artifacts.

INTEGRATION INTO A VOIP BRIDGE

In this section, we describe the integration of the masking algorithm into a VoIP bridge. Figure 4 illustrates the complete audio treatment chain, from the recording of the voice of one participant P1 to the listening of the spatialized sound of another one P4. Two other participants P2 and P3 are also part of the conference. In our final gaming application, we take into consideration the spatial position of the participants in the game. This is why the Virtools game client appears on the figure. However, it is not directly related to the masking procedure which would work similarly for a non-spatialized audio output. Each client-listener is processed as P4 and each clientspeaker is processed as P1. In the following, the different steps are presented from the client and server perspectives, but temporally occur in order 1 to 5.

3.1

Client Side

Steps 1 and 2

The sound is recorded with the microphone of participant P1, digitized and separated into time-frames (typically 960 samples at 16kHz). For each acquired frame, the client computes necessary audio descriptors (as described in Section 2.1). To avoid additional delay, no overlap was used when analyzing the input frames. Hence, audio descriptors are computed every 60 ms. Next, frames encoded by an audio coder (we used a France Telecom codec at 32 kbits/sec equivalent to standardized G.722.1), together with their side information and the position of the user given by the Virtools client, are multiplexed into a network packet and sent to the server.

Step 5

At the reception end, e.g., for participant P4, audio frames are decoded from audio packets and sent to the Virtools client for final mixing and restitution.

AES 30 th International Conference, Saariselkä, Finland, 2007 March 15-17 5

Server Side

Step 3

Each audio packet (e.g., originating from P1) arrives at the VoIP server which duplicates it and inserts it in the buffers of all the others participants. On this server, each client has a buffer queue where incoming packets are stored while awaiting transfer to their final destination.

When two packets of the same participant (e.g., P1) are present in a buffer of target participant (e.g., P4), the masking calculation is launched for this buffer. This treatment extracts perceptual descriptors of each packet available in the buffer except for the last received packet (e.g., the last one of P1). The masking algorithm, enables the selection of audible packets thanks to their associated descriptors and positions in the virtual environment (e.g., to account for distance attenuation). An order of importance is created and allows us to keep the audible packets at the output. In the situation illustrated by Figure 4, only two packets (those of P1 and P3) over three are audible for P4.

In order to avoid possible artifacts due to borderline signals rapidly alternating between audible and masked from one frame to the other, we implemented a smoothing function which remembers the two last results of the masking evaluation and decides to send the packets or not. Initially the algorithm does not send packets and it changes its state only if the current result of the algorithm and the last two are identical and opposed to the current state. A drawback of this approach is that it can erroneously discard the beginning of a sentence. Note that a packet can be heard by some participants but not by others. Hence, each masking evaluation is performed independently for each participant.

Step 4

Audio packets that have been accepted by the culling function are next sent to the participant (P4 in our case).

3.3

Implementation issues Some side effects might appear due to the recording quality on the various client platforms. In our case, tests were conducted with average consumer grade audio microphones soundcards and headphones. If no participant speaks at one time instant, the background noise can potentially be considered as a meaningful signal and will be transmitted. In fact, when there was no speaker, all the packets were sent because they all had roughly the same importance. In other cases, only packets with predominant background noise were sent. In those cases, we ideally do not want to transmit the data. However, the masking evaluation procedure does We first experimented with a fixed threshold to suppress the background noise frames but it was found to depend too much on the hardware. The implemented solution was to keep setting a very low threshold but to add a quality audio box on each sending terminal. This component located just after recording (e.g., see P1 on Figure 4) reduces noise and increases the audio level when speech is present, in order to help the masking algorithm. This noise reduction component contains:

• A high-pass filter with cut-off frequency set at 50Hz, due to the use of a wideband coder. • A Noise Reduction block,

• And an Automatic Gain Control block to equalize the level of the audio streams of each participant.

Alternate noise removal or gating strategies could be used to solve this problem [START_REF] Ephraim | Speech Enhancement Using a-Minimum Mean-Square Error Short-Time Spectral Amplitude Estimator[END_REF][START_REF] Mak | A robust speech/non-speech detection algorithm using time andfrequency-based features[END_REF][START_REF] Renevey | Entropy Based Voice Activity Detection in Very Noisy Conditions[END_REF][START_REF] Kang | Automatic gain control[END_REF] which is common in VoIP applications and not directly related to the proposed masking evaluation.

EVALUATION FOR IN-GAME 3D CHAT

In order to test our algorithm in a more realistic framework, we decided to integrate the France Telecom (FT) VoIP software into the game Flower Power Shooter (FPS) created by the company Virtools. The game FPS is available on "Virtools of Dassault Systèmes" website for test on http://www.virtools.com/applications/games-fps.asp. It is a multi-user game whose goal is to "shoot" the others, with paint-ball guns. Screenshots of in-game action are shown in Figure 5. Demonstration videos are available from: http://www-sop.inria.fr/reves/OPERA/videos.

The goal was to test our masking-based optimizations for in-game spatialized chat, which is likely to create simultaneous multi-talker dialogs.

Integration of the VoIP component in FPS

For audio conferencing applications, we saw that the best use of our masking algorithm is the case of a forwarding bridge.

Both FPS and FT VoIP software have client-server architectures. FT VoIP uses a forwarding bridge and its own signaling protocol to manage audio streams. Hence, we chose, as a preferred solution, to link the clients while keeping the two servers independent.

Experimental setup

In order to test the algorithm, we decided to play Flower Power Shooter using several configurations. All the participants had stereo headsets with microphones (e.g., Sennheiser HMD 280 pro). One platform was used as game server and another as VoIP forwarding bridge. Pilot tests were done for 3 to 5 simultaneous participants using WIFI and Ethernet connections. For each configuration, we used two test-modes: one where nobody talks and we can observe the effect of the algorithm only in the presence of background noise, and another where players play normally while talking to each other.

AES 30 th International Conference, Saariselkä, Finland, 2007 March 15- 17 7 We must highlight the fact that soundcards, headphones and microphones were not homogenous.

Results

Results are summarized in the following tables. "Frames normally sent" are frames which would have to be sent by a regular forwarding bridge, without our algorithm. If N participants are in the conference and one of them sends a frame, N-1 frames should be sent from the bridge to the others. Other ratios are based on this measure. "Frames accepted by the algorithm" correspond to the frames accepted by the masking algorithm but some of them can still be discarded as a result of the smoothing algorithm (see Section 3.3). First, we can see that during total silence, the algorithm enables a significant reduction of the output bandwidth. Of course, due to the quality of the audio hardware and its associated noise, some frames can still be sent because their energy level is more important. In this case, the bridge operates as a Discontinuous Transmission System. In periods of silence, talk or multi-talk, the reduction of output bandwidth is again quite important. We can observe the influence of the smoothing algorithm and notice that the percentage of accepted frames increases due to the augmentation of dialog possibilities. However, it does not tend to significantly grow with the number of participants. These results depend of course of the willingness of the players to chat, of the network, of the quality of the hardware, of the smoothing function, and of the location of the avatars.

DISCUSSION

The use of spatialized chat in a real-time environment depends of numerous factors such as the audio hardware, the network bandwidth, the noise rejection technique, the number of results memorized in the smoothing function of the masking procedure, the level of each speaker voice and his localization etc. All of them can affect audio quality but the obtained results are very positive. Due to positional audio rendering, all participants did not have the same restitution level. For instance, the sound of a distant participant is attenuated. As a result, intelligibility can be somewhat compromised but this is a gameplay issue that will probably receive further attention by game designers as spatialized chat capabilities become more widespread. For instance, allowing players to communicate over a walkie-talkie (intelligibility preserved but no spatial aspects) or through more physical positional audio (intelligibility can be compromised by obstacles, attenuation but direction of sound is perceived) could certainly be used to drastically modify gameplay for games where teamwork is required between participants. On the technical side, further improvements can be added to our system. We tested a Voice Activity Detection block (VAD) piloting a Discontinuous Transmission (DTX) in the VoIP system and found it to work quite well. However, the integration of the VAD and DTX blocks into the Virtools environment created some problems; hence our results are reported without these blocks. These tools would have helped the VoIP server selecting audible audio frames and reduce the problem highlighted in Section 3.3. Furthermore, a perceptually-based noise reduction model [START_REF] Tsoukalas | Speech enhancement based on audible noise suppression[END_REF] could be used instead of a simple energy-based model. Perceptual data and positions are currently just added in audio packets without trying to further compress them to reduce the bandwidth. For instance, audio frames of 60 ms in our case are coded with 240 bytes and 3D position with 12 bytes (one 32 bit-float for each dimension).

Moreover, in order to reduce the bandwidth from the server towards terminals, we could modify the audio packets with the goal of not transmitting the perceptual data and position. Assuming a fixed bandwidth from the VoIP server to terminals, the use of variable rate coder driven by the importance factor computed by the masking function would be an interesting extension. In fact, it will allow AES 30 th International Conference, Saariselkä, Finland, 2007 March 15- 17 8 the optimization of the available bandwidth in the spirit of [START_REF] Tsingos | Scalable Perceptual Mixing and Filtering of Audio Signals using an Augmented Spectral Representation[END_REF][START_REF] Kelly | The continuity illusion in virtual auditory space[END_REF][START_REF] Kelly | The continuity illusion revisited: coding of multiple concurrent sound sources[END_REF]. Streams output by a scalable variable rate coder can be cut anywhere and the audio quality depends on the length of the selected audio data. Hence, in this case, no decoding would have to be done to adapt the bandwidth, which remains consistent with the usual concept of a forwarding bridge.

In our application, we do not deal with sound effects coming from the game. Such sounds could mask or be masked by the on-line speech streams. A solution to this problem is to apply again our masking algorithm locally in each client in order to test speech signals against sound effects from the game. Our results using more general sound effects and additional masking tests presented in [START_REF] Tsingos | Perceptual Audio Rendering of Complex Virtual Environment {Proceedings of SIGGRAPH'04[END_REF] indicate that the technique would perform equally well in this case. Currently, the masking decision depends only on energy and tonality criteria. Others ways could certainly be explored to further prioritize the different streams, for instance the use of more perceptually oriented saliency metrics, as recently introduced in [START_REF] Kayser | Mechanisms for Allocating Auditory Attention: An Auditory Saliency Map[END_REF]. Further improvements could be added in the masking algorithm.

For instance, we could use a binaural version of this algorithm to better account for spatial audio effects, in a way similar to [START_REF] Tsingos | Perceptual Audio Rendering of Complex Virtual Environment {Proceedings of SIGGRAPH'04[END_REF]. However, designing a proper binaural masking model is still a challenging issue.

CONCLUSION

In this paper, we presented a novel approach to optimize bandwidth in the context of a voice over IP bridge for spatialized chat application. Our approach is based on an efficient on-the-fly auditory masking evaluation between all the signals generated by each participant. Masking evaluation is performed for all participants in turn so that only the audible audio frames are streamed to each client. We conducted a quality evaluation study showing that our masking algorithm yields transparent output when used to optimize mixing of a number of source signals.

In that context, our masking strategy can be used to typically remove between 20 and 80% of the original content while generating a perceptually-transparent mixture. Moreover, no delay is added by our algorithm, which is very useful in real-time application. We integrated this algorithm in a forwarding bridge and evaluated its performance in an in-game spatialized chat context. The algorithm was found to perform very well, by discarding almost 70% of the original data in our experiments. Its performance also seems to scale well with the number of participants although we could not conduct massive chat tests during this pilot study. We believe that such technology can be useful for massive chat applications, especially for future on-line games. Future work includes additional tests with larger numbers of participants and evaluation of a progressive streaming strategy based on the perceptual importance of each source signal.

Figure 1 :

 1 Figure 1: Illustration of the functionality of a forwarding bridge. This bridge receives a frame from a participant, duplicates and sends it towards all the other participants of the audio conference. E: Encoding / D: Decoding.

Figure 2 :

 2 Figure 2 : Overview of our masking algorithm

  AES 30 th International Conference, Saariselkä, Finland, 2007 March 15-17 4

Figure 3 :

 3 Figure 3 : Rates of identification of the processed mixtures (with masking) and results of the χ 2 test. Highlighted columns correspond to mixtures of speech signals.

Figure 4 :

 4 Figure 4: Overview of the VoIP /Virtools processing pipeline for a spatialized chat application.

Figure 5 :

 5 Figure 5 : Screenshots of the game Flower Power Shooter

Figure 6 :

 6 Figure 6 : Integration of the VoIP client in the Virtools client in order to provide some sound from each participant remote participant in the game This integration leads to the control of the VoIP client by the Virtools Client, as shown in Figure 6. When a player enters the game, the Virtools server informs the other Virtools clients already present the game of his arrival. The Virtools clients signal their VoIP clients to create a new incoming stream for the new participant. The creation of the conference on the VoIP server is done at the first demand from a VoIP client. Moreover, the Virtools client continuously informs its VoIP client of the position of the user in the virtual environment. The Virtools game client provides spatial audio rendering based on the location of each participant in the 3D game environment. The current implementation uses openAL hardware accelerated positional audio rendering (http://www.openal.org). Hence, upon reception, the VoIP client must feed its associated Virtools Client with audio frames received from the others participants.

Table 1 :

 1 Results for 3 participants in the game

		In period of	In period of silence
		silence	/ talk / multi-talk
	Frames normally	31163		169179
	sent
	Frames really	18	0%	27189	16%
	sent
	Frames accepted	208	0.6 % 37218	21.9%
	by the algorithm

Table 2 :

 2 Results for 4 participants in the game

		In period of	In period of silence
		silence		/ talk / multi-talk
	Frames normally	34528		182177
	sent
	Frames really	915	2.6%	30153	16.5%
	sent
	Frames accepted	2025	5.8% 41715	22.8%
	by the algorithm

Table 3 :

 3 Results for 5 participants in the game

		In period of	In period of silence
		silence		/ talk / multi-talk
	Frames normally	44161		201386
	sent
	Frames really	1000	2.2% 44628	22.1%
	sent
	Frames accepted	1679	3.8% 53831	26.73%
	by the algorithm

ACKNOWLEDGMENTS

This work was supported by the 2-year RNTL project OPERA, co-funded by the French Ministry of Research and Ministry of Industry. More information can be found at: http://www-sop.inria.fr/reves/OPERA.


Ka Tsingos

; Lefebvre

Sylvain

; Dachsbacher

Carsten

; Dellepiane
email: matteo.dellepiane@isti.cnr.it

EXTENDING GEOMETRICAL ACOUSTICS TO HIGHLY DETAILED ARCHITECTURAL ENVIRONMENTS PACS: 43.55

Geometrical acoustics (GA) is a widely used approximation for simulating sound propagation in virtual 3D environments. However, GA is a high-frequency approximation and therefore very detailed models, containing features small compared to audible wavelengths, may fall outside its validity domain. Including finer geometrical details might actually degrade the quality of the simulation, as supported by a number of previous studies. Furthermore, the cost of running GAbased simulations significantly increases with the geometrical complexity. In this paper, we propose an extension to GA for highly detailed environments. In particular, we pre-compute a representation of the scattering behavior off complex geometry using finite element techniques. We then use this representation within classical GA frameworks, such as radiosity or ray-casting, to compute impulse responses and auralize the corresponding acoustical effects.

INTRODUCTION

Geometrical acoustics (GA) is one of the most widespread approaches in architectural acoustic modeling. GA is a high-frequency approximation that models sound propagation along raypaths. The propagation paths can be constructed using techniques such as ray or beam tracing [START_REF] Lentz | Virtual reality system with integrated sound field simulation and reproduction[END_REF][START_REF] Funkhouser | A beam tracing approach to acoustic modeling for interactive virtual environments[END_REF] using spatial data structures (e.g., BSP trees) to maximize efficiency. However, the geometrical models used for architectural acoustic simulation tend to be of low complexity. Experimental studies have shown that low-resolution models might be more appropriate to evaluate acoustical criteria with GA approaches [START_REF] Rindel | Comparisons between computer simulations of room acoustical parameters and those measured in concert halls[END_REF], which is somewhat counter-intuitive. In interactive GA applications, the reflection of sound rays is usually modeled as purely specular assuming the size of the surfaces is large compared to the wavelength, which is often not the case in practice. In off-line acoustical simulations, additional lambertian or "glossy" reflections [START_REF] Iso | Acoustics -Sound-scattering properties of surfaces -Part 1: Measurement of the random-incidence scattering coefficient in a reverberation room[END_REF][START_REF] Christensen | A new scattering method that combines roughness and diffraction effects[END_REF][START_REF] Zeng | Practical methods to define scattering coefficients in a room acoustics computer model[END_REF][START_REF] Cox | A tutorial on scattering and diffusion coefficients for room acoustic surfaces[END_REF] have been classically used to approximate scattering off complex surfaces, which are usually replaced by a flat proxy geometry. Recent works have been devoted to level-ofdetail (LOD) approaches for GA [START_REF] Joslin | Significant fact retreival for real-time 3D sound rendering in complex virtual environments[END_REF][START_REF] Wang | Interactions of model detail level and scattering coefficients in room acoustic computer simulation[END_REF][START_REF] Siltanen | Geometry reduction in room acoustics modeling[END_REF] but to our knowledge no general simplification scheme that preserves the correct scattering properties has been proposed to date. The goal of our work is to compute scattering from very complex geometry, such as highly tesselated CAD-CAM models or those acquired through scanning techniques. We introduce a framework that can be used to approximate first-order scattering effects off arbitrary complex surfaces and pre-compute corresponding filters. These filters can be re-used within classical GA simulators. Our approach brings an alternative solution to the problem of surface-simplification. Below, we review existing approaches which can be used to improve classical GA models.

Edge diffraction models

GA simulations can be enhanced by introducing diffraction effects from wedges [START_REF] Mcnamara | Introduction to the Uniform Geometrical Theory of Diffraction[END_REF] in order to avoid audible discontinuities when the source or a strong reflection path becomes occluded from the receiver [START_REF] Tsingos | Modeling acoustics in virtual environments using the uniform theory of diffraction[END_REF]. As all GA models, the geometrical theory of diffraction (GTD) assumes edges to be large compared to the wavelength. Increasing geometrical complexity would imply using smaller primitives and eventually would fall outside the validity domain of GA. Thus, it is unclear how classical GA+GTD approaches could apply to more detailed scenes. Other approaches, such as the Biot-Tolstoy-Medwin model [START_REF] Svensson | Analytic secondary source model of edge diffraction impulse responses[END_REF][START_REF] Keiffer R | A time-domain rough surface scattering model based on wedge diffraction: Application to low-frequency backscattering from two-dimensional sea surfaces[END_REF][START_REF] Calamia | Fast time-domain edge-diffraction for interactive acoustic simulations[END_REF] can be used to accurately compute the impulse response due to scattering from finite-sized wedges in time-domain. However, due to its computational complexity it has found limited use in practical applications.

Finite element methods

Finite element methods (FEM) are numerical solutions to the wave (Helmholtz) equation and associated boundary conditions. They are classically solved in frequency domain by subdividing the environment into small elements (voxels) but alternative time-domain formulations can also be used [START_REF] Savioja | Simulation of room acoustics with a 3D finite difference mesh[END_REF]. Of special interest is the Green surface integral formulation. Using this formulation, the pressure solution P(R) to the Helmholtz equation can be obtained using an arbitrary surface Σ surrounding the receiver R [START_REF] Filippi | Acoustics, basic physics, theory and methods[END_REF]:

(1) where dS = ndS (n unit vector) and G(U,R) = -e ikr /4πr is the Green function corresponding to the propagation of a spherical wavefront in free-field (see Figure 1). Po(R) is the free-field pressure emitted by the source and the integral term, often called diffracted field, is the solution of the homogeneous Helmholtz equation associated with the boundary conditions on the surfaces of the environment. This surface integral is also called the Helmholtz-Kirchhoff integral theorem. The Green surface formulation serves as a basis for boundary element methods (BEM) techniques. In this case, Σ corresponds to the surfaces of the environment. BEM methods can account for full scattering effects in a unified way and are widely used to compute reference solutions. However, edge-length for surface elements must typically be smaller than 1/4th of the wavelength. This makes such approaches very time-consuming at high frequencies or for large-scale problems. They also require carefully-designed meshes with uniform elements to limit errors in the solution. They are thus hard to use with most 3D models, particularly those acquired through scanning.

The Kirchhoff approximation

The Kirchhoff approximation (KA) can be seen as a hybrid strategy between GA and wave acoustics [START_REF] Filippi | Acoustics, basic physics, theory and methods[END_REF]. It is based on Eq. 1 but imposes P(U) = Po(U) and ∇P(U) = ∇Po(U) on the "illuminated" side of the surfaces (visible to the source) and P(U) = ∇P(U) = 0 on the "shadowed" side. As a result, P(R) can be computed by a direct integration but surfaces facing away from the sound source will not contribute to the solution. Neglecting the contribution from occluded surfaces and higher-order scattering is the major source of error in the KA. As a result, the approximation will degrade at very low frequencies as the scattered component becomes more important in the regions occluded from the source. The KA will also lead to inaccurate results when second-order occlusions/reflections become prominent. Several studies have shown that it can introduce significant errors in the computed scattering from simple flat or randomly-rough surfaces when compared to reference solutions [START_REF] Jebsen | On the failure of the Kirchhoff assumption in backscatter[END_REF][START_REF] Thorsos | The validity of the Kirchhoff approximation for rough surface scattering using a Gaussian roughness spectrum[END_REF][START_REF] Norton | An evaluation of the Kirchhoff approximation in predicting the axial impulse response of hard and soft disks[END_REF][START_REF] Cox | Evaluations of methods for predicting the scattering from simple rigid panels[END_REF]. In particular, errors were found to be more important at grazing angles and near-field from the surface. However, some of these studies also used additional far field approximations which might also contribute to the observed errors. As stated in [START_REF] Thorsos | The validity of the Kirchhoff approximation for rough surface scattering using a Gaussian roughness spectrum[END_REF][START_REF] Embrechts | Determination of the scattering coefficient of random rough diffusing surfaces for room acoustics applications[END_REF], further work is still required to evaluate the validity of the KA which is still not well established even today. Despite these limitations, the KA has been widely used to solve scattering/occlusion problems in acoustics [START_REF] Sakurai | Sound reflections of a rigid plane panel and of the "live-end" composed by those panels[END_REF][START_REF] Ishimaru | Numerical simulation of the second-order Kirchhoff approximation from very rough surfaces and a study of backscattering enhancement[END_REF][START_REF] Cox | Evaluations of methods for predicting the scattering from simple rigid panels[END_REF][START_REF] Tsingos | Fast rendering of sound occlusion and diffraction effects for virtual acoustic environments[END_REF][START_REF] Embrechts | Simulation of first and second order scattering by rough surfaces with a sound-ray formalism[END_REF][START_REF] Embrechts | Determination of the scattering coefficient of random rough diffusing surfaces for room acoustics applications[END_REF]. In [START_REF] Sakurai | Sound reflections of a rigid plane panel and of the "live-end" composed by those panels[END_REF] it was shown that the KA can be used to simulate impulse response of first-order reflection/diffraction off rigid panels with good agreement to measured responses. In this paper, we evaluate whether the KA can be efficiently used to compute scattering effects off arbitrary complex geometries.

THE KIRCHOFF APPROXIMATION IN ARCHITECTURAL ACOUSTICS

To use the KA for complex architectural acoustics problems, we evaluate the surface integral on all geometry visible from the sound source. This integral can be computed in software using raycasting but it also maps very well to modern graphics hardware leading to a very efficient implementation. For more information and implementation details, we refer the reader to [START_REF] Tsingos | Instant Sound Scattering[END_REF]. To evaluate our approach, we computed scattering filters for large-scale real-world situations and compared them directly to corresponding recordings taken in the field. Example audio files can be found at: http://www-sop.inria.fr/reves/projects/InstantScattering. A first interesting example is the Kukulkan temple, a Maya staircase-pyramid located in Chichén Itzá, Mexico (Figure 2 left). The stairs of this pyramid act as a sound diffraction grating. They reflect a particular chirped echo which has been the object of a number of studies [START_REF] Declercq | A theoretical study of special acoustic effects caused by the staircase of the El Castillo pyramid at the maya ruins of Chichen-Itza in Mexico[END_REF][START_REF] Bilsen | Repetition pitch glide from the step pyramid at Chichen Itza[END_REF]. For additional information, please see http://www.ocasa.org/MayanPyramid.htm. We modeled a 856 polygons virtual replica of the pyramid, on which we applied our scattering algorithm. We used the same model for all frequencies without frequency-dependent adaptation of the tessellation. We computed a full-bandwidth (0-22KHz) transfer function in 2.45 Hz increments (8192 frequencies) in 92 sec. on a Pentium 4 3.4GHz workstation with a GeForce 8800 GTX graphics processor. We applied the obtained filter to the handclap used in the on-site recording available at the above URL. Figure 2 (right) compares the result of our simulation to the recording. Although the recording contains significant environmental noise, the comparison shows that our algorithm convincingly captures the chirped echo from the stairs.

Application to a complex range-scanned model

Our second, more challenging example models the scattering off the façade of the Duomo on the Piazza dei Miracoli in Pisa, Italy, also famous for its leaning bell-tower. We used a detailed model of the cathedral obtained from time-of-flight laser scanning (Figure 3 left) and containing 13 million triangles (a resolution of about 2 cm). Figure 3 (right) compares a simulation with an on-site recording of a handclap. The approach gives satisfying results although some components are missing, probably due to higher-order scattering or reflections from the ground (which was not acquired). The computing time for the solution is similar to the one of our pyramid example since we can calculate all geometrical information once (e.g., position of visible surface samples and normals) and re-use it for all frequencies. Creating the necessary renderings required only 2 additional seconds using our graphics hardware.

INTEGRATION WITH GEOMETRICAL ACOUSTICS ENGINES

In this section, we propose two solutions in order to integrate our KA-based approach into current GA simulators. First, the approach can be used to simplify complex geometry by introducing details as textures on a flat approximate geometry, as used in graphics rendering. Second, scattering filters from complex geometry can be pre-computed and stored in order to be directly used within classical GA frameworks.

th INTERNATIONAL CONGRESS ON ACOUSTICS -ICA2007MADRID

Scatter-preserving geometrical simplification

The surface integral approach of the KA offers the possibility to leverage level-of-detail schemes originally developed for graphics rendering [START_REF] Cohen | Appearance-preserving simplification[END_REF]. In this section, we propose a strategy combining normal mapping with displacement correction to model complex surface detail for acoustic scattering calculations. Displacement surfaces [START_REF] Cook R | The REYES image rendering architecture[END_REF] use textures to encode fine-grain surface detail which can be used at rendering time by a software ray-tracer or with the graphics hardware. To avoid costly ray-intersections with complex geometry, we propose a simpler approach that accounts for the correct propagation delay but omits accurate visibility calculations. To remove the contribution of back-facing fragments, which should not be contributing to the integral, we use an additional weighting term defined as -v.n, where v is the direction from the 3D location to the sound source and n is the surface normal. To evaluate our approach, we created several 4x4 meter surface samples from displacement textures. The amplitude of displacement was 0.5 meters. Figure 4 shows example surfaces and scattering impulse responses calculated with true displaced geometry. In this case, source and microphone were directly above the center of the surface respectively 10 and 20 meters away. We also created corresponding normal maps from the displaced geometry and performed a scattering calculation using a flat proxy surface with our displacement-corrected normal mapping and standard normal mapping only. Figure 4 compares the different results. As can be seen, normal maps without displacement correction result in very little difference compared to a flat surface. This demonstrates the importance of the interference phenomena which are paramount in modeling the proper scattering effect. Our displacement-corrected normal mapping results in a much better approximation. We also evaluated our simplification technique at oblique incidence. As can be seen in Figure 5, errors are more significant due to self occlusions but our approximation remains acceptable.

Using complex scattering filters with GA engines

We believe our approach can be used to enhance GA simulations with realistic pre-computed surface scattering functions or filters similar to Figures 5 and6. For instance, the filters could be convolved along the propagation paths obtained with an image-source/beam-tracing technique. However, such an approach would fail at modeling energy transfers in non-specular directions. Hence, the approach would probably better suit a radiosity framework to account for more diffuse transfer between surfaces. In this context, our approach could also be used to compute the impulse response of the form-factors when obstacles are present between surface patches.

CONCLUSIONS

In this paper, we proposed the first study aiming at computing early sound scattering off very detailed geometry as an alternative to using simplified models. We show that the Kirchhoff approximation (KA) is a good framework for the efficient implementation of a scattering integral which can be accelerated with commodity graphics hardware. Our approach shows for the first time that the KA can be used to compute scattering filters for detailed architectural models and offers a viable solution extending current GA techniques. It also provides the basis for geometrical simplification schemes preserving correct scattering properties of complex surfaces.

In the future, we plan to conduct further validation tests in order to assess the accuracy of our approach compared to traditional GA frameworks. Of special interest is also the perceptual evaluation of complex scattering effects in order to assess which level of accuracy is required for auralization. Although it remains an approximation to the physically-accurate solution, we believe that no other approach could currently offer a similar trade-off between accuracy and computing-time on our examples.

Figure 1 .

 1 Figure 1.-Notations for the Kirchhoff-Helmholtz theorem. S and R denote the source resp. receiver.

Figure 2 .

 2 Figure 2.-Left: Visualization of the scattering terms on the surface of a model of the Kukulkan temple for a 500Hz wave. The sound source is 15 meters in front of the stairs. Right: Comparison between spectrograms of a simulation and an on-site recording for the Kukulkan temple. The simulated response is convolved by the handclap of the original recording.

Figure 3 .

 3 Figure 3.-Left: A 3D model of the scanned façade of the Duomo in Pisa, Italy and close-ups on surface detail. Right: Comparison between spectrograms of a simulation and an on-site recording. The simulated response is convolved by the handclap of the original recording.

Figure 4 .

 4 Figure 4.-Comparison of true displaced geometry with a proxy flat quadrilateral enhanced with normalmap only or combined normal/displacement maps. Source and receiver are respectively 10 and 20 m directly above the center of the face. Note how the normal-map alone has little effect on the obtained response.

Figure 5 .

 5 Figure 5.-Comparison between displaced geometry and a combined normal+displacement map computed with the source and receiver respectively at 60 o and 45 o incidence relative to the surface normal and 5m away from the center of the face. The surface sample is 4x4m.

Figure 6 .

 6 Figure 6.-(a) Responses from different 4×4m surface samples. Each surface is composed of 131072 triangles and generated from displacement maps. Note the secondary scattering component due to the finite extent of the flat surface on the top row (green curve) and the increasingly "diffusing" nature of the surfaces from top to bottom. (b) Scattering patterns for a detailed surface. The figure compares sound pressure levels in a plane medial to the surface obtained by BEM and our approximation. Source is 5m directly above the center of the face and the pressure is plotted at a distance of 10m.

th INTERNATIONAL CONGRESS ON ACOUSTICS -ICA2007MADRID

Acknowledgements

This research was partially funded by the EU IST project CROSSMOD.


Nicolas Tsingos
email: nicolas.tsingos@sophia.inria.fr

PERCEPTUALLY-BASED AURALIZATION

Keywords: 43, 60, Dh

With increasingly complex environments, the cost of auralization can quickly become a significant bottleneck for interactive applications, such as video games or simulators. While limitations of the human auditory perception have been successfully leveraged for lossy audio compression, real-time auralization pipelines still implement brute-force processing, independent of the content to process and perceptive capabilites of a human listener. Several recent studies have shown that listeners are unlikely to perceive a complex auditory environment in its entirety. In this paper, we review a number of recent approaches which can be used to organize and simplify virtual auditory scenes, leveraging both the nature of the contents and limitations of human perception. Such approaches are independent of the spatial reproduction techniques and lead to scalable processing performance with minimal degradation in the perceived spatial and tonal quality. They can be applied to a variety of applications and allow for rendering highly complex environments at interactive speed.

INTRODUCTION

Handling 3D audio simulation is a key factor for creating convincing interactive virtual environments. The introduction of auditory cues associated to the different components of a virtual scene together with auditory feedback associated to the user interaction enhances the sense of immersion and presence [START_REF] Hendrix | The sense of presence within auditory virtual environments[END_REF][START_REF] Larsson | Better presence and performance in virtual environments by improved binaural sound rendering[END_REF]. Our spatial auditory perception will be solicited for localizing objects in direction and distance, discriminating between concurrent audio signals and analyzing spatial characteristics of the environment (indoor vs. outdoor contexts, size and materials of the room,…). Typical situations encountered in interactive applications such as video games and simulators require processing of hundreds or thousands of sources, which is several times over the capabilities of common audio dedicated hardware. The main computational bottlenecks are a per sound source cost, which relates to the different effects desired (various filtering processes, Doppler and source directivity simulation, etc.), and the cost of spatialization, which is related to the audio restitution format used (directional filtering, final mix of the different sources, reverberation, etc.). Although a scientifically authentic [START_REF] Lokki | A framework for evaluating virtual acoustic environments[END_REF][START_REF] Pellegrini | Quality assessment of auditory virtual environments[END_REF] result can be achieved through physical modeling of these steps, the processing of complex sound scenes, composed of numerous direct or indirect (reflections) sound sources, can take advantage of perceptually based optimizations in order to reduce both the necessary computer resources and the amount of audio data to be stored and processed. Several auditory perceptual properties may be exploited in order to simplify the rendering pipeline with limited impact on the overall perceived audio quality. The general approach is to structure the sound scene by (1) sorting the relative importance of its components, (2) distributing properly the computer resources on the different signal processing operations and (3) handling the spatial complexity of the scene (Figure 1). These techniques, derived from psycho-acoustics, perceptual audio-coding and auditory scene analysis introduce several concepts similar to those found in computer graphics: selective, progressive and scalable rendering (e.g., visibility/viewfrustum culling and geometrical/shading level-of-detail).

PRINCIPLES OF PERCEPTUALLY-BASED AURALIZATION

Masking and illusory continuity

Selective audio processing approaches build upon prior work from the field of perceptual audio coding that exploits auditory masking. When a large number of sources are present in the environment, it is very unlikely that all will be audible due to masking occurring in the human auditory system [START_REF] Moore | An introduction to the psychology of hearing[END_REF]. This masking mechanism has been successfully exploited in perceptual audio coding (PAC), such as the well known MPEG I Layer 3 (mp3) standard [START_REF] Painter | Perceptual coding of digital audio[END_REF] and several efficient computational models have been developed in this field. In the context of interactive applications, this approach is thus also linked to the illusion of continuity phenomena [START_REF] Kelly | The continuity illusion in virtual auditory space[END_REF], although current works do not generally include explicit models for this effect. This phenomenon is implicitly used together with masking to discard entire frames of original audio content without perceived artefacts or "holes" in the resulting mixtures. Importance and saliency of sound sources Evaluate all possible solutions to the optimization problem required for optimal rendering of a sound scene would be computationally untractable. An alternative is to use greedy approaches which first require estimating the relative importance of each sources in order to get a good starting point. A key aspect is also to be able to dynamically adapt to the content. Several metrics can be used for this purpose such as energy, loudness or the recently introduced saliency. Recent studies have compared some of these metrics showing that they might achieve different results depending on the nature of the signal (speech, music, ambient sound "textures"). Loudness has been found to be generally leading to better results while energy is a good compromise between complexity and quality.

Limitations of spatial hearing in complex soundscapes

Human spatial hearing limitations, as measured through perceivable distance and angular thresholds [START_REF] Begault | 3D Sound for Virtual Reality and Multimedia[END_REF] can be exploited for faster rendering independently of the subsequent signal processing operations. This is useful for applications where the reproduction format is not set in advance, Recent studies have also shown that our auditory localization is strongly affected in multi-source environments. Localisation performances decrease with increasing number of competing sources [START_REF] Brungart | Localization in the presence of multiple simultaneous sounds[END_REF] showing various effects such as pushing effect (the source localization is repelled from the masker) or pulling effects (the source localization is attracted by the masker) which depend on the time and frequency overlapping between the concurrent sources [START_REF] Best | Auditory spatial perception with sources overlapping in frequency and time[END_REF]. As a result, spatial simplification can probably be performed even more aggressively as the complexity of the scene, in particular the number of sound sources, grows.

TECHNIQUES FOR DYNAMIC MASKING OF SOUND SOURCES

Interactive applications bring several additional constraints since the masking does not concern a premixed audio stream but has to be evaluated between several concurrent sound signals in course of their processing. Since scenes are generally highly dynamic, masking thresholds have to be continuously predicted and updated according to the instantaneous characteristics of the source signals and their position in space relative to the listener. Dynamic masking of sound sources was introduced by Tsingos et al. [START_REF] Tsingos | Perceptual audio rendering of complex virtual environments[END_REF] in order to limit the spatial rendering only to audible sources. The method takes advantage of pre-computed signal characterisation (power spectrum distribution and tonality index [START_REF] Painter | Perceptual coding of digital audio[END_REF]) associated with each individual audio sample throughout its duration. At runtime, this information is accessed dynamically in order to predict the source's instantaneous loudness [START_REF] Moore | A model for the prediction of thresholds, loudness and partial loudness[END_REF] anticipating for subsequent frequency-dependent attenuation linked to the source directivity, source-listener distance and possible occlusion or scattering effects. At each frame, sources can thus be sorted according to their loudness contribution at the listener's ears and summed up until they mask the remaining sources which can then be simply discarded from the rendering pipeline.

SELECTIVE AND PROGRESSIVE SIGNAL PROCESSING

When large numbers of sound sources are still present in the scene after the auditory culling (masking) or for systems with limited processing power, per-source processing (e.g., studio-like effects [START_REF] Wand | Multi-resolution sound rendering[END_REF]; Doppler effect; distance, occlusion, reverberation filtering; sub-mixing, etc.) can still represent a strong bottleneck of the audio rendering pipeline. In recent years, several contributions were introduced that aim to bridge the gap between perceptual audio coding and audio processing in order to make audio signal processing pipelines more efficient. Fouad et al. [START_REF] Fouad | Perceptually based scheduling algorithms for real-time synthesis of complex sonic environments[END_REF] propose a level-of-detail progressive audio rendering approach in the time-domain; by processing every n th sample, artefacts are introduced at low budget levels. Wand and Straßer [START_REF] Wand | Multi-resolution sound rendering[END_REF] introduce an importance sampling strategy using random selection, but ignore the signal properties, thus potentially limiting the applicability of this method. A family of approaches proposed to directly process perceptually coded audio signals [START_REF] Lanciani | Psychoacousticallybased processing of mpeg-i layer 1-2 encoded signals[END_REF][START_REF] Darlington | Digital audio effects in the wavelet domain[END_REF][START_REF] Touimi | Efficient method for multiple compressed audio streams spatialization[END_REF] yields faster implementations than a full decode-process recode cycle. In the context of long FIR filtering for reverberation processing, the recent work by Lee et al. [START_REF] Lee | Perceptual convolution for reverberation[END_REF] shows that significant improvement can be obtained by estimating whether the result of the convolution is below hearing threshold, hence reducing the processing cost. Recent studies have proposed extensions of these approaches by concurrently prioritizing subparts of the original signals to process to guarantee a minimal degradation in the final result [START_REF] Kelly | The continuity illusion in virtual auditory space[END_REF][START_REF] Gallo | Prioritizing signals for selective realtime audio processing[END_REF][START_REF] Moeck | Progressive perceptual audio rendering of complex scenes[END_REF][START_REF] Tsingos | Scalable perceptual mixing and filtering of audio signals using an augmented spectral representation[END_REF] (see Figure 2). Key to such approaches is the choice of a signal representation that allows its progressive encoding and reconstruction. Tsingos et al. [START_REF] Tsingos | Scalable perceptual mixing and filtering of audio signals using an augmented spectral representation[END_REF] introduce a progressive signal processing technique where the coefficients of the short-time Fourier transform (STFT) of each signal are pre-computed and stored in decreasing modulus. In real-time during the simulation, the algorithm prioritizes the signals and allocates a number of coefficients to process for each source so that a predefined budget of operations is respected. Different perceptual metrics can be used to determine the cut-off point in the list of STFT coefficients, leading to different trade off between computer efficiency and perceptual quality (see Figure 3). Moeck et al. [START_REF] Moeck | Progressive perceptual audio rendering of complex scenes[END_REF], complements a loudness-based metrics with the tonality index [START_REF] Painter | Perceptual coding of digital audio[END_REF]. Loud tonal or speech signals, for which the STFT representation is sparser, will require fewer coefficients than a

th INTERNATIONAL CONGRESS ON ACOUSTICS -ICA2007MADRID

weaker noisier signal for transparent reconstruction. They compare the merits of this metrics with other models such as the audio saliency map proposed by Kayser et al [START_REF] Kayser | Mechanisms for allocating auditory attention: An auditory saliency map[END_REF]. The two metrics lead to very similar results. However, for intermediate budgets and for cases where both tonal and noisier signals are used, loudness-based prioritization improves perceived quality relative to the saliency-based alternative (Figure 3).

HIERARCHICAL CODING OF SPATIAL CUES IN SCENE-SPACE

One of the primary goals of spatial audio rendering is to reconstruct to the ears of the listeners the perceptual cues which are responsible for localizing sound in direction. The approaches described above can be used to simplify the signal processing operations required by specific 3D audio rendering algorithms. However, it is sometimes desirable to compress the spatial information of the scene in a way independent of the chosen reproduction technique to enable flexible reproduction. In this case, the coding/simplification of the spatial information must be performed in scene-space. This solution which is referred to as scalable spatial audio rendering can be divided into three categories: • Fixed basis functions/clustering: the first set of techniques encodes the spatial cues using a number of fixed basis functions or clusters. For instance, Ambisonics [START_REF] Malham | 3d sound spatialization using Ambisonic techniques[END_REF][START_REF] Daniel | Further study of sound field coding with higher order Ambisonics[END_REF] uses a spherical harmonics decomposition of the incoming sound pressure at the listening point. For binaural listening, there are approaches that decompose the HRTF onto a basis of eigen-filters corresponding to principal directions [START_REF] Larcher | Study and comparison of efficient methods for 3D audio spatialization based on linear decomposition of hrtf data[END_REF][START_REF] Jot | Approaches to binaural synthesis[END_REF], and approaches that operate in object space explicitly grouping neighboring sound sources belonging to the same cone of directions [START_REF] Herder | Optimization of sound spatialization resource management through clustering[END_REF], or using a hierarchical structure [START_REF] Wand | Multi-resolution sound rendering[END_REF].

• Dynamic per-object clustering: The clustering proposed by Sibbald is an object-based method [START_REF] Sibbald | MacroFX algorithms[END_REF]. Sound sources related to an object or an area are grouped according to their distance to the listener. In the near field, secondary sound sources are created and dynamically uncorrelated in order to improve the spatial sensation. In the far field, sources are clustered together, accelerating the spatial rendering. The drawback of the method is that the clustering is evaluated on a per-object basis and does not consider all the elements of the scene.

• Dynamic global clustering: Tsingos et al. [START_REF] Tsingos | Perceptual audio rendering of complex virtual environments[END_REF] introduced a dynamic source clustering method based on both the geometry of the scene and the signals emitted by each source (Figure 4). This is especially useful for scenes where sounds are frequently changing in time varying their shape, energy as well as in location. The algorithm flexibly allocates the required number of clusters; thus clusters are not wasted where they are not needed. The dynamic clustering is derived from the Hochbaum-Shmoys heuristic. The cost-function used for clustering combines instantaneous loudness, distance and angle. An equivalent signal for the cluster is then computed as a mixture of the signals of the clustered sound sources. A representative loudness-weighted centroid is used to spatialize the cluster according to the reproduction setup. This technique has been shown to lead to efficient rendering while maintaining very good rendering quality and minimal impact on localization-task performance, even with a small number of clusters.

APPLICATIONS Auralization for interactive virtual environments

Figure 1 illustrates the combined use of all previous techniques in the context of a 3D audio rendering engine for complex virtual environments, such as the ones found in video games or simulators. The set of signals for all sound sources are partially decoded so as to retrieve the descriptors and are first tested for masking. Audible sources are clustered and all per-source operations (or premixing) are performed using a progressive approach. Full or scalable decoding of the source signal can be delayed up to this stage, avoiding the cost of streaming/decompressing inaudible signals from the storage media. Finally the obtained signal from each cluster is spatialized using the location of its representative and mixed into the sound output. Note that the framework accommodates both primary sources and secondary sources arising from sound scattering off surfaces.

th INTERNATIONAL CONGRESS ON ACOUSTICS -ICA2007MADRID

Bandwidth management

The proposed dynamic masking strategy can also be used for bandwidth management, for instance to optimize voice streaming in voice-over-IP applications. In this case, the masking estimation is integrated into a forwarding bridge. Each participant sends coded frames of audio data together with additional descriptors (energy and tonality in a small set of frequency subbands) to the forwarding bridge. A dynamic masking procedure is performed for each participant and only the streams audible to each participant are sent downwards. This allows for optimizing bandwidth while maintaining flexible decoding on the client side. Details can be found in [START_REF] Nagle | On-the-fly Auditory Masking for Scalable VOIP Bridges[END_REF] and demos at the following URL: http://www-sop.inria.fr/reves/OPERA/

DISCUSSION AND CONCLUSIONS

Although the perceptually-motivated approaches use simple models rather than full binaural hearing models, they were found to work very well in practice, as supported by a number of perceptual studies in [START_REF] Gallo | Prioritizing signals for selective realtime audio processing[END_REF][START_REF] Moeck | Progressive perceptual audio rendering of complex scenes[END_REF][START_REF] Tsingos | Perceptual audio rendering of complex virtual environments[END_REF] and are already used in some commercial game applications [START_REF] Moeck | Progressive perceptual audio rendering of complex scenes[END_REF]. Their simplicity is key in providing a good trade-off between the time required for decisionmaking in order to optimize resources vs. the cost of the actual processing. Even with the increased computational power of recent multi-core processors, we believe scalable approaches are still likely to be required in order to tune computing resources or manage bandwidth. They make authoring easier by allowing sound designers to use as many sources as required by the application without thinking about rendering limitations. This is of particular interest for sources linked to physical simulations, such as contact sounds. A strong benefit of these approaches is also to be content-adaptive. An issue is the extra space required for storing additional sound descriptors which might become an issue on systems with limited memory. However, this information can be made quite compact (about 1 Kbyte per sec. of data) and required storage space should not be a strong limitation for a wide range of platforms. Several extensions could be made to the approaches. First, most approaches for masking or saliency do not account for the 3D location of the sources or use very crude approximations. Spatial unmasking effects inducing variations of masking thresholds due to the relative location of masker and maskee are likely to play a major role in the context of 3D audio perception. Extending current masking approaches to account for this phenomena would be of major interest. Extending the clustering techniques to account for the effect of reverberation would also be of primary interest. Finally, while the main target applications of 3D audio rendering are simulation and gaming, no spatial audio rendering work to date evaluates the influence of combined visual and audio restitution on the required quality of the simulation. However, a vast amount of literature in neurosciences suggests that cross-modal effects, such as ventriloquism, might significantly affect 3D audio perception [START_REF] Hairston | Visual localization ability influences cross-modal bias[END_REF]. A preliminary study has been conducted in [START_REF] Moeck | Progressive perceptual audio rendering of complex scenes[END_REF] but additional work is required in this domain.

th INTERNATIONAL CONGRESS ON ACOUSTICS -ICA2007MADRID

Figure 1 .

 1 Figure 1.-Overview of a perceptually-based auralization pipeline for VR applications.

Figure 2 .

 2 Figure 2.-(a) Loudness values (using Zwicker's loudness model) through time for the 17 tracks of a musical mix. Each track was selectively filtered and processed into 4 frequency sub-bands resulting in 68 signals to prioritize. (b) Priority map showing the first 12.5% most important frames highlighted in red. (c) First 25% most important frames and (d) first 50% most important frames.

Figure 3 .

 3 Figure 3.-Left: Average MUSHRA scores and 95% confidence intervals for our progressive processing tests. Right: Average MUSHRA scores and 95% confidence intervals as a function of budget. Note how perceived quality does not vary linearly with the processing budget and also varies depending on the type (i.e., sparseness) of the sounds.

Figure 4 .

 4 Figure 4.-Top row: note how the four clusters (in blue) adapt to the listener's location (shown in red). Bottom row: a clustering example with image-sources in a simple building environment (seen in top view). The audible image-sources, shown in green in the right-hand side image, correspond to the set of reflection paths (shown in white) in the left-hand side image

Acknowledgements

The author would like to thank all the people involved at various stages of this research: E. Gallo, G. Lemaitre, T. Moeck, N. Bonneel, F. Firsching, G. Drettakis and I. Viaud-Delmon. Parts of this research were funded by the EU IST projects CREATE and CROSSMOD and the French RNTL project OPERA. See also http://www-sop.inria.fr/reves/audile.