Nicolas Tsingos
email: nicolas.tsingos@sophia.inria.fr

SCALABLE PERCEPTUAL MIXING AND FILTERING OF AUDIO SIGNALS USING AN AUGMENTED SPECTRAL REPRESENTATION

Many interactive applications, such as video games, require processing a large number of sound signals in real-time. This paper proposes a novel perceptually-based and scalable approach for efficiently filtering and mixing a large number of audio signals. Key to its efficiency is a pre-computed Fourier frequency-domain representation augmented with additional descriptors. The descriptors can be used during the real-time processing to estimate which signals are not going to contribute to the final mixture. Besides, we also propose an importance sampling strategy allowing to tune the processing load relative to the quality of the output. We demonstrate our approach for a variety of applications including equalization and mixing, reverberation processing and spatialization. It can also be used to optimize audio data streaming or decompression. By reducing the number of operations and limiting bus traffic, our approach yields a 3 to 15-fold improvement in overall processing rate compared to brute-force techniques, with minimal degradation of the output.

INTRODUCTION

Many interactive applications such as video games, simulators and visualization/sonification interfaces require processing a large number of input sound signals in real-time (e.g., for spatialization). Typical processing includes sound equalization, filtering and mixing and is usually performed for each of the inputs individually. In modern video games, for instance, hundreds of audio samples and streams might have to be combined to re-create the various spatialized sound effects and background ambiance. This results in both a large number of arithmetic operations and heavy bus traffic. Although consumer-grade audio hardware can be used to accelerate some pre-defined effects, the limited number of simultaneous hardware voices calls for more sophisticated voice-management techniques. Besides, contrary to their modern graphics counterparts, consumer audio hardware accelerators still implement fixedfunction pipelines which might eventually limit the creativity of audio designers and programmers. Hence, designing efficient software solutions is still of major interest.

While perceptual issues have been a key aspect in the field of audio compression (e.g., mp3) [START_REF] Painter | Perceptual coding of digital audio[END_REF][START_REF] Herre | Audio coding -an all-round entertainment technology[END_REF], most software audio processing pipelines still use brute-force approaches which are completely independent of the signal content. As a result, the number of audio streams they can process is usually limited rapidly since the amount of processing cannot be adapted on-demand to satisfy a predefined time vs. quality tradeoff. This is especially true for multi-media or multi-modal applications where only a small fraction of the CPU-time can be devoted to audio processing.

In recent years, several contributions have been introduced that aim to bridge the gap between perceptual audio coding and audio processing in order to make audio signal processing pipelines more efficient. A family of approaches proposed to directly process perceptually-coded audio signals [START_REF] Lanciani | Psychoacousticallybased processing of MPEG-I layer 1-2 encoded signals[END_REF][START_REF] Lanciani | Subband-domain filtering of MPEG audio signals[END_REF][START_REF] Abdellatif | A generic framework for filtering in subband domain[END_REF][START_REF] Abdellatif | Efficient method for multiple compressed audio streams spatialization[END_REF][START_REF] Darlington | Digital audio effects in the wavelet domain[END_REF] yielding faster implementations than a full decode-process-recode cycle. Although they are well suited to distributed applications involving streaming over low-bandwith channels, they require specific coding of the filters and processing. Moreover, they cannot guarantee an efficient processing for a mixture of several signals, nor that they would produce an "optimal" processing for the mixture.

Others, inspired by psycho-acoustic research and audio coding work, tried to use perceptual knowledge to optimize various applications. For instance, a recent paper by Tsingos et al. [START_REF] Tsingos | Perceptual audio rendering of complex virtual environments[END_REF] proposed a real-time voice management technique for 3D audio applications which evaluates audible sound sources at each frame of the simulation and groups them into clusters that can be directly mapped to hardware voices. Necessary sub-mixing of all sources in each cluster is done in software at fixed-cost. Dynamic auditory masking estimation has also been successfully used to accelerate modal synthesis [START_REF] Lagrange | Real-time additive synthesis of sound by taking advantage of psychoacoustics[END_REF][START_REF] Kees Van Den Doel | Interactive simulation of complex audio-visual scenes[END_REF]. In the context of long FIR filtering for reverberation processing, the recent work by Lee et al. [START_REF] Lee | Perceptual convolution for reverberation[END_REF] also shows that significant improvement can be obtained by estimating whether the result of the convolution is below hearing threshold, hence reducing the processing cost. In this paper, we build on these approaches and propose a scalable, perceptually-based, audio processing strategy that can be applied to a frequency-domain processing pipeline performing filtering and mix-down operations on a large number of input audio signals [START_REF] Zölzer | DAFX -Digital Audio Effects[END_REF]. Key to our approach is the choice of a signal representation that allows its progressive encoding and reconstruction. In this paper, we use Fourier-transform domain as a convenient and widely used solution which satisfies these constraints. In this context, we present a set of techniques to:

• dynamically maintain features of the input audio signals to process (for instance, based on pre-computed information on the input audio samples in a way similar to [START_REF] Tsingos | Perceptual audio rendering of complex virtual environments[END_REF]),

• dynamically evaluate auditory masking between a number of input audio frames that have to be processed and mixeddown to produce a frame of audio output,

• implement a scalable processing pipeline by fitting a predefined budget of operations to the overall task based on the importance of each input audio signal. Figure 1: Overview of our scalable perceptual pipeline. All input signals frames at time t are first sorted according to their energy content and a masking estimation is performed. Audible frames are then sampled according to an importance metric so that only a subpart of their pre-computed STFT coefficients are processed to produce the output frame at t.

OVERVIEW OF OUR APPROACH

Our approach can be decomposed into four main stages (see Figure 1). The first stage builds a frequency domain representation of the audio signals based on a short-time Fourier transform (STFT). This representation is augmented by a set audio descriptors such as the root-mean-square (RMS) level of the signal in several frequency bands and the tonality [START_REF] Painter | Perceptual coding of digital audio[END_REF] of the signal. This kind of augmented description of audio signals is also similar in spirit to prior work in indexing and retrieval of audio [START_REF] Herrera | Audio descriptors and descriptors schemes in the context of MPEG-7[END_REF]. This first stage is usually performed off-line when the signals to process are known in advance.

The three remaining stages: masking evaluation, importance sampling and actual processing are performed on-line during the interactive application. Audio signals are processed using small frames of audio data (typically using windows of 20 to 40 ms) and, as a consequence, all three later steps are performed for each frame of the computed output stream.

The masking step determines which subset of the input audio frames will be audible in the final mixture. It is not mandatory but usually makes the importance sampling step more efficient. It can also be used to limit bus traffic since all inaudible signals can be directly discarded after the masking evaluation and do not have to go through the actual processing pipeline.

The importance sampling step determines the amount of data to select and process in each input signal in order to fit the predefined operation budget and minimize audible degradations. Finally, the actual processing step performs a variety of operations on the audio data prior to the final mix-down.

In the remainder of the paper, sections 3 to 5 detail these four stages while section 6 presents example applications of our techniques in the context of equalization/mixing, reverberation processing and 3D audio rendering.

PRE-PROCESSING AUDIO SIGNALS

The first stage of our approach aims at pre-computing a signal representation from which the later real-time operations can be efficiently performed. We chose a representation based on a STFT of the input signals augmented with additional information.

Constructing the representation

For each frame of the input audio signal, we first compute the STFT of the audio data. For 44.1 kHz signals, we use 1024 sam-ple Hanning-windowed frames with 50% overlap, resulting in 512 complex values in frequency domain. From the complex STFT, we then compute a number of additional descriptors:

• RMS level for a predefined set of i frequency bands (e.g., octave or Bark bands), • Tonality T calculated as a spectral flatness measure [START_REF] Painter | Perceptual coding of digital audio[END_REF];

tonality is a descriptor in [0, 1] encoding the tonal (when close to 1) or noisy (when close to 0) nature of the signal, • Reconstruction error indicator Err; this descriptor indicates how well the signal can be reconstructed from a small number of bins.

To compute the indicator Err, we first sort the FFT bins by decreasing modulus value. Then, several reconstructions (i.e., inverse Fourier transforms) are performed using an increasing number of FFT bins. The reconstruction error, calculated as the RMS level of the (time-domain) difference between the original and reconstructed frame, is then computed. For a N bin FFT, we perform k reconstructions using 1 to N FFT bins, in N /k increments. Err is calculated as the average of the k corresponding errors values. This indicator will be later used during the on-line importance sampling step.

Descriptors, together with the pre-sorted FFT bins, are computed for each frame of each input signal and pre-stored in a custom file-format. If required, descriptors can be stored separately from the FFT data used for the processing. They can be viewed as a compact representation of the signal, typically requiring a few additional kBytes of data per second of audio signals (e.g., 3kBytes/sec. at 44.1kHz for 1024 sample frames with 50% overlap and 8 frequency bands). Hence, for a set of short audio signals, they could easily fit into memory for fast random access over all signals.

Optimizing the representation

This representation can be further optimized if necessary during the pre-processing step. Frames whose energy is below audible threshold can be stored with a minimal amount of data. Basic masking calculations can also be performed while computing Err by examining the signal-to-noise ratio between the energy in the selected FFT bins and the resulting reconstruction error. The number of stored bins can then be limited as soon as the signal-to-noise exceeds a specified threshold, which can further depend on the tonality of the signal [START_REF] Painter | Perceptual coding of digital audio[END_REF]. Of course, any optimization made at this stage would imply that the signals are not going to be drastically modified during the processing step. However, this restriction applies to any approach applied to audio data encoded using a lossy audio compression strategy.

Although compression is not the primary goal of our paper, we also experimented with various strategies to optimize storage space. By quantizing the complex FFT data with non-uniform 16bit dynamic range and compressing all the data for each frame using standard compression techniques (e.g., zip), the size of the obtained sound files typically varies between 1.5 times (for wideband sounds) and 0.25 times the size (e.g. speech) of the original 16-bit PCM audio data. If more dynamic range is necessary, it is also possible to quantize the n first FFT bins, which contain most of the energy, over a 24-bit dynamic range and to represent the rest of the data using a more limited range with minimal impact on the size of the representation and quality of the reconstruction.

REAL-TIME MASKING EVALUATION

Once the input sound signals have been pre-processed, we can use the resulting information to optimize a real-time pipeline running during an interactive application.

The first step of our pipeline aims at evaluating which of the input signals are going to significantly contribute to a given frame of the output, which amounts to evaluating which input signals are going to be audible in the final mixture at a given time. Signals that have been identified as inaudible can be safely removed from the pipeline reducing both the arithmetic operations to perform and the bus traffic. Since the calculation must be carried on at each processing frame, it must be very efficient so that it does not result in significant overhead. The masking algorithm is similar to the one presented in [START_REF] Tsingos | Perceptual audio rendering of complex virtual environments[END_REF] and makes use of the the pre-computed descriptors (see Section 3.1) for maximum efficiency.

First, all input frames are sorted according to some importance metric. In [START_REF] Tsingos | Perceptual audio rendering of complex virtual environments[END_REF], a loudness metric was used but some of our recent experiments seem to indicate that the RMS level would perform equally well, if not better on average, for lack of a "ultimate" loudness metric [START_REF] Gallo | Prioritizing signals for selective real-time audio processing[END_REF]. If the signals must undergo filtering or equalization operations, we dynamically weight the RMS level values precomputed for several frequency-bands to account for the influence of the filtering operations in each band. We can then compute the importance as the sum of all weighted RMS values.

Second, all signals are considered in decreasing importance order for addition to the final mixture according to the following pseudo-code:

M mix = 200 P mix = 0 T = 0 PtoGo = k RMS k while (dB(PtoGo) > dB(P mix ) -Mmix) and (PtoGo > ATH) do tag signal S k as audible PtoGo -= RMS k P mix + = RMS k T + = P k * T k T mix = T/P mix M mix = 27 * T mix + 6 * (1 -T mix ) k++ end
This process basically adds the level RM S k of each source to an estimate of the level of the final result in each band Pmix (initially set to zero). Accordingly, it subtracts it from an estimate of the remaining level in each band PtoGo (initially set to the sum of all RMS levels for all signals). The process stops when the estimated remaining level in each band is below a given threshold Mmix from the estimated level of the final result. The process also stops if the remaining level is below the absolute threshold of hearing AT H [START_REF] Zwicker | Psychoacoustics: Facts and Models[END_REF]. Threshold Mmix is adjusted according to the estimated tonality of the final result Tmix, following rules similar to the ones used in perceptual audio coding [START_REF] Painter | Perceptual coding of digital audio[END_REF]. In our applications, a simple constant threshold of -27 dB also gave satisfying results indicating that pre-computing and estimating tonality values is not mandatory. Note that all operations must be performed for each frequency band, although we simplified the given pseudo-code for the sake of clarity (accordingly, all quantities should be interpreted as vectors whose dimension is the number of used frequency bands and all arithmetic operations as vector arithmetic). In particular the process stops when the masking threshold is reached for all frequency bands.

IMPORTANCE SAMPLING AND PROCESSING

The second step of our pipeline aims at processing the sub-set of audible input signals in a scalable manner while preserving the perceived audio quality. In our case this is achieved by performing the required signal processing using a target number of operations over a limited sub-set of the original signal data. Note that it is not mandatory to perform masking calculations (as described in the previous section) in order to implement the following importance sampling scheme. However, the masking step limits the number of samples going through the rest of the pipeline and ensures that no samples will be wasted since our sampling strategy itself does not ensure that masked signals will receive a zero-sample budget.

Selecting a budget number of FFT bins

We can assume a constant number of arithmetic operations will be required for each complex FFT coefficient (i.e., bin). Hence, fitting a budget number of operations for our pipeline at each processing frame directly amounts to selecting a budget number of FFT bins for each frame of input sound signal. We can take advantage of pre-storing our FFT in decreasing energy order by directly processing the ni first FFT bins for each input signal si so that i ni does not exceed our budget N .

We select ni as being directly proportional to an importance value Ii calculated for each audio signal. In our case, we define Ii as:

Ii = log(1 + Ei * (1 + Erri)), ( 1
)
where Erri is the error indicator of signal si and Ei is the summed RMS level of the signal in all bands (including possible effects of filtering). As for the masking calculation, we use the RMS level value as the primary importance value. We further weight this value according to the error indicator of the signal, so that signals requiring more FFT bins to achieve a good reconstruction get higher priority. The importance value Ii is then normalized so that i Ii = 1 and the number of bins to process for each signal is simply obtained as: ni = N Ii. Note that the overall target number is an upper bound and might not be exactly met. For instance, when an optimized input representation is used, some frames might contain a number of FFT bins smaller than their calculated ni. Currently, we do not re-affect the additional number of bins to another signal.

Processing and reconstruction

Once the most-important FFT bins have been selected for all signals according to our target budget, they are simply sent to the actual processing pipeline. All calculations are done in frequencydomain so that a single inverse FFT is required to obtain the final time-domain audio signal. Since we pre-compute FFT data for 50% overlapped frames, reconstructing a time-domain frame for playback requires processing two frequency-domain frames, involving two inverse FFTs and an overlap-add operation.

APPLICATIONS AND RESULTS

We implemented and tested our scalable processing algorithm for three applications: a simple mixing and equalization pipeline, an FIR reverberation pipeline and a massive spatial audio application which can render hundreds of simultaneous sound sources in realtime. Example results are available for listening at the following URL: www-sop.inria.fr/reves/projects/dafx05. All examples were implemented in C++ without any specific optimization and tested on a standard laptop computer with a Pentium 4m 1.8 GHz processor. All processing was done using 32-bit floating point arithmetic. Sampling rate was 44.1 kHz and we used output frames of 1024 samples. Hence, the output rate for audio data was 43 Hz. Masking calculations were performed using 15 frequency subbands.

Mixing and equalization

Our first application performs simple equalization and mix-down operations on a number of input sound signals. In this case, all data was streamed and decompressed from the disk in real-time. Even for a relatively small number of signals to process (in our test example we used 8), our approach shows a 3-fold speed-up compared to processing the entire data set. Overall processing speed, including streaming from disk and final time-domain reconstruction, is doubled. Table 1 shows a compute-time breakdown for various stages of the pipeline when processing the STFT data at several "resolutions", decreasing the number of target FFT bins. The results, expressed in Hz, correspond to the rate at which a full frame of output can be calculated. The load rate corresponds to the rate at which the audio data can be streamed from disk. For the load rate and processing rate, the figures are given per input signal. The total rate is given for the entire process applied to all signals, including streaming from disk. In this case, the total rate is streaming-bound. Hence, using an optimized representation, as described in section 3.2, brings a more than 2-fold improvement (as can be seen in the last column). In this case, near-transparent processing could be achieved in 1/10th of the duration of an output frame.

Block FIR filtering and reverberation

Our second application targets long FIR filters as used for reverberation processing. A common technique to implement low latency convolution with long filters is to decompose the filter in successive smaller blocks. These blocks can be of constant length [START_REF] Barry | Digital equalization using fourier transform techniques[END_REF][START_REF] Soo | Multidelay block frequency domain adaptive filter[END_REF] or can be adapted to optimize the number of arithmetic operations [START_REF] García | Optimal filter partition for efficient convolution with short input/output delay[END_REF]. Our technique extends the recent approach by Lee al. [START_REF] Lee | Perceptual convolution for reverberation[END_REF] by 1) determining which part of the reverberation filter will not be audible due to auditory masking and 2) allowing for scalable rendering of the reverberation. In fact, this application is very similar to our previous example: each block of the reverberation filter can be convolved with separate delayed copies of the original signal (by multiples of one frame) and the results are mixed together to produce one frame of reverberant output. We tested our perceptual reverberation algorithm with artificial reverberation FIR filters created from exponentially decaying white noise [START_REF]Applications of Digital Signal Processing to Audio and Acoustics[END_REF]. However, our approach could be applied to any measured impulse response.

Table 2 shows results of our approach applied to a stereo rendering of up to 12-second long reverberation filters (corresponding to about 1000 blocks of 512 time-samples). In the corresponding examples, the exponential decay was chosen to obtain a reverberation time of about 4.5 seconds. Here again, we show a compute-time breakdown for various stages of the pipeline when processing the STFT data at several "resolutions", decreasing the number of target FFT bins. The two left-most columns, denoted "all" and "all (mask)" correspond respectively to full processing (i.e., reference) and processing of all FFT bins of all audible audio frames (i.e., masking is turned on but all audible data is processed). As can be seen from the results, our approach can bring FIRbased reverberation to the efficiency level of IIR-based techniques [START_REF]Applications of Digital Signal Processing to Audio and Acoustics[END_REF] with an equivalent cost of about a few tens of operations/time sample. Our masking strategy also appears to be more efficient than the simpler hearing-threshold cut-off presented in [START_REF] Lee | Perceptual convolution for reverberation[END_REF] with a reverberation cut-down of up to 93%. Although the degradation becomes noticeable at low processing rates, we found the overall perceived room-effect to be well preserved. We believe an optimized implementation of our approach, including specific assembly language processing and FFT code (we used a simple implementation from [START_REF] Press | Numerical Recipes in C[END_REF]) could outperform efficient convolution engines such as [START_REF] Torger | BruteFIR software convolution engine[END_REF] for applications where lossy processing is acceptable.

Massive spatial audio rendering

We finally applied our technique to spatial audio rendering. In this case, we perform a complex multiply on our data to account for various filtering effects (e.g. Head Related Transfer Function [START_REF] Begault | 3D Sound for Virtual Reality and Multimedia[END_REF], atmospheric scattering, occlusions, source directivity) and to delay the audio data according to the distance and direction of each incoming source signal. As in the reverberation application, our pipeline computes a binaural output and thus requires four inverse FFTs to reconstruct a full output frame of stereo audio data. Table 3 shows results of our approach when used to process 1000 3D sound sources, using a variable number of coefficients for the entire pipeline. In our test case, all sound sources are instances of 8 primary audio signals, demonstrating possible application to auralizing sound reflection or diffraction resulting from a geometrical acoustics simulation [START_REF] Allen | Image method for efficiently simulating small room acoustics[END_REF][START_REF] Tsingos | Modeling acoustics in virtual environments using the uniform theory of diffraction[END_REF]. Refresh rates for a simultaneous basic 3D-graphics rendering of the sound sources are also provided. As we reduce the number of audio operations, more CPU-time can be devoted to graphics rendering resulting in increased frame-rates. For this application, our approach can lead to 8 to 15 fold-improvement over the brute force techniques while maintaining good output quality. In particular, we estimate that our unoptimized approach can render up to 6 times as many 3D sound sources as the approach of [START_REF] Tsingos | Perceptual audio rendering of complex virtual environments[END_REF]

DISCUSSION

One limitation of our importance sampling scheme is that it requires a fine-grain scalable model of sounds to be applicable. However, we also experimented with coarser-grain time-domain representations with promising results [START_REF] Gallo | Prioritizing signals for selective real-time audio processing[END_REF]. As with all frequencydomain processing approaches, it might require many inverse FFTs per frame to reconstruct multiple channels of output. This might be a limiting factor for applications requiring multi-channel output (e.g., 5.1 surround). However, in most cases the number of output channels is small. Another limiting factor is that we use pre-computed information to limit the overhead of our frequency domain processing and final reconstruction. In the case were the input signals are not known in advance (e.g., voice over IP, real-time synthesis,...), an equivalent representation would have to be constructed on-the-fly prior to processing. We believe that if a small number of such streams are present, our approach would still improve the overall performance.

Pre-sorting the FFT data also implies that the processing should not drastically affect the frequency spectrum of the input signals which might not be the case. However, any approach using signals encoded with a lossy algorithm would suffer from the same limitation since perceptual (e.g., masking) effects would be encoded a priori. A solution to the problem would be to store sorted FFT data associated to a number of subbands and re-order them in realtime according to how filtering operations might affect the level in each subband. Sorting the output frequency-domain data would also be necessary if several effects have to be chained together. Another solution to better account for filtering effects would be to extend our importance sampling strategy to account explicitly for frequency content (currently, importance is implied by the precomputed ordering of STFT data so that only the number of processed bins has to be determined). This might yield to an approach closer in spirit to [START_REF] Abdellatif | A generic framework for filtering in subband domain[END_REF] albeit we would still benefit from scalability and masking estimation (and would not require specific filter representation).

Although it can be optimized, our STFT data is not a compact representation (at least not as compact as standard perceptually coded representations, such as mp3 or AAC) which could be a limitation for streaming or bus transfers. However, masking calculations limit this problem, especially since they only require the additional descriptors to evaluate audible data.

Finally, the importance model used for selecting which FFT coefficients to process could be improved by using a more sophisticated loudness-related metric.

CONCLUSIONS

We presented a scalable approach to efficiently filter and mixdown a large number of audio signals in real-time. By pre-computing a Fourier frequency-domain representation of our input audio data augmented with a set of audio descriptors, we are able to concentrate our processing efforts on the most important components of the signals. In particular, we show that we can identify signals which will not be audible in the output at each processing timeframe. Such signals can be discarded thus reducing computational load and bus traffic. Remaining audible signals are sampled based on an importance metric so that only a subset of their representation is processed to produce a frame of output. Our approach yields a 3 to 15-fold improvement in overall processing rate compared to brute-force techniques with minimal degradation of the output. As future extensions, we plan to conduct perceptual validation studies to assess the auditory transparency of our approach at several "processing bit-rates" and further improve on our masking calculation and importance sampling metrics.

Table 1 :

 1 Breakdown performance figures for our test mixing and equalization pipeline. Load and processing rate are given per signal. All values are time-averaged over 29 seconds of processingtime.

	et

Table 2 :

 2 Breakdown performance figures for our test block FIR reverberation pipeline. All values are time-averaged over a processing-time equal to the duration of the input data.

	Rotor example
	target FFT bins	all	all (mask)	50000	10000	5000
	masking/sampling rate (Hz)	4754	2322	2263	2340	2387
	processing rate (Hz)	82	263	809	1965	3108
	FFT rate (Hz)	2109	2068	2124	2120	2110
	total rate (Hz)	38	105	230	350	405
	avg. masked frames (%)	0	87	87	87	87
	Song example
	target FFT bins	all	all (mask)	50000	10000	5000
	masking/sampling rate (Hz)	2452	1282	1273	1306	1316
	processing rate (Hz)	40	165	570	1666	2556
	FFT rate (Hz)	2126	2107	2084	2144	2129
	total rate (Hz)	19	68	164	270	304
	avg. masked frames (%)	0	80	80	80	80
	Voice example
	target FFT bins	all	all (mask)	50000	10000	5000
	masking/sampling rate (Hz)	13632	6387	6172	6004	6670
	processing rate (Hz)	269	531	1115	2081	3881
	FFT rate (Hz)	2130	2137	2162	2080	2200
	total rate (Hz)	116	197	324	435	567
	avg. masked frames (%)	0	93	93	93	93

  which relied on SSE-optimized code and hardware spatialization. Besides, it does not require specific spatial clustering. Example movie files demonstrating these results are also available on-line.

	target FFT bins	all (512000)	80000	50000	10000
	audio rate (Hz)	8	56	70	150
	graphics rate (Hz)	n.a.	8	15	27
	avg. masked frames (%)	0	24	24	24

Table 3 :

 3 Performance figures for our test 3D audio-visual rendering application processing 1000 sound sources. Masking is turnedon for all test-cases except the reference (i.e., first column). Results are averaged over 11 sec. of processing-time.

ACKNOWLEDGMENTS

The author would like to thank the anonymous reviewers for their useful comments. This research was supported in part by the 2year RNTL project OPERA, co-funded by the French Ministry of Research and Ministry of Industry. http://www-sop.inria.fr/reves/OPERA.


Emmanuel Gallo

Nicolas Tsingos

Guillaume Lemaitre

3D-Audio Matting, Post-editing and Re-rendering from Field Recordings

Keywords: Virtual Environments, Spatialized Sound, Audio Recording Techniques, Auditory Scene Analysis, Image-based rendering, Matting and compositing

Figure 1: Left: We use multiple arbitrarily positioned microphones (circled in yellow) to simultaneously record real-world auditory environments. Middle: We analyze the recordings to extract the positions of various sound components through time. Right: This high-level representation allows for post-editing and re-rendering the acquired soundscape within generic 3D-audio rendering architectures.

Introduction

While hardware capabilities allow for real-time rendering of increasingly complex environments, authoring realistic virtual audiovisual worlds is still a challenging task. This is particularly true for interactive spatial auditory scenes for which few content creation tools are available.

Current models for authoring interactive 3D-audio scenes often assume that sound is emitted by a set of monophonic point sources for which a signal has to be individually generated. In the general case, source signals cannot be completely synthesized from physics-based models and must be individually recorded, which requires enormous time and resources. Although this approach gives the user the freedom to control each source and freely navigate throughout the auditory scene, the overall result remains an approximation due to the complexity of real-world sources, limitations of microphone pick-up patterns and limitations of the simulated sound propagation models.

On the opposite end of the spectrum, spatial sound recordings which encode directional components of the sound-field can be directly used to acquire live auditory environments as a whole [START_REF] Malham | 3D sound spatialization using ambisonic techniques[END_REF][START_REF]Soundfield[END_REF]. They produce lifelike results but offer little control, if any, at the playback end. In particular, they are acquired from a single location in space, which makes them insufficient for walkthrough applications or rendering of large near-field sources. In practice, their use is mostly limited to the rendering of an overall ambiance. Besides, since no explicit position information is directly available for the sound sources, it is difficult to tightly couple such spatial recordings with matching visuals.

This paper presents a novel analysis-synthesis approach which bridges the two previous strategies. Our method builds a higherlevel spatial description of the auditory scene from a set of field recordings (Figure 1). By analyzing how different frequency components of the recordings reach the various microphones through time, it extracts both spatial information and audio content for the most significant sound events present in the acquired environment. This spatial mapping of the auditory scene can then be used for post-processing and re-rendering the original recordings. Rerendering is achieved through a frequency-dependent warping of the recordings, based on the estimated positions of several frequency subbands of the signal. Our approach makes positional information about the sound sources directly available for generic 3D-audio processing and integration with 2D or 3D visual content. It also provides a compact encoding of complex live auditory environments and captures complex propagation and reverberation effects which would be very difficult to render with the same level of realism using traditional virtual acoustics simulations.

Our work complements image-based modeling and rendering approaches in computer graphics [START_REF] Chen | View interpolation for image synthesis[END_REF][START_REF] Horry | Tour into the picture: using a spidery mesh interface to make animation from a single image[END_REF][START_REF] Buehler | Unstructured lumigraph rendering[END_REF][START_REF] Daniel | Plenoptic stitching: a scalable method for reconstructing 3d interactive walk throughs[END_REF]. Moreover, similar to the matting and compositing techniques widely used in visual effects production [START_REF] Porter | Compositing digital images[END_REF], we show that the various auditory components segmented out by our approach can be pasted together to create novel and spatially consistent soundscapes. For instance, foreground sounds can be integrated in a different background ambiance.

Our technique opens many interesting possibilities for interactive 3D applications such as games, virtual/augmented reality or off-line post-production. We demonstrate its applicability to a variety of situations using different microphone setups.

Related work

Our approach builds upon prior work in several domains including spatial audio acquisition and restitution, structure extraction from audio recordings and blind source separation. A fundamental difference between the approaches is whether they attempt to capture the spatial structure of the wavefield through mathematical or physical models or attempt to perform a higher-level auditory scene analysis to retrieve the various, perceptually meaningful, sub-components of the scene and their 3D location. The following sections give a short overview of the background most relevant to our problem.

Spatial sound-field acquisition and restitution

Processing and compositing live multi-track recordings is of course a widely used method in motion-picture audio production [START_REF] Yewdall | Practical Art of Motion Picture Sound[END_REF]. For instance, recording a scene from different angles with different microphones allows the sound editor to render different audio perspectives, as required by the visual action. Thus, producing synchronized sound-effects for films requires carefully planned microphone placement so that the resulting audio track perfectly matches the visual action. This is especially true since the required audio material might be recorded at different times and places, before, during and after the actual shooting of the action on stage. Usually, simultaneous monaural or stereophonic recordings of the scene are composited by hand by the sound designer or editor to yield the desired track, limiting this approach to off-line post-production. Surround recording setups (e.g., Surround Decca Trees) [START_REF] Streicher | The decca tree -it's not just for stereo anymore[END_REF][START_REF] Streicher | The new stereo soundbook, 2nd edition[END_REF], which historically evolved from stereo recording, can also be used for acquiring a sound-field suitable for restitution in typical cinemalike setups (e.g., 5.1-surround). However, such recordings can only be played-back directly and do not support spatial post-editing.

Other approaches, more physically and mathematically grounded, decompose the wavefield incident on the recording location on a basis of spatial harmonic functions such as spherical/cylindrical harmonics (e.g., Ambisonics) [START_REF] Gerzon | Ambisonics in multichannel broadcasting and video[END_REF][START_REF] Malham | 3D sound spatialization using ambisonic techniques[END_REF][START_REF] Daniel | Ambisonic encoding of other audio formats for multiple listening conditions[END_REF][START_REF] Leese | Ambisonic surround sound FAQ (version 2.8)[END_REF][START_REF] Merimaa | Applications of a 3D microphone array[END_REF] or generalized Fourier-Bessel functions [START_REF] Laborie | A new comprehensive approach of surround sound recording[END_REF]. Such representations can be further manipulated and decoded over a variety of listening setups. For instance, they can be easily rotated in 3D space to follow the listener's head orientation and have been successfully used in immersive virtual reality applications. They also allow for beamforming applications, where sounds emanating from any specified direction can be further isolated and manipulated. However, these techniques are practical mostly for low order decompositions (order 2 already requiring 9 audio channels) and, in return, suffer from limited directional accuracy [START_REF] Jot | A comparative study of 3D audio encoding and rendering techniques[END_REF]. Most of them also require specific microphones [START_REF] Abhayapala | Theory and design of high order sound field microphones using spherical microphone array[END_REF][START_REF] Meyer | Spherical microphone arrays for 3d sound recording. chap. 2 in Audio Signal Processing for next-generation multimedia communication systems[END_REF][START_REF]Soundfield[END_REF][START_REF] Laborie | High spatial resolution multi-channel recording[END_REF] which are not widely available and whose bandwidth usually drops when the spatial resolution increases. Hence, higher-order microphones do not usually deliver production-grade audio quality, maybe with the exception of Trinnov's SRP system [START_REF] Laborie | High spatial resolution multi-channel recording[END_REF] (www.trinnov.com) which uses regular studio microphones but is dedicated to 5.1-surround restitution. Finally, a common limitation of these approaches is that they use coincident recordings which are not suited to rendering walkthroughs in larger environments.

Closely related to the previous approach is wave-field synthesis/holophony [START_REF] Berkhout | Acoustic control by wave field synthesis[END_REF][START_REF] Boone | Spatial sound-field reproduction by wave-field synthesis[END_REF]. Holophony uses the Fresnel-Kirchoff integral representation to sample the sound-field inside a region of space. Holophony could be used to acquire live environments but would require a large number of microphones to avoid aliasing problems, which would jeopardize proper localization of the reproduced sources. In practice, this approach can only capture a live auditory scene through small acoustic "windows". In contrast, while not providing a physically-accurate reconstruction of the sound-field, our approach can provide stable localization cues regardless of the frequency and number of microphones.

Finally, some authors, inspired from work in computer graphics and vision, proposed a dense sampling and interpolation of the plenacoustic function [START_REF] Ajdler | The plenacoustic function and its sampling[END_REF][START_REF] Do | Toward sound-based synthesis: the far-field case[END_REF] in the manner of lumigraphs [START_REF] Gortler | The lumigraph[END_REF][START_REF] Levoy | Light field rendering[END_REF][START_REF] Buehler | Unstructured lumigraph rendering[END_REF][START_REF] Daniel | Plenoptic stitching: a scalable method for reconstructing 3d interactive walk throughs[END_REF]. However, these approaches remain mostly theoretical due to the required spatial density of recordings. Such interpolation approaches have also been applied to measurement and rendering of reverberation filters [START_REF] Pellegrini | Comparison of data and model-based simulation algorithms for auditory virtual environments[END_REF][START_REF] Horbach | Design and applications of a data-based auralization system for surround sound[END_REF]. Our approach follows the idea of acquiring the plenacoustic function using only a sparse sampling and then warping between this samples interactively, e.g., during a walkthrough. In this sense, it could be seen as an "unstructured plenacoustic rendering".

High-level auditory scene analysis

A second large family of approaches aims at identifying and manipulating the components of the sound-field at a higher-level by performing auditory scene analysis [START_REF] Bregman | Auditory Scene Analysis, The perceptual organization of sound[END_REF]. This usually involves extracting spatial information about the sound sources and segmenting out their respective content.

Spatial feature extraction and restitution

Some approaches extract spatial features such as binaural cues (interaural time-difference, interaural level difference, interaural correlation) in several frequency subbands of stereo or surround recordings. A major application of these techniques is efficient multi-channel audio compression [START_REF] Baumgarte | Binaural cue coding -part I: Psychoacoustic fundamentals and design principles[END_REF][START_REF] Faller | Binaural cue coding -part II: Schemes and applications[END_REF] by applying the previously extracted binaural cues to a monophonic down-mix of the original content. However, extracting binaural cues from recordings requires an implicit knowledge of the restitution system.

Similar principles have also been applied to flexible rendering of directional reverberation effects [START_REF] Merimaa | Spatial impulse response rendering[END_REF] and analysis of room responses [START_REF] Merimaa | Applications of a 3D microphone array[END_REF] by extracting direction of arrival information from coincident or near-coincident microphone arrays [START_REF] Pulkki | Directional audio coding in spatial sound reproduction and stereo upmixing[END_REF].

This paper generalizes these approaches to multi-channel field recordings using arbitrary microphone setups and no a priori knowledge of the restitution system. We propose a direct extraction of the 3D position of the sound sources rather than binaural cues or direction of arrival.

Blind source separation

Another large area of related research is blind source separation (BSS) which aims at separating the various sources from one or several mixtures under various mixing models [START_REF] Vincent | A tentative typologie of audio source separation tasks[END_REF][START_REF] O'grady | Survey of sparse and nonsparse methods in source separation[END_REF]. Most recent BSS approaches rely on a sparse signal representation in some space of basis functions which minimizes the probability that a Figure 2: Overview of our pipeline. In an off-line phase, we first analyze multi-track recordings of a real-world environment to extract the location of various frequency subcomponents through time. At run-time, we aggregate these estimates into a target number of clustered sound sources for which we reconstruct a corresponding signal. These sources can then be freely post-edited and re-rendered.

high-energy coefficient at any time-instant belongs to more than one source [START_REF] Rickard | Sparse sources are separated sources[END_REF]. Some work has shown that such sparse coding does exists at the cortex level for sensory coding [START_REF] Lewicki | Efficient coding of natural sounds[END_REF]. Several techniques have been proposed such as independent component analysis (ICA) [START_REF] Comon | Independent component analysis: A new concept[END_REF][START_REF] Sawada | Blind extraction of dominant target sources using ica and time-frequency masking[END_REF] or the more recent DUET technique [START_REF] Jourjine | Blind separation of disjoint orthogonal signals: Demixing n sources from 2 mixtures[END_REF][START_REF] Yilmaz | Blind separation of speech mixtures via timefrequency masking[END_REF] which can extract several sources from a stereophonic signal by building an inter-channel delay/amplitude histogram in Fourier frequency domain. In this aspect, it closely resembles the aforementioned binaural cue coding approach. However, most BSS approaches do not separate sources based on spatial cues, but directly solve for the different source signals assuming a priori mixing models which are often simple. Our context would be very challenging for such techniques which might require knowing the number of sources to extract in advance, or need more sensors than sources in order to explicitly separate the desired signals. In practice, most auditory BSS techniques are devoted to separation of speech signals for telecommunication applications but other audio applications include upmixing from stereo to 5.1 surround formats [START_REF] Avendano | Frequency-domain source identification and manipulation in stereo mixes for enhancement, suppresssion and re-panning applications[END_REF].

In this work, however, our primary goal is not to finely segment each source present in the recorded mixtures but rather to extract enough spatial information so that we can modify and re-render the acquired environment while preserving most of its original content. Closer in spirit, the DUET technique has also been used for audio interpolation [START_REF] Radke | Audio interpolation[END_REF]. Using a pair of closely spaced microphones, the authors apply DUET to re-render the scene at arbitrary locations along the line passing through the microphones. The present work extends this approach to arbitrary microphone arrays and rerendering at any 3D location in space.

Overview

We present a novel acquisition and 3D-audio rendering pipeline for modeling and processing realistic virtual auditory environments from real-world recordings.

We propose to record a real-world soundscape using arbitrarily placed omnidirectional microphones in order to get a good acoustic sampling from a variety of locations within the environment. Contrary to most related approaches, we use widely-spaced microphone arrays. Any studio microphones can be used for this purpose, which makes the approach well suited to production environments. We also propose an image-based calibration strategy making the approach practical for field applications. The obtained set of recordings is analyzed in an off-line pre-processing step in order to segment various auditory components and associate them with the position in space from which they were emitted. To compute this spatial mapping, we split the signal into short time-frames and a set of frequency subbands. We then use classical time-difference of arrival techniques between all pairs of microphones to retrieve a position for each subband at each time-frame. We evaluate the performance of existing approaches in our context and present an improved hierarchical source localization technique from the obtained time-differences.

This high-level representation allows for flexible and efficient on-line re-rendering of the acquired scene, independent of the restitution system. At run-time during an interactive simulation, we use the previously computed spatial mapping to properly warp the original recordings when the virtual listener moves throughout the environment. With an additional clustering step, we recombine frequency subbands emitted from neighboring locations and segment spatially-consistent sound events. This allows us to select and postedit subsets of the acquired auditory environment. Finally the location of the clusters is used for spatial audio restitution within standard 3D-audio APIs.

Figure 2 shows an overview of our pipeline. Sections 4, 5 and 6 describe our acquisition and spatial analysis phase in more detail. Section 7 presents the on-line spatial audio resynthesis based on the previously obtained spatial mapping of the auditory scene. Finally, Section 8 describes several applications of our approach to realistic rendering, post-editing and compositing of real-world soundscapes.

Recording setup and calibration

We acquire real-world soundscapes using a number of omnidirectional microphones and a multi-channel recording interface connected to a laptop computer. In our examples, we used up to 8 identical AudioTechnica AT3032 microphones and a Presonus Firepod firewire interface running on batteries. The microphones can be arbitrarily positioned in the environment. Section 8 shows various possible setups. To produce the best results, the microphones should be placed so as to provide a compromise between the signalto-noise ratio of the significant sources and spatial coverage.

In order to extract correct spatial information from the recordings, it is necessary to first retrieve the 3D locations of the microphones. Maximum-likelihood autocalibration methods could be used based on the existence of pre-defined source signals in the scene [START_REF] Moses | An autocalibration method for unattended ground sensors[END_REF], for which the time-of-arrival (TOA) to each microphone has to be determined. However, it is not always possible to introduce calibration signals at a proper level in the environment. Hence, in noisy environments obtaining the required TOAs might be difficult, if not impossible. Rather, we use an image-based technique from photographs which ensures fast and convenient acquisition on location, not requiring any physical measurements or homing device. Moreover, since it is not based on acoustic measurements, it is not subject to background noise and is likely to produce better results. We use REALVIZ ImageModeler (www.realviz.com) to extract the 3D locations from a small set of photographs (4 to 8 in our test examples) taken from several angles, but any standard algorithm can be applied for this step [START_REF] Faugeras | Three-Dimensional Computer Vision: A Geometric Viewpoint[END_REF]. To facilitate the process we place colored markers (tape or balls of modeling clay) on the microphones, as close as possible to the actual location of the capsule, and on the microphone stands. Additional markers can also be placed throughout the environment to obtain more input data for calibration. The only constraint is to provide a number of non-coplanar calibration points to avoid degenerate cases in the process. In our test examples, the accuracy of the obtained microphone locations was of the order of one centimeter. Image-based calibration of the recording setup is a key aspect of our approach since it allows for treating complex field recording situations such as the one depicted in Figure 3 where microphones stands are placed on large irregular rocks on a seashore.

Propagation model and assumptions for source matting

From the M recorded signals, our final goal is to localize and rerender a number J of representative sources which offer a good perceptual reconstruction of the original soundscape captured by the microphone array. Our approach is based on two main assumptions.

First, we consider that the recorded sources can be represented as point emitters and assume an ideal anechoic propagation model. In this case, the mixture x m (t) of N sources s 1 (t), .., s n (t) recorded by the m th microphone can be expressed as:

x m (t) = N ∑ n=1 a mn (t)s n (t -δ mn (t)), (1)
where parameters a mn (t) and δ mn (t) are the attenuation coefficients and time-delays associated with the n th source and the m th microphone.

Second, since our environments contain more than one active source simultaneously, we consider K frequency subbands, K >= J, as the basic components we wish to position in space at each time-frame (Figure 5 (a)). We choose to use non-overlapping frequency subbands uniformly defined on a Bark scale [START_REF] Brian | An introduction to the psychology of hearing[END_REF] to provide a more psycho-acoustically relevant subdivision of the audi-ble spectrum (in our examples, we experimented with 1 to 32 subbands).

In frequency domain, the signal x m filtered in the k th Bark band can be expressed at each time-frame as:

Y km (z) = W k (z) T ∑ t=1 x m (t)e -j(2πzt/T ) = W k (z)X m (z), (2)
where At each time-frame, we construct a new representation for the captured soundfield at an arbitrary listening point as:

W k ( f ) = 1 25k K < Bark( f ) < 25(k+1) K 0 otherwise (3) Bark( f ) = 13atan( 0.76 f 1000 ) + 3.5atan( f 2 7500 2 ), (4
x(t) ≈ J ∑ j=1 K ∑ k=1 α j km y km (t + δkm ), ∀m (5)
where y km (t) is the inverse Fourier transform of Y km (z), α j km and δkm are correction terms for attenuation and time-delay derived from the estimated positions of the different subbands. The term α j km also includes a matting coefficient representing how much energy within each frequency subband should belong to to each representative source. In this sense, it shares some similarity with the time-frequency masking approach of [START_REF] Yilmaz | Blind separation of speech mixtures via timefrequency masking[END_REF].

The obtained representation can be made to match the acquired environment if K >= N and if, following a sparse coding hypothesis, we further assume that the contents of each frequency subband belong to a single source at each time-frame. This hypothesis is usually referred to as W-disjoint orthogonality [START_REF] Yilmaz | Blind separation of speech mixtures via timefrequency masking[END_REF] and given N sources S 1 , .., S N in Fourier domain, it can be expressed as:

S i (z)S j (z) = 0 ∀i = j (6)
When the two previous conditions are not satisfied, the representative sources will correspond to a mixture of the original sources and Equ. 5 will lead to a less accurate approximation.

Spatial mapping of the auditory scene

In this step of our pipeline, we analyze the recordings in order to produce a high-level representation of the captured soundscape. This high-level representation is a mapping, global to the scene, between different frequency subbands of the recordings and positions in space from which they were emitted (Figure 5).

Following our previous assumptions, we consider each frequency subband as a unique point source for which a single position has to be determined. Localization of a sound source from a set of audio recordings, using a single-propagation-path model, is a well studied problem with major applications in robotics, people tracking and sensing, teleconferencing (e.g, automatic camera steering) and defense. Approaches rely either on time-difference of arrival (TDOA) estimates [START_REF] Aarabi | The fusion of distributed microphone arrays for sound localization[END_REF][START_REF] Knapp | The generalized correlation method for estimation of time delay[END_REF][START_REF] Huang | Microphone arrays for video camera steering[END_REF], high-resolution spectral estimation (e.g., MUSIC) [START_REF] Schmidt | Multiple emitter location and signal parameter estimation[END_REF][START_REF] Krim | Two decades of array signal processing research[END_REF] or steered response power using a beamforming strategy [START_REF] Dibiase | Microphone Arrays, Signal Processing Techniques and Applications[END_REF][START_REF] Chen | Acoustic source localization and beamforming: Theory and practice[END_REF][START_REF] Mungamuru | Enhanced sound localization[END_REF]. In our case, the use of freely positioned microphones, which may be widely spaced, prevents from using a beamforming strategy. Besides, such an approach would only lead to direction of arrival information and not a 3D position (unless several beamforming arrays were used simultaneously). In our context, we chose to use a TDOA strategy to determine the location of the various auditory events. Since we do not know the directivity of the sound sources nor the response of the microphones, localization based on level difference cannot be applied.

Figure 4 details the various stage of our source localization pipeline.

Time-frequency correlation analysis

Analysis of the recordings is done on a frame by frame basis using short time-windows (typically 20ms long or 1024 samples at CD quality). For a given source position and a given pair of microphones, the propagation delay from the source to the microphones generates a measurable time-difference of arrival. The set of points which generate the same TDOA defines an hyperboloid surface in 3D (or an hyperbola in 2D) which foci are the locations of the two microphones (Figure 5 (b)).

In our case, we estimate the TDOAs, τmn , between pairs of microphones < m, n > in each frequency subband k using standard generalized cross-correlation (GCC) techniques in frequency domain [START_REF] Knapp | The generalized correlation method for estimation of time delay[END_REF][START_REF] Rabinkin | Estimation of wavefront arrival delay using the cross-power spectrum phase technique[END_REF][START_REF] Chen | Time delay estimation in room acoustic environments: An overview[END_REF]:

τmn = arg max τ GCC mn (τ), (7)
where the GCC function is defined as:

GCC nm (τ) = Z ∑ z=1 ψ nm (z)E {Y kn (z)Y * km (z)} e j(2πτz/Z) . ( 8
)
Y kn and Y km are the 2Z-point Fourier transforms of the subband signals (see Eq. 2), E Y kn (z)Y * km (z) is the cross spectrum and . * denotes the complex conjugate operator.

For the weighting function, ψ, we use the PHAT weighting which was shown to give better results in reverberant environments [START_REF] Chen | Time delay estimation in room acoustic environments: An overview[END_REF]:

ψ mn (z) = 1 |Y n (z)Y * m (z)| (9)
Note that phase differences computed directly on the Fourier transforms, e.g. as used in the DUET technique [START_REF] Jourjine | Blind separation of disjoint orthogonal signals: Demixing n sources from 2 mixtures[END_REF][START_REF] Yilmaz | Blind separation of speech mixtures via timefrequency masking[END_REF], cannot be applied in our framework since our microphones are widely spaced.

We also experimented with an alternative approach based on the average magnitude difference function (AMDF) [START_REF] Merimaa | Applications of a 3D microphone array[END_REF][START_REF] Chen | Performance of GCC-and AMDF-based time-delay estimation in practical reverberant environments[END_REF]. The TDOAs are then given as:

τnm = arg min τ AMDF nm (τ), (10)
where the AMDF function is defined as:

AMDF nm (τ) = 1 Z Z ∑ z=1 |y kn (τ) -y km (k + τ)| (11)
We compute the cross-correlation using vectors of 8192 samples (185 ms at 44.1KHz). For each time-frame, we search the highest correlation peaks (or lowest AMDF values) between pairs of recordings in the time-window defined by the spacing between the corresponding couple of microphones. The corresponding timedelay is then chosen as the TDOA between the two microphones for the considered time-frame.

In terms of efficiency, the complexity of AMDF-based TDOA estimation (roughly O(n 2 ) in the number n of time-domain samples) makes it unpractical for large time-delays. In our test-cases, running on a Pentium4 Xeon 3.2GHz processor, AMDF-based TDOA estimations required about 47 s. per subband for one second of input audio data (using 8 recordings, i.e., 28 possible pairs of microphones). In comparison, GCC-based TDOA estimations require only 0.83 s. per subband for each second of recording.

As can be seen in Figure 8, both approaches resulted in comparable subband localization performance and we found both approaches to perform reasonably well in all our test cases. In more reverberant environments, an alternative approach could be the adaptive eigenvalue decomposition [START_REF] Huang | Microphone arrays for video camera steering[END_REF]. From a perceptual point-of-view, listening to virtual re-renderings, we found that the AMDF-based approach lead to reduced artifacts, which seems to indicate that subband locations are more perceptually valid in this case. However, validation of this aspect would require a more thorough perceptual study.

Position estimation

From the TDOA estimates, several techniques can be used to estimate the location of the actual sound source. For instance, it can be calculated in a least-square sense by solving a system of equations [START_REF] Huang | Microphone arrays for video camera steering[END_REF] or by aggregating all estimates into a probability distribution function [START_REF] Rui | New direct approaches to robust sound source localization[END_REF][START_REF] Aarabi | The fusion of distributed microphone arrays for sound localization[END_REF]. Solving for possible positions in a least-square sense lead to large errors in our case, mainly due to the presence of multiple sources, several local maxima for each frequency subband resulting in an averaged localization. Rather, we choose the latter solution and compute a histogram corresponding to the probability distribution function by sampling it on a spatial grid (Figure 6) whose size is defined according to the extent of the auditory environment we want to capture (in our various examples, the grid covered areas ranging from 25 to 400 m 2 ). We then pick the maximum value in the histogram to obtain the position of the subband.

For each cell in the grid, we sum a weighted contribution of the distance function D i j (x) to the hyperboloid defined by the TDOA for each pair of microphones < i, j >: where M i resp. M j is the position of microphone i resp. j, x is the center of the cell and DDOA i j = T DOA i j /c is the signed distancedifference obtained from the calculated TDOA (in seconds) and the speed of sound c.

D i j (x) = |(||M i -x|| -||M j -x||) -DDOA i j |, ( 12
)
The final histogram value in each cell is then obtained as :

H(x) = ∑ i j e (γ(1-D i j (x))) e γ (1 -DDOA i j /||M i -M j ||) if D i j (x) < 1, 0 otherwise . ( 13
)
The exponentially decreasing function controls the "width" of the hyperboloid and provides a tradeoff between localization accuracy and robustness to noise in the TDOA estimates. In our examples, we use γ = 4. The second weighting term reduces the contribution of large TDOAs relative to the spacing between the pair of microphones. Such large TDOAs lead to "flat" ellipsoids contributing to a large number of neighboring cells in the histogram and resulting into less accurate position estimates [START_REF] Ajdler | Acoustic source localization in distributed sensor networks[END_REF].

The histogram is re-computed for each subband at each timeframe based on the corresponding TDOA estimates. The location of the k th subband is finally chosen as the center point of the cell having the maximum value in the probability histogram (Figure 5 (c)):

B k = arg max x H(x) (14)
In the case where most of the sound sources and microphones are located at similar height in a near planar configuration, the histogram can be computed on a 2D grid. This yields faster results at the expense of some error in localization. A naive calculation of the histogram at each time-frame (for a single frequency band and 8 microphones, i.e., 28 possible hyperboloids) on a 128 × 128 grid requires 20 milliseconds on a Pentium4 Xeon 3.2GHz processor. An identical calculation in 3D requires 2.3 s. on a 128 × 128 × 128 grid. To avoid this extra computation time, we implemented a hierarchical evaluation using a quadtree or octree decomposition [START_REF] Samet | The Design and Analysis of Spatial Data Structures[END_REF].

We recursively test only a few candidate locations (typically 16 to 64), uniformly distributed in each cell, before subdividing the cell in which the maximum of all estimates is found. Our hierarchical localization process supports real-time performance requiring only 5 ms to locate a subband in a 512 × 512 × 512 3D grid. In terms of accuracy, it was found to be comparable to the direct, nonhierarchical, evaluation at maximum resolution in our test examples.

Figure 7: Indoor validation setup using 8 microphones. The 3 markers (see blue, yellow, green arrows) on the ground correspond to the location of the recorded speech signals.

Indoor validation study

To validate our approach, we conducted a test-study using 8 microphones inside a 7m×3.5m×2.5m room with limited reverberation time (about 0.3 sec. at 1KHz). We recorded three people speaking while standing at locations specified by colored markers. Figure 7 depicts the corresponding setup. We first evaluated the localization accuracy for all subbands by constructing spatial energy maps of the recordings. As can be seen in Figure 8, our approach properly localizes the corresponding sources. In this case, the energy corresponds to the signal captured by a microphone located at the center of the room.

Figure 11 shows localization error over all subbands by reference to the three possible positions for the sources. Since we do not know a priori which subband belongs to which source, the error is simply computed, for each subband, as the minimum distance between the reconstructed location of the subband and each possible source position. Our approach achieves a maximum accuracy of one centimeter and, on average, the localization accuracy is of the order of 10 centimeters. Maximum errors are of the order of a few meters. However, listening tests exhibit no strong artefacts showing that such errors are likely to occur for frequency subbands containing very little energy. Figure 11 also shows the energy of one of the captured signals. As can be expected, the overall localization error is also correlated with the energy of the signal.

We also performed informal comparisons between reference binaural recordings and a spatial audio rendering using the obtained locations, as described in the next section. Corresponding audio files can be found at: http://www-sop.inria.fr/reves/projects/audioMatting.

They exhibit good correspondence between the original situation and our renderings showing that we properly assign the subbands to the correct source locations at each time-frame.

3D-audio resynthesis

The final stage of our approach is the spatial audio resynthesis. During a real-time simulation, the previously pre-computed subband positions can be used for re-rendering the acquired soundfield while changing the position of the sources and listener. A key aspect of our approach is to provide a spatial description of a realworld auditory scene in a manner independent of the auditory restitution system. The scene can thus be re-rendered by standard 3Daudio APIs: in some of our test examples, we used DirectSound 3D accelerated by a CreativeLabs Audigy2 NX soundcard and also implemented our own software binaural renderer, using head-related The top map is computed using AMDF-based TDOA estimation while the bottom map is computed using GCC-PHAT. Both maps were computed using 8 subbands and corresponding energy is integrated over the entire duration of the sequence.

transfer function (HRTF) data from the LISTEN HRTF database 1 .

Inspired by binaural-cue coding [START_REF] Faller | Binaural cue coding -part II: Schemes and applications[END_REF], our re-rendering algorithm can be decomposed in two steps, that we detail in the following sections:

• First, as the virtual listener moves throughout the environment, we construct a warped monophonic signal based on the original recording of the microphone closest to the current listening position.

• Second, this warped signal is spatially enhanced using 3Daudio processing based on the location of the different frequency subbands.

These two steps are carried out over small time-frames (of the same size as in the analysis stage). To avoid artefacts we use a 10% overlap to cross-fade successive synthesis frames.

Warping the original recordings

For re-rendering, a monophonic signal best matching the current location of the virtual listener relative to the various sources must be synthesized from the original recordings.

At each time-frame, we first locate the microphone closest to the location of the virtual listener. To ensure that we remain as faithful as possible to the original recording, we use the signal captured by this microphone as our reference signal R(t).

We then split this signal into the same frequency subbands used during the off-line analysis stage. Each subband is then warped to the virtual listener location according to the pre-computed spatial mapping at the considered synthesis time-frame (Figure 9). This warping involves correcting the propagation delay and attenuation of the reference signal for the new listening position, according to our propagation model (see Eq.1). Assuming an inverse distance attenuation for point emitters, the warped signal R i (t) in subband i is thus given as:

R i (t) = r i 1 /r i 2 R i (t + (δ i 1 -δ i 2 )), ( 15
)
where r i 1 ,δ i 1 are respectively the distance and propagation delay from the considered time-frequency atom to the reference microphone and r i 2 ,δ i 2 are the distance and propagation delay to the new listening position.

Clustering for 3D-audio rendering and source matting

To spatially enhance the previously obtained warped signals, we run an additional clustering step to aggregate subbands which might be located at nearby positions using the technique of [START_REF] Tsingos | Perceptual audio rendering of complex virtual environments[END_REF]. The clustering allows to build groups of subbands which can be rendered from a single representative location and might actually belong to the same physical source in the original recordings. Thus, our final rendering stage spatializes N representative point sources corresponding to the N generated clusters, which can vary between 1 and the total number of subbands. To improve the temporal coherence of the approach we use an additional Kalman filtering step on the resulting cluster locations [START_REF] Kalman | A new approach to linear filtering and prediction problems[END_REF].

With each cluster we associate a weighted sum of all warped signals in each subband which depends on the Euclidean distance between the location of the subband B i and the location of the cluster representative C k . This defines matting coefficients α k , similar to alpha-channels in graphics [START_REF] Porter | Compositing digital images[END_REF]:

α(C k , B i ) = 1.0/(ε + ||C k -B i ||) ∑ i α(C k , B i ) . ( 16
)
In our examples, we used ε = 0.1. Note that in order to preserve the energy distribution, these coefficients are normalized in each frequency subband. These matting coefficients control the blending of all subbands rendered at each cluster location and help smooth the effects of localization errors. They also ensure a smoother reconstruction when sources are modified or moved around in the re-rendering phase.

The signal for each cluster S k (t) is finally constructed as a sum of all warped subband signals R i (t), as described in the previous section, weighted by the matting coefficients α(C k , B i ) :

S k (t) = ∑ i α(C k , B i )R i (t). (17)
The representative location of each cluster is used to apply the desired 3D-audio processing (e.g., HRTFs) without a priori knowledge of the restitution setup. Figure 10 summarizes the complete re-rendering algorithm.

Applications and results

Our technique opens many interesting application areas for interactive 3D applications, such as games or virtual/augmented reality, and off-line audio-visual post-production. Several example renderings demonstrating our approach can be found at the following URL:

http://www-sop.inria.fr/reves/projects/audioMatting.

Modeling complex sound sources

Our approach can be used to render extended sound sources (or small soundscapes) which might be difficult to model using individual point sources because of their complex acoustic behavior.

For instance, we recorded a real-world sound scene involving a car which is an extended vibrating sound radiator. Depending on the point of view around the scene, the sound changes significantly due to the relative position of the various mechanical elements (engine, exhaust, etc.) and the effects of sound propagation around the body of the car. This makes an approach using multiple recordings very interesting in order to realistically capture these effects. Unlike other techniques, such as Ambisonics O-format [START_REF] Malham | Spherical harmonic coding of sound objects -the ambisonic 'O' format[END_REF], our approach captures the position of the various sounding components and not only their directional aspect. In the accompanying examples, we demonstrate a re-rendering with a moving listening point of a car 8. computed over 8 subbands. Averaged error over all subbands is displayed in blue, maximum error in green and minimum error in red. The top (magenta) curve represents the energy for one of the input recordings and shows its correlation with the localization error (clearly larger when the energy drops out). scenario acquired using 8 microphones surrounding the action (Figure 12). In this case, we used 4 clusters for re-rendering. Note in the accompanying video available on-line, the realistic distance and propagation effects captured by the recordings, for instance on the door slams. Figure 13 shows a corresponding energy map clearly showing the low frequency exhaust noise localized at the rear of the car and the music from the on-board stereo audible through the driver's open window. Engine noise was localized more diffusely mainly due to interference with the music. Figure 12: We capture an auditory environment featuring a complex sound source (car engine/exhaust, passengers talking, door slams and on-board stereo system) using 8 microphones surrounding the action.

Figure 13: Energy localization map for a 15 sec.-long recording of our car scenario featuring engine/exhaust sounds and music (on the on-board stereo system and audible through the open driverwindow). Positions were computed over 8 subbands using GCC-PHAT-based TDOA estimation. Energy is integrated over the entire duration of the input audio sequence.

Spatial recording and view-interpolation

Following binaural cue coding principles, our approach can be used to efficiently generate high-resolution surround recordings from monophonic signals. To illustrate this application we used 8 omnidirectional microphones located in a circle-like configuration about 1.2 meters in diameter (Figure 14) to record three persons talking and the surrounding ambiance (fountain, birds, etc.). Then, our preprocessing was applied to extract the location of the sources. For re-rendering, the monophonic signal of a single microphone was used and respatialized as described in Section 7.1, using 4 clusters (Figure 16). Please, refer to the accompanying video provided on the web site to evaluate the result. Another advantage of our approach is to allow for re-rendering an acquired auditory environment from various listening points. To demonstrate this approach on a larger environment, we recorded two moving speakers in a wide area (about 15 × 5 meters) using the microphone configuration shown in Figure 1 (Left). The recording also features several background sounds such as traffic and roadwork noises. Figure 15 shows a corresponding spatial energy map. The two intersecting trajectories of the moving speakers are clearly visible. Applying our approach, we are able to re-render this auditory scene from any arbitrary viewpoint. Although the rendering is based only on the monophonic signal of the microphone closest to the virtual listener at each time-frame, the extracted spatial mapping allows for convincingly reproducing the motion of the sources. Note in the example video provided on the accompanying web-site how we properly capture front-to-back and left-to-right motion for the two moving speakers.

Spatial audio compositing and post-editing

Finally, our approach allows for post-editing the acquired auditory environments and composite several recordings.

Source re-localization and modification

Using our technique, we can selectively choose and modify various elements of the original recordings. For instance, we can select any spatial area in the scene and simply relocate all clusters included in the selected region. We demonstrate an example interactive interface for spatial modification where the user first defines a selection area then a destination location. All clusters entering the selection area are translated to the destination location using the translation vector defined by the center of the selection box and the target location. In the accompanying video, we show two instances of source re-localization where we first select a speaker on the left-hand side of the listener and move him to the right-hand side. In a second example, we select the fountain at the rear-left of the listener and move it to the front-right (Figure 16).

Compositing

Since our recording setups are spatially calibrated, we can integrate several environments into a single composite rendering which preserves the relative size and positioning of the various sound sources. For instance, it can be used to integrate a close-miked sound situation into a different background ambiance. We demonstrate an example of sound-field compositing by inserting our previous car example (Figure 12) into the scene with the two moving speakers (Figure 1). The resulting composite environment is rendered with 8 clusters and the 16 recordings of the two original soundscapes. Future work might include merging the representations in order to limit the number of composite recordings (for instance by "reprojecting" the recordings of one environment into the recording setup of the other and mixing the resulting signals). Figure 16: An example interface for source re-localization. In this example we select the area corresponding to the fountain (in purple) and translate it to a new location (shown as a yellow cross). The listener is depicted as a large red sphere, the microphone array as small yellow spheres and the blue spheres show cluster locations.

Real/Virtual integration

Our approach permits spatially consistent compositing of virtual sources within real-world recordings. We can also integrate virtual objects, such as walls, and make them interact with the original recordings. For instance, by performing real-time ray-casting between the listener and the location of the frequency subbands, we can add occlusion effects due to a virtual obstacle using a model similar to [START_REF] Tsingos | Fast rendering of sound occlusion and diffraction effects for virtual acoustic environments[END_REF]. Please, refer to the accompanying examples at the previously mentioned URL for a demonstration. Of course, perfect integration would also require correcting for the reverberation effects between the different environments to composite. Currently, we experimented only in environments with limited reverberation but blind extraction of reverberation parameters [START_REF] Baskind | Methods for blind computational estimation of perceptual attributes of room acoustics[END_REF] and blind deconvolution are complementary areas of future research in order to better composite real and virtual sound-fields.

Discussion

Although it is based on a simple mixing model and assumes Wdisjoint orthogonality for the sources, we were able to apply our approach to real-world recording scenarios. While not productiongrade yet, our results seem promising for a number of interactive and off-line applications.

While we tested it for both indoor and outdoor recordings, our approach is currently only applicable to environments with limited reverberation. Long reverberations will have a strong impact on our localization process since existing cross-correlation approaches are not very robust to interfering sound reflections. Other solutions based on blind channel identification in a reverberant context could lead to improved results [START_REF] Chen | Time delay estimation in room acoustic environments: An overview[END_REF].

Errors in localization of the frequency subbands can result in noticeable artefacts especially when moving very close to a source. These errors can come from several factors in our examples particularly low signal-to-noise ratio for the source to localize, blurring from sound reflections, correlation of two different signals in the case of widely spaced microphones or several sources being present in a single frequency subband. As a result, several overlapping sources are often fused at the location of the louder source. While the assumption of W-disjoint orthogonality has been proven to be suitable for speech signals [START_REF] Rickard | On the approximate w-disjoint orthogonality of speech[END_REF], it is more questionnable for more general scenarios. It will only be acceptable if this source can perceptually mask the others. However, recent approaches for efficient audio rendering have shown that masking between sources is significant [START_REF] Tsingos | Perceptual audio rendering of complex virtual environments[END_REF], which might explain why our approach can give satisfying results quite beyond the validity domain of the underlying models. Alternate decompositions [START_REF] Mallat | Matching pursuits with time-frequency dictionaries[END_REF][START_REF] Lewicki | Learning overcomplete representations[END_REF] could also lead to sparser representations and better results within the same framework.

The signal-to-noise ratio of the different sound sources is also directly linked to the quality of the result when moving very close to the source since our warping is likely to amplify the signal of the original recording in this case.

We are working on several improvements to alleviate remaining limitations of the system and improve the rendering quality: Currently, we do not interpolate between recordings but select the signal of the microphone closest to the listener location for subsequent warping and re-rendering. This provides a correct solution for the case of omnidirectional anechoic point sources. In more general situations, discontinuities might still appear when switching from one microphone to the next. This can be caused, for instance, by the presence of a sound source with a strong directionality. A solution to this problem would be to warp the few microphones closest to the listener and blend the result at the expense of a higher computing cost. Note that naive blending between microphone signals before warping would introduce unwanted interferences, very noticeable in the case of widely-spaced microphones. Another option would be to experiment with morphing techniques [START_REF] Slaney | Automatic audio morphing[END_REF] as an alternative to our position-based warping. We could also use different microphones for each frequency subband, for instance choosing the microphone closer to the location of each subband rather than the one closest to the listener. This would increase the signal-tonoise ratio for each source and could be useful to approximate a close-miking situation in order to edit or modify the reverberation effects for instance.

The number of bands also influences the quality of the result. More bands are likely to increase the spatial separation but since our correlation estimates are significantly noisy, it might also make artefacts more audible. In our case, we obtained better sounding results using a limited number of subbands (typically 8 to 16). Following the work of Faller et al. [START_REF] Baumgarte | Binaural cue coding -part I: Psychoacoustic fundamentals and design principles[END_REF][START_REF] Faller | Binaural cue coding -part II: Schemes and applications[END_REF][START_REF] Faller | Source localization in complex listening situations: Selection of binaural cues based on interaural coherence[END_REF], we could also keep track of the inter-correlation between recordings in order to precisely localize only the frames with high correlation. Frames with low correlation could be rendered as "diffuse", forming a background ambiance which cannot be as precisely located [START_REF] Merimaa | Spatial impulse response rendering[END_REF]. This could be seen as explicitly taking background noise or spatially extended sound sources into account in our mixing model instead of considering only perfect anechoic point sources. We started to experiment with an explicit separation of background noise using noiseremoval techniques [START_REF] Ephraim | Speech enhancement using a minimum mean-square error short-time spectral amplitude estimator[END_REF]. The obtained foreground component can then be processed using our approach while the background-noise component can be rendered separately at a lower spatial resolution. Example renderings available on the web site demonstrate improved quality in complex situations such as a seashore recording.

Sound source clustering and matting also strongly depends on the correlation and position estimates for the subbands. An alternative solution would be to first separate a number of sources using independent component analysis (ICA) techniques and then run TDOA estimation on the resulting signals [START_REF] Saruwatari | Blind source separation combining independent component analysis and beamforming[END_REF][START_REF] Huang | Multiple acoustic sources location based on blind source separation[END_REF]. However, while ICA might improve separation of some sources, it might still lead to signals where sources originating from different locations are combined.

Another issue is the microphone setup used for the recordings. Any number of microphones can be used for localization starting from two (which would only give directional information). If more microphones are used, the additional TDOA estimates will increase the robustness of the localization process. From our experience, closely spaced microphones will essentially return directional information while microphone setups surrounding the scene will give good localization accuracy. Microphones uniformly spaced in the scene provide a good compromise between signal-to-noise ratio and sampling of the spatial variations of the sound-field. We also experimented with cardioid microphone recordings and obtained good results in our car example. However, for larger environments, correlation estimates are likely to become noisier due to the increase in separation between different recordings, making them difficult to correlate. Moreover, it would make interpolating between recordings more difficult in the general case. Our preferred solution was thus to use a set of identical omnidirectional microphones. However, it should be possible to use different sets of microphones for localization and re-rendering which opens other interesting possibilities for content creation, for instance by generating consistent 3D-audio flythroughs while changing the focus point on the scene using directional microphones. Finally, our approach currently requires an off-line step which prevents it from being used for real-time analysis. Being able to compute cross-correlations in real-time for all pairs of microphones and all subbands would make the approach usable for broadcast applications.

Conclusions

We presented an approach to record, edit and re-render real-world auditory situations. Contrary to most related approaches, we acquire the sound-field using an unconstrained, widely-spaced, microphone array which we spatially calibrate using photographs. Our approach pre-computes a spatial mapping between different frequency subbands of the acquired live recordings and the location in space from which they were emitted. We evaluated standard TDOA-based techniques and proposed a novel hierarchical localization approach. At run-time, we can apply this mapping to the frequency subbands of the microphone closest to the virtual listener in order to resynthesize a consistent 3D sound-field, including complex propagation effects which would be difficult to simulate. An additional clustering step allows for aggregating subbands originating from nearby location in order to segment individual sound sources or small groups of sound sources which can then be edited or moved around. To our knowledge, such level of editing was impossible to achieve using previous state-of-the-art and could lead to novel authoring tools for 3D-audio scenes.

We believe our approach opens many novel perspectives for interactive spatial audio rendering or off-line post-production environments, for example to complement image based rendering techniques or free-viewpoint video. Moreover, it provides a compact encoding of the spatial sound-field, which is independent of the restitution system. In the near future, we plan to run more formal perceptual tests in order to compare our results to binaural or high-order Ambisonics recordings in the case of fixed-viewpoint scenarios and to evaluate its quality using various restitution systems. From a psychophysical point of view, this work suggests that real-world sound scenes can be efficiently encoded using limited spatial information.

Other promising areas of future work would be to exploit perceptual localization results to improve localization estimation [START_REF] Wilson | Learning a precedence effect-like weighting function for the generalized cross-correlation framework[END_REF] and apply our analysis-synthesis strategy to the real-time generation of spatialized audio textures [START_REF] Lu | Audio textures: Theory and applications[END_REF]. Finaly, making the calibration and analysis step interactive would allow the approach to be used in broadcasting applications (e.g., 3D TV).

Figure 3 :

 3 Figure 3: We retrieve the position of the microphones from several photographs of the setup using a commercial image-based modeling tool. In this picture, we show four views of a recording setup, position of the markers and the triangulation process yielding the locations of the microphone capsules.

  ) f = z/Z f s is the frequency in Hertz, f s is the sampling rate and X m (z) is the 2Z-point Fourier transform of x m (t). We typically record our live signals using 24-bit quantization and f s = 44.1KHz. The subband signals are computed using Z = 512 with a Hanning window and 50% overlap before storing them back into timedomain for later use.

Figure 4 :

 4 Figure 4: Overview of the analysis algorithm used to construct a spatial mapping for the acquired soundscapes.

Figure 5 :

 5 Figure 5: Illustration of the construction of the global spatial mapping for the captured sound-field. (a) At each time-frame, we split the signals recorded by each microphone into the same set of frequency subbands. (b) Based on time-difference of arrival estimation between all pairs of recordings, we sample all corresponding hyperbolic loci to obtain a position estimate for the considered subband. (c) Position estimates for all subbands at the considered time-frame (shown as colored spheres).

Figure 6 :

 6 Figure 6: (a) A 2D probability histogram for source location obtained by sampling a weighted sum of hyperbolas corresponding to the time-difference of arrival to all microphone pairs (shown in blue). We pick the maximum value (in red) in the histogram as the location of the frequency band at each frame. (b) A cut through a 3D histogram of the same situation obtained by sampling hyperboloid surfaces on a 3D grid.

Figure 8 :

 8 Figure 8: Energy localization map for a 28s.-long audio sequence featuring 3 speakers inside a room (indicated by the three yellow crosses). Light-purple dots show the location of the 8 microphones.The top map is computed using AMDF-based TDOA estimation while the bottom map is computed using GCC-PHAT. Both maps were computed using 8 subbands and corresponding energy is integrated over the entire duration of the sequence.

Figure 9 :

 9 Figure 9: In the resynthesis phase, the frequency components of the signal captured by the microphone closest to the location of the virtual listener (shown in red) is warped according to the spatial mapping pre-computed in the off-line stage.

Figure 10 :

 10 Figure 10: Overview of the synthesis algorithm used to re-render the acquired soundscape based on the previously obtained subband positions.

Figure 11 :

 11 Figure11: Localization error for the same audio sequence as in Figure8. computed over 8 subbands. Averaged error over all subbands is displayed in blue, maximum error in green and minimum error in red. The top (magenta) curve represents the energy for one of the input recordings and shows its correlation with the localization error (clearly larger when the energy drops out).

Figure 14 :

 14 Figure 14: Microphone setup used to record the fountain example. In this case the microphones are placed at the center of the action.

Figure 15 :

 15 Figure 15: Energy map for a recording of our moving speaker scenario. The arrows depict the trajectory of the two speakers. Energy is integrated over the entire duration of the input audio sequence. Note how the two intersecting trajectories are clearly reconstructed.

Acknowledgments

This research was made possible by a grant from the région PACA and was also partially funded by the RNTL project OPERA (http://www-sop.inria.fr/reves/OPERA). We acknowledge the generous donation of Maya as part of the Alias research donation program, Alexander Olivier-Mangon for the initial model of the car, and Frank Firsching for the animation.