Thibaud Necciari

Peter Balazs

Richard Kronland-Martinet

Sølvi Ystad

Bernhard Laback

Sophie Savel

Sabine Meunier

Auditory time-frequency masking: psychoacoustical data and application to audio representations

Keywords: auditory masking, time-frequency representation, Gabor, wavelets

come   L'archive ouverte pluridisciplinaire

Introduction

The main goal of this study was to collect data on time-frequency (TF) auditory masking for stimuli with maximal concentration in the TF plane. The results of such measurements served as a basis to examine the accuracy of simple TF masking models currently implemented in some perceptual audio codecs like MP3 and develop a perceptually relevant and perfectly invertible audio signal representation. The latter aspect was achieved by improving an existing algorithm [START_REF] Balazs | Time-frequency sparsity by removing perceptually irrelevant components using a simple model of simultaneous masking[END_REF] designed to remove the inaudible components in a TF transform while causing no audible difference to the original sound after re-synthesis.

Representations of Sound Signals in Audio Signal Processing

In the field of audio signal processing, many applications involving sound analysissynthesis (e.g., virtual reality, sound design, sonification, perceptual audio coding) require specific tools enabling the analysis, processing, and re-synthesis of non-stationary signals. Most of them are based on linear TF representations such as the Gabor and wavelet transforms. These transforms allow decomposing any natural sound into a set of elementary functions or "atoms" that are well localized in the TF plane (for a review on TF analysis see, e.g., [START_REF] Daubechies | Ten Lectures on Wavelets[END_REF][START_REF] Gröchening | Foundations of time-frequency analysis[END_REF][START_REF] Vetterli | Wavelets and subband coding[END_REF]). For the cited applications, obtaining a perceptually relevant (i.e., providing a good match between signal representation and human auditory perception) and perfectly invertible signal representation would be of great interest. To that end, the long-term goal of the present study is to propose a signal representation being as close as possible to "what we see is what we hear ". Because the achievement of such a sparse representation would facilitate the extraction and reconstruction of perceptually relevant sound features, it would also be of great interest for music information retrieval applications.

To date, two approaches exist to obtain a perceptually motivated time versus frequency representation of an audio signal. The first approach includes models of auditory processing providing an "internal" representation of sound signals like in [START_REF] Jepsen | A computational model of human auditory signal processing and perception[END_REF][START_REF] Patterson | Time-domain modeling of peripheral auditory processing: A modular architecture and a software platform[END_REF][START_REF] Plack | Linear and nonlinear processes in temporal masking[END_REF]. While this approach is useful to improve our understanding of auditory processing, it does not enable reconstruction of the input signal. Thus, it is not useful as a TF analysis-synthesis tool. The second approach includes TF transforms whose parameters are tuned to mimic the spectro-temporal resolution of the auditory system [START_REF] Agerkvist | A time-frequency auditory model using wavelet packets[END_REF][START_REF] Jeong | Implementation of a new algorithm using the STFT with variable frequency resolution for the time-frequency auditory model[END_REF][START_REF] O'donovan | Perceptually motivated time-frequency analysis[END_REF]. While this approach is useful for audio signal analysis, the cited algorithms feature some limitations. In particular, [START_REF] Agerkvist | A time-frequency auditory model using wavelet packets[END_REF][START_REF] Jeong | Implementation of a new algorithm using the STFT with variable frequency resolution for the time-frequency auditory model[END_REF] can only approximate the auditory resolution because the temporal and spectral resolutions cannot be set independently. The use of a bilinear transform in [START_REF] O'donovan | Perceptually motivated time-frequency analysis[END_REF] overcomes this limitation but, on the other hand, this method does not allow reconstruction of the input signal. More recently, Balazs et al. [START_REF] Balazs | Time-frequency sparsity by removing perceptually irrelevant components using a simple model of simultaneous masking[END_REF] proposed a new approach to obtain a perceptually relevant and perfectly invertible signal representation. They introduced the concept of the "irrelevance filter", which consists in removing the inaudible atoms in a perfectly invertible Gabor transform while causing no audible difference to the original sound after re-synthesis. To identify the irrelevant atoms, a simple model of auditory spectral masking was used. A perceptual test performed with 36 normal-hearing listeners in [START_REF] Balazs | Time-frequency sparsity by removing perceptually irrelevant components using a simple model of simultaneous masking[END_REF] revealed that, on average, 36% of the atoms could be removed without causing any audible difference to the original sound after re-synthesis. The work described in the present paper can be considered as an extension of the irrelevance filter. Mostly, we attempt to improve it in two aspects: (i) overcome the fixed resolution in the Gabor transform by using a wavelet transform and (ii) replace the simple spectral masking model by using psychoacoustical data on auditory TF masking for stimuli with maximal concentration in the TF plane.

State-of-the-Art on Auditory Masking

Auditory masking occurs when the detection of a sound (referred to as the "target") is degraded by the presence of another sound (the "masker"). This effect is quantified by measuring the degree to which the detection threshold of the target increases in the presence of the masker. 3 In the literature, masking has been extensively investigated with simultaneous and non-simultaneous presentation of masker and target (for a review see, e.g., [START_REF] Moore | An introduction to the psychology of hearing[END_REF]).

In simultaneous masking, the masker is present throughout the presentation time of the target (i.e., the temporal shift between masker and target, ∆T , equals zero) and the frequency shift (∆F ) between masker and target is varied, resulting in the spectral masking function. To vary the ∆F parameter, either the target frequency (F T ) is fixed and the masker frequency (F M ) is varied, or vice versa. When F T is fixed the masking function measures the response of a single auditory filter (i.e., the filter centered on F T ). 4 As a result, such functions (called "psychoacoustical tuning curves" or "filter functions" depending on whether the masker or the target is fixed in level) are commonly used as estimates of auditory frequency selectivity [START_REF] Moore | An introduction to the psychology of hearing[END_REF]Chap. 3]. When F M is fixed it is common to plot the target level at threshold (L T ) or amount of masking (see Footnote 3) as a function of F T for a fixed-level masker, which is called a "masking pattern". Because a masking pattern measures the responses of different auditory filters (i.e., those centered on the individual F T s), it can be interpreted as an indicator of the spectral spread of masking produced by the masker. The physiological mechanisms of simultaneous masking can be of two origins [START_REF] Delgutte | Physiological mechanisms of psychophysical masking: Observations from auditory-nerve fibers[END_REF]: excitation and suppression. Excitation refers to the spread of excitation produced by the masker to the place responding to the target on the basilar membrane (BM). In other terms, the spread of excitation masks the BM response to the target. Suppression refers to the suppression or "inhibition" of the BM response to the target by the masker, even if the masker does not produce excitation at the place responding to the target (for an illustration of the excitatory and suppressive masking phenomena see [START_REF] Delgutte | Physiological mechanisms of psychophysical masking: Observations from auditory-nerve fibers[END_REF]Fig. 1]). Excitation and suppression are not mutually exclusive. Their relative contributions depend on the frequency and level relationships between masker and target.

In non-simultaneous masking, ∆F most often equals zero and ∆T is varied, resulting in the temporal masking function. Backward masking (the target precedes the masker, ∆T 0) is weaker than forward masking (the masker 3 The detection threshold of the target measured in presence of the masker represents the masked threshold, whereas the detection threshold of the target measured in quiet represents the absolute threshold. The difference between the masked threshold and the absolute threshold (in dB) represents the "amount of masking". 4 The spectral resolution in the auditory system can be approximated by a bank of bandpass filters with a constant relative bandwidth. Each of these filters, named auditory filters, is characterized by its equivalent rectangular bandwidth (ERB) in Hz. This concept led to the definition of the so-called "ERB scale" that allows to plot signals or psychoacoustical data on a frequency scale related to human auditory perception [START_REF] Moore | An introduction to the psychology of hearing[END_REF]Chap. 3].

precedes the target, ∆T 0). The amounts of backward and forward masking depend on masker duration [START_REF] Duifhuis | Consequences of peripheral frequency selectivity for nonsimultaneous masking[END_REF][START_REF] Fastl | Temporal masking effects: III. Pure tone masker[END_REF][START_REF] Zwicker | Dependence of post-masking on masker duration and its relation to temporal effects in loudness[END_REF]. Although the mechanisms underlying backward masking remain unclear, the most accepted explanation is that it is of peripheral origin. Backwards masking would be caused by the temporal overlap of the BM responses to masker and target at the outputs of the auditory filters [START_REF] Duifhuis | Consequences of peripheral frequency selectivity for nonsimultaneous masking[END_REF]. The amount of overlap depends on the "ringing" time of the BM, i.e., the length of the impulse response of the BM, which itself depends on signal frequency. 5 Note, however, that backward masking studies often reported large inter-listener differences (e.g., [START_REF] Fastl | Temporal masking effects: III. Pure tone masker[END_REF]) and that trained listeners often show little or no backward masking. Thus, backward masking may also reflect some confusion effects between masker and target [START_REF] Moore | An introduction to the psychology of hearing[END_REF]. Forward masking can be attributed to three mechanisms. The first is the temporal overlap of the BM responses to masker and target at the outputs of the auditory filters as a consequence of the filters' ringing. This phenomenon is more likely to be involved with small values of ∆T . The second is short-term adaptation or "fatigue": the exponential decay of masker-induced excitation over time in the cochlea and in the auditory nerve can reduce the response to a target presented shortly after the extinction of the masker [START_REF] Duifhuis | Consequences of peripheral frequency selectivity for nonsimultaneous masking[END_REF][START_REF] Fastl | Temporal masking effects: III. Pure tone masker[END_REF][START_REF] Soderquist | Backward, simultaneous, and forward masking as a function of signal delay and frequency[END_REF][START_REF] Zwicker | Dependence of post-masking on masker duration and its relation to temporal effects in loudness[END_REF]. The third is temporal integration or "persistence" of masker excitation: the neural representation of the masker is smoothed over time by an integration process so that the representation of the masker overlaps with the representation of the target at some stage in the auditory system [START_REF] Plack | Linear and nonlinear processes in temporal masking[END_REF][START_REF] Plack | Basilar-membrane nonlinearity and the growth of forward masking[END_REF]. To date, however, the distinction between short-term adaptation and temporal integration as the most probable explanation to forward masking is still a matter of debate [START_REF] Oxenham | Forward masking: Adaptation or integration?[END_REF].

Because of the specific demands in the simultaneous and non-simultaneous masking experiments reported in the literature, the experimental stimuli were almost always broad either in the temporal domain (e.g., long-lasting sinusoids), the frequency domain (e.g., clicks), or both.

A few studies investigated how masking spreads in the TF domain (i.e., by measuring masking patterns for various ∆T s) [START_REF] Fastl | Temporal masking effects: III. Pure tone masker[END_REF][START_REF] Kidd | Patterns of residual masking[END_REF][START_REF] Moore | Behavioural measurement of leveldependent shifts in the vibration pattern on the basilar membrane[END_REF][START_REF] Soderquist | Backward, simultaneous, and forward masking as a function of signal delay and frequency[END_REF]. Those studies involved relatively long (duration 100 ms) sinusoidal maskers, that is, maskers with good concentration in frequency but not in time. Overall, little is known about the spread of TF masking for a masker with good concentration both in time and frequency.

Models of Auditory Masking

The results of the spectral and temporal masking experiments reported in the literature were used to develop models of either spectral, temporal, or TF masking. Masking models are useful to both the fields of psychoacoustics and sound signal processing. In psychoacoustics they allow to improve our understanding of auditory processing. In signal processing they allow to exploit auditory masking in some applications, for instance in perceptual audio coding. To reduce the digital size of audio files, audio codecs decompose sounds into TF segments and use masking models to reduce the bit rates in those segments (for a review on audio coding techniques see [START_REF] Spanias | Audio Signal Processing and Coding[END_REF]).

Masking models can be classified into two groups: excitation pattern-based models and auditory processing-based models. Excitation pattern-based models transform the short-term spectrum of the input signal into an excitation pattern reflecting the spread of excitation induced by the signal on the BM. This approach is based on the power-spectrum model of masking in which the auditory periphery is conceived as a bank of bandpass filters (see Footnote 4). Masking is then determined by the target-to-masker ratio at the output of each filter. This group of models mostly includes spectral masking models (e.g., [START_REF] Glasberg | Development and evaluation of a model for predicting the audibility of time-varying sounds in the presence of background sounds[END_REF][START_REF] Van Der Heijden | Using an excitation-pattern model to predict auditory masking[END_REF]) and is the technique most frequently employed in audio codecs [START_REF] Spanias | Audio Signal Processing and Coding[END_REF].

In contrast, auditory processing-based models attempt to simulate the effective signal processing in the auditory system. Such models consist of a series of processing stages and a decision stage on which the prediction of masking is based [START_REF] Jepsen | A computational model of human auditory signal processing and perception[END_REF][START_REF] Plack | Linear and nonlinear processes in temporal masking[END_REF]. The model described in [START_REF] Plack | Linear and nonlinear processes in temporal masking[END_REF] is able to predict temporal and TF masking data. The model described in [START_REF] Jepsen | A computational model of human auditory signal processing and perception[END_REF] is able to predict temporal and spectral masking data but has not been tested on TF conditions. Because auditory models usually have a high computational complexity and are not invertible, they are rarely used in audio processing algorithms.

To obtain a perceptually relevant audio signal representation based on a perfectly invertible transform, we propose to exploit masking in TF representations of sounds, that is, predict the audibility of each TF atom in the signal decompositions. To do so, a model of TF masking is required. There exist some models of TF masking that are currently implemented in audio coding algorithms [START_REF] Hamdi | Low bit rate high quality audio coding with combined harmonic and wavelet representations[END_REF][START_REF] He | Psychoacoustic music analysis based on the discrete wavelet packet transform[END_REF][START_REF] Huang | A new audio coding scheme using a forward masking model and perceptually weighted vector quantization[END_REF][START_REF] Vafin | Exploiting time and frequency masking in consistent sinusoidal analysis-synthesis[END_REF]. In the cited studies, the predictions of TF masking are based on a simple superposition of spectral and temporal masking functions (typically, only forward masking is considered). The decay of forward masking is modeled with a linear function of logÔ∆T Õ [START_REF] Hamdi | Low bit rate high quality audio coding with combined harmonic and wavelet representations[END_REF][START_REF] He | Psychoacoustic music analysis based on the discrete wavelet packet transform[END_REF][START_REF] Vafin | Exploiting time and frequency masking in consistent sinusoidal analysis-synthesis[END_REF], or with an exponential function of the form e ¡Ô∆TßτÕ where τ is a time constant depending both on frequency (∆F ) and level [START_REF] Huang | A new audio coding scheme using a forward masking model and perceptually weighted vector quantization[END_REF]. Given the highly nonlinear behavior of cochlear mechanics (e.g., [START_REF] Robles | Mechanics of the mammalian cochlea[END_REF]), such a simple combination of spectral and temporal masking functions is unlikely to correctly predict TF masking. Accordingly, the results presented in Sec. 2.3 reveal that such approaches are not adequate for predicting the audibility of TF atoms. To do so, it seems more appropriate to use masking functions that are based on the spread of TF masking produced by a maximally-compact masker. Because previous psychoacoustical studies mostly focused either on temporal or on spectral masking and used stimuli with temporally and/or spectrally broad supports, the spread of TF masking for a signal that is maximally compact in the TF plane cannot easily be derived from available data and therefore has to be measured.

Outline of the Present Study

The present paper consists of two main parts. In the first part, the results of psychoacoustical experiments on masking using maximally-compact stimuli are presented. To best fulfill the requirement of maximum compactness in the TF plane, we used Gaussian-shaped sinusoids (referred to as Gaussians) as masker and target stimuli. Three experiments were conducted. The spectral and temporal masking functions for Gaussian maskers were measured in Experiments 1 and 2, respectively. In Experiment 3, the TF spread of masking was measured. We then tested with which accuracy the results from Exp. 3 can be predicted based on the results from Exps. 1 and 2 (assuming a simple superposition of spectral and temporal masking effects). This allowed us to examine the accuracy of simple TF masking models currently implemented in some perceptual audio codecs.

In the second part, the "extended" irrelevance filter based on psychoacoustical data on TF masking is described. Then, preliminary results are presented and discussed.

Psychoacoustical Measurements of Masking Using

Gaussian Stimuli

General Methods

Stimuli. Masker and target were Gaussian-shaped sinusoids (Gaussians) defined by [START_REF] Necciari | Auditory time-frequency masking: Psychoacoustical measures and application to the analysis-synthesis of sound signals[END_REF][START_REF] Van Schijndel | Intensity discrimination of Gaussian-windowed tones: Indications for the shape of the auditory frequency-time window[END_REF] sÔtÕ Γ sin

¡ 2πf 0 t π 4 © e ¡πÔΓtÕ 2 (1)
where f 0 is the carrier frequency, Γ defines the equivalent rectangular bandwidth (ERB), and Γ ¡1 defines the equivalent rectangular duration (ERD) of sÔtÕ. In our experiment, Γ was set to 600 Hz, corresponding to Γ ¡1 = 1.7 ms. The f 0 value varied depending on ∆F . By introducing the πß4 phase shift, the energy of the signal is independent of f 0 . Since a Gaussian window has infinite duration, the signals were windowed in the time domain using a Tukey window. The "effective duration" (defined as the 0-amplitude points duration) of the stimuli was 9.6 ms and the cutoff in the frequency domain was located at the 220-dB down points. The sound pressure level (SPL) of the Gaussian was specified by measuring the SPL of a long-lasting sinusoid having the same frequency (f 0 ) and maximum amplitude as the carrier tone of the Gaussian.

Procedure. Thresholds were estimated using a three-interval, three-alternative forced-choice procedure with a 3-down-1-up criterion that estimates the 79.4%correct point on the psychometric function. Each trial consisted of three 200-ms observation intervals visually indicated on the response box, with a betweeninterval gap of 800 ms. The masker was presented in the three intervals and the target was presented with the masker in one of those intervals, chosen randomly. The listener indicated in which interval he/she heard the target by pressing one of three buttons on the response box. Immediate feedback on the correctness of the response was visually provided to the listener. The target level varied adaptively by initial steps of 5 dB and 2 dB following the second reversal. Twelve reversals were obtained. The threshold estimate was the mean of the target levels at the last 10 reversals. A threshold estimate was discarded when the standard deviation of these 10 reversals exceeded 5 dB. Two threshold estimates were obtained for each condition. If the standard deviation of these two estimates exceeded 3 dB, up to four additional estimates were completed. The final threshold was the average across all estimates (maximum = 6).

Apparatus. A personal computer was used to control the experiments and generate the stimuli. Stimuli were output at a 48-kHz sampling rate and a 24-bit resolution using an external digital-to-analog converter (Tucker-Davis Technologies (TDT) System III), attenuated (TDT PA5) and passed to a headphone buffer (TDT HB7), and to the right ear-pad of a circumaural headphone (Sennheiser HD545).The headphones were calibrated so that levels were considered as SPL close to the eardrum. Listeners were tested individually in a double-walled, sound-attenuated booth.

Listeners. Six normal-hearing listeners participated in Exps. 1 and 2. Four of the listeners (L1-L4) participated in Exp. 3.

Experimental Conditions. Throughout the experiments, the carrier frequency of the masker was fixed to 4 kHz. Its sensation level (i.e., the level above the absolute threshold of the masker for each listener, see Footnote 3) was fixed to 60 dB, which corresponded to SPLs of 81-84 dB across listeners.

Spectral Masking. Masker and target were presented simultaneously (∆T = 0). Masking patterns were measured for 11 values of ∆F , defined in the ERB scale: -4, -3, -2, -1, 0, +1, +2, +3, +4, +5, and +6 ERB units. 6 To prevent cochlear combination products from being detected and thus from producing irregularities in the masking patterns [START_REF] Moore | An introduction to the psychology of hearing[END_REF], a continuous background noise was added for all ∆F s 0 [START_REF] Necciari | Auditory time-frequency masking: Psychoacoustical measures and application to the analysis-synthesis of sound signals[END_REF]. Each session contained conditions measured either with or without background noise. The order of sessions (with noise; without) was counterbalanced over days. Within a session, ∆F was chosen randomly.

Temporal Masking. Masker and target had the same carrier frequency (∆F = 0). Because a pilot experiment indicated very little backward masking for such short maskers, we focused on forward masking. ∆T , defined as the time shift between masker onset and target onset, was 0, 5, 10, 20, or 30 ms. Within a session, ∆T was chosen randomly. Time-Frequency Masking. Both ∆T and ∆F were varied. Masked thresholds were measured for 30 out of 40 possible ∆T ¢ ∆F combinations (i.e., 5 ∆T s from Exp. 2 ¢ 8 ∆F s from Exp. 1). Although the effect of cochlear combination products is usually ignored in forward masking studies, we used a background noise identical to that of Exp. 1 to mask potential cochlear combination products (when ∆F 0) because of the small ∆T values. The whole set of conditions was split into two groups: frequency separations measured with and without background noise. Then, experimental blocks were formed that contained the ∆T conditions for each ∆F . The order of blocks and groups was randomized across sessions. Within a session, the target frequency was fixed and ∆T was chosen randomly.

Results

This section presents the data with respect to their applications described in Secs. 2.3 and 3. A more thorough description and interpretation of the data can be found in [START_REF] Necciari | Auditory time-frequency masking: Psychoacoustical measures and application to the analysis-synthesis of sound signals[END_REF].

Experiment 1: Spectral Masking. Figure 1 presents the individual and mean amounts of masking (in dB) as a function of ∆F (in ERB units). First, in some listeners a dip (L1, L3 and L4) or a plateau (L5) was observed instead of a peak at ∆F = 0.7 It has to be considered that this represents a special condition, where masker and target were exactly the same stimuli presented at the same time. Thus, the listeners could only use as a cue the intensity increase in the interval containing the target. In other words, the listeners performed an intensity discrimination task in this condition [START_REF] Florentine | Level discrimination of tones as a function of duration[END_REF][START_REF] Moore | Masking patterns for sinusoidal and narrow-band noise maskers[END_REF][START_REF] Necciari | Auditory time-frequency masking: Psychoacoustical measures and application to the analysis-synthesis of sound signals[END_REF].

Second, for all listeners and ∆F 2 ERB units, the amount of masking decreased as ∆F increased. The decrease was more abrupt for F T F M than for F T F M . Regression lines computed for each side of the masking patterns and listener (straight lines in Fig. 1) indeed show that, on average, the slopes for F T F M (mean slope = +60 dB/octave) are 1.6 times those for F T F M (mean slope = -39 dB/octave). This steeper masking decay for F T F M is consistent with that reported in classical spectral masking studies (see, e.g., [START_REF] Moore | An introduction to the psychology of hearing[END_REF][START_REF] Moore | Masking patterns for sinusoidal and narrow-band noise maskers[END_REF] for a review).

Experiment 2: Temporal Masking. Figure 2 presents the individual and mean amounts of masking as a function of ∆T on a logarithmic scale. On average, masking decreased from 50 dB for ∆T = 0 to about 6 dB for ∆T = 30 ms. The data for ∆T 0 are well fitted with straight lines, a result consistent with almost all previous forward masking studies using various types of maskers (e.g., [START_REF] Duifhuis | Consequences of peripheral frequency selectivity for nonsimultaneous masking[END_REF][START_REF] Fastl | Temporal masking effects: III. Pure tone masker[END_REF][START_REF] Zwicker | Dependence of post-masking on masker duration and its relation to temporal effects in loudness[END_REF]). A straightforward description of these data is provided by Fig. 1: Results of Experiment 1: amount of masking (in dB) as a function of ∆F (in ERB units). Data were fitted with linear regression lines on each side of the masking patterns (excluding the point at ∆F = 0). The bottom panel shows the mean data with ¨1 standard deviation bars [START_REF] Necciari | Auditory time-frequency masking: Psychoacoustical measures and application to the analysis-synthesis of sound signals[END_REF].

AM α logÔ∆T Õ β (2)
where AM is the amount of masking, α is the slope of the forward masking decay, and β is the offset of the forward masking decay. Table 1 lists the values of α and β determined by applying a weighted-least-squares fit of (2) to the data for ∆T 0. To take the variability of each data point into account in the estimation of parameters α and β, the weight of each data point corresponded to the reciprocal of the variance of the measurement. Experiment 3: Time-Frequency Masking. Figure 3 presents the results as simultaneous and forward masking patterns, that is, the amount of masking as a function of ∆F with ∆T as the parameter. For all ∆F s, the largest amount Table 1: Values of parameters α (in dB/logÔ∆T Õ) and β (in dB) determined by fitting [START_REF] Balazs | Theory, implementation and applications of nonstationary Gabor frames[END_REF] to the data for ∆T 0 in Fig. 2 using a weighted-least-squares criterion. Note that because the ∆T axis is logarithmically scaled in Fig. 2, the values of β correspond to the y-intercepts at ∆T = 1 ms. The last column indicates r 2 values. of masking was obtained in the simultaneous condition (∆T = 0). Masking dropped as ∆T increased to 5 ms. For ∆T s 10 ms, masking was generally less than 10 dB for all ∆F s. To assess whether the patterns broadened or narrowed with increasing ∆T , we estimated the quality factors at the -3-dB bandwidth (Q 3dB ) [START_REF] Kidd | Patterns of residual masking[END_REF]. The mean values of Q 3dB are 12, 3, and 2 for ∆T = 0, 5, and 10 ms, respectively, i.e., the patterns flattened as ∆T increased. The mean masking patterns in Fig. 3 are asymmetric for all ∆T s. Finally, the dip/plateau observed in listeners L1, L3, and L4 at ∆F = 0 for ∆T = 0 (see also Fig. 1) disappeared when ∆T increased. For ∆T 0, listeners L1 and L3 exhibited a peak at ∆F = +1 instead of 0. In other terms, these two listeners revealed a shift in the maximum masking frequency towards F T s F M .

Our pattern of results is consistent with the few preceding studies that measured TF masking patterns with long maskers [START_REF] Fastl | Temporal masking effects: III. Pure tone masker[END_REF][START_REF] Kidd | Patterns of residual masking[END_REF][START_REF] Moore | Behavioural measurement of leveldependent shifts in the vibration pattern on the basilar membrane[END_REF][START_REF] Soderquist | Backward, simultaneous, and forward masking as a function of signal delay and frequency[END_REF] in that (i) masking patterns flatten with increasing ∆T , (ii) the masking patterns' asymmetry remains for ∆T 0, and (iii) a shift in the maximum masking frequency towards F T s F M is observed in some listeners for ∆T 0. However, because TF masking is affected by nonlinear processes in the cochlea [START_REF] Laback | Additivity of nonsimultaneous masking for short Gaussian-shaped sinusoids[END_REF][START_REF] Moore | An introduction to the psychology of hearing[END_REF][START_REF] Necciari | Auditory time-frequency masking: Psychoacoustical measures and application to the analysis-synthesis of sound signals[END_REF], the present data could not have been deduced from existing data for long maskers.

Our results are summarized in the three-dimension plot in Fig. 4. To provide a smooth and "complete" representation of TF masking (i.e., one that reaches 0 dB of masking), the ∆T axis was sampled at 1 kHz and the data for ∆F s below -4 and above +6 ERB units were then extrapolated based on a twodimensional cubic spline fit along the TF plane. Overall, the function shown in Fig. 4 represents the TF spread of masking produced by a Gaussian TF atom.

Accuracy of Simple Time-Frequency Masking Models Used in Perceptual Audio Codecs

To examine the accuracy of simple TF masking models currently used in some audio codecs, we tested two prediction schemes assuming a linear combination Fig. 4: Mean TF masking data extrapolated and plotted in the TF plane [START_REF] Necciari | Auditory time-frequency masking: Psychoacoustical measures and application to the analysis-synthesis of sound signals[END_REF] of spectral and temporal masking. Specifically, we tested with which accuracy the results of Exp. 3 can be predicted based on the results of Exps. 1 and 2. The general idea of the prediction is that the spread of TF masking caused by a masker can be described by the spectral masking pattern combined with the decay of forward masking from each point of the masking pattern. In the following, let AM Ô∆T, ∆F Õ denote the amount of masking produced by the masker on a target separated from the masker by ∆T and ∆F in the TF plane (∆T 0, ∆F 0).

Simple Superposition of Spectral and Temporal Masking Functions.

We first considered a simple superposition of the spectral and temporal masking functions to predict TF masking. A similar approach is used in [START_REF] Hamdi | Low bit rate high quality audio coding with combined harmonic and wavelet representations[END_REF][START_REF] He | Psychoacoustic music analysis based on the discrete wavelet packet transform[END_REF][START_REF] Vafin | Exploiting time and frequency masking in consistent sinusoidal analysis-synthesis[END_REF]. This prediction scheme, referred to as "Prediction A", is given by

AM Ô∆T, ∆F Õ AM Ô0, ∆F Õ ¡ AM Ô0, 0Õ ¡ AM Ô∆T, 0Õ ¨(3)
where AM Ô0, ∆F Õ represents the "initial" spread of masking produced by the masker at the target frequency (read from Fig. 1) from which is subtracted the temporal decay of forward masking over time ∆T (read from Fig. 2). The mean masking patterns predicted with Prediction A for ∆T values of 5, 10, and 20 ms are depicted in Fig. 5 (crosses, solid lines). It is clear from the figure that Prediction A overestimates the amount of masking for small frequency separations ( ∆F 2 ERB units) and underestimates masking for larger ∆F s. One obvious reason for the inefficiency of Prediction A is the fact that it does not take into account the ∆F dependency of the forward masking decay.

Superposition of Spectral Masking Function and Level-Dependent

Temporal Masking Function. An approach which takes into account the ∆F dependency of forward masking is used in [START_REF] Huang | A new audio coding scheme using a forward masking model and perceptually weighted vector quantization[END_REF]. In this approach (referred to as "Prediction B"), each point of the spectral masking pattern with F M F T is considered as a hypothetical forward masker with F M F T but with a lower level. This consideration follows from the analogy reported between the ∆F -and level-dependency of forward masking [START_REF] Necciari | Auditory time-frequency masking: Psychoacoustical measures and application to the analysis-synthesis of sound signals[END_REF]. Prediction B has the form

AM Ô∆T, ∆F Õ AM Ô0, ∆F Õ ¡ α ½ logÔ∆T Õ (4)
with α ½ AM Ô0, ∆F Õß logÔ∆T 0dB Õ, ∆T 0dB being the ∆T -axis intercept, or 0-dB masking point, at which the forward masking functions converge. Given the parameters α and β determined by fitting (2) to the temporal masking data presented in Fig. 2, ∆T 0dB 10 ¡βßα (see Tab. 1). As for Prediction A, AM Ô0, ∆F Õ is determined from the spectral masking data. The mean masking patterns predicted with Prediction B for ∆T values of 5, 10, and 20 ms are depicted in Fig. 5 (filled diamonds, dashed lines). It can be seen that the shape of the masking patterns with Prediction B is more similar to the data than that with Prediction A. Nevertheless, it is clear that Prediction B overestimates the amount of masking in almost all conditions, the overestimation being particularly large for small frequency separations ( ∆F 2 ERB units). Overall, both prediction schemes failed in accurately predicting TF masking data for Gaussian stimuli. This confirms that TF masking is not predictable by assuming a simple combination of temporal and spectral masking functions.

Interim Summary

To obtain a measure of the TF spread of masking produced by a signal with maximal concentration in the TF plane, three experiments were conducted that involved Gaussian-shaped sinusoids with fixed bandwidth (ERB = 600 Hz) and duration (ERD = 1.7 ms) both as masker and target. In all experiments, the masker had a carrier frequency of 4 kHz and a sensation level of 60 dB. The target was shifted relative to the masker either in frequency, in time, or both.

The results of the frequency-and time-shift conditions showed that the superposition of spectral and temporal masking effects, as currently implemented in some perceptual audio codecs, does not provide an accurate representation of the measured TF masking effects for Gaussian maskers. These results suggest that audio coding algorithms using such an approach provide rather erroneous predictions of TF masking.

The results of the TF conditions provide the TF spread of masking produced by a Gaussian TF atom. These new data constitute a crucial basis for the prediction of auditory masking in audio TF representations. This is addressed in the following section.

Exploiting Time-Frequency Masking in a Time-Frequency Transform: Improvement of the Irrelevance Filter Algorithm

The concept of the irrelevance filter was first introduced in [START_REF] Balazs | Time-frequency sparsity by removing perceptually irrelevant components using a simple model of simultaneous masking[END_REF]. It consists in removing the inaudible atoms in a Gabor transform while causing no audible difference to the original sound after re-synthesis. The algorithm first determines an estimation of the masked threshold based on a simple model of spectral masking. The masked threshold is then shifted in level by an amount determined experimentally, which results in the "irrelevance threshold". This shift gives a conservative way to deal with uncertainty effects resulting from removing TF atoms and with inaccuracies in the masking model. Next, all TF atoms falling below threshold are removed. Although a perceptual test performed in [START_REF] Balazs | Time-frequency sparsity by removing perceptually irrelevant components using a simple model of simultaneous masking[END_REF] with 36 normal-hearing listeners indicated that, on average, 36% of the atoms can be removed without causing any audible difference to the original sound after resynthesis, the irrelevance filter algorithm can be improved. The main limitations of the algorithm are the fixed resolution in the Gabor transform and the use of a simple spectral masking model to predict masking in the TF domain.

In this section, a preliminary version of the extended irrelevance filter is presented. Because the algorithm presented below differs from the original algorithm in many aspects, including signal representation, masking model, and irrelevance threshold calculation, no direct comparison can be established between the two versions. Moreover, because the new algorithm is still being developed, it has not been formally evaluated yet (e.g., by conducting perceptual listening tests with natural sounds). Thus, we merely evaluate the performance of the new algorithm based on preliminary results with deterministic signals and informal listening by the authors.

Choice of the Signal Representation: Wavelet Transform

To mimic the spectral resolution of the human auditory system, a signal representation allowing a variable frequency resolution is required. The continuous wavelet transform (CWT) fulfills this requirement, unlike the Gabor transform that allows only a fixed TF resolution (e.g., [START_REF] Gröchening | Foundations of time-frequency analysis[END_REF]). Thus, the CWT was chosen as the TF analysis-synthesis scheme used in this paper. In the following we summarize some general theory on wavelets (for a more detailed description see, e.g., [START_REF] Daubechies | Ten Lectures on Wavelets[END_REF][START_REF] Vetterli | Wavelets and subband coding[END_REF]) and describe the practical implementation of the CWT used in our algorithm.

The CWT results from the decomposition of a signal into a family of functions that are scaled versions of a prototype function ("mother wavelet") gÔtÕ according to

g a ÔtÕ 1 a g ¢ t a ª Ôa È IR ¦ Õ ( 5
)
where a is a scale factor allowing to compress (a 1) or dilate (a 1) the mother wavelet gÔtÕ (a 1). This parameter defines the time and frequency resolution in the TF plane. The scale is linked to the frequency according to ω ω 0 ßa where ω 0 is the pulsation of the mother wavelet. Then, for any signal where xÔωÕ and ĝa ÔωÕ denote the Fourier transforms of xÔtÕ and g a ÔtÕ, respectively. The CWT is invertible if and only if gÔtÕ fulfills the admissibility condition

xÔtÕ, CW T x Ôb, aÕ Ü g a,b , x Ý 1 a ÷ ¡ xÔtÕ g ¢ t ¡ b a ª dt ( 6
C g 1 2π ÷ ¡ ĝÔωÕ 2 ω dω (8)
which implies that gÔtÕ be of finite energy. This is usually fulfilled in practice since gÔtÕ must be a function oscillating in time (hence the name "wavelet"). Additionally, gÔtÕ must be of zero mean. Finally, the reconstruction formula is

xÔtÕ 1 C g ø a 0, IR CW T x Ôb, aÕ g a,b ÔtÕ dadb a 2 (9)
which reflects the "atomic" decomposition of xÔtÕ into wavelets.

The CWT has the properties of being linear and ensuring energy conservation. Another property is the "reproducing kernel", which states that

CW T Ôa ½ , b ½ Õ 1 C g ø IR K g Ôa ½ , b ½ , a, bÕCW T Ôa, bÕ dadb a 2 (10)
where K g Ôa ½ , b ½ , a, bÕ Ü g a,b , g a ½ ,b ½ Ý is called the reproducing kernel. Equa- tion [START_REF] Gröchening | Foundations of time-frequency analysis[END_REF] means that the reproducing kernel ensures a strong correlation between all components in the CWT. In other words, any component at location a ½ , b ½ depends upon the remote component at location a, b through the reproducing kernel. This is reflected by the fact that the CWT is highly redundant. The numerical implementation of the CWT requires the discretization of the time and scale variables b, a and the choice of the mother wavelet gÔtÕ. In our implementation we chose the following discretization Ôa j , b k Õ Ôa j 0 , kb 0 Õ where a 0 2 and b 0 1ßF S (F S being the sampling frequency) are two constants defining the size of the sampling grid. Furthermore, we opted for a sub-sampling of the scale in voices and octaves such that

a j 0 2 m Dv n a m,n
where m È Ö0, . . . ,

D v ¡ 1×, n È Ö0, . . . , D o ¡ 1×, and j È Ö0, . . . , D v D o ¡ 1×, D v
and D o being the number of voices and octaves, respectively. This discretization yields scale factors a j 0 1 with increment steps of 2 1ßDv . Moreover, it provides two parameters, D v and D o , for determining the total number of scales in the representation.

Regarding the choice of the mother wavelet, the accurate prediction of masking in the time-scale domain requires that the spectro-temporal characteristics of the wavelets match the spectro-temporal characteristics of the masker used in the psychoacoustical experiment. Thus, a straightforward solution for gÔtÕ was to use a Gaussian-shaped sinusoid similar to that defined in [START_REF] Agerkvist | A time-frequency auditory model using wavelet packets[END_REF]. Because the CWT was computed in the frequency domain according to [START_REF] Fastl | Temporal masking effects: III. Pure tone masker[END_REF], we defined the following function for the mother wavelet

ĝÔωÕ 1 2i Γ e ¡πÔ ω¡ω 0 Γ Õ 2 (11)
where Γ µf 0 µ ω0 2π , µ being the shape factor of the Gaussian window. To provide a Γ value of 600 Hz at f 0 = 4 kHz as used in the psychoacoustical experiment (see Sec. 2), µ was set to 0.15. Note that ĝÔωÕ corresponds to the positive-frequency components of the Fourier transform of sÔtÕ in [START_REF] Agerkvist | A time-frequency auditory model using wavelet packets[END_REF].

The frequency of the mother wavelet (f 0 ) was set to 3F S ß8. Because we used only scale factors a j 0 1 (i.e., we used only compressed versions of ĝÔωÕ), f 0 defines the highest center frequency in the signal representation. To cover the whole spectrum of audible frequencies (i.e., 0.02-20 kHz) while maintaining a large overlap between wavelets, thus to avoid loosing details in the signals, we used 108 scales split into 9 octaves (D o = 9) and 12 voices per octave (D v = 12). At F S = 44.1 kHz and µ = 0.15, the highest-frequency analysis filter had a center frequency f 0 = 16.5 kHz and a bandwidth of 2.5 kHz. The lowest-frequency filter had a center frequency f 0 ßa DvDo¡1 0 = 33.8 Hz and a bandwidth of 5 Hz.

Implementation of the Irrelevance Filter

The gathered TF masking data were used to predict masking in the time-scale domain. More specifically, the TF masking function in Fig. 4 was used as a masking kernel in the time-scale domain. Accordingly, this function had to be discretized in time and scales. Because we conserved all time samples of the signal, the ∆T axis was sampled at F S . The ∆F axis (in ERB units) had to be matched to the scale axis (in voices and octaves). Considering that the ERB of an auditory filter corresponds to approximately one third of octave [START_REF] Moore | An introduction to the psychology of hearing[END_REF] and the present analysis counts 12 voices per octave, one ERB unit was associated with 4 voices. The TF masking kernel in Fig. 4 covers a range of 15 ERB units (∆F = -5 to +10). Thus, the ∆F axis should be divided into 61 voices. This was achieved by interpolating the ∆F axis at a sampling rate of 4 voices per ERB unit based on a two-dimensional cubic spline fit along the TF plane.

In the following, we denote by XÔa, bÕ, a

Øa j ; j 0, . . . , D o D v ¡ 1Ù the discrete wavelet transform (DWT) of the input signal xÔkÕ, k being the discrete time variable such that t kT S (all signals were sampled at F S = 44.1 kHz). The representation from which components have been removed is referred to as Ö XÔa, bÕ. Accordingly, the output signal (reconstructed from Ö XÔa, bÕ) is referred to as xÔkÕ. MÔa, bÕ refers to the discrete masking kernel in dB.

The structure of the irrelevance time-scale filter is shown in Fig. 6. The algorithm includes three main steps:

1. Scale the modulus of the DWT in dB SPL. The difficulty in the SPL normalization is that the actual playback level remains unknown during the entire signal processing. We considered that an amplitude variation of ¨1 bit in the signal is associated with a SPL of 0 dB, while a full-scale signal is associated with a SPL close to 92 dB [START_REF] Spanias | Audio Signal Processing and Coding[END_REF]. 2. Identify local maskers, i.e., local maxima in the transform that fulfill XÔa, bÕ T qÔa, ¤Õ 60 (in dB SPL) where T qÔaÕ is an analytic function approximating the absolute threshold of a normal-hearing listener in dB SPL. It is given by [START_REF] Terhardt | Calculating virtual pitch[END_REF] T q ÔaÕ 3.64 Ôf 0 ßaÕ ¡0.8 ¡ 6.5 e ¡0.6Ô f 0 a ¡3.3Õ Note that a more straightforward approach could have consisted in applying MÔa, bÕ to the whole time-scale domain, that is, without identifying local maskers in the transform (Step 2). However, we opted for a local application of MÔa, bÕ because the amount of masking highly depends on level [START_REF] Moore | An introduction to the psychology of hearing[END_REF]. Thus, applying the TF masking kernel derived from data measured with an average masker SPL of 84 dB to components with SPLs below 84 dB is likely to result in an overestimation of masking. This would in turn result in the removal of audible components. To process all components in the transform, a level-dependent masking kernel is required.

Results

We present below the results obtained when the irrelevance filter was applied on deterministic and musical signals. 8 Two conditions measured in Exp. 3 were tested: one condition ("Condition 1": ∆F = +4 ERB units, ∆T = 10 ms, target SPL (L T ) = 50 dB) where the target is not masked and another condition ("Condition 2": ∆F = -2 ERB units, ∆T = 5 ms, L T = 15 dB) where the target is masked. A test signal xÔtÕ composed of two Gaussians (see [START_REF] Agerkvist | A time-frequency auditory model using wavelet packets[END_REF]) with time and frequency shifts was synthesized as follows: [START_REF] He | Psychoacoustic music analysis based on the discrete wavelet packet transform[END_REF] with g l ÔtÕ A l sin 2πf l t π 4 ¨e¡πÔΓ tÕ 2 , l ØM, T Ù where A l allows to control the signal SPLs9 . Let x 1 ÔtÕ and x 2 ÔtÕ denote the test signals for Conditions 1 and 2, respectively. Their parameters are listed in Tab. 2.

xÔtÕ g M ÔtÕ ÐÓ ÓÑÓ ÓÒ Masker g T Ôt ¡ ∆T Õ ÐÓÓÓÓÓÑÓÓÓÓÓÒ Target
Consider first x 1 ÔtÕ. Because the target is not masked, the representation of g T ÔtÕ should not be removed from X 1 Ôa, bÕ. Figure 7 depicts the original (Fig. 7a) and modified (Fig. 7b) representations of x 1 ÔtÕ in dB SPL. As expected, it can be seen that the target was not removed from X 1 Ôa, bÕ. However, the representation of g M ÔtÕ was roughly altered by the filter. To evaluate the amount of components filtered out from X 1 Ôa, bÕ, we computed the binary representations associated with X 1 Ôa, bÕ and Ö X 1 Ôa, bÕ . These representations (not shown) comprise pixels '1' where X 1 Ôa, bÕ (respectively, Ö X 1 Ôa, bÕ ) -10 dB SPL and pixels '0' elsewhere. Comparing the DWT of the input representation X 1 Ôa, bÕ and the output representation Ö X 1 Ôa, bÕ indicated that about 45% components were removed. It has to be considered, however, that Ö X 1 Ôa, bÕ in Fig. 7b is not the actual representation of x1 ÔtÕ. Because of the reproducing kernel (see [START_REF] Gröchening | Foundations of time-frequency analysis[END_REF]), reconstructing the signal from the modified representation restores some of the removed components. To illustrate this effect, the modulus of the DWT of x1 ÔtÕ, i.e., the analysis of the reconstructed signal, is represented in Fig. 7c. It can be seen that the masker components removed by the filter were restored by the reproducing kernel. Informal listening revealed no perceptual difference between x1 ÔtÕ and x 1 ÔtÕ, and the reconstruction error (x 1 ÔtÕ ¡ x 1 ÔtÕ) was 10 ¡4 . Consider next x 2 ÔtÕ. In this case the target is masked, and thus the representation of g T ÔtÕ should be removed from X 2 Ôa, bÕ. This was the case, as depicted in Fig. 8b. Computations of the binary representations indicated that about 57% components were removed. As for x1 ÔtÕ, the reproducing kernel restored the masker components removed by the filter. Informal listening revealed no perceptual difference between x2 ÔtÕ and g M ÔtÕ.

Finally, we applied the irrelevance time-scale filter to a musical sound (referred to as x 3 ÔtÕ), namely a clarinet sound representing the note A3. The results are depicted in Fig. 9. Computations of the binary representations indicated that about 50% components were removed. Although the reconstruction error was 10 ¡4 , in that case, informal listening revealed some perceptual differences between x3 ÔtÕ and x 3 ÔtÕ. More specifically, the filter altered the attack of the note, which became noisy.

Discussion

The preliminary results obtained with deterministic sounds indicated that the irrelevance time-scale filter removes information (as predicted from experimental data) in the signal representation while causing little or no audible difference to the original sound after re-synthesis. The result obtained with a musical sound is more challenging: although the filter removed about 50% components, informal listening indicated that the output signal was not perceptually identical to the input signal. This suggests that the filter removed some relevant information. A possible explanation can be attributed to the methods employed itself. Indeed, the present algorithm removes components by setting their amplitudes to zero (Step 3). This operation affects the phase relationships between components, which can in turn result in audible effects in the reconstructed signal. Accordingly, the noisy attack of the re-synthesized clarinet sound is likely to result from phase effects. Moreover, it has to be considered that removing a component in a CWT is tricky. Because of the strong correlation between components (as a consequence of the reproducing kernel), removing a component in a CWT affects other components remote from that component and leads to a new representation which is not a CWT anymore (i.e., it does not satisfy the reproducing kernel property). We were conscious of this problem when developing the algorithm but we found worth trying it. In [START_REF] Balazs | Time-frequency sparsity by removing perceptually irrelevant components using a simple model of simultaneous masking[END_REF], this problem was addressed using a conservative approach: to compensate for the inaccuracies of the masking model, the masking function was shifted in level by an amount determined experimentally. An alternative approach could consist in encoding the masked components on a smaller number of bits than the audible components, as currently done in perceptual audio codecs [START_REF] Spanias | Audio Signal Processing and Coding[END_REF].

Furthermore, the determination of the irrelevance threshold is roughly dependent on the discretization of the CWT. The highly redundant sampling grid we opted for in the present study is likely to have caused an overestimation of masking. In future works, a more appropriate discretization should be chosen so as to represent a Gaussian with a shape factor µ = 0.15 by a single atom. Such a discretization could be, for example, a dyadic grid [START_REF] Vetterli | Wavelets and subband coding[END_REF]. Another possibility could be to exploit the recent theory on non-stationary Gabor frames [START_REF] Balazs | Theory, implementation and applications of nonstationary Gabor frames[END_REF][START_REF] Jaillet | Nonstationary Gabor frames[END_REF]. To overcome the limitation of fixed resolution in the Gabor transform, the nonstationary Gabor transform provides some freedom of evolution of resolution either in time or frequency and allows perfect reconstruction. This constitutes an interesting background for implementing our TF masking data and will be investigated in future works.

To avoid the constraint on the discretization, the discrete masking kernel could be replaced by an explicit function of TF masking allowing the prediction of masking for any TF coordinates. Furthermore, the masking kernel was designed based on TF masking data for a single Gaussian masker with fixed frequency and level. Because masking is highly dependent on frequency and level, additional data are required to develop a model able to accurately predict masking in realworld sounds. Studies are currently underway (e.g., [START_REF] Laback | Additivity of nonsimultaneous masking for short Gaussian-shaped sinusoids[END_REF]) that investigate the additivity of masking arising from multiple Gaussian maskers shifted in time and frequency. It would be interesting to explore the extent to which these new data on the additivity of TF masking can be incorporated in the current algorithm. Combining data on the frequency-and level-dependency of spectral masking for Gaussian atoms gathered in [START_REF] Necciari | Auditory time-frequency masking: Psychoacoustical measures and application to the analysis-synthesis of sound signals[END_REF] and literature data on the level-dependency of temporal masking may allow designing a level-dependent TF masking kernel or function.

Summary and Conclusions

In this paper, the question of the development of a perfectly invertible audio signal representation being as close as possible to "what we see is what we hear" was addressed, with specific considerations to TF representations and auditory masking. The proposed approach consisted in predicting the audibility of each TF atom in the TF representations of sounds based on psychoacoustical data on TF masking. To achieve this approach, data on the spread of TF masking produced by a TF atom (i.e., a signal with maximal concentration in the TF plane) were required. Because (i) a few psychoacoustical studies investigated TF masking and (ii) those studies used stimuli with temporally broad supports, their results could not be used to derive the spread of TF masking for one atom.

Therefore, three psychoacoustical experiments were conducted that involved Gaussian-shaped sinusoids with fixed bandwidth (ERB = 600 Hz) and duration (ERD = 1.7 ms) both as masker and target stimuli. The target was shifted either along the time axis, the frequency axis, or both relative to the masker. The same group or subgroup of listeners participated in all three experiments. The conclusions that can be drawn from our data are:

(i) The superposition of the temporal and spectral masking functions does not provide an accurate representation of the measured TF masking function for a Gaussian masker; (ii) This suggests that audio coding algorithms using such an approach provide rather erroneous predictions of TF masking and that our data may allow to improve the estimation of TF masking in these systems; (iii) These new data constitute a crucial basis for the prediction of auditory masking in the TF representations of sounds.

We proposed an algorithm (referred to as the "extended irrelevance filter") for removing the inaudible atoms in the wavelet transform of a sound while causing little or no audible difference to the original sound after re-synthesis. Preliminary results obtained with deterministic and musical signals are promising. Future works will include: development of a level-dependent model of TF masking, incorporation of the nonlinear additivity of masking, replacement of the CWT by the non-stationary Gabor transform, refinement of the methods to remove the inaudible components, and perceptual validation of the algorithm with calibrated natural sounds.

Fig. 2 :

 2 Fig. 2: Results of Experiment 2: amount of masking (in dB) as a function of ∆T (in ms) on a logarithmic scale with straight-line fits to the data for ∆T 0 according to (2). The fit parameters are listed in Tab. 1. Error bars in the individual panels indicate ¨1 standard deviation across measurements. The bottom panel shows the mean data with ¨1 standard deviation bars [23].

Fig. 3 :

 3 Fig. 3: Results of Experiment 3: amount of masking (in dB) as a function of ∆F (in ERB units) obtained for five ∆T s [23].

Fig. 5 :

 5 Fig. 5: Mean forward masking patterns for ∆T = 5, 10, and 20 ms predicted with Prediction A (¢, solid lines) and Prediction B (¬, dashed lines). Because only one point was measured for ∆F 0 and ∆T = 20 ms, only symbols are used in the bottom left panel. Error bars show ¨1 standard deviation.

  ) provides a two-dimensional representation of xÔtÕ in the time-scale plane, b È IR being the time variable. Using Parseval's relation, (6) can also be written in the frequency domain CW T x Ôb,

Fig. 6 :

 6 Fig. 6: Structure of the irrelevance time-scale filter

Fig. 7 :

 7 Fig. 7: Modulus of the DWT (in dB SPL) of test signal x 1 ÔtÕ (see Tab. 2) (a) at the input and (b) at the output of the irrelevance time-scale filter. (c) Modulus of the DWT (in dB SPL) of the reconstructed signal x1 ÔtÕ.

Fig. 8 :Fig. 9 :

 89 Fig. 8: As in Fig. 7 but for test signal x 2 ÔtÕ.

2

  Ôf 0 ßaÕ 4 1000 with f 0 in kHz. More precisely, Step 2 selects the components whose SPL exceeds the absolute threshold by 60 dB. This selection rule follows from the masker sensation level of 60 dB used in the experiment (see Sec. 2). Let Ω M denote the set of maskers selected in Step 2. 3. Apply the masking kernel MÔa, bÕ (in dB) to each masker in order of descending SPL and iteratively compute the output wavelet transform as Ö

	XÔa, bÕ	XÔa, bÕ 0	if otherwise	XÔa, bÕ	T qÔa, ¤Õ MÔa, bÕ (dB SPL)
	until Ω M is empty.

Table 2 :

 2 Parameters used for test signals x 1 ÔtÕ and x 2 ÔtÕ to simulate experimental Conditions 1 and 2.

		x1ÔtÕ	x2ÔtÕ
	Γ	600	600
	fM (kHz)	4.0	4.0
	LM (dB SPL)	80	80
	fT (kHz)	6.3	3.2
	LT (dB SPL)	50	15
	∆T (ms)	10	5
	Target status not masked masked

An estimation of the ringing time at a given frequency can be obtained by considering the inverse of the ERB (in Hz) of the auditory filter centered on that frequency.

The target frequencies corresponding to these ∆F s were 2521, 2833, 3181, 3568, 4000, 4480, 5015, 5611, 6274,

7012, and 7835 Hz, respectively.

In simultaneous masking, the greatest amount of masking, also referred to as the "maximum masking frequency", is classically located at FT FM , which results in a peak in the masking pattern at ∆F = 0[START_REF] Moore | Masking patterns for sinusoidal and narrow-band noise maskers[END_REF].

The sound files corresponding to each of the results can be downloaded as wav files at: http://www.lma.cnrs-mrs.fr/ ~kronland/cmmr2011.

The SPL of the test signal was controlled by setting the signal amplitudes A l

ÔL l ¡92Õß20 where L l is the desired SPL and 92 dB corresponds to the amplitude of a full-scale signal.

Acknowledgments. This work was supported by grants from Egide (PAI "Amadeus" WTZ 1/2006), the ANR (project "SenSons"), and the WWTF (project "MulAc" MA07025).


Mitsuko Aramaki
email: aramaki@lma.cnrs-mrs.fr

Richard Kronland-Martinet

Sølvi Ystad
email: ystad@lma.cnrs-mrs.fr

Perceptual Control of Environmental Sound Synthesis

Keywords: perceptual control, synthesis, analysis, acoustic descriptors, environmental sounds, abstract sounds

In this article we explain how perceptual control of synthesis processes can be achieved through a multidisciplinary approach relating physical and signal properties of sound sources to evocations induced by sounds. This approach is applied to environmental and abstract sounds in 3 different experiments. In the first experiment a perceptual control of synthesized impact sounds evoking sound sources of different materials and shapes is presented. The second experiment describes an immersive environmental synthesizer simulating different kinds of environmental sounds evoking natural events such as rain, waves, wind and fire. In the last example motion evoked by abstract sounds is investigated. A tool for describing perceived motion through drawings is proposed in this case.

Introduction

The development and the optimization of synthesis models have been important research issues since computers produced the first sounds in the early sixties [START_REF] Mathews | The digital computer as a musical instrument[END_REF]. As computers became increasingly powerful, real-time implementation of synthesis became possible and new research fields related to the control and the development of digital musical instruments appeared. One of the main challenges linked to such fields is the mapping between the control parameters of the interface and the synthesis parameters. Certain synthesis algorithms such as additive synthesis [START_REF] Kronland-Martinet | The use of the wavelet transform for the analysis, synthesis and processing of speech and music sounds[END_REF][START_REF] Kleczkowski | Group additive synthesis[END_REF][START_REF] Aramaki | Analysis-synthesis of impact sounds by realtime dynamic filtering[END_REF] allow for a very precise reconstruction of sounds, but contain a large number of parameters (several hundreds in the case of piano synthesis), which makes the mapping between the control device and the synthesis model complicated. Other synthesis approaches such as global or non-linear approaches (e.g. frequency modulation (FM) or waveshaping [START_REF] Chowning | The synthesis of complex audio spectra by means of frequency modulation[END_REF][START_REF] Brun | Digital waveshaping synthesis[END_REF]) are easier to implement and to control since they contain fewer synthesis parameters, but do not allow for a precise resynthesis. This means that the control device cannot be dissociated from the synthesis model when conceiving a digital musical instrument and even more, a genuine musical interface should go past the technical stage to integrate the creative thought [START_REF] Gobin | From sounds to music: Different approaches to event piloted instruments[END_REF]. So far, a large number of control devices have been developed for musical purposes [START_REF] Miranda | New Digital Musical Instruments: Control And Interaction Beyond the Keyboard[END_REF], but only a few are being actively used in musical contexts either because the control is not sufficiently well adapted to performance situations or because they do not offer an adequate sound control. This means that the control of digital musical instruments is still an issue that necessitates more investigations. Nowadays, sounds are used in a large number of applications (e.g. car industry, video games, radio, cinema, medicine, tourism, . . . ) since new research domains where sounds are investigated to inform or guide persons (e.g. auditory display, sound design, virtual reality, . . . ) have developed. Researchers within these domains have traditionally made use of prerecorded sounds, but since important progress has been achieved concerning the development of efficient and realistic synthesis models, an increasing interest for synthesis solutions has lately been observed [START_REF] Bzat | Car door closure sounds : Characterization of perceptual properties through analysissynthesis approach[END_REF][START_REF] Sciabica | Towards the timbre modeling of interior car sound[END_REF][START_REF] Verron | Spatialized additive synthesis[END_REF]. The control requirements related to such applications differ from musical control devices since the role of the sounds in this case is to provide specific information to the end users. Hence, a perceptual control that makes it possible to control sounds from semantic labels, gestures or drawings would be of great interest for such applications. Such control implies that perceptual and cognitive aspects are taken into account in order to understand how a sound is perceived and interpreted. Why are we for instance able to recognize the material of falling objects simply from the sounds they produce, or why do we easily accept the ersatz of horse hooves made by the noise produced when somebody is knocking coconuts together? Previous studies [START_REF] Aramaki | Sound categorization and conceptual priming for nonlinguistic and linguistic sounds[END_REF][START_REF] Schn | The evocative power of sounds: Conceptual priming between words and nonverbal sounds[END_REF] have shown that the processing of both linguistic and non-linguistic target sounds in conceptual priming tests elicited similar relationships in the congruity processing. These results indicate that it should be possible to draw up a real language of sounds. A certain number of questions have to be answered before such a language can be defined, in particular whether the identification of a sound event through the signal is linked to the presence of specific acoustic morphologies, so-called invariants that can be identified from signal analyses [START_REF] Mcadams | Thinking in Sound: The cognitive psychology of human audition[END_REF]. If so, the identification of signal invariants should make it possible to propose a perceptual control of sound synthesis processes that enables a direct evocative control. To develop perceptual control strategies of synthesis processes, it is in the first place necessary to understand the perceptual relevance of the sound attributes that characterize the sound category that is investigated. The sound attributes can be of different types and can either be linked to the physical behavior of the source [START_REF] Giordano | Material identification of real impact sounds: Effects of size variation in steel, wood, and plexiglass plates[END_REF], to the signal parameters [START_REF] Kronland-Martinet | Modelling of natural sounds by time-frequency and wavelet representations[END_REF] or to timbre descriptors obtained from perceptual considerations [START_REF] Mcadams | Perspectives on the contribution of timbre to musical structure[END_REF]. In this paper we focus on the perceptual control of environmental sounds and evoked motion by describing how such control can be defined from the identification of signal invariants obtained both from the considerations of physical behavior of the sound generating sources and the perceptual impact of the sounds on the listeners. The general approach proposed to obtain perceptual control strategies is shown in Figure 1. In the first section of this article, we describe how the perceptual control of an impact sound synthesizer enabling the definition of the sound source through verbal labels can be defined. Then a tool for controlling 3D environmental im-!"#$%"&'()*+,)'' -.+/+$0&'+#1$%23)#$14'5"617,814'3"7"#',09$2%)4':;' <,"217,',=0%0,$)%+>07"#'-?+$,=4'&"2(#)114'73@%);' A"2#(' A6#$=)1+1'9%",)11'"%'1"2#('$)B$2%)''' ?)%,)9$20&',"#$%"&'$=%"2/='/)1$2%)1'-(6#03+,1;4'' 1)30#7,'&0@)&1'-1+>)4'30$)%+0&;4'(%0C+#/14':' mersive auditory scenes with verbal labels based on a synthesizer adapted to environmental sounds is described. Finally an investigation on perceived motion and how intuitive control parameters for this specific type of evocation can be defined is presented.

Impact sound synthesizer

From the physical point of view, impact sounds are typically generated by an object undergoing free oscillations after being excited by an impact, or by a collision with other solid objects. These vibrations are governed by a wave equation and the natural frequencies of the system are obtained from the solution of this equation. These natural frequencies correspond to the frequencies for which the objet is capable of undergoing harmonic motion. The wave propagation depends on the characteristics of the object that influences two physical phenomena, i.e. dispersion (due to the stiffness of the material) and dissipation (due to loss mechanisms). Dispersion results from the fact that the wave propagation speed varies with the frequency and introduces inharmonicity in the spectrum. Dissipation is directly linked to the damping of the sound and is generally frequency-dependent. The perceptual relevance of these phenomena and how they contribute to the identification of impact sounds will be discussed in the next section.

Invariant sound structures characterizing impact sounds

Impact sounds have been largely investigated in the literature. In particular, some links between the physical characteristics of actions (impact, bouncing . . . ) and sound sources (material, shape, size, cavity . . . ) and their perceptual correlates were established (see [START_REF] Aramaki | Controlling the perceived material in an impact sound synthesizer[END_REF][START_REF] Aramaki | Timbre perception of sounds from impacted materials: behavioral, electrophysiological and acoustic approaches[END_REF] for a review). For instance, the perception of the hardness of a mallet impacting an object is related to the characteristics of the attack time. The perception of material seems to be linked to the characteristics of the damping that is generally frequency-dependent: high frequency components are damped more heavily than low frequency components. In addition to the damping, we concluded that the density of spectral components which is directly linked to the perceived roughness, is also relevant for the distinction between metal versus glass and wood categories [START_REF] Aramaki | Controlling the perceived material in an impact sound synthesizer[END_REF][START_REF] Aramaki | Timbre perception of sounds from impacted materials: behavioral, electrophysiological and acoustic approaches[END_REF]. The perceived shape of the object is related to the distribution of the spectral components of the produced sound. It can therefore be assumed that both the inharmonicity and the roughness determine the perceived shape of the object. From a physical point of view, large objects vibrate at lower eigenfrequencies than small ones. Hence, the perceived size of the object is mainly based on the pitch. For complex sounds, the determination of pitch is still an open issue. In some cases, the pitch may not correspond to an actual component of the spectrum and both spectral and virtual pitches are elicited [START_REF] Terhardt | Pitch of complex signals according to virtualpitch theory: Tests, examples, and predictions[END_REF]. However, for quasi-harmonic sounds, we assume that the pitch is linked to the fundamental frequency.

These considerations allowed us to identify signal morphologies (i.e. invariants) conveying relevant information on the perceived material, size, shape and type of impact on an object. A mapping strategy defining a link between synthesis parameters, acoustic descriptors and perceptual control parameters can then be defined, as described in the next section.

Control of the impact sound synthesizer

To develop a perceptual control of impact sounds based on semantic description of the sound source, a mapping strategy between synthesis parameters (low level), acoustic descriptors (middle level) and semantic labels (high level) characterizing the evoked sound object was defined. The mapping strategy that was chosen is based on a three level architecture as seen in Figure 2) [START_REF] Aramaki | A percussive sound synthetizer based on physical and perceptual attributes[END_REF][START_REF] Aramaki | Imagine the sounds : an intuitive control of an impact sound synthesizer[END_REF][START_REF] Aramaki | Controlling the perceived material in an impact sound synthesizer[END_REF]. The top layer is composed of verbal descriptions of the object (nature of the material, size and shape, etc.). The middle layer concerns the control of acoustic descriptors that are known to be relevant from the perceptual point of view as described in section 2.1. The bottom layer is dedicated to the control of the parameters of the synthesis model (amplitudes, frequencies and damping coefficients of components). The mapping strategy between verbal descriptions of the sound source and sound descriptors is designed with respect to the previous considerations described in section 2.1. The control of the perceived material is based on the manipulation of damping but also that of spectral sound descriptors such as inharmonicity and roughness. Since the damping is frequency dependent, a damping law was arbitrarily defined and we proposed an exponential function: The perception of size is controlled by the frequency of the first component and the perception of shape by the spectral distribution of components defined from inharmonicity and roughness. As for damping, an inharmonicity law characterized by a few parameters was proposed. Some pre-defined presets give direct access to typical inharmonicity profiles, such as those of strings, membranes and plates. The roughness is created by applying amplitude and frequency modulations on the initial sound and can be controlled separately for each Bark band.

The mapping between sound descriptors and synthesis parameters is organized as follows. The damping coefficients of the components are determined from the damping law α(ω) and their amplitudes from the envelope modulations introduced by the excitation point. The spectral distribution of components (frequency values) are defined from the inharmonicity law and the roughness. A direct control at low level allows for readjustments of this spectral distribution of components by acting separately on the frequency, amplitude and damping coefficients of each component. This mapping between middle and bottom layer depends on the synthesis model and should be adapted with respect to the chosen synthesis process.

What the control of action is concerned, the hardness of the mallet is controlled by the attack time and the brightness while the perceived force is related to the brightness: the heavier the applied force is, the brighter the sound. The timbre of the generated sound is also strongly influenced by the excitation point of the impact that creates envelope modulations in the spectrum due to the cancellation of modes presenting a node at the point of excitation. From a synthesis point of view, the location of the impact is taken into account by shaping the spectrum with a feed forward comb filter.

Immersive environmental sound synthesizer

Impact sounds constitute a specific category of environmental sounds. In this section an immersive synthesizer simulating various kinds of environmental sounds is proposed. These sounds are divided in three main categories according to W. W. Gaver's taxonomy of everyday sound sources: vibrating solids (impact, bouncing, deformation . . . ) liquid (wave, drop, rain . . . ) and aerodynamic (wind, fire . . . ) objects [START_REF] Gaver | What in the world do we hear? an ecological approach to auditory event perception[END_REF][START_REF] Gaver | How do we hear in the world? explorations in ecological acoustics[END_REF]. Based on this taxonomy, we proposed a synthesizer to create and control immersive environmental scenes intended for interactive virtual and augmented reality and sonification applications. Both synthesis and spatialization engines were included in this tool so as to increase the realism and the feeling of being immersed in virtual worlds.

Invariant sound structures characterizing environmental sounds

In the case of impact sounds, we have seen that physical considerations reveal important properties that can be used to identify the perceived effects of the generated sounds (cf. section 2.1). For other types of environmental sounds such as wave, wind or explosion sounds, the physical considerations involve complex modeling and can less easily be taken into account for synthesis perspective with interactive constraints. Hence the identification of perceptual cues linked to these sound categories was done by the analyses of sound signals representative of these categories. From a perceptual point of view, these sounds evoke a wide range of different physical sources, but interestingly, from a signal point of view, some common acoustic morphologies can be highlighted across these sounds. To date, we concluded on five elementary sound morphologies based on impacts, chirps and noise structures [START_REF] Verron | Controlling a spatialized environmental sound synthesizer[END_REF]. This finding is based on a heuristic approach that has been verified on a large set of environmental sounds. Actually, granular synthesis processes associated to the morphologies of these five grains have enabled the generation of various environmental sounds such as solid interactions and aerodynamic or liquid sounds. Sounds produced by solid interactions can be characterised from a physical point of view. When a linear approximation applies (small deformation of the structure), the response of a solid object to external forces can be viewed as the convolution of these forces with the modal response of the object. Such a response is given by a sum of exponentially damped sinusoids, defining the typical "tonal solid grain". Nevertheless, such a type of grain cannot itself account for all kinds of solid impact sounds. Actually, rapidly vanishing impact sounds or sounds characterized by a strong density of modes may rather be modelled as exponentially damped noise. This sound characterization stands for both perceptual and signal points of views, since no obvious pitch can be extracted from such sounds. Exponentially damped noise constitutes the socalled "noisy impact grain". Still dealing with physical considerations, we may design a "liquid grain" that takes into account cavitation phenomena occurring in liquid motion. Cavitation leads to local pressure variations that, from an acoustic point of view, generate time varying frequency components such as exponentially damped linear chirps. Exponentially damped chirps then constitute our third type of grain: the "liquid grain". Aerodynamic sounds generally result from complicated interactions between solids and gases. It is therefore difficult to extract useful information from corresponding physical models. The construction of granular synthesis processes was therefore based on heuristic perceptual expertise defining two kinds of aerodynamic grains: the "whistling grain" consisting in a slowly varying narrow band noise; and the "background aerodynamic grain" consisting in a broadband filtered noise. By combining these five grains using an accurate statistics of appearance, various environmental sounds can be designed such as rainy ambiances, seacoast ambiances, windy environments, fire noises, or solid interactions simulating solid impacts or footstep noises. We currently aim at extracting the parameters corresponding to these grains from the analysis of natural sound, using matching pursuit like methods.

Control of the environmental sound synthesizer

To develop a perceptual control of the environmental sound synthesizer based on semantic labels, a mapping strategy that enabled the design of complex auditory scenes was defined. In particular, we took into account that some sound sources such as wind or rain are naturally diffuse and wide. Therefore, the control included the location and the spatial extension of sound sources in a 3D space. In contrast with the classical two-stage approach, which consists in first synthesizing a monophonic sound (timbre properties) and then spatializing the sound (spatial position and extension in a 3D space), the architecture of the proposed synthesizer yielded control strategies based on the overall manipulation of timbre and spatial attributes of sound sources at the same level of sound generation [START_REF] Verron | A 3d immersive synthesizer for environmental sounds[END_REF].

For that purpose, we decided to bring the spatial distribution of the sounds to the lowest level of the sound generation. Indeed, the characterization of each elementary time-localized sound component, that is generally limited to its amplitude, frequency and phase, was augmented by its spatial position in the 3D space. This tremendous addition leads to an increasing number of control possibilities while still being real time compatible thanks to an accurate use of the granular synthesis process in the frequency domain [START_REF] Verron | Analysis/synthesis and spatialization of noisy environmental sounds[END_REF]. We then showed that the control of the spatial distribution of the partials together with the construction of decorrelated versions of the actual sound allowed for the control of the spatial position of the sound source together with the control of its perceived spatial width. These two perceptual spatial dimensions have shown to be of great importance in the design of immersive auditory scenes.

Complex 3D auditory scenes can be intuitively built by combining spatialized sound sources that are themselves built from the elementary "grain" structures (cf. section 3.1). The fire is for instance built from three elementary grains that are a whistling grain (simulating the hissing), a background aerodynamic grain (simulating the background combustion) and noisy impact grains (simulating the cracklings). The grains are generated and launched randomly with respect to time using an accurate statistical law that can be controlled. A global control of the fire intensity, mapped with the control of the grain generation (amplitude and statistical law), can then be designed. The overall control of the environmental scene synthesizer is effectuated through a graphical interface (see figure 3) where the listener is positioned in the center of the scene. Then the user selects the sound sources to be included in the auditory scene among a set of available sources (fire, wind, rain, wave, chimes, footsteps . . . ) and places them around the lis-tener by graphically defining the distance and the spatial width of the source. In cases of interactive uses, controls can be achieved using either MIDI interfaces, from data obtained from a graphical engine or other external data sources.

WIND

Synthesis of evoked motion

A third approach aiming at developing perceptual control devices for synthesized sounds that evoke specific motions is presented in this section. The definition of perceptual control necessitates more thorough investigations in this case than in the two previous cases due to the rather vague notion of perceived motion. Although physics of moving sound sources can to some extent give indications on certain morphologies that characterize specific movements [START_REF] Kronland-Martinet | Real-time perceptual simulation of moving sources: Application to the leslie cabinet and 3d sound immersion[END_REF], it cannot always explain the notion of perceived motion. In fact, this notion does not only rely on the physical displacement of an object, but can also be linked to temporal evolutions in general or to motion at a more metaphoric level. It is therefore necessary to improve the understanding of perceived dimension of motion linked to the intrinsic properties of sounds. Therefore, an investigation of perceived motion categories obtained through listening tests was effectuated before signal morphologies that characterize the perceptual recognition of motion could be identified.

Invariant structures of evoked motion

As already mentioned, motion can be directly linked to physical moving sound sources, but can also be considered in more metaphoric ways. Studies on the physical movement of a sound source and the corresponding signal morphologies have been widely described in the literature [START_REF] Chowning | The simulation of moving sound sources[END_REF][START_REF] Rosenblum | Relative effectiveness of three stimulus variables for locating a moving sound source[END_REF][START_REF] Neuhoff | The doppler illusion: the influence of dynamic intensity change on perceived pitch[END_REF][START_REF] Warren | Perception of soundsource motion by the human brain[END_REF][START_REF] Kronland-Martinet | Real-time perceptual simulation of moving sources: Application to the leslie cabinet and 3d sound immersion[END_REF]. One aspect that links physics and perception is the sound pressure that relates the sound intensity to the loudness. The sound pressure is known to vary inversely with the distance between the source and the listener. This rule is highly important from the perceptual point of view [START_REF] Rosenblum | Relative effectiveness of three stimulus variables for locating a moving sound source[END_REF], and it is possibly decisive in the case of slowly moving sources. It is worth noting that only the relative changes in the sound pressure should be considered in this context. Another important aspect is the timbre and more specifically the brightness variations, which can be physically accounted for in terms of the air absorption [START_REF] Chowning | The simulation of moving sound sources[END_REF]. A third phenomenon which is well known in physics is the Doppler effect which explains why frequency shifts can be heard while listening to the siren of an approaching police car [START_REF] Neuhoff | The doppler illusion: the influence of dynamic intensity change on perceived pitch[END_REF]. Actually, depending on the relative speed of the source with respect to the listener, the frequency measured at the listener's position varies and the specific time-dependent pattern seems to be a highly relevant cue enabling the listener to construct a mental representation of the trajectory. Finally, the reverberation is another aspect that enables the distinction between close and distant sound sources [START_REF] Jot | A real-time spatial sound processor for music and virtual reality applications[END_REF]. A close sound source will produce direct sounds of greater magnitude than the reflected sounds, which means that the reverberation will be weaker for close sound sources than for distant ones.

When considering evoked motion at a more metaphoric level, like for instance in music and cartoon production processes, signal morphologies responsible for the perceived motion cannot be directly linked to physics and must be identified in other ways, for instance through listening tests. The selection of stimuli for such investigations is intricate, since the recognition of the sound producing source might influence the judgement of the perceived motion. For instance, when the sound from a car is presented, the motion that a listener will associate to this sound will most probably be influenced by the possible motions that the car can make, even if the sound might contain other interesting indices that could have evoked motions at more metaphoric levels. To avoid this problem, we therefore decided to investigate motion through a specific sound category, so-called "abstract sounds" which are sounds that cannot be easily associated to an identifiable sound source. Hence, when listeners are asked to describe evocations induced by such sounds, they are forced to concentrate on intrinsic sound properties instead of the sound source. Such sounds, that have been explored by electroacoustic music composers, can be obtained from both recordings (for instance with a microphone close to a sound source) or from synthesized sounds obtained by for instance granular synthesis [START_REF] Merer | Abstract Sounds and Their Applications in Audio and Perception Research, chap[END_REF]. In a previous study aiming at investigating semiotics of abstract sounds [START_REF] Schn | The evocative power of sounds: Conceptual priming between words and nonverbal sounds[END_REF], subjects often referred to various motions when describing these sounds. This observation reinforced our conviction that abstract sounds are well adapted to investigate evoked motion.

As a first approach toward a perceptual control of evoked motion, perceived motion categories were identified through a free categorization test [START_REF] Merer | Computer Music Modeling and Retrieval[END_REF]. Subjects were asked to categorize 68 abstract sound and further give a verbal description of each category. Six main categories were identified through this test, i.e. "rotating", "falling down", "approaching", "passing by", "going away", "going up". The extraction of signal features specific to each category revealed a systematic presence of amplitude and frequency modulations in the case of sounds belonging to the category"turning", a logarithmic decrease in amplitude in the category "passing" and amplitude envelopes characteristic of impulsive sounds for the category "falling". Interestingly, several subjects expressed the need to make drawings to describe the perceived motions. This tends to indicate that a relationship between the dynamics of sounds and a graphic representation is intuitive. This observation was decisive for our control strategy investigation presented in the next section.

Control of evoked motion

In the case of evoked motion the definition of a perceptual control is as previously mentioned less straightforward than in the case of impact sounds and environmental sounds. From the free categorization test described in the previous section, categories of motion were identified along with suitable signal invariants corresponding to each category. However, this test did not directly yield any perceptual cues as to how these evocations might be controlled in a synthesis tool. Therefore, to identify perceptually relevant control parameters corresponding to evoked dynamic patterns, further experiments were conducted in which subjects were asked to describe the evoked trajectories by drawings. Since hand made drawings would have been difficult to analyze and would have been influenced by differences in people's ability to draw, a parametrized drawing interface was developed, meaning that subjects were given identical drawing tools that required no specific skills. The control parameters available in the interface were based on the findings obtained in the free categorization test, and the accuracy of the drawing was limited to prevent the interface from becoming too complex to handle. The interface is shown in Figure 4. Two aspects, i.e. shape and dynamics, enabled the subjects to define the motion. Six parameters were available to draw the shape of the trajectory (shape, size, frequency oscillation, randomness, angle, initial position) and three parameters were available to define the dynamics (initial and final velocity and number of returns). Each time a sound was presented, the subject made a drawing that corresponded to the trajectory he or she had perceived. No time constraint was imposed and the subject could listen to the sound as often as he/she wanted. The dynamics was illustrated by a ball that followed the trajectory while the sound was played.

Results showed that although the subjects used various drawing strategies, equivalent drawings and common parameter values could still be discerned. As far as the shape was concerned, subjects showed good agreement on the distinction between linear and oscillating movements and between wave-like and circular oscillations. This means that these three aspects give a sufficiently exact control of the perceived shape of sound trajectories. As far as the orientation of the trajectory was concerned, only the distinction between horizontal and vertical seems to be relevant. While there was agreement among subjects about the distinction between the upward/downward direction, the difference between the left/right direction was not relevant. As far as the velocity was concerned, the subjects distinguished between constant and varying velocities, but they did not show good agreement in the way they specified the velocity variations they perceived. This might have been related to the graphical user interface which did not provide a sufficiently precise control of the dynamics according to several subjects. The identification of perceptually relevant parameters enabled the definition of a reduced number of control possibilities. Hence 3 kinds of shapes (linear, circular and regular), 3 different directions (south, north and horizontal), and various degrees of oscillation frequencies (high and low), randomness (non, little, much), size (small, medium, large) and dynamics (constant, medium an high speed) were found to be important control parameters that enabled the definition of perceived trajectories. Based on these findings, a generic motion control strategy could hereby be defined as shown in Figure 5. This strategy could be separated in three parts, i.e. a perceptual control level based on drawings, an image processing level dividing the drawings in elementary patterns (i.e. waves, lines, direction, etc) and a third level containing the synthesis algorithm or a sound texture. This article describes perceptual control strategies of synthesis processes obtained from the identification of sound structures (invariants) responsible for evocations induced by sounds. In the case of impact sounds, these sound structures are obtained by investigating the perceptual relevance of signal properties related to the physical behavior of the sound sources. Variations of physical phenomena such as dispersion and dissipation make perceptual distinctions possible between different types of objects (i.e. strings versus bars or plates versus membranes) or materials (wood, glass, metal,. . . ). The spectral content of the impact sound, in particular the eigen-frequencies that characterize the modes of a vibrating object, is responsible for the perception of its shape and size. In cases where the physical behavior of sound sources are not known (e.g. certain categories of environmental sounds) or cannot explain evocations (e.g. metaphoric description of motion), recorded sounds are analyzed and linked to perceptual judgements. Based on the invariant signal structures identified (chirps, noise structures, . . . ), various control strategies that make it possible to intuitively control interacting objects and immersive 3-D environments are developed. With these interfaces, complex 3-D auditory scenes (the sound of rain, waves, wind, fire, etc.) can be intuitively designed. New means of controlling the dynamics of moving sounds via written words or drawings are also proposed.

These developments open the way to new and captivating possibilities for using non-linguistic sounds as a means of communication. Further extending our knowledge in this field will make it possible to develop new tools for generating sound metaphors based on invariant signal structures which can be used to evoke specific mental images via selected perceptual and cognitive attributes. This makes it for instance possible to transform an initially stationary sound into a sound that evokes a motion that follows a specific trajectory.

Fig. 1 .

 1 Fig. 1. Synoptics of the perceptual control strategy.

Fig. 2 .

 2 Fig. 2. Three level control strategy of impact sounds

Fig. 3 .

 3 Fig. 3. Auditory scene of a windy day (wind source surrounding the listener) on a beach (wave coming towards the listener) and including a BBQ sound (fire located at the back right of the listener).

Fig. 4 .

 4 Fig. 4. Graphical User Interface

Fig. 5 .

 5 Fig. 5. Generic motion control